Using Data Fabric for Real-Time Processing

Charlene O’Hanlon and Scott Gnau from InterSystems talk about silos in organizations and how data fabric enables organizations to process, transform, secure and orchestrate data from disparate sources in real-time. The video is below followed by a transcript of the conversation.

Announcer: This is Digital Anarchist.

O’Hanlon: Hey, everybody, welcome back to Tech Strong TV. I’m Charlene O’Hanlon, and I’m here now with Scott Gnau, who is the VP of Data Platforms at InterSystems. Scott, thank you so much for taking the time and getting on the call with me, I really appreciate it.

Gnau: And likewise—thanks for having me. It’s great to be here.

O’Hanlon: Alright. So, tell me a little bit about InterSystems.

Gnau: InterSystems is a creative data technology company, we’re based in Cambridge, Massachusetts. We’ve been around for about 40 years, and we actually build data integration, data platform technology that’s used by a lot of very sophisticated applications around the world—in fact, more than half of medical records in North America go through our systems, 15 percent of global equity trades touch an InterSystems data platform or interface along the way.

So, we’re known for solving very large, complicated, business critical problems that are centered around data.

O’Hanlon: Well, data is the new oil, right, is that what I’ve heard? Time and time again, so many companies now rely on the data to make their business decisions and find areas of opportunity and so much more just, you know, ways to do their business better. So, data—you guys are definitely in the right space, for sure.

Gnau: Absolutely.

O’Hanlon: [Laughter] So, let’s talk a little bit about, then, the issue of silos when we’re talking about data and access to data, especially when we’re talking about things such as customer experience or uncovering those opportunities within the business, whether it’s improving internal processes or external applications and issues and things that do touch end users.

So, we’ve been talking for a long time about breaking down silos when it comes to data, but do you think that that’s something that’s feasible or do you think that it’s kind of one of those wishes that will never really come true?

Gnau: [Laughter] I’ve kind of given up on the breaking down the silos, and I’ll back into that with, you know, I’ll hit on a couple of things that you said. Because I think some of the things you said are actually really, extremely relevant, right? Data being the new oil and, you know, people have been saying that, and it’s absolutely the case. If you look at the wealth that was created in the 18th Century, it was railroads, and if you look at the wealth that was created in the 20th Century, it was energy and oil, right? Look at the wealth that’s been created in the last 20 years. It’s all about data centric kinds of companies, right?

O’Hanlon: Right.

Gnau: And that’s a really cool thing. And the difference really is, you know, companies can leverage that data not only to understand their business and have a rearview mirror view of what happened in the business and be able to make better decisions about it, but actually synthesize that data into building new businesses for themselves and new ways to interact with customers and so on. So, it’s absolutely critical and important.

And sadly, I’ve been in industry, I guess, about 35 years. It feels like just five, but a long time. And throughout that career, we’ve been talking about how do we break down silos in data and silos in data processing? And, you know, silos crepe up for many different reasons. They can creep up because you’ve deployed an application that’s very specialized to go solve a problem that’s extremely important and the data that gets collected is kind of in that application and not shared easily, and that becomes, eventually, a silo in and of itself.

Certainly, there are business model and political silos that get created, right? I’ve got a business unit model and the mortgage folks don’t wanna talk to the consumer lending folks don’t wanna talk to the deposit folks and so on. And it’s also very easy for silos to build up because of geographic differences in the business, right?

O’Hanlon: Mm-hmm.

Gnau: And so, this has been going on for a long time, and for a long time, we in technology have tried to solve and advise our clients, “Hey, don’t let silos control you.”

O’Hanlon: Right.

Gnau: I would say that the problem of silos and how they come up has actually gotten larger and is increasing at almost an unimaginable pace. Certainly, with the advent of the cloud and microservices and easy to spin up applications, more and more silos can come up very, very quickly. The fact that data and relevant data, the oil of our businesses used to be created inside the business, inside the firewall by our application, so we actually had some semblance of control over it. Now, a lot of the data that we need to use, understand, consume, and drive decisions from may live its entire life cycle outside the firewall of our business and out of our control.

So, you think about one of the drivers of silos, and it just has now gone kind of exponentially very different.

O’Hanlon: Mm-hmm.

Gnau: Certainly, the pandemic has created more geographic silos, but also, the need to be able to deploy data and decision making very quickly as many of us have been remanded to our home offices and trying to manage customer relationships and run our businesses remotely in a new and a different way.

So, I think kind of the drive for silos is going to continue to increase. And so, I think it’s also then unimaginable to think that we will live in a world without data silos. So, if that were the case, it’s kinda like, how do we come up with an approach that lets us leverage and build beyond that?

O’Hanlon: Yeah. So, you know, it’s a very good point that you make, the fact that we’ve had so many silos, the number and the velocity of silos has increased multifold, I would imagine, since the onset of the pandemic. And with this remote and hybrid work environment that we’re now entering into, I don’t think those silos are gonna be going away.

So, short of I don’t know what, short of whatever an organization needs to do to break down those silos, how can an organization work effectively to access that information and to get the information they need and really kind of make the most of it without being able to break down those silos? Is there a way that they can still utilize that information as effectively as they could if there was—you know, if everything was kinda linear and they could just kind of skip across the data meadow, if you will, and kinda pick the flowers as they went along? I don’t know where that analogy came from, but [Laughter].

Gnau: Well, listen, it would be less interesting if it was equally kind of a serial thing when you go pick up those flowers.

O’Hanlon: Right, right.

Gnau: And certainly, we talk and work with our clients, really, about this concept that we call the data fabric and a lot of technology ins and outs and all that kinda stuff with data fabric, which we probably won’t get into a lot today, but it’s almost at a mindset. And the way I describe it, being a pilot myself, is—think about being able to have a control tower kind of impact to how you manage and integrate your silos.

So, think about control towers in flight, right? What is the purposes of having control towers? Well, obviously, the mission is to provide safe flights. How do they do that? Well, they connect different kinds of data together and synthesize that data for decision making, right? They don’t consolidate it, they don’t fly the planes, necessarily, but they collect that data and then disseminate it in a way to kind of control and coordinate the flow.

O’Hanlon: Right.

Gnau: They also, it’s not a single point of contact kind of consolidation, right? There’s a control tower at every airport, right? And they connect to each other as well to manage traffic between their segments and sectors in that grid. And so, the notion of control tower is a way to think about, how do you apply that technologically speaking into thinking about your infrastructure more as a data fabric and the things that are necessary from a mindset—again, not a specific technology or a vendor, necessary—but just a mindset of how to manage through the notion of silos.

And so, the first thing, really, is something that’s also kind of nontraditional in data processing, at least during my career, it’s all about connecting data and not consolidating data, right? And for many years, it was like, “How do I consolidate all this stuff into one place to make it more manageable to take cost out?” and all that kinda stuff. Again, in a cloud-centric world, almost unmanageable.

O’Hanlon: Mm-hmm.

Gnau: So, instead of thinking about consolidating it, really think about how do you connect the different data across different places? And that obviously becomes a key part of your data fabric—the connectivity, the pipes, and the widgets, and how they fit together. Connectivity isn’t enough without context.

O’Hanlon: Mm-hmm.

Gnau: So, data that’s created in the silo may have a certain context inside of the silo, but that context needs to be reflected externally just like the control tower needs to understand what kind of plane you’re flying, because they’ll understand the operating capacity of that plane—how fast does it go, how high can it go, and what are its maneuvering capabilities, right?

Same kind of thing in a control tower kind of aspect to data fabric—connect the data and provide context for the data so that, as the data get connected, there’s some relevance to what is happening and something could actually be derived from it.

O’Hanlon: That makes a whole lot of sense, and to your point, having the data remain kinda siloed and just being able to kinda select the data that you’re looking for, you know, that’s most relevant to you at that time, I imagine actually, is a little bit easier than kinda sifting through an entire haystack for one little needle. So, that’s a really interesting concept.

Is this a technology that is currently being used within the data management space, the data science space?

Gnau: You know, there are more and more people talking about it, and I’m sure there’s some folks that maybe have different words to describe it. We’re thinking, and some of the analysts that we work with as well, are talking about the data fabric. I’d say first and foremost, it’s a mentality, but like I said, connect, don’t consolidate. Provide the context. Have the control towers communicate amongst themselves so it’s not a single control tower but multiple different control towers for the context or the application that’s required and kinda think about the network effect of being able to build that out.

Once you have that mentality, then it comes down to a checklist of, you know, certainly a list of what is the existing technology that we have and what is the existing technology that we use, how are the data created, how are the data stored—you know, kind of just a roadmap of what ya got. And then, think about the connectivity and the context provisioning that you want to create and look at other technologies that might need to be added to the mix to make that work in an end to end state.

So, it’s a mindset first, and then it’s a collection of some technologies to enable that, and it’s a combination of the existing stuff that you’ve got, and certainly, an application that’s been deployed in a business-critical nature, you’re not gonna yank that out for some technology decision. So, again, it’s all about that connectivity and fitting it together and piecing it together.

The other thing that most of the clients that we’re talking with about this are very pragmatic, it’s not a—like anything ever really should be, it’s not a 10-year plan, but it’s an, okay, how do I map this sector? How do I put a control tower at this airport and this airport and how do I leverage that, right? So, how do I go connect this silo and that silo, and then let’s go get the next silo and let’s go get the other business unit and figure out how to kinda build that out over time.

And what’s really cool is, the value accelerates as you start to collect more of these things together. And, you know, you’ve got three or four and then you add the fifth one—the value doesn’t go up 20 percent, the value actually, of what you can do with that can go up dramatically, much higher, because of being able to understand all of the customer interaction points and create new programs, or leverage that data to actually go build a new line of business that your customers weren’t expecting.

O’Hanlon: Mm-hmm. Right, right. Interesting. That’s really, really fascinating to me. Now, does this also kinda democratize the data? Does it enable other parts of an organization to access the value from a lot of this data by using this data fabric without the need for maybe a data scientist or somebody who’s a specialist in this in an organization?

Gnau: Yeah, absolutely, absolutely. And again, it’s a mentality and needs to be kind of thought of with a design going in, right?

O’Hanlon: Mm-hmm.

Gnau: So—right, in today’s world, thinking about security and privacy on one plane and then democratization of data on the other plane, they’re almost kind of at odds. So, there needs to be some forethought that goes into what is appropriate for the appropriate eyeballs in these different spaces, and then again, making sure that the technology stack that’s been built out has the capability to deliver that. But certainly, companies that—in my experience, at last, companies that have the most success in kind of refining their oil, their data, are the ones who are actually getting data anywhere that the data need to be for a decision that’s local, right?

So, you know, people talk about real time decisions and real time AI models and all that kinda stuff. All really compelling, all really valuable, but all completely worthless if you can’t leverage that to deliver a yes to a customer when the customer asks a question, right? And so, democratization and the architectural democratization of the data is an important key component to making your data fabric really successful and really showing up with a lot of value for the business.

O’Hanlon: And I would imagine that AI, machine learning—you know, those technologies can actually add a whole lot to making this a much more intelligent way of gathering data as well.

Gnau: Yeah, and it’s a symbiotic relationship, right? So, you know, machine learning and AI are all the rage and all really good stuff. A lot of the math behind it is not particularly new. What’s interesting is, the math behind it dictates that the more data you add to the model, the more accurate the prediction becomes, right? And that is very different than some of the previous kind of data mining and data science stuff that went on.

And so—okay, that means you need lots of data. How do you get lots of data? Well, you don’t if it’s in a bunch of silos that don’t talk to each other with no context and no connectivity.

O’Hanlon: Right.

Gnau: So, I actually think they’re kind of symbiotic in that an effective data fabric could actually enhance the capabilities of your AI programs, and your AI programs can actually enhance the value of your data fabric in return.

O’Hanlon: That’s really, really fascinating. [Laughter] I love this topic. I think this is definitely something that more organizations should be considering, because you know, especially if you think about just the sheer volume of data that’s out there and, I mean, really, what organization today is not basically putting their entire business on the shoulders of data without having that information that they need to make those business decisions to uncover those opportunities to just basically survive and thrive in today’s environment. I really—short of having that data, I just don’t know how organizations today can do it.

So, I think anything that’s going to make it easier for them to uncover those areas of opportunity and find those nuggets of information that’s in all of that data, I think, is gonna be a huge, huge value add for organizations.

Gnau: Yeah. I mean, and the last piece, really, of success for a data fabric and why I think data fabric is so important—I tried to come up with something that started with a C, because I said connect and then context. But the last one really is lineage, because lineage creates trust.

O’Hanlon: Mm-hmm.

Gnau: So, you know, it’s interesting, the other interesting thing about some of the AI models out there is that the answers they create are actually sometimes not intuitive.

O’Hanlon: Right.

Gnau: And as human beings going to execute these things, we’ve gotta really trust that the models and the data are effectively connected, so that we’re gonna risk our business on these decisions, right?

O’Hanlon: Yep, yep.

Gnau: And so, you know, connectivity, context, and ultimately lineage or trust of the data are kind of the three key components, again, of the mentality. Nothing to do with technology, but you’ve gotta kinda build in what’s your framework, what’s your appetite in your business for these pieces and then go look at the technology landscape to start building it out.

O’Hanlon: That’s a very intelligent way of going about that, too, so—I love that. Scott, thank you so much for having the conversation with me about data fabrics. I know we’re gonna be hearing more about it in the future. I think it’s a great mindset and a great way to approach data access and data use in an organization. So, thanks again for your time and your expertise, I really do appreciate it.

Gnau: Thank you.

O’Hanlon: Yeah. Alright. Alright, everybody, please stick around, we’ve got lots more Tech Strong TV coming up, so stay tuned.

[End of Audio]