Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Blockchain has become famous in recent years for powering the cryptocurrency craze. But the fundamentals behind blockchain could prove helpful in other areas of the data economy, too. Because, as the value of data rises, so does the need for accurate, real-time data sharing across multiple partners.
As we saw recently, global trade is surprisingly fickle. If a 1,300 ft long cargo ship can end up blocking the Suez Canal for an entire week, imagine how many data record incongruities the shipping and logistics industries have to sort out each day. Similarly, other partner networks, such as food distribution or airline affiliates, for example, must constantly share data back and forth to remain in business.
Since a distributed ledger is external yet sharable with partners, it opens some interesting data-sharing possibilities. Similar to merges for collaborative software, partners would retrieve and manipulate the distributed ledger, which would act as a source of truth for status on a physical object or a piece of data. Essentially, this process could reduce manual reconciliation efforts and avoid disputes when something goes wrong.
I recently spoke with “father of serverless” and AWS Lambda inventor Tim Wagner. Wagner, now heading Vendia, shared with me why he foresees distributed ledgers as the future for real-time, business-to-business data sharing. “Supply chains are surprisingly brittle,” Wagner said. To solve this problem, we need an “automation of that workflow that isn’t centralized any longer,” he said.
Role of a Distributed Ledger
A distributed ledger could be especially relevant for a multi-party supply chain. Within multi-partner agreements, data tracking is complex, and many partners struggle to come together. “Companies are struggling for a lack of truth,” said Wagner. “Everyone wants their own copy of the data, but they want it to be locked to what everyone is seeing at the same time.”
For example, take supply chain management. Suppose an automotive parts manufacturer produces a chassis. They send it to a shipping logistics provider, who then sends it to an automaker to complete assembly of an automobile. If the chassis breaks at some point during these handoffs, whose fault is it? Without a system of record, you open a big problem of distrust, Wagner explained.
Or, consider an airline ecosystem. Partner airlines must often coordinate multi-carrier flight plans, apply discounts or settle unused travel vouchers. Often, it’s difficult to track the journey of these transactions between organizations. An externalized ledger could help partner airlines follow the route of a single passenger, helping to ease financial settlements.
“It’s not just about sharing, which is hard enough, but it’s really about sharing with control,” said Wagner. These days, tracing the lineage of data is critical, as is having real-time visibility into an object’s state. A distributed ledger could help these partner ecosystems have a record they can all trust. Wagner describes this as a tamper-proof, immutable record — an “unassailable source of truth.”
Would This Replace Partner APIs?
What’s especially interesting to the integration-minded is that the distributed ledger concept could supplant established business-to-business communication models. Over the last couple of decades, most partner integrations have moved to RESTful API connections. One company opens up data access, and another authenticated partner makes HTTP API calls to integrate data into their own application.
Yet, the issue here is that the API is only a layer on top of a private database owned by the provider. Located behind closed doors, there’s no industry-wide real-time data synchronization. API calls are good for data retrieval and manipulation, but stringing a sequence of thousands of transactions together is difficult, described Wagner. Retaining the time-series integrity of a complex database is a little too much for synchronous calls to handle.
Instead of opening access to an internal platform via API, Wagner believes that a distributed ledger is a better fit for certain business-to-business machine automation. “Any place where you see API-level protocols developed to maturity is a good sign; there is a good chance distributed ledgers will find greater speed,” he said. “Trying to pull that information through APIs and pray you’re finding the right data is difficult to manage.”
For example, Wagner referred to how some animal shelters are creating a global repository to avoid siloed records. In doing so, they are making a global animal rescue schema for a unified database that could hopefully help every pet find a home.
Communicating with the ledger would still involve calls, but likely not your typical RESTful requests. Wagner advocates for the use of GraphQL as a better method to involve clients around changes. With a GraphQL API, stakeholders can query, mutate and subscribe against changes in a distributed system. GraphQL isn’t perfect, but its easy data mechanics make it a Goldilocks-style middle ground between REST and SQL, said Wagner.
Getting Buy-In
A synchronized distributed ledger could help resolve disputes and reduce manual checks. But, how do we get buy-in? How can you convince an entire partner ecosystem to align around a distributed ledger as a source of truth?
Wagner says there is usually a leader or single manufacturer leading the charge. This figure is likely a more prominent, well-respected player in the process. But for a digital ledger system to be successful, it must also satisfy other criteria too, Wagner said:
- Data schema forethought: Agreeing upon a shared data schema may be a challenge. Ecosystems will require forethought when designing a data model for the long haul.
- Compliance: There are many new data regulations to consider, such as GDPR, CCPA and PCI. “You can’t blindly copy data the way you used to,” said Wagner.
- Governance: As enterprises deal with data crossing boundaries, they must apply governance measures to satisfy compliance and maintain security. This governance needs to be tightly associated with data and offer fine-grained control.
- High performance: A distributed ledger requires operational flexibility for large objects, such as a 5 TB file, noted Wagner.
- Redaction capabilities: Whereas some partners must see all fields, others may, by law, be limited in what they can view. Thus, a ledger must include the ability to redact fields. Wagner described this as “intentional redaction or visibility control of individual customer records.”
Of course, different industries are at very different points along their digital journey, some more advanced than others. Change may be easier for, say, financial services supporting ACH payment technology, which Wagner described as being “ripe for natural organic expansion into a data-centric methodology.”
The Poor Man’s Blockchain
Tapped into the same feed, an extensive partner ecosystem could use a blockchain-esque arrangement to stay on the same page. This could help reduce manual reconciliation typically handled through email or over the phone.
Yet, as distributed systems experts are few and far between, implementing such technologies will most likely involve a cloud platform solution, said Wagner. He jokingly calls this the “poor man’s blockchain.” Such a solution must be cloud-friendly, enterprise-focused and easy to integrate and deploy.
As more and more critical data begins to reside outside company walls, “organizations have to reconstitute that data to work with partners,” said Wagner. A distributed ledger could, in effect, act as a sort of outsourced centralized IT, enabling a data model for serverless environments.
