This article was originally published on the layline.io blog.
The difference between "near-real-time" and actually-real-time is wider than most teams realize — and it's getting wider as customer expectations accelerate. A major European retailer lost €4.7 million on Black Friday 2024 not because their website crashed, but because their "real-time" inventory system was running four hours behind.
This post explains what "real-time" actually means, why the shift from batch to streaming is accelerating, and what well-architected real-time systems look like in practice.
The €4.7 million delay
A major European retailer lost €4.7 million on Black Friday 2024. Not because their website crashed. Not because they ran out of stock. Because their "real-time" inventory system was running four hours behind.
340,000 customers placed orders for items that had already sold out. The system showed availability. The warehouse had none. By the time the discrepancy surfaced, the damage was done. Refunds issued. Customer service overwhelmed. Brand reputation dented. The post-mortem revealed something awkward: the pipeline was never designed for real-time. It was designed for "near-real-time," a distinction that sounded technical in architecture reviews and turned out to be catastrophic in production.
I've heard versions of this story dozens of times. The gap between what "real-time" promises and what most systems deliver is wider than most teams realize. And it's getting wider, not narrower, as customer expectations accelerate.
Like a Formula 1 pit stop, real-time data processing requires precision, coordination, and the right infrastructure.
What "real-time" actually means (and doesn't)
The industry has muddied this water. Three categories get conflated under the same label.
Batch means hours or days between updates. Your nightly ETL job. Your weekly report. Clear boundaries, predictable windows, well-understood failure modes.
Near-real-time means minutes between updates. The system checks every five, fifteen, thirty minutes. Most "real-time dashboards" fall here. Good for many use cases. Not good for the ones that matter most.
Real-time means seconds or sub-second. The event happens. The system knows. The downstream action triggers immediately.
The retailer didn't have a real-time problem. They had a near-real-time system marketed as real-time, and nobody questioned the difference until it cost them four million euros.
Three forces driving the shift
The Amazon effect. Customers expect instant everything. Not because they analyzed the technical requirements. Because that's what they've been trained to expect. A 2022 Shopify study of 12,000 consumers found 73% expect checkout, inventory, and shipping updates in real time. Not "within the hour." Real time.
Operational windows are shrinking. Fraud detection after the transaction isn't detection. It's notification. The money's already gone. Manufacturing lines that wait for batch quality reports produce bad units for hours before someone notices. The cost of delay compounds faster than most spreadsheets capture.
Competitive pressure. If your competitor updates pricing every thirty seconds and you update every six hours, you're not competing. You're spectating. This isn't theoretical. E-commerce platforms, travel aggregators, financial services. The companies winning in these spaces made real-time data infrastructure a strategic priority, not a technical nice-to-have.
Speed without control is dangerous. Real-time systems need to handle velocity while maintaining accuracy and reliability.
The hidden complexity
Moving from batch to streaming is harder than it looks. The surface seems simple: instead of waiting, react immediately. Underneath, everything changes.
State management. Batch jobs process bounded datasets. You know the input size when you start. Streaming processes unbounded streams. You need to track windows, handle late-arriving data, manage state across events that may arrive out of order.
Exactly-once processing. Run a batch job twice by accident? You get duplicate output, fix it, move on. Run a streaming pipeline twice? You double-charge customers, double-count inventory, double-notify systems. The semantics matter in ways they didn't before.
Backpressure. What happens when your source produces faster than your sink can consume? In batch, this shows up as a slow job. In streaming, it shows up as dropped messages, cascading failures, or systems that simply stop responding.
These aren't rare edge cases. They're Tuesday. Teams that underestimate this complexity end up with pipelines that work in demos and fail in production.
What good looks like
Well-architected real-time systems share traits.
Resilience by default. Not bolted on. The system expects components to fail and continues operating. Circuit breakers. Graceful degradation. Bounded queues that shed load rather than crash.
Observable. You need to see what's happening inside a pipeline that processes thousands of events per second. Metrics that matter. Tracing that follows events through the system. Alerting that fires on symptoms, not just component failures.
Growth-ready. The system that handles ten thousand events per minute should handle ten million without a rewrite. Horizontal scaling. Partition-aware design. No single points of contention.
Accessible. Real-time data integration shouldn't require a PhD in distributed systems. The tools exist. The documentation is clear. The concepts are learnable. Teams should be productive in days, not quarters.
This last point matters more than the others. The teams that succeed with real-time infrastructure aren't the ones with the most sophisticated technology. They're the ones that made it approachable enough for their existing teams to operate.
The accessibility gap
There's a two-tier market forming. Tier one: companies with dedicated streaming teams, Kafka expertise, infrastructure engineers who understand partition rebalancing and exactly-once semantics. Tier two: everyone else, stuck with batch because real-time seems too complex to attempt.
This is backwards. Real-time data integration should be as accessible as batch processing. Same team. Same skill level. Same time-to-production. The technology is there. What's missing is the packaging. Tools that handle the complexity so teams don't have to.
At layline.io, we're building for the second tier. Unified workflows that handle both batch and streaming with the same interfaces. Resilience and observability built in. Scaling that happens automatically. The goal isn't to make streaming simple. It's complex, and pretending otherwise helps nobody. The goal is to make it accessible.
Because the retailers and manufacturers and financial services companies that need real-time data already have smart teams. They don't need different people. They need better tools.
I am a founder of layline.io, building enterprise data processing infrastructure for batch and real-time workloads.
Top comments (0)