If you're new to Neon Postgres, I've put together a quick little explainer video on what exactly "Serverless Postgres" means.

Post #4 in my observability series is live.   Last week: We have seen how bad observability impacts organization.   This week: Let's look at various key components of observability.   In this blog, I have tried to cover three key pillars and a key fourth pillar "Event".  https://lnkd.in/e-7fQwKN Comment if you see any other factor!!   #Observability #SRE #DevOps #PlatformEngineering #EngineeringLeadership Minutus Computing

We scaled a FastAPI service from 2 pods to 8.   Throughput didn’t change. No errors. 😑   No alerts.  🤔 Just… no improvement. 🥲 The issue wasn’t Kubernetes. It was one early decision: synchronous database calls where async should have been used. But this isn’t an “always use async” post. Both have a place. Knowing when to use each is what separates systems that scale from ones that quietly bottleneck. Have you run into this before? https://lnkd.in/eVhQyBPW

I recently worked on optimizing a backend system where API latency had become a serious bottleneck. At peak load, response times were inconsistent — affecting user experience and overall system reliability. Instead of scaling blindly, I focused on identifying the real issues: • Inefficient database queries (missing indexes, heavy aggregations) • Repeated calls to external services • Lack of caching for frequently accessed data Here’s what I did: → Optimized MongoDB queries and added proper indexing → Introduced caching for high-read endpoints → Reduced unnecessary external API calls → Refactored parts of the service for better separation of concerns Result: ~40% reduction in latency and much more stable performance under load. One thing I’ve learned: Scaling isn’t always about adding more resources — sometimes it’s about fixing what’s already there.

If you want to learn how databases really work behind the scenes, check out this free YouTube series by Ben Dicken He has covered the entire "Database Internals" book videos where you will learn: • How databases store data using B-Trees • How storage engines work • LSM Trees and write optimization • Distributed systems and leader election • etc Over 10 hours of free content. Highly recommended for anyone into backend development. checkout: https://lnkd.in/gNNSbeWZ #Databases #BackendDevelopment #SoftwareEngineering

AWS shipped something last week that I think is a bigger deal than people are giving it credit for. It's called S3 Files. The short version: you can now mount any S3 bucket like a regular file system. ~1ms latency on cached data, the same bucket reachable through both the S3 API and POSIX, and it's already live in 34 regions. The reason this matters is that most tooling — data science notebooks, build systems, log processors, even the bash scripts you wrote five years ago — expects a normal file system. Until now you'd copy data out of S3 onto EFS or a local disk, run your job, and copy it back. S3 Files just deletes that step. And if you can co-locate it next to compute, there's no data egress at all. One nice side effect: agents get easier. A coding agent already knows ls, cat, and grep, so instead of writing a custom S3 tool wrapper, you can point it at a mount path and let it work the way it already wants to. The bigger thing I'd zoom in on is the pattern. Object storage stays the source of truth, and a faster layer sits on top as a cache. We use the same shape at Mixpeek for vectors — S3 Vectors underneath, Qdrant on top, retrieval pipeline on top of that. Same thesis, applied to two different abstractions: object storage is truth, the fast layer is a cache. mixpeek.com #AWS #S3 #CloudInfrastructure #DataEngineering #AIInfrastructure #ObjectStorage

We were at a breaking point. Our main orders table was hitting 100M+ rows, and query latency was spiking to 5 seconds. The team was already drafting a plan for Database Sharding We thought we had outgrown our infrastructure. We were wrong. Our dashboard was crawling. Every time a user searched for their order history, the CPU on our RDS instance pegged at 99%. We assumed the data volume was simply too high for a single node to handle. Before committing to the massive architectural overhead of Sharding, we ran a simple EXPLAIN ANALYZE on the offending query What we found was embarrassing: A Sequential Scan. The database was reading every single row on the disk to find a specific customer_id. Instead of a multi-month migration to a sharded architecture We added a composite index on (customer_id, created_at). Old way: The DB searched the entire "library" for one book. New way: The DB went straight to the "Index" at the back of the book and found the page number instantly. The results were immediate and saved us thousands in engineering hours and infrastructure costs: Latency: Dropped from 5s to 15ms. CPU Usage: Fell from 99% to 12% under peak load. Cost: We actually downsized our instance (Vertical Scaling in reverse!), saving 30% on our monthly cloud bill. The Lesson: Scaling is a ladder. Don't jump to Sharding Replication until you have absolutely mastered Indexing Query Optimization. Software engineering is often about doing the simple things right, not the complex things first. #SystemDesign #Database #SoftwareEngineering #Postgres #Scalability #Backend

Most “Simple” APIs Fail the Same Way, Here’s What Actually Breaks First I took a minimal Flask API (one endpoint + PostgreSQL) and load tested it with k6 until it started to crack. The goal wasn’t to build a high‑performance system, but to understand how a typical service degrades under load and what scaling actually means. Setup: • Flask app with Gunicorn (4 workers) • PostgreSQL with a single indexed table • k6 running on a local machine • One endpoint: GET /data → simple SELECT query What Broke First (and Why): • CPU saturation - At ~350 req/s, CPU spiked to 100%. Response times increased linearly, why because the API logic itself request parsing, JSON serialization became the bottleneck before the database even saw the traffic. • Database contention - After increasing workers, the bottleneck shifted. PostgreSQL showed high wait events and connection queueing. Why? because Concurrent connections overwhelmed the database’s pool. Even simple reads slowed down under concurrency. Tradeoffs I Observed: • Caching → reduced DB load by ~70% but added ~200MB memory overhead • More workers → improved throughput until CPU became the limit again • Database indexes → lowered per‑query latency but didn’t solve concurrency spikes Have you load tested your own services? What was the first bottleneck you hit? #BackendEngineering #DevOps #SystemDesign #Performance #LoadTesting

🚀 HOW TO TAMING S3 SHUFFLE AT SCALE In the latest edition of Data Engineering Weekly, we break down the Zero-to-One Guide to optimizing your data pipelines. 🛠 What’s Holding Your S3 Shuffle Back? 🔹Over-reliance on Local Disk: Traditional shuffle relies on local storage, which leads to "Disk Full" errors and prevents independent scaling of compute. 🔹Ignoring S3 Request Limits: Without proper partitioning and pacing, you’ll hit S3 503 Slow Down errors. 🔹Poor Data Serialization: Inefficient formats like Java default increase payload size, driving up network I/O costs and slowing down transfers. 🔹Lack of Adaptive Execution: Static partitioning fails with data skew. You need Adaptive Query Execution (AQE) to handle skewed keys and coalesce small partitions. 🔹Neglecting Cost Monitoring: API calls (PUT/GET) can cost more than the storage itself. Without lifecycle policies to clean up temporary files, your AWS bill will snowball. 💡 Deep Dive: This breakdown was inspired by the insightful work of Manoj Babu and Meghanath Macha in their recent article on optimizing S3 shuffle performance. 🔗 Read the full breakdown here: https://lnkd.in/gPmjEuDv #DataEngineering #AWS #S3 #BigData #CloudComputing #Spark #DataArchitecture #DataEngineeringWeekly