Muhammad Absar’s Post

View profile for Muhammad Absar
Muhammad Absar

Enigmatix•302 followers

Your Django app is probably opening and closing database connections on every single request. This is Django's default behavior. A connection is established on the first query and torn down when the request finishes. For low-traffic apps, this is fine. But it becomes a serious bottleneck under load. The overhead of the TCP handshake and PostgreSQL authentication for every single request adds up quickly. It consumes CPU on both your app and database servers and can easily lead to connection exhaustion, where your database simply refuses new clients. Your app starts throwing errors, and users see failures. The solution is connection pooling. A connection pooler like PgBouncer sits between your Django application and your database. It maintains a pool of persistent connections to Postgres. When your app needs a connection, PgBouncer hands it a ready-to-use one from the pool instantly, bypassing the expensive setup process. When the request is done, the connection is returned to the pool, not closed. Setting Django's `CONN_MAX_AGE` for persistent connections is a good first step, but an external pooler gives you far more control and insight for serious production environments. Have you ever been bitten by database connection exhaustion in production? What was the fix? Connect with me if you're navigating similar scaling challenges. #Django #PostgreSQL #SystemDesign

  • This is Django's default behavior. A connection is established on the first query and torn down when the request finishes. For low-traffic apps, this is fine. But it becomes a serious bottleneck under load. The overhead of the TCP handshake and PostgreSQL authentication for every single request adds up quickly. It consumes CPU on both your app and database servers and can easily lead to connection exhaustion, where your database simply refuses new clients. Your app starts throwing errors, and users see failures. The solution is connection pooling. A connection pooler like PgBouncer sits between your Django application and your database. It maintains a pool of persistent connections to Postgres. When your app needs a connection, PgBouncer hands it a ready-to-use one from the pool instantly, bypassing the expensive setup process. When the request is done, the connection is returned to the pool, not closed. Setting Django's `CONN_MAX_AGE` for persistent connections is a good first step, but an external pooler gives you far more control and insight for serious production environments. Have you ever been bitten by database connection exhaustion in production? What was the fix? Connect with me if you're navigating similar scaling challenges. #Django #PostgreSQL #SystemDesign
  • This is Django's default behavior. A connection is established on the first query and torn down when the request finishes. For low-traffic apps, this is fine. But it becomes a serious bottleneck under load. The overhead of the TCP handshake and PostgreSQL authentication for every single request adds up quickly. It consumes CPU on both your app and database servers and can easily lead to connection exhaustion, where your database simply refuses new clients. Your app starts throwing errors, and users see failures. The solution is connection pooling. A connection pooler like PgBouncer sits between your Django application and your database. It maintains a pool of persistent connections to Postgres. When your app needs a connection, PgBouncer hands it a ready-to-use one from the pool instantly, bypassing the expensive setup process. When the request is done, the connection is returned to the pool, not closed. Setting Django's `CONN_MAX_AGE` for persistent connections is a good first step, but an external pooler gives you far more control and insight for serious production environments. Have you ever been bitten by database connection exhaustion in production? What was the fix? Connect with me if you're navigating similar scaling challenges. #Django #PostgreSQL #SystemDesign

To view or add a comment, sign in

More Relevant Posts

  • View profile for PREM S. B.
    PREM S. B.

    PIPRA Solutions•3K followers

    🐘 PostgreSQL Installation (Ubuntu – Step by Step) If you're working on Node.js / backend apps, PostgreSQL setup is a must 🔥 ✅ Steps: 1. Install PostgreSQL 2. Set password for postgres user 3. Update pg_hba.conf (peer → md5) 4. Restart service 5. Test login 💡 Common Issue: “password authentication failed” → fix config file 🎯 Pro Tip: Always verify connection using psql before integrating with backend #PostgreSQL #Backend #NodeJS #Database #Developers

    To view or add a comment, sign in

  • View profile for Girish Sawant
    Girish Sawant

    Xebia•681 followers

    🚀 Excited to share a project I've been building: PgIDE — an open-source PostgreSQL IDE. As someone who works with PostgreSQL daily, I was frustrated with switching between multiple tools for querying, performance analysis, and schema visualization. So I built one tool that does it all. What makes it different: 📊 Visual EXPLAIN Analyzer — not just a tree view, but critical path detection, join strategy analysis, parameter testing (tweak work_mem and see plan changes instantly), and plan history with side-by-side comparison. 🧠 Index Advisor — paste a query, get missing index suggestions with ready-to-run SQL. 🤖 pgvector Support — with AI embeddings becoming mainstream, PgIDE has first-class support for vector columns, HNSW/IVFFlat index recommendations, and code templates for similarity search and RAG patterns. 🗺️ Interactive ER Diagrams, Schema Diff, and Migration Generator. Built with React 18, TypeScript, Monaco Editor (VS Code engine), Node.js, and Express. Runs in the browser or as an Electron desktop app. MIT licensed. GitHub: https://lnkd.in/dZvBqC8f If you work with PostgreSQL, I'd love your feedback. And if you find it useful, a ⭐ on GitHub would mean a lot! #PostgreSQL #OpenSource #WebDevelopment #React #TypeScript #DatabaseTools #pgvector #AI

    To view or add a comment, sign in

  • View profile for Amos Makokha
    Amos Makokha

    Freelance | Self-Employed•159 followers

    Our Django app ran on SQLite in production for 8 months without any complaints. Then my client needed two servers running the same app, and both had to share data in real time. SQLite is just a file sitting on one server, so you can't share it between two machines 2,000 kilometers apart. We had to migrate to PostgreSQL. I exported the data and tried to import it into Postgres, but it failed. Three text fields had values longer than their max_length setting, and SQLite had been storing them anyway while Postgres rejected every single one. Also, our post_save signal was auto-creating user profiles, so when I ran loaddata it triggered saves for every user and the signal fired before the fixture could insert its own profile record. This caused duplicate key violations everywhere. SQLite had been hiding both problems for months. I fixed it by increasing max_length on those fields and temporarily disconnecting the signal during import. After that I loaded 803,897 objects with zero errors. The migration itself was easy, but the scary part was discovering what SQLite had been silently tolerating that Postgres would not. Your database might be lying to you right now, and you won't know until you try to move it. This migration added an additional $18 per month to the overral AWS bill but it is worth it.

    To view or add a comment, sign in

  • View profile for Muhammad Absar
    Muhammad Absar

    Enigmatix•302 followers

    Your Django `manage.py runserver` setup is lying about your database performance. The default SQLite setup is great for local development. It's fast, zero-config, and just works. But it creates a dangerous blind spot: it completely hides the realities of network latency and connection management. In production, your Django app and your database live on separate machines. Every single query pays a network round-trip cost. Django's default behavior of opening a new database connection for each request, which is trivial locally, becomes a significant performance bottleneck under load. This is where a connection pooler like PgBouncer becomes non-negotiable. It sits between your Django application servers and your PostgreSQL database, maintaining a pool of ready-to-use connections. Instead of the expensive TCP handshake for every request, your app grabs a warm connection from the pool, drastically reducing latency. For read-heavy applications, the next step is implementing read replicas. You can configure Django's database router to send all `GET` requests to one or more replica databases, leaving your primary database free to handle writes. This simple separation is one of the most effective ways to scale a Django backend. What's the first database optimization you implement when a Django project starts to scale? Let's connect — I share lessons on building scalable backends. #Django #PostgreSQL #SystemDesign #Database

    • The default SQLite setup is great for local development. It's fast, zero-config, and just works. But it creates a dangerous blind spot: it completely hides the realities of network latency and connection management. In production, your Django app and your database live on separate machines. Every single query pays a network round-trip cost. Django's default behavior of opening a new database connection for each request, which is trivial locally, becomes a significant performance bottleneck under load. This is where a connection pooler like PgBouncer becomes non-negotiable. It sits between your Django application servers and your PostgreSQL database, maintaining a pool of ready-to-use connections. Instead of the expensive TCP handshake for every request, your app grabs a warm connection from the pool, drastically reducing latency. For read-heavy applications, the next step is implementing read replicas. You can configure Django's database router to send all `GET` requests to one or more replica databases, leaving your primary database free to handle writes. This simple separation is one of the most effective ways to scale a Django backend. What's the first database optimization you implement when a Django project starts to scale? Let's connect — I share lessons on building scalable backends. #Django #PostgreSQL #SystemDesign #Database
    • The default SQLite setup is great for local development. It's fast, zero-config, and just works. But it creates a dangerous blind spot: it completely hides the realities of network latency and connection management. In production, your Django app and your database live on separate machines. Every single query pays a network round-trip cost. Django's default behavior of opening a new database connection for each request, which is trivial locally, becomes a significant performance bottleneck under load. This is where a connection pooler like PgBouncer becomes non-negotiable. It sits between your Django application servers and your PostgreSQL database, maintaining a pool of ready-to-use connections. Instead of the expensive TCP handshake for every request, your app grabs a warm connection from the pool, drastically reducing latency. For read-heavy applications, the next step is implementing read replicas. You can configure Django's database router to send all `GET` requests to one or more replica databases, leaving your primary database free to handle writes. This simple separation is one of the most effective ways to scale a Django backend. What's the first database optimization you implement when a Django project starts to scale? Let's connect — I share lessons on building scalable backends. #Django #PostgreSQL #SystemDesign #Database

    To view or add a comment, sign in

  • View profile for Kamil Wojtalak
    Kamil Wojtalak

    Startup Community Poznan (SCP)•7K followers

    Found a bug in the Laravel 13 installer. No time to fix it - so I'm giving it away: The bug: When you run laravel new with --database=pgsql and choose a starter kit (React, Vue, Svelte), your database choice gets ignored. The .env file is overridden by the starter kit's default - SQLite. You asked for pgsql. You get SQLite. Silently. - it is an issue with ci cd script also. Is it intentional? Probably not fully. The Laravel team prioritizes SQLite as the default so you can see your app instantly without any local database setup. But ignoring an explicit --database flag you passed yourself? :/ If you took the time to type --database=pgsql, the installer should respect it. The title for the PR is basically already written: Fix: Ensure --database option is respected when installing Starter Kits Laravel community prefers PRs over issues. So - no issue, no complaining. Just a clear description, a reproduction case, and a fix waiting to happen. - the issue is env file, and the tests yml file - since lack of the database driver If you want an easy open source contribution, this one's yours. If you decide to do it, you can share a link to your solution in the comments.

    • No alternative text description for this image

    To view or add a comment, sign in

  • View profile for Haris Furqan
    Haris Furqan

    Learning Web development•417 followers

    🎩 Hat Store - Full-Stack Inventory Management App Built a complete CRUD application for managing a hat store inventory using Node.js, Express, PostgreSQL, and EJS. 🔗 Live Demo: https://lnkd.in/duP45keG ✨ Features: - Full CRUD operations for items and categories - PostgreSQL database with relational data modeling - Deployed on Render with live database 🔧 Tech Stack: Node.js | Express | PostgreSQL | EJS | Render This project helped me strengthen my understanding of backend development, database design, and deployment workflows. 💻 GitHub: https://lnkd.in/dR2_kKGF Note: First load may take ~30s due to free tier cold start. #WebDevelopment #FullStack #NodeJS #PostgreSQL #CodingJourney

    To view or add a comment, sign in

  • View profile for Raj Shrivansh
    Raj Shrivansh

    Zehntech•1K followers

    Rails 8 brought Solid Queue to the table — and the community is divided. Sidekiq killer? Or just a neat addition for small apps? Here's my honest breakdown after working with both 👇 🔧 What Solid Queue actually brings: • Zero extra infrastructure — runs on your existing DB (PostgreSQL / MySQL / SQLite) • Native Rails 8 integration — no gem juggling, no separate config • Supports concurrency, priorities, recurring jobs & pausing queues • Swappable with ActiveJob in one line of config 📊 Numbers that matter: Solid Queue handles ~1,000 jobs/second on standard PostgreSQL. Sidekiq with Redis? 10,000+ easily. For most apps — 1,000/sec is more than enough. For high-throughput platforms — Sidekiq is still king. 🤔 The question nobody asks: How many Rails apps actually NEED 10,000 jobs/second? Most apps send emails, process uploads, generate reports. Not run real-time trading systems. ⚠️ Where Solid Queue still needs work: • No built-in Web UI yet (Sidekiq's dashboard is still unmatched) • No equivalent for Sidekiq Pro's unique jobs & batches • At extreme scale, DB-backed queues add write pressure to your primary DB 🏆 Verdict: → Greenfield Rails 8 app? Start with Solid Queue. Add Sidekiq only if you need it. → Existing Sidekiq app? No rush to migrate. Sidekiq is rock solid. → High-scale app? Sidekiq + Redis remains the battle-tested choice. The best tool is the one that fits YOUR scale — not the one in every tutorial. Are you switching to Solid Queue or sticking with Sidekiq? 👇 #RubyOnRails #Rails8 #SolidQueue #Sidekiq #BackendDevelopment #Ruby #SoftwareEngineering #WebDevelopment #AWS

    • No alternative text description for this image

    To view or add a comment, sign in

  • View profile for Bishwajit Paul
    Bishwajit Paul

    Fiverr•242 followers

    Mastering Mongoose Queries: Find, FindOne, and Beyond 🚀 Ever spent hours wondering why your MongoDB query returned an empty array or crashed your Node app? Been there. 😅 Here’s the quick cheat sheet that saved my sanity: find(filter) → fetch multiple documents → always returns an array []. findOne(filter) → fetch a single document → returns the doc or null. findById(id) → shortcut for findOne({_id: id}). findByIdAndUpdate(id, update, { new: true }) → update in one step, returns updated doc. findByIdAndDelete(id) → delete by ID, returns deleted doc or null. Pro tip: If you’re using .populate() with filters, always use optional chaining to avoid null crashes: const posts = bookmarks.map(b => b?.blogId?.toObject()).filter(Boolean); That little ? saved me from hours of debugging… and a LOT of stress. Mongoose is powerful — but knowing which query method to use and handling nulls properly is half the battle. #MongoDB #Mongoose #NodeJS #WebDevelopment #LearningByDoing #DevTips

    To view or add a comment, sign in

  • View profile for Kenneth H.
    Kenneth H.

    KeyQ, Inc.•499 followers

    How much Docker to actually use in development... Early on, we tried containerizing everything. It worked… but dev experience suffered: slow hot reloads file sync issues unnecessary complexity What’s worked better for us: 👉 Containerize the database 👉 Run backend/frontend locally You still get consistency where it matters (stateful services), but keep fast feedback loops for actual development. It’s a small shift, but it makes onboarding and day-to-day dev way smoother. Wrote up the setup here: 👉 https://lnkd.in/gApNdWtx #Docker #DevOps #FullStackDevelopment #SoftwareEngineering #DeveloperExperience #SystemDesign #StartupLife #EngineeringLeadership

    View organization page for KeyQ, Inc.
    KeyQ, Inc.

    50 followers

    Most teams overcomplicate Docker in development. They either: • Containerize everything → slow, frustrating dev experience • Or skip it entirely → inconsistent environments We’ve landed on something simpler that’s worked across multiple projects: 👉 Containerize the stateful parts (like MySQL) 👉 Run stateless services (NestJS, React) locally with hot reload That balance gives you: • Fast local development • Consistent environments • Easy onboarding (new devs up and running in minutes) One detail that made a bigger difference than expected: Avoiding Docker volume mounts for Node apps on macOS/Windows — the performance hit isn’t worth it for dev workflows. This setup has become our default for full-stack projects. We broke it down here (with configs): 👉 https://lnkd.in/g44Y6-BW #Docker #WebDevelopment #DevOps #FullStackDevelopment #NestJS #ReactJS #MySQL #DeveloperExperience #SoftwareEngineering

    To view or add a comment, sign in

Image
Muhammad Absar

302 followers

View Profile

Explore content categories