Forem

PHP to Go: The Mental Model Shift Nobody Warns You About

Gabriel Anhaia — Sat, 18 Apr 2026 10:36:13 +0000

Book: Observability for LLM Applications · Ebook from Apr 22
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

The Laravel container. Eloquent. Facades. Magic methods. Thirty years of PHP have taught you that the framework does the composition for you. You type User::find(1) and something, somewhere, boots an ORM, resolves a connection, hydrates a model, and hands it back. You never had to ask who wired what.

Go hands the composition back. Every dependency is a parameter. Every request is a goroutine. There is no container that calls new on your behalf, no __construct metadata, no framework that reads your route annotations at boot.

This is the part of the move that actually hurts. Not the syntax. The syntax is small. The mental model is the thing. Here is what that looks like in practice, feature by feature, for a PHP developer who already ships.

flowchart LR
    subgraph PHP["PHP / PHP-FPM"]
        R1[Request] --> W1[Spawn worker]
        W1 --> P1[Boot framework<br/>Load config]
        P1 --> H1[Handle request]
        H1 --> D1[Die]
    end
    subgraph GO["Go / net/http"]
        R2[Request] --> G2[Go routine]
        G2 --> H2[Handle request]
        H2 --> F2[Return<br/>routine freed]
    end

The request lifecycle stops dying

In PHP-FPM, every request is a short-lived process. It boots the framework, handles one HTTP call, and dies. State does not survive. Caches need Redis. Background work needs a queue worker. You never had to think about a request that outlives its handler because nothing ever did.

A Go server is one long-lived process. Every request is a goroutine — roughly 2 KB of stack, scheduled on top of a small pool of OS threads by the Go runtime. The process boots once. Globals survive. In-memory caches work. The http.Server you start in main runs until you SIGTERM it.

package main

import (
    "log"
    "net/http"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hello"))
    })
    log.Fatal(http.ListenAndServe(":8080", mux))
}

That is the whole server. No index.php, no public/ directory, no .htaccess. The Laravel equivalent has a router, a kernel, middleware pipeline, service providers, and PHP-FPM in front of it. Go skips all of that because it doesn't need to reboot every request.

RoadRunner and FrankenPHP narrow the gap on the PHP side. They keep the worker alive between requests. But the default PHP model is still die-on-response, and most Laravel code is written as if that's true — globals are suspect, singletons need ->singleton() bindings, and memory leaks "don't exist" because the process gets flushed. In Go, none of that applies. A leaked goroutine will still be there at 3 a.m.

Eloquent vs `database/sql` + `sqlc`

This is the habit that breaks hardest. Eloquent and Doctrine let you write:

$users = User::where('active', true)
    ->with('orders')
    ->orderBy('created_at', 'desc')
    ->limit(20)
    ->get();

And you mostly stop thinking about the SQL. The ORM lazy-loads, eager-loads, and builds the query for you.

Go has ORMs (GORM, ent, Bun). Most production Go code does not use them. The pattern most teams coming from Laravel converge on is sqlc — write SQL by hand, generate typed Go functions from it, call the functions.

-- queries.sql
-- name: ListActiveUsersWithOrders :many
SELECT u.id, u.email, u.created_at
FROM users u
WHERE u.active = true
ORDER BY u.created_at DESC
LIMIT $1;

// generated by sqlc — you do not write this file
users, err := q.ListActiveUsersWithOrders(ctx, 20)
if err != nil {
    return err
}
for _, u := range users {
    // u.ID is int64, u.Email is string, u.CreatedAt is time.Time
}

The SQL lives in a file. The types live in generated Go. You can grep for every query in the codebase. No lazy loading, no N+1 hidden behind a property access, no tap and dd to inspect what the ORM did — the query is literally the file you opened.

If you've been fighting Eloquent for years over whereHas generating awful joins, this feels like leaving a loud room.

Dependency injection: the container is gone

Laravel's service container is one of its best ideas. You bind an interface, type-hint a constructor, and the framework resolves the graph for you. It feels like magic because it reads reflection metadata at runtime.

Go has no reflection-based container in the standard library. You wire your graph by hand, in main, and pass dependencies down as arguments.

func main() {
    db, err := sql.Open("postgres", os.Getenv("DATABASE_URL"))
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    queries := sqlc.New(db)
    mailer := smtp.NewMailer(os.Getenv("SMTP_URL"))

    userSvc := user.NewService(queries, mailer)
    handler := httpapi.NewHandler(userSvc)

    log.Fatal(http.ListenAndServe(":8080", handler))
}

That main is your composition root. Every dependency is visible. Nothing is auto-wired. If you want to swap the mailer for a test fake, you pass a different mailer into user.NewService.

Laravel devs usually hate this for a week and then find it restful. You stop grepping for ->bind() calls in twelve different service providers. You stop wondering whether a test is getting the real Mailer or a spy. The graph is the code in main.

Tools like wire exist to generate this wiring at compile time when the graph gets big, but the generated output is still plain func main() code you can read.

Middleware: `http.Handler` is the whole pattern

Laravel middleware looks like this:

public function handle(Request $request, Closure $next): Response
{
    if (!$request->user()) {
        return redirect('/login');
    }
    return $next($request);
}

Go's http.Handler is the same idea with one less layer of abstraction. A middleware is a function that takes a handler and returns a handler.

func RequireUser(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        uid := r.Header.Get("X-User-ID")
        if uid == "" {
            http.Error(w, "unauthorized", http.StatusUnauthorized)
            return
        }
        ctx := context.WithValue(r.Context(), userKey, uid)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

// wire the chain
handler := RequireUser(Logging(apiHandler))

No $kernel->pushMiddleware(...), no priority ordering, no group names. The chain is function composition. You can read the whole pipeline in the file where you build it.

The tradeoff: Laravel's named middleware groups and route-level middleware declarations are genuinely more ergonomic for a team of 20 people who all need to add auth to a new route. Go needs a router like chi or echo before you get that back.

Concurrency: goroutines are not workers

PHP's async story is fragmented. Fibers landed in PHP 8.1. Amphp and ReactPHP exist. Swoole and OpenSwoole give you coroutines. Most production PHP apps still dispatch long work to a queue and let Horizon or a custom worker eat it.

In Go, concurrency is a language feature. You write:

func fetchAll(ctx context.Context, urls []string) ([]string, error) {
    results := make([]string, len(urls))
    errs := make(chan error, len(urls))
    var wg sync.WaitGroup

    for i, u := range urls {
        wg.Add(1)
        go func(i int, u string) {
            defer wg.Done()
            body, err := fetch(ctx, u)
            if err != nil {
                errs <- err
                return
            }
            results[i] = body
        }(i, u)
    }
    wg.Wait()
    close(errs)

    for err := range errs {
        if err != nil {
            return nil, err
        }
    }
    return results, nil
}

No queue. No worker. No serialization. Twelve HTTP calls fan out, run concurrently on the Go scheduler, and come back.

This is the capability PHP genuinely cannot match without bolt-ons. If your current Laravel code dispatches 12 jobs and polls for completion, the Go version is one function.

The warning: go somefunc() with no coordination is how Go services leak goroutines and eat memory. Every goroutine needs a way to finish — usually a context.Context with a deadline, or a channel close, or a sync.WaitGroup. "Just spawn a goroutine" is the Go equivalent of "just fire and forget a queue job" and it causes similar classes of bugs.

Typing: PHP 8.4 is close, but not the same thing

PHP 8.4 has declare(strict_types=1), typed properties, union types, readonly classes, and asymmetric visibility. You can write PHP that looks a lot like TypeScript.

Go's type system is cruder but fully static and fully compiled. No strings to int coercion, no array as both list and map, no variadic arrays of whatever. A []User is a slice of User. A map[string]int is a map from string to int. The compiler refuses to build if the types don't fit.

What PHP has that Go doesn't: named arguments, default parameter values, optional arguments. In Go, you build a struct.

type CreateUserParams struct {
    Email    string
    Password string
    Role     string // defaults to "member" if empty
}

func CreateUser(ctx context.Context, p CreateUserParams) (*User, error) {
    if p.Role == "" {
        p.Role = "member"
    }
    // ...
}

// call site
u, err := CreateUser(ctx, CreateUserParams{
    Email: "a@b.com",
    Password: "hunter2",
})

Verbose compared to createUser(email: 'a@b.com', password: 'hunter2') in PHP. You get used to it. The payoff is that refactoring a function signature across a 200-file codebase is a compiler job, not a grep job.

Testing: the stdlib is enough

PHPUnit and Pest are mature, opinionated, and do a lot for you — data providers, mocking, snapshot testing, beautiful output. Pest in particular reads nicely.

Go's testing package is deliberately plain. No framework. No mocking library in the stdlib. Table tests are the idiom.

func TestNormalizeEmail(t *testing.T) {
    cases := []struct {
        in, want string
    }{
        {"A@B.com", "a@b.com"},
        {"  x@y.io  ", "x@y.io"},
        {"", ""},
    }
    for _, c := range cases {
        got := NormalizeEmail(c.in)
        if got != c.want {
            t.Errorf("NormalizeEmail(%q) = %q, want %q", c.in, got, c.want)
        }
    }
}

Run with go test ./.... No config file. The test is a function. Subtests, benchmarks, fuzzing, race detection, coverage — all in the stdlib or one flag away.

Teams coming from Pest usually miss the expressiveness for the first month. Then they stop noticing.

Five pitfalls PHP devs hit in their first Go week

Ignoring errors. In PHP you throw and catch. In Go, every call that can fail returns an error, and the compiler lets you drop it with _. Do not drop it. A _ = json.Unmarshal(...) is a silent data bug waiting to ship.
Sharing a struct across goroutines without a mutex. Every PHP request gets its own process. Shared state was Redis, period. A Go handler runs concurrently with every other handler, and a map you read without a lock will eventually panic under load.
Treating nil like null. A typed nil inside an interface is not equal to a plain nil. var err *MyError = nil; var e error = err; e == nil is false. This trips every PHP dev once.
Over-using packages. Laravel encourages small service classes. Go packages are heavier — each is a compilation unit and an import. A PHP app with 200 classes in 40 namespaces maps to maybe 6 Go packages, not 40.
Building a container. Someone on the team, by week two, will try to port Laravel's container to Go using reflection. Do not. Pass dependencies as arguments. The one-time refactor pain saves a year of debugging "which binding did the container resolve at 3 a.m."

What Laravel still does better

Be honest about this. Laravel has things Go will not give you back:

Artisan. php artisan make:controller, make:migration, tinker. Go has go run, go generate, and whatever CLI you build yourself. There is no REPL you can fire up to poke production data.
Eloquent for CRUD apps. If you are building an admin panel and 80% of your code is Model::where(...)->update(...), Eloquent is faster to write than any Go ORM.
Blade. Go's html/template is safe and decent. It is not Blade.
The ecosystem. Laravel Nova, Filament, Livewire, Inertia, Horizon, Forge, Vapor. The Go ecosystem has nothing like the batteries-included admin and deploy story.
Conventions. A new Laravel dev can find the UserController in any app because it is always in the same place. Go projects disagree about project layout — there is one community proposal and plenty of teams who reject it.

What Go does that PHP genuinely cannot

The list is shorter but load-bearing:

A single static binary. go build produces one file. Ship it. No PHP version, no extensions, no FPM pool config, no Composer install on the server.
Real concurrency. See the fan-out example above.
Predictable memory and GC. A Go service at 200 RPS uses 80 MB and stays there. A Laravel FPM pool at the same load needs to fork workers and eats multiples of that.
The compile step. Most of the "what does this function take" questions are answered before runtime. This is the thing that scales a codebase past 20 engineers.
Standard tooling. go fmt is not negotiable. go vet, go test -race, go test -cover are all stdlib. PHP has PHP-CS-Fixer, PHPStan, Psalm, Rector, PHPUnit, Xdebug, Pest — pick your stack and argue about it for a year.

The honest summary for a PHP dev eyeing Go

You are not learning a better PHP. You are learning a language designed by people who wanted to delete most of what PHP gives you for free, because at a certain scale the magic costs more than it saves.

The first week hurts. You will type User::find(1) into an empty file and stare at it. You will write twelve lines of explicit wiring where Laravel would have written zero. You will forget to check an error and spend 40 minutes debugging silent data.

The second week, something clicks. The wiring you wrote by hand is the wiring. The function signature is the contract. The test is a function. The server is one process. And when you push to prod, one binary goes with it.

If you want the long version of this — the full "write a production Go service from scratch" arc, with the patterns Laravel devs specifically need unlearned and the Go idioms that replace them — the Thinking in Go series is written for exactly this reader.

flowchart TD
    subgraph Laravel["Laravel container"]
        Bind[service bindings] --> Reflect[Reflection resolves<br/>dependencies]
        Reflect --> Magic[auto-wired Controller]
    end
    subgraph GoMain["Go main.go"]
        Cfg[load config] --> DB[open DB]
        DB --> Repo[NewUserRepo]
        Repo --> Svc[NewUserService]
        Svc --> Handler[NewUserHandler]
    end

If this was useful

Thinking in Go is the 2-book series built for developers coming to Go from a framework-heavy background. The Complete Guide to Go Programming walks through the language end to end — the one you want when Eloquent habits keep showing up in your Go code. Hexagonal Architecture in Go is the follow-up for when you need a real project layout.

Observability for LLM Applications is the other book, for engineers running LLM features in production (many of those services are written in Go).

Thinking in Go (series): Complete Guide to Go Programming · Hexagonal Architecture in Go
Observability for LLM Applications: Amazon · Ebook from Apr 22
Hermes IDE: hermes-ide.com — an IDE for developers shipping with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Data Validation Using Early Return in Python

Mee Mee Alainmar — Sat, 18 Apr 2026 10:36:13 +0000

While working with data, I find validation logic tends to get messy faster than expected.

It usually starts simple then a few more checks get added, and suddenly everything is wrapped in nested if statements.
That pattern works, but it doesn’t feel great to read or maintain.

That's how I learned Early return (or guard clause) pattern.

Note: In programming, return means sending a value back from a function to wherever that function was called and stopping the function’s execution right there. Meaning return acts as a checkpoint.

Think of it like saying, “I’m done; here’s my answer.”

So I tried a different approach, combining:

early return
rules defined as simple dictionaries

The result turned out surprisingly clean.

The Problem

Let's say a validation task might involve checks like:

age should be at least 18
email should contain @
user_id should be an integer

The usual way often ends up looking like this:

id="a1k29d"

if "age" in record:
    if record["age"] >= 18:
        ...

It works, but the structure quickly becomes hard to follow as more rules are added.

The Pattern

Instead of hardcoding each condition, the rules can be defined as data:

id="ab123"
rules = [
    {"field": "user_id", "type": "type", "value": int},
    {"field": "age", "type": "min", "value": 18},
    {"field": "email", "type": "contains", "value": "@"},
]

Then a single function applies these rules.


id="ab123"
def validate_record(record: dict, rules: list) -> dict:
    for rule in rules:
        field = rule["field"]

        # early return: missing field
        if field not in record:
            return {
                "status": "ERROR",
                "field": field,
                "issue": "Missing field"
            }

        value = record[field]

        # type check
        if rule["type"] == "type":
            if not isinstance(value, rule["value"]):
                return {
                    "status": "FAIL",
                    "field": field,
                    "issue": f"Expected {rule['value'].__name__}"
                }

        # minimum value
        elif rule["type"] == "min":
            if value < rule["value"]:
                return {
                    "status": "FAIL",
                    "field": field,
                    "issue": f"Must be >= {rule['value']}"
                }

        # contains (for strings)
        elif rule["type"] == "contains":
            if rule["value"] not in value:
                return {
                    "status": "FAIL",
                    "field": field,
                    "issue": f"Must contain '{rule['value']}'"
                }

    return {"status": "OK"}

Example

id="d4hf80"
record = {
    "user_id": 1,
    "age": 16,
    "email": "testemail.com"
}

result = validate_record(record, rules)
print(result)

Output:

id="d4hf80"
{
    "status": "FAIL",
    "field": "age",
    "issue": "Must be >= 18"
}

Diagram

 ┌─────────────┐
 │   Function  │
 └──────┬──────┘
        │
        ▼
 ┌─────────────┐
 │ Validate    │
 │ Input       │
 └──────┬──────┘
        │Invalid?
        ├── Yes → Return Error
        │
        ▼
 ┌─────────────┐
 │ Check Pre-  │
 │ conditions  │
 └──────┬──────┘
        │Fail?
        ├── Yes → Return Early
        │
        ▼
 ┌─────────────┐
 │ Main Logic  │
 │ Execution   │
 └──────┬──────┘
        │
        ▼
 ┌─────────────┐
 │ Return      │
 │ Success     │
 └─────────────┘

What is better about this

You can see a few things stood out after using this pattern:

The logic stays flat, no deep nesting
Rules are easy to scan and update
Adding a new validation doesn’t require touching the core function
Early return keeps the flow straightforward

It feels closer to describing what to validate instead of how to validate it step by step.

This example shows the pattern scales nicely. Running this pattern across a dataset and turning the results into a table would be a natural next step. In a way, it feels like a tiny version of larger data validation tools, just stripped down to the core idea.

For Schema Validation, Pydantic is the best no doubt for this. It ensures that the data entering the system is the right shape, type, and format. Meanwhile, Early Return pattern is to handle edge cases or invalid states immediately, preventing deeply nested if/else blocks.

Microlearning for developers: learn new concepts in 15 minutes

Tdvh yfdg — Sat, 18 Apr 2026 10:29:56 +0000

Developers are expected to keep up with an industry that never slows down. New frameworks, languages, and tools appear every few months, and falling behind even slightly can feel overwhelming. The good news is that you don't need to block out entire evenings to stay current. Platforms like SmartyMe are built around the idea that focused, short learning sessions fit naturally into a developer's lifestyle and actually produce better results than marathon study sessions.

The developer's learning problem

Technology moves fast. Every year brings new libraries, updated best practices, cloud services to explore, and paradigms to understand. What was relevant three years ago may already be considered outdated today. Keeping pace is not optional if you want to stay competitive in the job market or grow within your current role.

The bigger challenge, though, is time and energy. After a full day of writing code, debugging, attending standups, and reviewing pull requests, most developers simply don't have the mental bandwidth for a two-hour course. You open a video lecture with the best intentions, but by minute 20, you've lost focus entirely.

This pattern repeats constantly: we enroll in courses, make it through the first few modules, and then quietly abandon them when a deadline hits. According to data from online learning platforms, course completion rates often sit below 15%. That's not a motivation problem, it's a format problem.

The format of traditional online education is simply not built for developers who are already cognitively loaded. Long-form content demands sustained attention that most working professionals can't reliably offer. What the industry needs is a smarter approach to continuous education, one that respects limited time and works with human cognitive patterns rather than against them.

Why microlearning works for technical minds

Developers already think in modules. Breaking a complex problem into smaller, manageable pieces is literally a core skill of the job. So it makes complete sense that microlearning developers aligns naturally with the way technical minds are already trained to operate.

A 15-minute learning session delivers exactly one focused concept. That constraint actually helps. When the scope is defined, it's easier to stay engaged, process the material, and walk away with something concrete. There's no vague endpoint where your attention starts drifting.

The science backs this up. Research in cognitive psychology, including studies related to what's known as the "spacing effect," confirms that information absorbed in smaller chunks over repeated sessions is retained far better than material consumed in long, single sittings. Hermann Ebbinghaus's research on memory showed that spaced repetition can improve retention by up to 200%. Short sessions spaced over days are more effective than a single long session.

For developers, the accumulation adds up quickly:

5 days a week x 15 minutes = 75 minutes of focused learning
In a month, that's roughly 5-6 hours of quality study time
Each session builds on the last, reinforcing what you already know
Over a year, you can cover multiple entirely new skill areas

The key is consistency, not intensity. One concept a day, five days a week, compounds into serious expertise over time.

Beyond coding: skills developers often overlook

Strong technical skills will get you hired, but they won't always get you promoted or help you build something independently. Learning coding fundamentals is essential, but it's only part of what makes a developer genuinely effective in real-world environments.

Communication is one of the most underrated skills in tech. Can you explain a complex architectural decision to a non-technical stakeholder? Can you write documentation that another developer can actually follow? The ability to communicate clearly, both in writing and in conversation, directly affects your impact on a team.

Critical thinking shapes how you approach problems beyond syntax and logic. It's about evaluating tradeoffs, questioning assumptions, and making decisions under uncertainty. Developers who think critically write better code and make fewer costly architectural mistakes.

Here's what well-rounded developers typically invest in beyond technical knowledge:

Communication: Clearer code reviews, better documentation, smoother collaboration
Critical thinking: Better decisions during architecture planning and debugging
Logic: Understanding cognitive biases helps in UX decisions and team dynamics
Finance basics: Essential if you're considering freelance work or launching your own product
Personal productivity: Time management and focus skills directly improve output quality

Developers who invest in these areas become more than just coders. They become people their teams rely on for judgment, clarity, and leadership. That shift in value is significant, and it doesn't require a degree in business or psychology. A consistent microlearning habit covering these topics gets you there gradually and without overwhelm.

How to fit learning into a developer's day

Finding time for learning isn't about having free time. It's about using the time you already have more intentionally. Most developers have several natural gaps in their day that work perfectly for a 15-minute session.

Morning is one of the most effective windows. Before your inbox fills up and your brain is pulled in six directions, a single focused lesson with your coffee sets a productive tone. Many developers find that morning learning sticks better because the mind is fresh and there are fewer interruptions.

The lunch break is another overlooked opportunity. Scrolling social media during lunch is a default habit for many, but swapping just part of that time for a lesson is an easy upgrade. You're already stepping away from work, so the mental context switch is natural.

For those who commute, audio-based learning formats are a practical fit. Listening to a lesson on the way to the office means you arrive already having learned something, before the workday even begins. After work, lighter topics like art or finance can serve as a natural wind-down that still feels productive.

Time of day	Format that works best	Duration
Morning (pre-work)	Text or video lesson	15 min
Lunch break	Short interactive module	10-15 min
Commute	Audio lesson	15-20 min
Evening	Light reading or review	10-15 min

The trick is to anchor your learning to an existing habit. Attach the lesson to something you already do every day, and it becomes automatic rather than a decision you have to make.

Building a learning habit that survives deadlines

The most common reason developers stop learning mid-streak is a crunch period at work. When a project is on fire, learning is the first casualty. This is where the format of microlearning genuinely outperforms traditional courses.

It's hard to justify skipping a 2-hour course when you're exhausted. It's much harder to justify skipping 15 minutes. That small size is the feature, not a limitation. Even during the most intense sprint weeks, a single short lesson is almost always possible.

Streaks and progress tracking serve as powerful motivators. When you can see a visible chain of completed days, breaking it feels costly. Many learners report that the desire to maintain a streak keeps them coming back even on days when motivation is low.

The "never miss twice" rule is a practical tool for managing guilt and maintaining momentum:

Miss one day? That's fine. Life happens.
Miss two in a row? The habit starts to dissolve.
One missed day is a pause. Two is the beginning of quitting.
Resuming after a skip matters more than the skip itself.

Letting go of perfectionism around the habit is part of making it sustainable. You don't need a flawless record, you need a resilient one.

What to learn first: practical recommendations

Starting is often the hardest part, especially when the options seem endless. A useful framework is to begin with skills that immediately improve your day-to-day work, then layer in broader knowledge over time.

For most developers, logic and critical thinking offer the fastest return. These skills directly improve how you approach debugging, system design, and code review. Communication skills follow closely, since they affect how your work is perceived by others and how effectively you collaborate.

Here's a suggested learning order based on practical impact:

Logic and critical thinking - Immediately useful in problem-solving and design decisions
Communication skills - Improves code reviews, documentation, and meetings
Personal development - Helps build self-awareness and reduce cognitive biases in decision-making
Finance - Essential groundwork for freelancing or building a product
Art and History - Broader perspective that improves creative thinking and problem framing

This sequence isn't rigid. If you're already planning a freelance move, finance might belong at the top. The goal is to learn in a direction that's relevant to where you are and where you want to go.

Start small, stay consistent

You don't need to overhaul your schedule or commit to hours of study every week. What you need is 15 minutes a day and the decision to start. One lesson is one step forward, and those steps build into something real over months.

Developers understand the value of incremental progress better than most. A feature isn't built in a single commit. A codebase isn't refactored overnight. The same principle applies to skills: slow, steady progress done consistently always beats occasional bursts of effort. Start today, keep it small, and let the habit do the work.

I Built a Circle QR Code Generator — Yes, With Curved Border Text

monkeymore studio — Sat, 18 Apr 2026 10:23:52 +0000

Standard square QR codes are everywhere. But what if you want something that stands out? I built a circle QR code generator that wraps your brand message around the QR code itself—completely client-side, no server required.

Why Circle QR Codes?

A circle QR code isn't just a different shape. It's a complete visual reimagining. The circular frame with curved text creates something that looks less like a tech utility and more like a designed brand asset.

Think about where QR codes are actually used: product packaging, business cards, event posters, restaurant menus. A circle QR with "Call now" or "Visit our site" curved along the border does what a plain square code can't—it communicates while staying scannable.

The best part? It's still a real, scannable QR code. The underlying data encodes exactly like a standard QR. Only the visual presentation changed.

How the Circle QR Code Generator Works

The flow is similar to the square version, but with some interesting additions for circular rendering:

Base QR Generation with Circle Shape

The library handles the circular module pattern:

const qrCode = new QRCodeStyling({
  width: size,
  height: size,
  type: "canvas",
  shape: "circle",  // The key difference from square
  data,
  image: imageUrl,
  dotsOptions: {
    color: dotColor,
    type: "square",  // Inner modules stay square for reliability
  },
  imageOptions: {
    crossOrigin: "anonymous",
    margin: 10,
    hideBackgroundDots: true,
    imageSize: Math.max(0.01, Math.min(1, logoSize / 100)),
  },
});

This generates the QR with circular data modules—the dots themselves are arranged in rings rather than a square grid. It's a visual effect, but the encoding remains standard QR.

Circle Masking

After generating the QR, we apply a circular clip to ensure clean edges:

export function makeCircleFromCanvas(source: HTMLCanvasElement, bgColor?: string): HTMLCanvasElement {
  const size = source.width;
  const canvas = document.createElement("canvas");
  canvas.width = size;
  canvas.height = size;
  const ctx = canvas.getContext("2d")!;

  // Draw circular background if color is set
  if (bgColor && bgColor !== "transparent") {
    ctx.beginPath();
    ctx.arc(size / 2, size / 2, size / 2, 0, Math.PI * 2);
    ctx.closePath();
    ctx.fillStyle = bgColor;
    ctx.fill();
  }

  // Clip to circle
  ctx.beginPath();
  ctx.arc(size / 2, size / 2, size / 2, 0, Math.PI * 2);
  ctx.closePath();
  ctx.clip();

  // Draw source inside the circle
  ctx.drawImage(source, 0, 0, size, size);
  return canvas;
}

The function creates a new canvas, optionally fills a circle background, then clips and draws the QR inside. Simple, but produces that clean circular badge look.

Curved Border Text — The Interesting Part

This is where it gets creative. Drawing text along a curve isn't built into canvas—here's how it's done:

export function addCircularTextToCanvas(
  canvas: HTMLCanvasElement,
  topText: string,
  bottomText: string,
  color: string,
  fontSize: number,
  bgColor?: string,
  contentScale = 0.85
): HTMLCanvasElement {
  const padding = Math.ceil(fontSize * 1.5);
  const qrSize = canvas.width;
  const newSize = qrSize + padding * 2;
  const newCanvas = document.createElement("canvas");
  newCanvas.width = newSize;
  newCanvas.height = newSize;
  const ctx = newCanvas.getContext("2d")!;

  // Clear background
  ctx.clearRect(0, 0, newSize, newSize);

  // Draw circular background
  if (bgColor && bgColor !== "transparent") {
    ctx.beginPath();
    ctx.arc(newSize / 2, newSize / 2, newSize / 2, 0, Math.PI * 2);
    ctx.closePath();
    ctx.fillStyle = bgColor;
    ctx.fill();
  }

  // Clip to circle
  ctx.beginPath();
  ctx.arc(newSize / 2, newSize / 2, newSize / 2, 0, Math.PI * 2);
  ctx.closePath();
  ctx.clip();

  // Scale QR to fit inside with room for text
  const scaledQrSize = qrSize * contentScale;
  const offset = (newSize - scaledQrSize) / 2;
  ctx.drawImage(canvas, offset, offset, scaledQrSize, scaledQrSize);

  // Draw curved text
  ctx.save();
  ctx.translate(newSize / 2, newSize / 2);
  ctx.font = `bold ${fontSize}px sans-serif`;
  ctx.fillStyle = color;

  const radius = scaledQrSize / 2 + fontSize * 0.8;

  if (topText) {
    drawArcText(ctx, topText, radius, -Math.PI / 2, false);  // Top text reads left-to-right
  }
  if (bottomText) {
    drawArcText(ctx, bottomText, radius, Math.PI / 2, true);  // Bottom text inverted
  }

  ctx.restore();
  return newCanvas;
}

The algorithm:

Create a larger canvas with padding for text
Draw the circular background and clip to it
Scale the QR code down slightly to make room
Calculate a text radius (slightly outside the QR)
Draw top text counterclockwise, bottom text clockwise

The Arc Text Math

This is the core curved text algorithm:

function drawArcText(
  ctx: CanvasRenderingContext2D,
  text: string,
  radius: number,
  centerAngle: number,
  invert: boolean
) {
  ctx.save();
  ctx.textAlign = "center";
  ctx.textBaseline = "middle";

  // Split and measure each character
  const chars = text.split("");
  const widths = chars.map((c) => ctx.measureText(c).width);
  const totalWidth = widths.reduce((a, b) => a + b, 0);

  // Calculate starting angle based on text width
  const direction = invert ? -1 : 1;
  let currentAngle = centerAngle - direction * (totalWidth / 2) / radius;

  // Draw each character rotated to follow the arc
  for (let i = 0; i < chars.length; i++) {
    const w = widths[i];
    const angle = currentAngle + direction * (w / 2) / radius;

    ctx.save();
    ctx.translate(Math.cos(angle) * radius, Math.sin(angle) * radius);
    ctx.rotate(angle + (invert ? -Math.PI / 2 : Math.PI / 2));
    ctx.fillText(chars[i], 0, 0);
    ctx.restore();

    currentAngle += direction * (w / radius);
  }

  ctx.restore();
}

The key insight: instead of trying to draw curved text directly, rotate each character individually to follow the arc. For a character at angle θ on a circle of radius r, the rotation needed is θ + π/2 (for top) or θ - π/2 (for bottom).

User Controls

Similar to square QR, but with border text options:

// Circle border text inputs
const [circleBorderTopText, setCircleBorderTopText] = useState("");
const [circleBorderBottomText, setCircleBorderBottomText] = useState("");
const [circleBorderTextColor, setCircleBorderTextColor] = useState("#000000");
const [circleBorderTextSize, setCircleBorderTextSize] = useState(16);

Users can specify text for the top and bottom of the circle, customize color and size. Typical use cases:

Top: brand name or promo ("SCAN ME")
Bottom: website or phone ("example.com")

Real-World Usage

This tool shines in practical scenarios:

Use Case	Top Text	Bottom Text
Restaurant menu	"MENU"	"Scan to order"
Business card	"CALL NOW"	"+1-555-0123"
Product packaging	"VIDEO"	"Watch demo"
Event flyer	"REGISTER"	"example.com/events"

Batch Processing

Like the square version, supports multiple URLs at once:

const handleDownloadAll = async () => {
  const zip = new JSZip();
  qrCodes.forEach(({ url, name }) => {
    const base64Data = url.replace(/^data:image\/png;base64,/, "");
    zip.file(`${name}.png`, base64Data, { base64: true });
  });
  const content = await zip.generateAsync({ type: "blob" });
  saveAs(content, "qrcodes.zip");
};

Perfect for printing shops or marketing teams generating dozens of QR codes at once.

Try It Yourself

Check out the live generator at Circle QR code generator. Upload your logo, add some border text, pick your colors—generates instantly in your browser.

Your content never goes to a server. What's entered stays on your device, processed entirely locally.

Where Do Interfaces Live? IVP Answers What SOLID Cannot

Yannick Loth — Sat, 18 Apr 2026 10:23:51 +0000

If you know a software architect, a professor who teaches SOLID, or anyone who takes design principles seriously — I'd appreciate you sharing this with them.

Two companion papers established the structural biconditional between ISP and per-client DIP (paper 1) and proved that Martin's packaging principles are jointly unsatisfiable for the resulting configuration — specifically, CCP and REP cannot both be satisfied (paper 2).

This third paper applies the Independent Variation Principle (IVP) to the same five-element setup and obtains two results from a single application.

Result 1: DIP → ISP, formalized in IVP terms

The first companion paper proved the structural biconditional: ISP ↔ per-client DIP. This paper formalizes the forward direction (DIP implies ISP) in IVP terms. Each interface slice shares its client's change-driver assignment. The IVP biconditional (elements co-reside iff they share driver assignments) closes the proof: the slices are disjoint because the clients belong to different modules. IVP adds formalism to one direction of a valid structural biconditional; it is useful here but not indispensable.

Result 2: A unique modularization

This is where IVP is essential.

Five elements: provider A, clients B and C, interface slices I_B and I_C. Three distinct driver assignments: {γ_ord} for B and I_B, {γ_rep} for C and I_C, {γ_ord, γ_rep, γ_impl} for A.

IVP-3 (elements with different driver assignments cannot co-reside) and IVP-4 (elements with identical driver assignments must co-reside) together admit exactly one partition:

Module B: B and I_B (both driven by γ_ord)
Module C: C and I_C (both driven by γ_rep)
Module A: A alone (composite: γ_ord, γ_rep, γ_impl)

Each interface slice lives with its client — not as a design preference, but as a consequence of the axioms applied to the causal structure of change.

What this settles

The packaging paper showed that eight principles — DIP, ISP, and six packaging principles (REP, CCP, CRP, ADP, SDP, SAP) — give three incompatible answers (with client, with provider, own module) and no meta-criterion for choosing. IVP eliminates two of the three by deriving the answer from Γ.

If two engineers disagree about where I_B belongs, the disagreement traces to a disagreement about Γ — about what can cause I_B to require modification. That is an epistemic question about the system, not a design preference. IVP converts a judgment call into a falsifiable claim.

What IVP's placement produces

The paper verifies that the unique IVP partition delivers:

Change isolation: a change to γ_ord modifies Module B (containing B and I_B) and nothing else. Module C is untouched.
No shotgun surgery: the pathology that scattered interface placement creates is eliminated by construction.
Acyclic dependencies in the provider–client subgraph: A depends on Module B and Module C (for the interface specifications it implements). Neither client module depends on A. No cycle exists in this subgraph.
Independent deployability: each module can be released independently.

The paper

"The Interface Segregation Principle Is a Corollary of the Dependency Inversion Principle --- A Formal Proof via the Independent Variation Principle"

Paper: https://zenodo.org/records/19641242
Companion 1 (ISP = DIP, structural): https://zenodo.org/records/19633560
Companion 2 (packaging unsatisfiability): https://zenodo.org/records/19636635

Companion 1 dev.to article: SOLID: ISP Is Just DIP Applied Twice
Companion 2 dev.to article: SOLID's Packaging Principles Are Jointly Unsatisfiable

The first paper showed ISP and DIP produce the same structure. The second showed SOLID's principles can't agree on where to put it. This one shows IVP can.

If you think the unique partition is wrong — that I_B belongs somewhere other than with B — I'd like to hear the argument. What change driver does I_B respond to, if not B's?

Claude + Groq Hybrid LLM — AI University Memory Agent

kanta13jp1 — Sat, 18 Apr 2026 10:22:32 +0000

Claude + Groq Hybrid LLM — AI University Memory Agent

After each learning session in AI University, a Memory Agent automatically builds a structured learner profile — weak providers, strong providers, preferred learning style. Next session, quizzes are personalized based on that profile.

The trick: two models, two jobs — Claude Sonnet for deep profile extraction, Groq Llama for real-time quiz scoring.

Architecture

Session ends
  → learner.update_profile (Edge Function)
    → Claude Sonnet → structured JSON profile
      → UPSERT into ai_university_learner_profiles

Quiz answer submitted
  → quiz.evaluate (Edge Function)
    → Groq Llama 3.3 70B → JSON score {result, confidence}
    → fallback: string match

Memory Agent — Claude Extracts the Profile

// supabase/functions/ai-hub — learner.update_profile
const prompt = `Extract a structured learner profile from this session data.
Session summary: ${sessionSummary}
Score data: ${JSON.stringify(scores).slice(0, 2000)}
Return JSON: {"weak_providers":["..."],"strong_providers":["..."],"preferred_style":"visual|text|voice","insights":"..."}`;

const claudeResp = await fetch("https://api.anthropic.com/v1/messages", {
  body: JSON.stringify({
    model: "claude-sonnet-4-6",
    max_tokens: 512,
    messages: [{ role: "user", content: prompt }],
  }),
});

// Strip code fences before parsing
const rawText  = claudeData.content[0].text;
const profile  = JSON.parse(rawText.replace(/```
{% endraw %}
json\n?|\n?
{% raw %}
```/g, "").trim());

Save to Supabase:

await admin.from("ai_university_learner_profiles").upsert({
  user_id,
  weak_providers:  profile.weak_providers  ?? [],
  strong_providers: profile.strong_providers ?? [],
  preferred_style: profile.preferred_style  ?? "text",
  profile_json:    profile,
  total_sessions:  (existing?.total_sessions ?? 0) + 1,
}, { onConflict: "user_id" });

Quiz Evaluator — Groq Scores Answers Fast

// quiz.evaluate — Groq Llama 3.3 70B, JSON mode
const groqResp = await fetch("https://api.groq.com/openai/v1/chat/completions", {
  body: JSON.stringify({
    model: "llama-3.3-70b-versatile",
    max_tokens: 100,
    temperature: 0,
    response_format: { type: "json_object" },  // guaranteed JSON
    messages: [{
      role: "user",
      content: `Question: ${question}\nExpected: ${correctAnswer}\nUser: ${userAnswer}
Score: {"result":"correct|incorrect|partial","confidence":0-100}`,
    }],
  }),
}).catch(() => null);

// Groq failure → fallback to exact string match
if (!groqResp || !groqResp.ok) {
  const match = userAnswer.trim().toLowerCase() === correctAnswer.trim().toLowerCase();
  return json({ result: match ? "correct" : "incorrect", confidence: 100, fallback: true });
}

Why Two Models?

Task	Model	Reason
Learner profile extraction	Claude Sonnet 4.6	Complex reasoning, structured JSON quality
Quiz scoring	Groq Llama 3.3 70B	Low latency, high volume, free tier

Claude runs once at session end. Groq runs on every quiz answer. Matching the model to the task cuts costs without sacrificing quality.

DB Schema

CREATE TABLE ai_university_learner_profiles (
  user_id          uuid PRIMARY KEY REFERENCES auth.users,
  weak_providers   text[] DEFAULT '{}',
  strong_providers text[] DEFAULT '{}',
  preferred_style  text   DEFAULT 'text',
  profile_json     jsonb  DEFAULT '{}',
  total_sessions   int    DEFAULT 0,
  updated_at       timestamptz DEFAULT now()
);

Key Takeaways

Assign models by strength — Claude for deep analysis, Groq for speed
response_format: json_object — eliminates JSON parse errors from Groq
Strip code fences — Claude wraps JSON in json → strip before JSON.parse
Always fallback — Groq outage shouldn't break quiz scoring

Building in public: https://my-web-app-b67f4.web.app/

FlutterWeb #Supabase #buildinpublic #LLM #FlutterTips

ServiceKit V2 — The Async Service Locator for Unity

Paul Stamp — Sat, 18 Apr 2026 10:21:06 +0000

I just shipped V2 of ServiceKit, my lightweight dependency management package for Unity. Before I get into what's new, I want to address the thing some of you are already typing into the comments: yes, ServiceKit is a service locator. On purpose. I think that's the right shape for Unity, and V2 is where I've stopped hedging about it.

Why a service locator, not a "proper" DI framework

The "service locator is an anti-pattern" mantra largely comes from enterprise .NET, where you control object construction, processes are long-lived, and writing an installer for two hundred services is routine. Unity isn't that world. Unity instantiates your components for you, so something has to do late-binding resolution regardless of what you call it. The heavyweight DI frameworks respond to that by adding more ceremony, not less. Contexts, scopes, installers, factories, binding chains. Adopting Zenject is a lifestyle decision. You don't use Zenject so much as restructure your project around it.

Service locators are the opposite trade. Low friction, no framework imposed on the rest of your codebase, incremental. You can drop ServiceKit in to solve one pain point without rewriting everything around it.

The two classic critiques of service locators are hidden dependencies and async timing. V2 is where I've tried to put both to bed.

Hidden dependencies, un-hidden

[InjectService] on fields surfaces the dependency graph at the class level. It's not quite a constructor signature, but it's visible to code review, tooling, and the new Roslyn analyzers. That covers most of the "you can't see what a class needs" complaint.

[Service(typeof(IPlayerController))]
public class PlayerController : ServiceKitBehaviour, IPlayerController
{
    [InjectService] private IPlayerService _playerService;
    [InjectService] private IAudioService  _audioService;

    protected override void InitializeService()
    {
        _playerService.LoadPlayer();
    }
}

Two compile-time analyzers ship in V2:

SK003 flags [Service(typeof(IFoo))] attributes on classes that don't actually implement IFoo.
SK005 catches ServiceKitBehaviour subclasses that forget base.Awake(). Both catch the kind of bug that's annoying to hunt down at runtime.

Async resolution, the real headache fixed

This is the one I'm most pleased with. The traditional service locator's real failure mode in Unity isn't philosophical, it's the timing problem. You ask for IAudioService in Awake on the wrong GameObject, it isn't registered yet, you get null, and you're debugging initialisation order for three hours.

V2 resolves this cleanly. Field injection is a single await:

// V1
await _serviceKit.Inject(this)
    .WithErrorHandling()
    .WithTimeout()
    .ExecuteWithCancellationAsync(token);

// V2
await _serviceKit.InjectAsync(this, destroyCancellationToken);

And under the hood there's a new atomic 3-state resolution primitive that performs registration and readiness checks inside a single lock:

var status = locator.TryResolveService(typeof(IMyService), out var service);
// ServiceResolutionStatus.Ready
// ServiceResolutionStatus.RegisteredNotReady
// ServiceResolutionStatus.NotRegistered

Previously, consumers had to check registration, then readiness, then fetch, giving the world three chances to change between steps. V2 closes that window.

Alongside that: task forwarding in GetServiceAsync now happens inside locks, double-registration is blocked with Interlocked operations, and circular-dependency detection uses types rather than string comparison. Less likely to bite you in production.

Generics out, attributes in

The other big ergonomic change. V1 asked you to inherit from ServiceKitBehaviour<T>, which got painful fast when your own classes needed generics too. V2 replaces that with a plain base class and an explicit attribute:

// V1
public class AudioManager : ServiceKitBehaviour<IAudioService>, IAudioService { }

// V2
[Service(typeof(IAudioService))]
public class AudioManager : ServiceKitBehaviour, IAudioService { }

Registration intent is declarative instead of tangled in type parameters, and abstract base classes can keep their own generics without fighting ServiceKit's.

Migrating from V1

Mostly mechanical: drop the generic parameter on ServiceKitBehaviour, add [Service(typeof(IYourInterface))], and replace builder chains with InjectAsync(this, token). The README has the full migration guide, including how to handle abstract base classes that mix ServiceKit generics with their own.

Try it

Install via Package Manager → Add package from git URL:

https://www.pkglnk.dev/servicekit.git

Repo and docs: github.com/PaulNonatomic/ServiceKit

MIT licensed. Issues and PRs welcome. If you think I'm wrong about the locator vs DI trade-off, I want to hear it. V2 is the foundation I want to build on, so I'm happy to argue about it.

Rust vs Go for AI Infrastructure in 2026: Here's What the Benchmarks Actually Say

Gabriel Anhaia — Sat, 18 Apr 2026 10:18:05 +0000

Book: Observability for LLM Applications · Ebook from Apr 22
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

You have already picked Python for your model serving. The question in 2026 is what sits in front of it: the gateway, the router, the span collector, the streaming layer. This is not a research question. You benchmark it.

Most of the Rust-vs-Go argument on HackerNews treats the comparison like a team sport. Rust-shaped people point at the Techempower numbers and declare victory. Go-shaped people point at Kubernetes and declare victory. Neither side is answering the question you actually have, which is whether the language delta matters on the specific workloads that sit between a user and a model in a production AI product.

The honest answer is narrower than either camp admits. Rust wins raw throughput and tail latency. Go wins developer velocity, deployment story, and the mental overhead of shipping something that has to be read by a rotating team next year. And on the workloads AI infrastructure actually runs, the gap is smaller than the synthetic benchmarks suggest.

This post walks through the five workloads that matter: LLM gateway throughput, token streaming, OpenTelemetry span processing, vector similarity hot paths, and prompt templating under concurrency. Where a public benchmark exists, it is cited. Where one does not, the text says "rough estimate" and explains what it is estimating from. Nothing is made up.

The workloads, and why these five

The serving layer of an AI product is not one workload. It is at least five.

Gateway / reverse proxy — accepts a request, checks a key, picks a vendor, forwards, streams back.
Token streaming — long-lived SSE or WebSocket connections, many concurrent, each trickling bytes.
OpenTelemetry span processing — ingest, batch, sanitize, forward spans at high cardinality.
Vector similarity hot path — the per-query CPU work around an ANN index, plus hybrid-search reranking.
Prompt templating under concurrency — string assembly, tokenization, context packing on every request.

Training, fine-tuning, and inference kernels are not on this list. Those are Python's home and no sane person is porting them. The argument here is about the tier above the model.

Gateway throughput

The Techempower benchmarks are the most cited public numbers for web-framework throughput. On the plaintext and JSON serialization rounds (latest 2024 results at techempower.com/benchmarks), Rust frameworks like actix-web and axum consistently sit near the top of the table. Go frameworks like fiber and the stdlib net/http sit further down, often at 50–70% of the top Rust result, depending on the round and the hardware.

That is real. It is also not your workload.

A Techempower "plaintext" round measures how many times per second a process can respond to a noop HTTP request. An LLM gateway spends the overwhelming majority of its wall time waiting on an upstream model call that takes 800–2000ms. The local CPU cost of the request terminator is a rounding error against that. If your gateway can do 50k noop RPS in Rust vs 25k in Go, but every real request spends 1200ms in Anthropic, the throughput you observe in production is bottlenecked by connection pool size and upstream concurrency, not by which language terminated the TLS.

The place the language gap actually bites is the tail. Go's GC is excellent, but it is a GC. Rust has no stop-the-world pauses because it has no GC. For a p99.9 SLO measured in milliseconds, Rust's tail is flatter. Rough estimate: in a gateway doing 5k RPS with streaming fanout, you will see Go tail latency wobble 2–5ms under GC pressure; Rust will not. For most AI products, where model latency dwarfs everything, this does not matter. For frontier products with hard tail-latency commitments, it does.

Verdict on gateways. Go is fine. Rust is better if tail latency is a product requirement. The choice is dominated by the team, not the language.

Token streaming

This is the one most people get wrong about both languages.

The workload is: one process holding N concurrent long-lived connections, each streaming a slow byte trickle from an upstream model to a downstream client. Memory per connection matters. Context-switch cost matters. The C10K problem, basically, applied to SSE.

Go was designed for this. Goroutines cost ~2KB at start, the scheduler multiplexes them onto OS threads for you, and the stdlib net/http handles SSE without ceremony. You can hold 10k concurrent streaming connections on a single 2-core 4GB container and not break a sweat. This is well-documented in Cloudflare's engineering posts about their Go-based proxies (blog.cloudflare.com) and in Discord's writeup of their Go-to-Rust migration for a specific service (discord.com/blog/why-discord-is-switching-from-go-to-rust).

Rust does this too, via tokio. The async story is more explicit, the compile times are worse, and the abstraction cost of pinning, borrow checking across await points, and lifetime management in a connection struct is real. You pay for it once, in developer time. Rough estimate: a mid-level engineer building a streaming SSE proxy hits a working first cut in Go in a day, and in Rust in two to three days. The Rust version, once it compiles, will use less memory per connection. Not an order of magnitude less. Call it 30–50% less, heavily dependent on what you pack into each connection's state.

The Discord post is the honest reference point here. They switched one service from Go to Rust because a specific GC pause in a specific code path was hurting a specific user-facing metric. They did not switch their whole stack. They would not have switched at all if the metric had been less sensitive. Read the post; it is a good lesson in what the language delta actually buys you.

Verdict on streaming. Go is the default. Rust is the answer when you have measured a GC pause in the streaming path and it is hurting something you care about. If you have not measured it, you do not have the problem.

OpenTelemetry span processing

The reference OpenTelemetry Collector is Go (open-telemetry/opentelemetry-collector). There is also a Rust collector effort (open-telemetry/opentelemetry-rust) but it is not a drop-in replacement for the Go collector; it is the Rust SDK, not the full collector pipeline.

The Collector is the span-processing workload in practice. It ingests OTLP over gRPC, runs spans through a chain of processors (batch, filter, attribute sanitization, sampler), and forwards them to exporters. Throughput depends on the processor chain more than the language.

Published numbers from the collector project's own benchmarks (github.com/open-telemetry/opentelemetry-collector/tree/main/testbed) show the Go collector sustaining spans in the tens-of-thousands-per-second range on modest hardware with a realistic processor chain. Rough estimate: a hand-written Rust equivalent with a similar processor chain would gain 30–60% on raw throughput, mostly from zero-copy deserialization and less GC pressure. That is meaningful if you are ingesting a million spans per second. It is a rounding error if you are ingesting ten thousand.

The other side of this workload is memory. A span processor that batches in-memory before flushing has to decide how much to hold. The Go collector's GC behavior under bursty ingest has been the subject of multiple issues and tuning guides. Rust's memory story is more predictable. Again, rough estimate: at steady state the Rust version will sit at maybe 40–60% of the Go version's RSS, under the same ingest rate, with a flatter variance.

Verdict on OTel. Ship the Go collector unless your span ingest is genuinely at hyperscale. Most teams who think they are there, are not. If you are, Rust is the right answer and there are people writing the plumbing for it.

Vector similarity hot path

The top vector databases are written in the languages you would expect. Qdrant is Rust (github.com/qdrant/qdrant). Weaviate is Go (github.com/weaviate/weaviate). Milvus is Go with C++ kernels (github.com/milvus-io/milvus). LanceDB is Rust.

The benchmark question here is split. The per-query ANN search kernel is SIMD-heavy numeric code. Rust and C++ both have excellent SIMD stories. Go's SIMD story is getting better via the assembler and the recently proposed simd package, but the ecosystem is behind. If the workload is dominated by the ANN kernel — a high-QPS search against a large index, no reranking, no filtering — Rust will beat Go on per-query latency, probably by a factor of 2–3x on a cold kernel. That is a rough estimate based on the consistent pattern in numeric-code microbenchmarks.

But most production RAG workloads are not dominated by the ANN kernel. They are dominated by the glue: embedding the query, filtering by metadata, calling the vector DB over a network, reranking with a cross-encoder, assembling the final context. The ANN search is frequently less than 20% of the total latency. Go is fine for the glue. Rust is not required.

The ANN-benchmarks project (github.com/erikbern/ann-benchmarks) publishes standardized numbers across vector indices. The Rust-implemented indices (Qdrant's HNSW, for example) consistently post competitive recall/QPS curves. So do the Go-based ones. The choice of vector DB should turn on the feature set and the operational story, not the implementation language.

Verdict on vector search. The engine you pick is usually not the one you write. Rust is the right answer if you are building the engine. Go is the right answer if you are building the surrounding product.

Prompt templating under concurrency

This is the least-discussed workload and the one most teams under-engineer.

On every request, the gateway has to: load a template, interpolate user variables, sanitize, tokenize for budget checks, pack context windows, serialize to the vendor's JSON schema. That is a pile of string work, often running in the critical path. Under concurrent load, the allocator matters.

Go's garbage collector has to deal with the string churn. Rust has no GC; String allocations go through the normal allocator and drop when they go out of scope. In microbenchmarks of pure string assembly, Rust typically runs 1.5–3x faster than Go, with a flatter memory profile. Rough estimate from public Criterion vs testing.B numbers in various language-shootout repos; these are not AI-specific but the pattern holds.

What this translates to in a real gateway is a few milliseconds of CPU time per request, and a few hundred bytes of transient allocation. At 1k RPS, both languages handle it without breathing hard. At 50k RPS with complex template logic and heavy context assembly, you might measure the difference in p99. You are unlikely to measure it in p50.

For tokenization specifically, both ecosystems have production-grade wrappers around the same tiktoken and tokenizers native libraries. github.com/tiktoken-go/tokenizer on the Go side and tiktoken-rs on the Rust side both call into the same optimized Rust kernels underneath. The language at the call site is not the bottleneck.

Verdict on templating. Not a language-choice workload. Write it carefully in either language. If you are allocating on every request instead of reusing buffers, your problem is not Go vs Rust.

What this actually adds up to

Five workloads, five verdicts, and a pattern.

Rust wins when:

Tail latency under 5ms is a product requirement.
Span ingest or streaming is at hyperscale (hundreds of thousands per second sustained).
You are writing the CPU-bound kernel yourself — a vector index, a rerank scorer, a SIMD-heavy filter.
The operational story can absorb longer compile cycles and a smaller hiring pool.

Go wins when:

The team needs to onboard new engineers quickly.
The deployment story is Kubernetes-heavy, because the Go-in-Kubernetes ecosystem is decisive.
Most of the real latency is in the upstream model call, which describes every AI product that uses a hosted LLM.
You are building the surrounding product, not the engine.

The HackerNews framing of this debate, that Rust is objectively superior and Go is what happens when you pick the inferior option, does not survive contact with the workloads. On an LLM gateway spending 95% of its wall time in an Anthropic call, the 2x raw-throughput advantage of Rust is a rounding error. On a streaming proxy that has to be rewritten by a new hire in six months, the compile-time advantage of Go is the product feature.

There is also a middle answer that a lot of teams end up at: Go for the serving layer, Rust for the single hottest path inside it, bridged by gRPC or an FFI call. Discord's migration is one shape of this. The OpenTelemetry project's two-collector approach is another. The right question is not "which language" but "which workload, and is the language the bottleneck."

Pick the one that matches your team and your failure modes. Measure before you migrate. If your production evidence does not show GC pauses in the path you care about, you do not have a Go-to-Rust problem. If it does, you have a migration to plan.

The benchmarks tell you what the language can do. Your traces tell you whether the language is the thing to fix.

If this was useful

The observability side of this story, the traces, the tail-latency histograms, the span attributes that tell you whether you have a language problem or an upstream problem, is the subject of the book I just finished. It is language-agnostic on the surface and has examples in Go. If you are picking a stack for the AI serving layer and want a field guide for instrumenting whichever one you pick, that book is the most direct thing I can point you at.

Observability for LLM Applications — Amazon — paperback live, ebook from Apr 22.
Thinking in Go (2-book series) — Complete Guide to Go + Hexagonal Architecture in Go.
Hermes IDE — hermes-ide.com — an IDE built for developers who ship with Claude Code and other AI coding tools.
More writing — xgabriel.com and GitHub.

From $50K/Year of Datadog to $0/Year of Self-Hosted Observability: The Migration Every Team Is Doing in 2026

Gabriel Anhaia — Sat, 18 Apr 2026 10:17:40 +0000

Book: Observability for LLM Applications — paperback and hardcover on Amazon · Ebook from Apr 22
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Your Datadog bill crossed $50K in March. Your finance team flagged it. Your engineering team has been eyeing the LLM traces line on the invoice for a year. The number on that line is not the problem on its own. The problem is the slope. Spans are growing faster than users, because every agent call fans out into four tool spans and a retrieval span and an embedding span, and the invoice tracks the fan-out.

Here is the 2026 migration path teams are actually taking, and the one invisible cost that keeps some of them paying.

Why the bill grew that fast

An LLM feature looks cheap at the model layer and expensive at the telemetry layer. A single user message to an agent produces, on a normal day, one chat span, three to five execute_tool spans, a retrieval span with a half dozen child embeddings, and a parent span to hold them together. Ten spans per user turn is a floor, not a ceiling.

Datadog's APM list price, as documented on their pricing page, is $31 per host per month for Pro and $40 for Enterprise, plus $1.27 per million ingested spans and $2.55 per million indexed spans at their published rates. Datadog LLM Observability is a separate SKU that meters per LLM span. When a product that was billed at one span per request starts emitting ten, the invoice grows tenfold without a single new user arriving.

The teams I have talked to over the last six months all describe the same curve. A fintech that shipped an agent in Q3 2025 watched their Datadog bill go from $18K to $62K per year across two quarters. A European SaaS vendor hit an $80K renewal quote and asked their platform team for an alternative. None of this is Datadog doing anything wrong. It is a pricing model designed for HTTP services meeting a workload that emits an order of magnitude more spans per unit of user value.

The stack teams are migrating to

The 2026 self-hosted stack is narrow and boring in a good way. Three containers, one afternoon, zero SaaS contracts.

OpenTelemetry Collector ingests OTLP over gRPC or HTTP, batches, drops what you do not need, and forwards.
ClickHouse stores the spans. Columnar, compressed, queryable in SQL.
Grafana reads ClickHouse via the official datasource plugin and draws the dashboards.

Optional fourth container: Langfuse, which was acquired by ClickHouse in January 2026. Langfuse already ran on ClickHouse as its trace store, so the acquisition is a clarification of the architecture rather than a change to it. OSS Langfuse is MIT, runs in its own container, and points at the same ClickHouse instance the Collector writes to. You get a polished trace explorer on top of your own data.

The reason this shape dominates 2026 is OpenTelemetry's GenAI semantic conventions v1.37, which stabilized late 2025 and is now emitted natively by OpenAI's Python SDK, Anthropic's SDK, the Traceloop SDK, and by Datadog LLM Observability itself. Every tool in the category reads the same wire format. Migration is a DNS change, not a rewrite.

The Collector config

This is the whole pipeline. OTLP in, ClickHouse out.

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    timeout: 5s
    send_batch_size: 10000
  # Keep every error, keep every slow trace,
  # sample 5% of the rest. Saves 80% of volume.
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: sample
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

exporters:
  clickhouse:
    endpoint: tcp://clickhouse:9000?dial_timeout=10s
    database: otel
    username: otel
    password: ${env:CH_PASSWORD}
    traces_table_name: otel_traces
    create_schema: true
    ttl: 720h
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [clickhouse]

Two things worth calling out. The Collector you want is otel/opentelemetry-collector-contrib, not the base distribution. The ClickHouse exporter lives in contrib only. Second, tail_sampling is what makes this stack financially viable at scale. You keep every error, every slow trace, and a representative sample of the rest. The rows you drop are the ones you would not have looked at anyway.

The Grafana datasource

File-based provisioning. Drop one YAML under grafana/provisioning/datasources/ and the instance boots with the datasource wired.

# grafana/provisioning/datasources/clickhouse.yaml
apiVersion: 1
datasources:
  - name: ClickHouse
    type: grafana-clickhouse-datasource
    access: proxy
    uid: clickhouse-otel
    isDefault: true
    user: otel
    jsonData:
      host: clickhouse
      port: 9000
      protocol: native
      defaultDatabase: otel
    secureJsonData:
      password: ${CH_PASSWORD}

Install the grafana-clickhouse-datasource plugin via GF_INSTALL_PLUGINS in the Grafana container's env, and the datasource is ready on first boot. One query later, you are drawing token spend by model.

The cost math, with real numbers

Two workloads. Both assume 30 days of retention, prompts and completions captured, tail sampling at 5% for the self-hosted path.

Workload A: 10M spans per month. This is a small-to-medium LLM product. One agentic feature, a few thousand active users, moderate tool use.

Line item	Datadog (list)	Self-hosted
Ingest / host	~$12,700 ingest + ~$4,800 host (10 hosts Pro)	$0
LLM Obs per-span	~$9,000 (metered)	$0
Storage	included	~$60/mo (EBS)
Compute	included	~$140/mo (one `m7i.2xlarge`)
Backups to S3	included	~$15/mo
Annual	~$312,000	~$2,600

Datadog list prices are public. Real Datadog bills at this volume are usually negotiated 20–40% below list, which gets the annual closer to $180K–$250K. Even at the discounted number, the gap is two orders of magnitude.

Workload B: 100M spans per month. Larger product, multiple agent surfaces, heavy tool use.

Line item	Datadog (list)	Self-hosted
Ingest / host	~$127K ingest + ~$20K host (40 hosts)	$0
LLM Obs per-span	~$90K (metered)	$0
Storage	included	~$400/mo (EBS)
Compute	included	~$900/mo (three-node CH cluster)
Backups + egress	included	~$150/mo
Annual	~$2.85M	~$17.5K

The self-hosted numbers come from teams I have talked to running roughly this shape at AWS list prices, with tail sampling doing most of the heavy lifting. Without tail sampling, you push 20x the data into ClickHouse and the storage line grows, but the compute line stays remarkably flat because ClickHouse compresses spans at 8–12x on the wire.

You will notice the self-hosted column has no "Datadog salesperson discount." It also has no renewal cycle. The invoice is from AWS, and the AWS invoice does not care whether your span volume doubled this quarter.

What Datadog still does better

The dollar column is one slide. A complete migration plan has three more.

Alert routing and PagerDuty integration. Datadog has fifteen years of accumulated alerting UX. Monitor composition, flaky alert suppression, the entire library of pre-built integrations with PagerDuty, Opsgenie, Slack with context-aware message formatting. Grafana Alerting is competent in 2026 and covers the 80% case, but the last 20% — multi-condition composite monitors with suppression windows and PagerDuty context payloads — is where Datadog still wins. Budget a week of your platform team's time to port alerts, and a quarter of maintenance to get parity on the sharp-edged cases.

SaaS UX for non-engineers. Your product manager and your CFO both have Datadog logins. They will not have ClickHouse logins. Grafana is the replacement surface for most of what they were looking at, but Grafana dashboards optimized for engineers are not the same artifact as Datadog dashboards optimized for execs. You will build two sets of dashboards in Grafana, one per audience, and it will be more work than you estimated.

The ecosystem of integrations. Datadog has agents for every piece of infrastructure you have ever deployed. Every new AWS service ships with Datadog support in its first quarter. Self-hosting means you own the instrumentation for anything outside the OpenTelemetry contrib repo, and the contrib repo is good but not exhaustive.

Incident retrospective tooling. Datadog's Watchdog and anomaly detection have real signal. Grafana's built-in anomaly detection is weaker. Teams that migrate and still want this end up adding a Grafana ML plugin or a separate Metrics Advisor pipeline, which is another stateful service to run.

Those are the things that keep some teams paying Datadog after they priced the alternative. The teams that migrate anyway do it because the dollar gap is bigger than the UX gap, and because the UX gap shrinks every release.

What you give up: the ops burden

This is the line on the migration plan that teams underestimate. Running three stateful services is real work.

ClickHouse upgrades. ClickHouse ships fast. Pin the version, upgrade on a schedule, read the release notes when you cross LTS branches. Minor bumps inside 25.3 are safe. Crossing 24.8 to 25.3 is not automatic. Budget an engineer day per quarter for upgrades.

Backups. BACKUP TABLE to S3 works. Nobody sets it up on day one. The first outage is also the first time most teams learn their backup cron never ran. Wire it up before you need it.

Disk planning. At 100M spans per month with prompts captured, you are looking at 200–400 GB per month of compressed data. EBS throughput matters; a small gp3 volume will bottleneck ClickHouse on ingest spikes. Provision intentionally.

On-call for the observability stack itself. This is the cost that keeps some of the teams I have talked to paying for SaaS. When your observability is hosted, the observability company is on-call for their own stack. When you self-host, you are on-call for it. An outage at 03:00 on the ClickHouse instance means the engineer who would have been debugging production is now debugging the thing they use to debug production. That is a bad place to stand.

The teams that handle this well have a platform team of three or more engineers with one of them owning the observability stack as a named responsibility. The teams that handle this badly are the ones that rolled the stack because it looked cheap on a Thursday, and still have an unupgraded ClickHouse 25.3 running a year later because nobody owns it.

Who should migrate

Three conditions, all three required.

You have a platform team. Not a fractional engineer, not "a backend engineer who handles infra on Fridays." A named person or team whose job includes keeping stateful services alive. If your company is under 20 engineers and has no platform function, keep paying Datadog and spend that engineering capacity on the product.

Your volume is predictable and growing. The break-even against Datadog LLM Observability lands somewhere around 2–5M spans per month on list pricing, earlier against Langfuse Cloud. Below that, you are paying your platform engineer more than a vendor would charge. Above it, the math flips and keeps flipping.

Data sovereignty is a constraint, not a preference. A regulated industry, a customer contract that forbids prompts leaving your VPC, an internal policy about user content. If your reason for self-hosting is "I do not like SaaS," you will hate this stack in six months.

Who should not migrate

If you are a team of 12 engineers, with no platform function, whose Datadog bill is $40K a year and whose product is growing, the arithmetic of this migration does not work. You will spend the annual savings on platform engineering time in the first quarter, and you will learn that the Datadog bill you were trying to avoid is the smallest line item in the real cost of running observability.

The honest version of the recommendation: migrate when the Datadog bill is bigger than the cost of a half-time engineer, and you have the half-time engineer. Otherwise, negotiate with Datadog on the renewal.

The migration, step by step

If the three conditions hold, here is the shape the migration takes. Budget two sprints.

Week one. Stand up the compose stack in a staging environment. OTel Collector, ClickHouse, Grafana, Langfuse on top if you want the polished trace UI. Point a single service at it. Verify spans land. Write the first four panels: token usage per model, cost per hour, latency percentiles, error rate per operation. Chapter 15 of the observability book walks through the exact queries.

Week two. Point 10% of production traffic at the new Collector in parallel with Datadog. Run both for a week. Compare dashboards side by side. Find the metrics that look different and figure out why (usually a missing attribute or a sampling config drift). Port your top ten Datadog monitors to Grafana Alerting.

Week three. Cut over 100% of traffic. Keep Datadog running for the LLM traces for one more billing cycle as a safety net. Write the runbook for the three failure modes that matter: ClickHouse disk fills up, Collector OOMs, Grafana cannot reach the datasource.

Week four. Cancel the LLM Observability SKU on Datadog. Watch the next invoice.

The full decision tree

Is your Datadog LLM spend > $30K/year?
  No  → negotiate your renewal, stop reading.
  Yes → continue.
Do you have a platform team with one named owner for this?
  No  → keep paying Datadog.
  Yes → continue.
Is your span volume growing faster than your headcount?
  No  → keep paying Datadog; the math won't flip.
  Yes → continue.
Is data sovereignty a real constraint?
  Yes → migrate now.
  No  → migrate when the bill exceeds a half-time engineer's loaded cost.

The migration is not a question of whether self-hosted is possible. It has been possible since 2023. The question in 2026 is whether the operational maturity of your team matches the operational demands of a stateful columnar store.

If it does, the stack is three containers, the config is above, and the book has the rest.

If this was useful

Chapter 15 of Observability for LLM Applications is the full version of this post — the compose file, the ClickHouse schema walkthrough, the four panels with their SQL, the operator notes that come from having run the stack past the easy months. Chapter 16 picks up where this one stops, on cost tracking and token accounting once the telemetry is in your own warehouse. The rest of Part IV compares Langfuse, LangSmith, Phoenix, Braintrust, DeepEval, and Helicone against the same decision matrix this post uses for Datadog.

Book: Observability for LLM Applications — paperback and hardcover now; ebook April 22.
Thinking in Go: 2-book series on Go programming and hexagonal architecture.
Hermes IDE: hermes-ide.com — the IDE for developers shipping with Claude Code and other AI tools.
Me: xgabriel.com · github.com/gabrielanhaia.

I built an accessibility compliance SaaS in a day with Claude Code

asafamos — Sat, 18 Apr 2026 10:17:31 +0000

TL;DR — I'm not a primary coder. I spent ~12 hours on Saturday with Claude Code and shipped axle: an accessibility compliance CI that scans every PR for WCAG 2.1/2.2 AA violations and proposes AI-generated code fixes. It's live on GitHub Marketplace, npm, and a hosted web UI, and processed its first real $49 transaction the same day.

This is the honest play-by-play — the decisions, the dead ends, the Stripe-doesn't-serve-Israel detour, and what I'd tell someone trying to build a SaaS with AI as the primary contributor.

Why build this

Three things clicked at the same time:

The European Accessibility Act went into force in June 2025. Every company selling into the EU now needs WCAG 2.1 AA. Fines up to €1M per violation.
The FTC fined accessiBe $1 million in January 2025 for misleading AI claims about their overlay widget. That whole product category is legally radioactive.
Claude Sonnet 4.6 is good enough at targeted code fixes that the "AI auto-fix" pitch isn't marketing anymore — it actually works for ~60-75% of WCAG violations on realistic pages.

The wedge: real code fixes proposed as PR diffs, bundled with the legal artifacts a compliance officer actually needs (audit trail, accessibility statement, monitoring). Scanners are a commodity. The workflow isn't.

The stack I settled on

Every piece was chosen for "fastest path to first real revenue":

Next.js 16 on Vercel — App Router, server components. vercel deploy --prod in 30 seconds.
Playwright + axe-core 4.11 — the industry-standard scanner, wrapped in a headless Chromium.
@sparticuz/chromium — the serverless-compatible Chromium build. Needed once I realized Vercel functions don't ship a full browser.
Claude Sonnet 4.6 via the Anthropic SDK, with ephemeral prompt caching. Per-fix cost is ~$0.008.
Upstash Redis (via Vercel Marketplace, one click) for API-key storage and rate limiting.
Resend for transactional email.
Polar.sh as the merchant of record. This one deserves its own section.

The Stripe problem (and why Polar saved the day)

I'm Israeli. Stripe's dropdown for Business Location on a new account does not include Israel. I stared at it for ten minutes. I scrolled twice. No Israel.

Paddle was option B. I tried — and discovered my old Paddle account had been rejected years ago for an unrelated product (Swap-video, a face-swap tool that violated their acceptable use policy). Appeals exist, but take days.

Polar.sh (an MoR on top of Stripe Connect) was option C. Clean signup, no prior baggage, full KYC in one afternoon including identity verification. Bank account linked directly to my Israeli bank in USD. First real $49 charge processed end-to-end inside of six hours after signup.

If you're a non-US founder building SaaS in 2026: Polar.sh is the path.

How Claude Code actually feels as the primary contributor

I drove with a conversational loop: describe the next chunk, review the diff, ask for iteration. Claude wrote ~90% of the characters. What mattered from my side:

Knowing what I wanted at the product level (the pricing tiers, the legal positioning, the anti-overlay stance).
Catching when Claude over-engineered (it loves defensive programming; I rejected about a third of the try/catch blocks).
Forcing narrow scope (one feature at a time; no "while we're at it" refactors).
Recovering from dead ends fast.

The part Claude does not do for you: talking to customers, writing positioning copy that actually converts, choosing which channel to launch on.

Distribution, day one

GitHub Marketplace — asafamos/axle-action@v1.
npm — axle-cli and axle-netlify-plugin published.
Web UI — scan widget + Hebrew accessibility statement generator + embeddable compliance badge, all at axle-iota.vercel.app.
Scaffolds ready for Raycast, Chrome, Cloudflare Pages — publishing over the week.

Every output carries a "Powered by axle" trail: PR comments, generated statements, the badge itself (which is a backlink by definition). The product is the funnel.

What I learned

Legal pressure beats feature richness. The EAA/ADA/Israeli-law positioning converts better than any feature list.
KYC is the real bottleneck for non-US founders. Not coding. Not marketing. The paperwork.
Claude Code without product taste produces plausible-looking mush. With product taste, it produces shippable software in hours.
The "AI auto-fix" claim is defensible only with honest confidence scoring. Every fix axle generates ships with a confidence score and a manual-review flag.

What's next

More marketplace integrations — Cloudflare, Raycast, Chrome, VS Code.
Multi-language accessibility statements (French, German, Spanish) for full EAA coverage.
Custom domain, ProductHunt launch, and the inevitable "I scanned the top 100 SaaS landing pages" content marketing piece.

If you want to try axle (free tier: 1 repo, unlimited scans, bring your own Anthropic key):

Scan any URL: axle-iota.vercel.app
Install the Action: github.com/marketplace/actions/axle-accessibility-compliance-ci
CLI: npx axle-cli scan https://your-site.com

axle provides remediation assistance, not a compliance certificate. Automated tools catch ~57% of WCAG issues; manual human review is still recommended.

How to Build an AI Vocal Remover in Python with the StemSplit API (2026)

StemSplit — Sat, 18 Apr 2026 10:17:23 +0000

I needed to add vocal removal to an app last week without shipping a 300 MB Demucs model with the binary. The shortest path I found was StemSplit's API — three endpoints, one model (HTDemucs), and a free tier big enough to prototype on.

This is the working tutorial I wish I'd had. Single file, batch, webhooks, and a Flask wrapper at the end. All copy-paste runnable.

What You'll Learn

✅ How to remove vocals from any audio file in 5 lines of Python
✅ How the async job pattern works (upload → poll → download)
✅ Batch processing with bounded concurrency
✅ Webhook callbacks instead of polling
✅ Wrapping the API as a Flask backend for your own frontend
✅ Cost math, rate limits, and the production gotchas

Prerequisites

pip install requests python-dotenv tenacity flask

You'll need a free API key from the AI vocal remover dashboard — sign up gives you 10 minutes of processing, no card. Drop it in .env:

STEMSPLIT_API_KEY=your_key_here

# config.py
import os
from dotenv import load_dotenv

load_dotenv()

API_KEY  = os.environ["STEMSPLIT_API_KEY"]
API_BASE = "https://api.stemsplit.io/v1"
HEADERS  = {"Authorization": f"Bearer {API_KEY}"}

The 5-Line Version

If you just want to see it work before committing to anything:

import requests, time

job = requests.post(
    "https://api.stemsplit.io/v1/separate",
    headers=HEADERS,
    files={"audio": open("song.mp3", "rb")},
    json={"stems": 2, "format": "wav"},
).json()
status = {"status": "processing"}
while status["status"] not in ("completed", "failed"):
    time.sleep(3)
    status = requests.get(f"https://api.stemsplit.io/v1/jobs/{job['job_id']}", headers=HEADERS).json()
open("instrumental.wav", "wb").write(requests.get(status["stems"]["instrumental"]).content)

That's the entire flow. Upload, poll, download. The rest of this article is making that production-grade.

How the API Works

Three endpoints. That's the whole surface area:

Method	Path	Purpose
`POST`	`/v1/separate`	Upload audio + start a job
`GET`	`/v1/jobs/{id}`	Poll job status
`POST`	`/v1/webhooks`	Register a callback URL (skip the polling)

The stems field on the upload controls what you get back:

`stems`	Output
`2`	`vocals` + `instrumental` (vocal removal)
`4`	`vocals` + `drums` + `bass` + `other`
`6`	adds `guitar` + `piano`

For vocal removal you want stems: 2. The instrumental is the "everything except vocals" file.

📝 BPM and musical key come back in every job response at no extra cost. Useful if you're piping into a DJ tool, practice app, or recommender.

Single File: The Production Version

Same flow as the 5-liner, but with timeouts, retries on flaky network calls, exponential backoff for polling, and proper file streaming for large files.

import requests
import time
from pathlib import Path
from tenacity import retry, stop_after_attempt, wait_exponential

from config import API_BASE, HEADERS


@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=2, max=20))
def _post_with_retry(url: str, **kwargs):
    resp = requests.post(url, headers=HEADERS, timeout=60, **kwargs)
    resp.raise_for_status()
    return resp


def remove_vocals(
    audio_path: str,
    output_dir: str = "output",
    poll_interval: float = 3.0,
    timeout: float = 300.0,
) -> dict:
    """
    Remove vocals from a single audio file using the StemSplit API.

    Args:
        audio_path:    Path to input file. MP3, WAV, FLAC, M4A, OGG, WEBM up to 100 MB.
        output_dir:    Where to write the downloaded stems.
        poll_interval: Seconds between status checks.
        timeout:       Maximum seconds to wait before giving up.

    Returns:
        Dict with 'vocals' and 'instrumental' file paths, plus 'bpm' and 'key'.
    """
    Path(output_dir).mkdir(parents=True, exist_ok=True)

    with open(audio_path, "rb") as f:
        job = _post_with_retry(
            f"{API_BASE}/separate",
            files={"audio": (Path(audio_path).name, f)},
            data={"stems": "2", "format": "wav"},
        ).json()

    job_id = job["job_id"]
    deadline = time.time() + timeout

    while time.time() < deadline:
        status = requests.get(f"{API_BASE}/jobs/{job_id}", headers=HEADERS, timeout=30).json()

        if status["status"] == "completed":
            break
        if status["status"] == "failed":
            raise RuntimeError(f"Job {job_id} failed: {status.get('error')}")

        time.sleep(poll_interval)
    else:
        raise TimeoutError(f"Job {job_id} did not complete within {timeout}s")

    out = {}
    for stem, url in status["stems"].items():
        path = Path(output_dir) / f"{Path(audio_path).stem}_{stem}.wav"
        with requests.get(url, stream=True, timeout=120) as r:
            r.raise_for_status()
            with open(path, "wb") as f:
                for chunk in r.iter_content(chunk_size=8192):
                    f.write(chunk)
        out[stem] = str(path)

    out["bpm"] = status.get("bpm")
    out["key"] = status.get("key")
    return out

Usage:

result = remove_vocals("song.mp3")
print(result)
# {
#   'vocals': 'output/song_vocals.wav',
#   'instrumental': 'output/song_instrumental.wav',
#   'bpm': 124.0,
#   'key': 'C# minor'
# }

Notes on what this gives you that the 5-liner doesn't:

Streaming download. Stems can be 30–80 MB each. Don't .content them into memory if you're processing many files.
Bounded wait. A timeout parameter means a stuck job can't hang your worker forever.
Retried uploads. tenacity wraps the upload in three attempts with exponential backoff — survives transient network blips.

Batch Processing

The naive batch loop is slow because most of the wall-clock time is the model running on the server. You want to upload several files, then poll them all in parallel.

from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import glob


def remove_vocals_batch(
    input_dir: str,
    output_dir: str = "output",
    max_concurrent: int = 3,
) -> list[dict]:
    """
    Remove vocals from every audio file in a directory.

    Keep max_concurrent low — the free tier rate-limits aggressively.
    Bumped to 5–10 once you're on a paid plan.
    """
    files = []
    for ext in ("mp3", "wav", "flac", "m4a", "ogg", "webm"):
        files.extend(glob.glob(f"{input_dir}/*.{ext}"))

    print(f"Found {len(files)} files to process")
    results = []

    with ThreadPoolExecutor(max_workers=max_concurrent) as ex:
        futures = {ex.submit(remove_vocals, f, output_dir): f for f in files}
        for future in as_completed(futures):
            src = futures[future]
            try:
                result = future.result()
                results.append({"input": src, **result, "status": "ok"})
                print(f"✅ {Path(src).name} ({result.get('bpm', '?')} BPM)")
            except Exception as e:
                results.append({"input": src, "status": "error", "error": str(e)})
                print(f"❌ {Path(src).name}: {e}")

    ok = sum(1 for r in results if r["status"] == "ok")
    print(f"\nDone. {ok}/{len(files)} succeeded.")
    return results


# Usage
results = remove_vocals_batch("./music")

A 50-file batch on the free tier with max_concurrent=3 finishes in around 12–15 minutes wall-clock for ~4-minute songs. Most of that is wait time on the GPU queue, not network.

⚠️ Set max_concurrent to your tier's allowed parallel jobs. The free tier rate-limits at 3 concurrent. You'll get HTTP 429 above that, which will retry-storm if you don't catch it.

Skip the Polling: Webhooks

Polling is fine for scripts. For a backend, you want webhooks so you're not burning HTTP requests waiting around.

def remove_vocals_async(audio_path: str, callback_url: str) -> str:
    """
    Start a vocal removal job. The API will POST to callback_url when done.
    Returns the job_id immediately.
    """
    with open(audio_path, "rb") as f:
        resp = requests.post(
            f"{API_BASE}/separate",
            headers=HEADERS,
            files={"audio": (Path(audio_path).name, f)},
            data={
                "stems": "2",
                "format": "wav",
                "webhook_url": callback_url,
            },
            timeout=60,
        )
    resp.raise_for_status()
    return resp.json()["job_id"]

The callback payload looks like:

{
  "job_id": "job_a8f2c1",
  "status": "completed",
  "stems": {
    "vocals": "https://stems.stemsplit.io/...",
    "instrumental": "https://stems.stemsplit.io/..."
  },
  "bpm": 124.0,
  "key": "C# minor",
  "duration_seconds": 218.5
}

Receive it in Flask:

from flask import Flask, request, jsonify
import requests

app = Flask(__name__)


@app.route("/webhook/stemsplit", methods=["POST"])
def stemsplit_webhook():
    payload = request.get_json()

    if payload["status"] != "completed":
        # log + alert; payload['error'] has the reason
        return "", 204

    job_id = payload["job_id"]
    instrumental_url = payload["stems"]["instrumental"]

    # Download in a background task — don't block the webhook response
    queue_download.delay(job_id, instrumental_url)

    return "", 200

Two rules for webhooks that bite people:

Always return 2xx fast. The webhook caller won't wait for you to download the stem. Queue the download, return, then process out-of-band.
Verify the payload. Set a webhook secret in your dashboard and check the X-StemSplit-Signature header. The format matches Stripe-style HMAC-SHA256 over the raw body.

Wrapping It as Your Own API

If you're building a frontend for vocal removal, you usually don't want browsers calling the StemSplit API directly — your key would leak. Wrap it.

from flask import Flask, request, jsonify, send_file
import io
import requests
from pathlib import Path

app = Flask(__name__)


@app.route("/api/remove-vocals", methods=["POST"])
def remove_vocals_endpoint():
    """
    POST a multipart audio file. Returns the instrumental as the response body.
    """
    if "audio" not in request.files:
        return jsonify({"error": "missing 'audio' file"}), 400

    upload = request.files["audio"]

    if upload.content_length and upload.content_length > 100 * 1024 * 1024:
        return jsonify({"error": "file too large (max 100 MB)"}), 413

    job = requests.post(
        f"{API_BASE}/separate",
        headers=HEADERS,
        files={"audio": (upload.filename, upload.stream)},
        data={"stems": "2", "format": "wav"},
        timeout=60,
    ).json()

    job_id = job["job_id"]

    while True:
        status = requests.get(f"{API_BASE}/jobs/{job_id}", headers=HEADERS, timeout=30).json()
        if status["status"] == "completed":
            break
        if status["status"] == "failed":
            return jsonify({"error": status.get("error", "job failed")}), 500

    instrumental_url = status["stems"]["instrumental"]
    audio_bytes = requests.get(instrumental_url, timeout=120).content

    return send_file(
        io.BytesIO(audio_bytes),
        mimetype="audio/wav",
        as_attachment=True,
        download_name=f"{Path(upload.filename).stem}_instrumental.wav",
    )


if __name__ == "__main__":
    app.run(port=8000)

For real production, swap the inline polling for a Celery task and webhook the result back to the user via WebSocket or SSE. The pattern in the Stem Splitter API with FastAPI and Celery article drops in cleanly here — same architecture, different upstream.

Cost Math

The pricing is $0.10 per minute of input audio. So a 4-minute song costs $0.40 to process. The 10-minute free tier on signup works out to ~2–3 average tracks, enough to wire up the integration before you commit a card.

A few back-of-envelope numbers I ran for the side project:

Workload	Math	Monthly
Personal app, 50 songs/month	50 × 4 min × $0.10	$20
Side project, 1,000 songs/month	1000 × 4 × $0.10	$400
Production batch, 10,000 songs/month	10000 × 4 × $0.10	$4,000

Above ~5,000 songs/month, running Demucs yourself on a $0.40/hr GPU starts to make sense if you have someone willing to babysit the queue. Below that, the API is the cheaper option because you're not paying for idle GPU time.

Credits don't expire, so buying $50 of credits and burning through them slowly is fine — useful for hobby projects.

Common Issues

"Job stuck on `processing` for >2 minutes"

A 4-minute track typically completes in 40–60 seconds. If it's been over two minutes, the file is probably long (10+ min mixes are slower) or the queue is backed up. The API has a hard 5-minute SLA per job; my polling code times out at 300 s for a reason.

"HTTP 429 on every other request"

You're hitting the concurrency cap. Drop max_concurrent to 3 and add a backoff:

from tenacity import retry, retry_if_exception_type, wait_exponential

@retry(retry=retry_if_exception_type(requests.HTTPError), wait=wait_exponential(min=2, max=60))
def safe_post(*args, **kwargs):
    r = requests.post(*args, **kwargs)
    r.raise_for_status()
    return r

"The instrumental still has a faint vocal in the chorus"

Heavily layered choruses are the hard case for any AI vocal remover. Two things help:

Try stems: 6 instead of 2 and re-mix without the vocal stem. Backing harmonies sometimes get bucketed into other or guitar when the model isn't sure they're lead vocal.
Convert your input to WAV first if it's an MP3 below 192 kbps. Lossy compression strips frequency detail the model relies on.

"Webhook never fires"

Three things to check, in order:

Is your webhook URL publicly reachable? (Use a tunnel like cloudflared for local dev.)
Are you returning 2xx within 10 seconds? Slow handlers get marked failed.
Is the URL on HTTPS? The API won't POST to plain HTTP.

"Big files time out on upload"

Default timeout=60 for requests.post covers maybe a 50 MB upload on a decent connection. For 100 MB files, bump to 300:

requests.post(..., timeout=300)

Or stream the upload from disk so you're not loading the whole file into memory first (which the example code already does via open(path, "rb")).

Summary

Use Case	Pattern
Quick script to remove vocals from one file	The 5-line version above
CLI tool for a folder of files	`remove_vocals_batch` with bounded concurrency
Backend for a vocal-removal frontend	Flask wrapper + webhook callbacks
Production pipeline	Celery + webhooks + retry-with-backoff
5,000+ songs/month	Reconsider local Demucs on your own GPU

The whole AI vocal remover workflow boils down to three HTTP calls. The interesting part is what you build around them — error handling, batching, and how you stream results back to the user without blocking your workers.

Debugging an LLM Bug at 3 AM: The Runbook I Wish I'd Had

Gabriel Anhaia — Sat, 18 Apr 2026 10:17:23 +0000

Book: Observability for LLM Applications — paperback and hardcover on Amazon · Ebook from Apr 22
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Your pager goes off at 03:14. The chat says llm_judge_score_avg is down 0.14 against the 24h baseline. Latency is fine. Cost is fine. Every dashboard except one is green. You have thirty minutes before this becomes an executive problem. Here is the order.

This is the runbook I wish I'd had when I started writing the book on LLM observability. The five triage branches come from the book's incident-response chapter, which pulls from two years of published postmortems: Anthropic's August 2025 three-bug cascade, the April 6 2026 ten-hour outage, the $47K LangChain loop, the GPT-4o sycophancy rollback, Air Canada's invented fare policy. Each branch has a first check, the commands to run, and the fallback move.

00:00 — The page fires

The first instinct is to open the model playground and start poking. Do not. You cannot debug a distributed system by asking it one question at a time. Open the incident channel. Paste the alert. Read the last five messages in #deploys and #llm-gateway.

One sentence in the channel, not a thread:

Paged on llm_judge_score_avg down 0.14 vs 24h baseline. Investigating. No hypothesis yet.

This is the entire value of the incident channel for the first minute. It tells the next person who shows up that you are on it and what you know.

00:02 — Shape of the change

Do not look at the model. Look at the shape of the change. There are exactly five shapes an LLM incident can take, and getting the shape right in the first five minutes saves you the whole half hour.

Provider availability. HTTP errors, elevated latency, outright outage.
Provider quality. 200 OK everywhere, silently worse outputs.
Self-inflicted quality. You shipped something. The model did not change; your inputs did.
Cost. Dollars move. Latency and quality do not.
Regulatory / reputational. Nothing in your stack misbehaved by its own lights. The output is the problem.

The triage question is never "is it up." The triage question is: what changed, and on whose side.

00:05 — The three commands that pick a branch

Three commands. In this order. Do not skip one because you think you know the answer.

# 1. Are the upstream providers healthy?
curl -s https://status.anthropic.com/api/v2/status.json \
  | jq '.status.indicator'
curl -s https://status.openai.com/api/v2/status.json \
  | jq '.status.indicator'

Indicator values are none, minor, major, critical. If anything is non-none on a provider you depend on, you are likely in branch 1. Pin the page in the channel and keep going — you still need the next two commands to rule out a compounding incident on your side.

# 2. What does the last hour of your own traffic look like?
promtool query range \
  --start="$(date -u -v-1H +%s)" \
  --end="$(date -u +%s)" --step=60s \
  'sum by (gen_ai_provider_name, gen_ai_request_model) (
     rate(gen_ai_client_operation_duration_count[5m])
   )'

You are looking for three patterns. Error rate spiking on one provider but not the others, which points at branch 1. Error rate flat but judge score falling, which points at branch 2 or 3. Judge score flat but cost per request climbing, which points at branch 4.

# 3. What did your own team ship in the last hour?
git log --since='1 hour ago' --oneline \
  -- prompts/ configs/ tools/ retrieval/

If that command returns a commit, your prior just shifted hard toward branch 3. An LLM incident that coincides with a prompt or retrieval change is almost always self-inflicted until proven otherwise.

State your hypothesis in the channel as a sentence. "Judge score regressed on summarize-v7; prompts/ had a commit 42 minutes ago; rolling back." The sentence is the incident.

00:10 — Branch 1: provider availability

Signal: gen_ai.response.status_code distribution shifted to 5xx or overloaded_error. The status page is yellow or red.

First check:

# Per-provider error rate over the last 15 minutes.
promtool query instant \
  'sum by (gen_ai_provider_name) (
     rate(gen_ai_client_operation_duration_count{
       gen_ai_response_status_code=~"5.."
     }[15m])
   )
   /
   sum by (gen_ai_provider_name) (
     rate(gen_ai_client_operation_duration_count[15m])
   )'

Fallback move: trip the router. If you run LiteLLM, Portkey, or OpenRouter, the fallback tier is a config flip, not a deploy. If you have never exercised it, this is the incident you learn whether it works. The April 6, 2026 Anthropic outage separated teams into two groups: the ones whose fallback was committed and tested, and the ones whose fallback was committed and untested. The gap is one practice drill.

If you do not have a router, your only fallback is a prepared canned response on the user-facing surface. Serve it. Update the public status page within ten minutes of declaring the incident.

00:10 — Branch 2: provider quality

Signal: 200 OK everywhere, but your online judge score or a character-distribution signal shifted. The canonical example is Anthropic's August–September 2025 bug cascade — three defects shipped as HTTP 200s, peaking at 16% of Sonnet 4 requests on August 31. No availability SLO fired. Only quality drift showed it.

First check:

# Judge score, last hour vs same hour yesterday,
# bucketed by model.
promtool query range \
  --start="$(date -u -v-1H +%s)" \
  --end="$(date -u +%s)" --step=60s \
  'avg by (gen_ai_response_model) (
     llm_judge_score
   )'

Then scan a sample of traces for character-distribution anomalies — Thai or Chinese characters in English responses was one of the August 2025 symptoms. A one-liner against your trace store:

# Percent of outputs containing non-ASCII where they
# shouldn't. Adjust the feature filter to yours.
curl -s "$TRACE_API/search?feature=summarize&limit=500" \
  | jq '[.traces[].output
         | test("[^\\x00-\\x7F]")] | map(select(.)) | length'

Fallback move: route traffic to the secondary tier and open a support ticket with trace IDs attached. Providers are faster to confirm a quality bug when you hand them span data, not screenshots. Keep the primary in shadow mode so you can confirm recovery without rolling back.

00:10 — Branch 3: self-inflicted quality

Signal: git log --since='1 hour ago' returned something, or your feature flag system shows a rollout that started in the last two hours. The judge score regression is localized to the feature that changed.

First check:

# What's running, per feature, right now.
curl -s "$FLAGS_API/state" \
  | jq '.features[]
        | select(.updated_at > (now - 7200))'

Then bisect. You have four axes of change: the prompt version, the model version, the retrieval index, and the tool definitions. A trace backend worth using (Langfuse, Phoenix, Braintrust, Datadog LLM Observability) lets you filter on each independently and diff the last hour against the same hour yesterday in a three-click operation. If your backend cannot do that, the incident is now also a retrospective action item.

Fallback move: roll back the prompt behind its feature flag. No redeploy. The rollback budget should be under ten minutes. If it takes longer, flip to the kill-switch prompt. The kill-switch is a known-good conservative baseline, reviewed quarterly, and exists for exactly this moment. Cursor's April 2025 "Sam" incident is the canonical recovery: they flipped to a baseline prompt and recovered because the flip existed as a single action.

00:10 — Branch 4: cost

Signal: latency flat, judge score flat, but cost per request or cost per tenant has climbed. The dashboards for this branch accumulate rather than ring, which is why cost incidents are almost always found late.

First check:

# Cost per hour, per tenant, last 6 hours vs last 7d avg.
promtool query range \
  --start="$(date -u -v-6H +%s)" \
  --end="$(date -u +%s)" --step=300s \
  'sum by (app_tenant_id) (
     rate(app_llm_cost_usd[10m])
   )'

If a single tenant is three times their seven-day rolling average, you likely have a loop. The July 2025 Claude Code recursion loop burned 1.67B tokens in five hours before detection. Look at invoke_agent spans and count execute_tool children under a single parent:

# Tool-call fanout per agent run, top 10 offenders.
curl -s "$TRACE_API/query?span=invoke_agent&limit=1000" \
  | jq '[.traces[]
         | {id: .trace_id,
            tool_calls: ([.spans[]
                          | select(.name == "execute_tool")]
                         | length)}]
        | sort_by(-.tool_calls) | .[0:10]'

Fallback move: trip the circuit breaker at the gateway. A hard per-tenant budget cap that refuses requests past threshold is the only brake that works in real time. Cap cumulative tokens per agent session. If you do not have a gateway-level cap, your rollback is the per-feature flag that disables the agent surface until the loop is diagnosed.

00:10 — Branch 5: regulatory / reputational

Signal: nothing in your metrics fired. Someone on the support team or a customer on social media surfaced a specific output that is legally or reputationally bad. Air Canada's Moffatt v. Air Canada chatbot invented a bereavement-fare policy. The BC tribunal held the airline liable for the invented policy. What your LLM says, your company said.

First check:

# Pull the exact trace by user_id and timestamp window.
curl -s "$TRACE_API/search?\
user_id=$REPORTED_USER&since=$REPORTED_TS_MINUS_10M&\
until=$REPORTED_TS_PLUS_10M" | jq '.traces[0]'

You need three things from the trace: the prompt version that was live, the retrieval context that was pulled, and the full model output as stored. PII redaction should have scrubbed sensitive data at write time. If it did not, the incident has a data-handling component too.

Fallback move: flip the feature to the kill-switch prompt or disable the surface entirely. This is the one branch where "serve a canned response" is the correct first action, not the last. Legal gets looped in immediately, not after the bisect. For user-facing LLM output, an AI disclosure label and an output moderation layer are the controls that prevent the next occurrence — if they are not live, the postmortem action item writes itself.

00:25 — Communicate, then fix

By the 25-minute mark you should have declared the branch, stated the hypothesis, and applied the mitigation. The public status page is updated. The incident channel has a single pinned message with the current state. If the mitigation is a rollback, the rollback is complete. If the mitigation is a router flip, the flip is in effect and you are watching the secondary tier's judge score to confirm the fallback is actually producing acceptable output.

One detail most teams miss: the fallback tier needs its own online judge, running in steady state, not only during the incident. A secondary model that scores 0.55 against your rubric on a quiet Tuesday is a latent incident — you find out about it the first time primary fails over. Run the online judge on the fallback. Confirm the number before you need it.

00:30 — The postmortem template

When it is over, write it down. Within 48 hours. Memory degrades on the wrong axis for postmortems: you remember the fix, you forget the false leads, and the false leads are where the organizational learning lives.

# Postmortem: <incident title>

**Was this detectable from metrics alone? If not, what signal
would have caught it, and what do we need to build to capture
that signal next time?**

## Summary
<Two sentences. What broke, how long, user impact.>

## Timeline (UTC)
- HH:MM — first signal
- HH:MM — page fired
- HH:MM — hypothesis stated
- HH:MM — mitigation applied
- HH:MM — resolved

## Root cause
<Technical cause. Link traces, dashboards, provider status.>

## What went well
## What went badly
## Action items
<Each item has an owner and a date. No exceptions.>

The bolded question at the top is the entire point. LLM incidents fail the metrics-alone test more often than traditional incidents do, and every time the answer is "no, not from metrics alone," you have found a gap in your observability that the next incident will walk through. Treat every "no" as a ticket. Assign it. Close it before the next quarter.

The five branches on one page

Pin this to the wall next to the on-call laptop:

Branch	First signal	First check	Fallback
Provider availability	5xx / `overloaded_error` rate up	`curl status.anthropic.com`	Router flip to secondary tier
Provider quality	200 OK, judge score down	Char-distribution scan, per-model judge	Route to secondary + support ticket with trace IDs
Self-inflicted quality	`git log --since='1h ago'` returns	Bisect prompt / model / retrieval / tools	Feature-flag rollback to baseline
Cost	$/tenant above 3x 7d avg	`execute_tool` fanout per `invoke_agent`	Gateway per-tenant cap, kill agent surface
Regulatory / reputational	Support ticket, social media	Pull trace by user + timestamp	Kill-switch + AI disclosure + legal

The order you read this table matters. Availability and cost are visible in metrics. Quality and regulatory are not. Most teams wire alerts for the first two and learn about the second two from customers. The job is to close that gap before the next incident does it for you.

If this was useful

The playbook above is a one-page compression of the book's chapter on incident response. The full version covers the OTel span schema that makes the bisect a three-click operation, the LiteLLM router config that makes the failover a config flip, the quality-aware circuit breaker that trips on judge score and not just HTTP status, and the fifty-item production readiness checklist that blocks a launch. That is what the book is for.

Book: Observability for LLM Applications — paperback and hardcover now; ebook April 22.
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go.
Hermes IDE: hermes-ide.com — the IDE for developers shipping with Claude Code and other AI tools.
Me: xgabriel.com · github.com/gabrielanhaia.

Forem

PHP to Go: The Mental Model Shift Nobody Warns You About

The request lifecycle stops dying

Eloquent vs database/sql + sqlc

Dependency injection: the container is gone

Middleware: http.Handler is the whole pattern

Concurrency: goroutines are not workers

Typing: PHP 8.4 is close, but not the same thing

Testing: the stdlib is enough

Five pitfalls PHP devs hit in their first Go week

What Laravel still does better

What Go does that PHP genuinely cannot

The honest summary for a PHP dev eyeing Go

If this was useful

Data Validation Using Early Return in Python

The Problem

The Pattern

Example

Diagram

What is better about this

Microlearning for developers: learn new concepts in 15 minutes

The developer's learning problem

Why microlearning works for technical minds

Beyond coding: skills developers often overlook

How to fit learning into a developer's day

Building a learning habit that survives deadlines

What to learn first: practical recommendations

Start small, stay consistent

I Built a Circle QR Code Generator — Yes, With Curved Border Text

Why Circle QR Codes?

How the Circle QR Code Generator Works

Base QR Generation with Circle Shape

Circle Masking

Curved Border Text — The Interesting Part

The Arc Text Math

User Controls

Real-World Usage

Batch Processing

Try It Yourself

Where Do Interfaces Live? IVP Answers What SOLID Cannot

Result 1: DIP → ISP, formalized in IVP terms

Result 2: A unique modularization

What this settles

What IVP's placement produces

The paper

Claude + Groq Hybrid LLM — AI University Memory Agent

Claude + Groq Hybrid LLM — AI University Memory Agent

Architecture

Memory Agent — Claude Extracts the Profile

Quiz Evaluator — Groq Scores Answers Fast

Why Two Models?

DB Schema

Key Takeaways

FlutterWeb #Supabase #buildinpublic #LLM #FlutterTips

ServiceKit V2 — The Async Service Locator for Unity

Why a service locator, not a "proper" DI framework

Hidden dependencies, un-hidden

Async resolution, the real headache fixed

Generics out, attributes in

Migrating from V1

Try it

Rust vs Go for AI Infrastructure in 2026: Here's What the Benchmarks Actually Say

The workloads, and why these five

Gateway throughput

Token streaming

OpenTelemetry span processing

Vector similarity hot path

Prompt templating under concurrency

What this actually adds up to

If this was useful

From $50K/Year of Datadog to $0/Year of Self-Hosted Observability: The Migration Every Team Is Doing in 2026

Why the bill grew that fast

The stack teams are migrating to

The Collector config

The Grafana datasource

The cost math, with real numbers

What Datadog still does better

What you give up: the ops burden

Who should migrate

Who should not migrate

Eloquent vs `database/sql` + `sqlc`

Middleware: `http.Handler` is the whole pattern

"Job stuck on `processing` for >2 minutes"