The incident shape
Picture an orders service on a busy afternoon. Traffic climbs, and the Kubernetes Horizontal Pod Autoscaler (HPA) adds pods, exactly as configured. Each new pod opens its own pool of connections to PostgreSQL. A few pods into the scale-up, the database starts refusing new connections. Requests fail, and every pod the autoscaler adds from here asks for connections that do not exist.
Nothing in that story is a Kubernetes defect or a PostgreSQL defect. The autoscaler scaled. The database protected itself by enforcing its limit. Each layer passed its own checks. The failure lives in a number nobody owns: how many connections the whole deployment can open, against how many the database will accept.
Site reliability engineers (SREs) and platform engineers know this failure well. Two current changes add new ways to trigger it. Autoscaling changes replica counts without a person in the loop, and coding agents can now edit Deployment manifests, HPA objects and pool settings in the same change as application code, without anyone re-running the arithmetic.
What each layer guarantees
Kubernetes documents the HPA as a control loop. By default the controller runs every 15 seconds, compares the current metric with the target, and computes the desired replica count from their ratio. It never goes above the maximum you configure. Scaling up has no stabilization window by default: the controller may add 4 pods or double the running count every 15 seconds, whichever is more. Its inputs are CPU, memory, custom and external metrics. The database's connection limit is not one of them unless you build that metric yourself.
A rolling update adds a second multiplier. A Deployment's maxSurge setting allows extra pods above the desired count during a rollout, and its default is 25%, rounded up.
PostgreSQL enforces its own ceiling. In the PostgreSQL 18 documentation, max_connections defaults to "typically 100", and you can only change it at server start. Part of that number is not yours. superuser_reserved_connections holds back 3 slots by default for superusers, and reserved_connections (default 0) can hold back more for roles granted pg_use_reserved_connections. Once free slots fall to those reserves, the server admits only those roles.
Both systems behave as documented. The HPA object has no field for the database limit, and PostgreSQL has no idea how many pods exist.
The connection budget, worked through
The figures below are hypothetical. They show the shape of the calculation, not a measured system.
| Input (hypothetical) | Value | Where it lives |
|---|---|---|
| HPA maximum replicas | 12 | HPA object |
| Connection pool size per pod | 12 | Application config |
| Potential connections at maximum scale | 12 × 12 = 144 | Nowhere |
PostgreSQL max_connections | 100 | Database config |
| Left after 3 superuser-reserved slots | 97 | Derived |
| Pods during a rollout at maximum (25% surge) | 12 + 3 = 15, so 180 connections | Deployment strategy |
At full scale the service can ask for 144 connections from a database that accepts 97 ordinary ones, and 180 during a rollout. That is before migration jobs, scheduled workers, admin sessions or a second service that shares the database take their share. Their numbers live in other files, or in nobody's file.
Run the arithmetic the other way and the decision becomes concrete. With 12 connections per pod and 97 slots, at most 8 pods fit: 8 replicas use 96 connections. Count the 25% rollout surge as well and the HPA maximum drops to 6 (6 plus 2 surge pods is 8). Cut the pool to 8 connections per pod and the cap rises to 9, surge included (9 plus 3 surge pods is 12, using 96 connections). Either answer is a trade-off a team has to choose and write down.
What a pooler changes
A connection pooler such as PgBouncer changes the terms. It does not remove the budget. Pods connect to PgBouncer, and PgBouncer holds a smaller set of server connections to PostgreSQL. In transaction pooling, a client holds a server connection only for the length of a transaction.
The budget now has three terms instead of one. Server connections are bounded per user and database pair by default_pool_size (default 20), and per database by max_db_connections, which defaults to 0, meaning unlimited. Client connections are bounded by max_client_conn, default 100. The hypothetical 144 client connections would exceed that default on a single PgBouncer instance. Run three PgBouncer replicas with a pool of 20 each for one user and database, and PostgreSQL sees up to 60 server connections from them, plus anything that bypasses the pooler.
Transaction pooling also constrains the application. PgBouncer's feature map lists session features that transaction mode does not support, including SET/RESET, LISTEN and session-level advisory locks. Choosing the pooling mode is a decision the code has to respect too.
The budget is an architecture decision
A connection budget has the properties of an architecture decision: a reason, an owner, a scope and consequences across several components. Written down, it reads something like this:
- The orders service reaches PostgreSQL only through its data adapter: the one module that opens database connections and owns the connection pool.
- Potential server connections from every client, including rollout surge, stay below the database limit minus reserved slots.
- The service connects through PgBouncer in transaction mode and does not depend on session state.
Once the decision exists as a record, each part has a home. Architecture as Code describes the intent in a form tools can read. Policy as Code turns parts of it into rules that engines evaluate. Keeping those rules tied to the current version of the decision is a lifecycle problem of its own, covered in Policy as Code Enforces Rules. Who Governs the Decisions Behind Them?
Static controls and runtime controls
Some parts of the budget are deterministic: you can compute them from files before anything deploys. Others only exist at runtime. A static check cannot see a client that leaks connections, and a dashboard cannot stop a bad manifest from deploying, so the budget needs both kinds.
| Control | Kind | Enforced by |
|---|---|---|
| Application code reaches the database only through the adapter | Static, deterministic | Mneme, an open-source tool that enforces recorded decisions on application code (shipped, with limits below) |
| HPA maximum × pool size, plus surge, stays within the budget | Static, deterministic | A continuous integration (CI) step over rendered manifests (the final Kubernetes objects after templating), or a policy engine such as Open Policy Agent (OPA) or the command-line tool of Kyverno, a Kubernetes policy engine |
| The cluster rejects a manifest that breaks the budget | Static, at admission | An admission controller (a hook that approves or rejects objects before the cluster stores them), such as Kyverno or OPA Gatekeeper |
| PgBouncer pool sizes and mode match the decision | Static, deterministic | Review and CI on the pooler config |
| Connections in use, other clients, leaks | Runtime, measured | Monitoring on pg_stat_activity and pooler stats |
The static check proves the deployment cannot request more than the budget on paper. It cannot prove what is open right now. PostgreSQL's pg_stat_activity view shows one row per server process, and that is where a forgotten cron job or a leaking client appears. You need both: arithmetic before deploy, measurement after.
The arithmetic check has to see the HPA object, the Deployment, the application's pool setting and the database limit together. Rendered manifests in CI give it all four in one place. Kyverno, which runs as a dynamic admission controller, and Gatekeeper, a validating and mutating webhook that enforces policies executed by Open Policy Agent (OPA), are the right tools when the cluster itself must refuse the change.
Where Mneme fits, and where it does not
Mneme is an open-source decision and control layer for agentic software development, and it enforces the first row today, in the agent and in CI on every pull request, in warn or strict mode. The decision lives in an architecture decision record (ADR) with a typed rule: a constraint written in a fixed format that Mneme can evaluate mechanically. This one forbids the direct client import inside request handlers:
## Constraints
- FORBID_LITERAL:
value: from app.db.client import
include_paths:
- "app/handlers/**"Mneme returns one of three verdicts: PASS, WARN or FAIL. A handler that contains that text gets FAIL, the strongest verdict (exit code 2 in strict mode; warn mode reports it without failing the job), and the result names the decision. Mneme runs the same check before a supported agent's built-in file edit is saved (for example Claude Code's Edit and Write tools, as opposed to shell commands), and on pull requests in CI.
This rule matters to the budget because the pool size only bounds connections if every query goes through the pool. A handler that opens its own client adds connections no manifest declares, and the static arithmetic above becomes wrong without any infrastructure file changing.
The rule has documented limits. FORBID_LITERAL matches exact text, so import app.db.client as db passes; you add one rule per import form or run an import linter beside it.
No Mneme rule type checks replica counts and pool sizes against a connection budget. Doing that from a recorded decision is a concept, not a shipped feature. Mneme does not enforce infrastructure policy or admit deployments; Kyverno, OPA and similar tools do that work.
One decision, several enforcement points
The budget decision feeds at least four enforcement points: the code rule, the CI arithmetic, the admission policy and the alert. When one fails, the on-call engineer should be able to answer which decision it protects, which version, which rule fired, on which artifact, and with what verdict. That is the evidence question engineering decision governance asks.
Part of that chain ships today. A Mneme violation names the decision and rule behind it, and the Decision Index, the versioned record of decisions that Mneme keeps in the repository's .mneme/project_memory.json file, records each decision's versions, lifecycle and source. Linking the CI budget check, the Kyverno policy and the alert to the same decision version is something teams do by convention, for example by citing the ADR identifier in each policy. Mneme does not link them across layers for you.
The urgency comes from who edits the numbers. An agent that raises an HPA maximum from 12 to 20 to fix a latency ticket has made a reasonable local change. The budget breaks in a file it never opened. As When Agents Launch the Database argued, agents now act on infrastructure directly, so the decision has to be enforceable wherever they act.
What to do now
- Write the budget down. Record the arithmetic as a decision: database limit, reserved slots, other clients, pool size per pod, HPA maximum and surge.
- Compute it in CI. Fail the build when rendered manifests can request more than the budget.
- Add admission only where you need it. Use Kyverno or Gatekeeper when the cluster must refuse the change, not only the pipeline.
- Measure the runtime side. Alert on connections in use against the same budget, so the static number and the observed one stay comparable.
- Hold the code boundary. Enforce the adapter rule in application code, or the pool size stops meaning anything.