The first boot in the deployment environment failed: the service would not start, and the logs showed the database connection pointing at a container hostname that had only ever existed on one colleague's laptop. The string was nowhere in the code, the deploy scripts or the config templates. The hunt started there.
The timeline
| Phase | Observation | Conclusion |
|---|---|---|
| Symptom | Boot fails; connection error names an unknown host | Treat as networking first |
| Hypothesis 1 | Network/firewall blocked | Ruled out: other services on the same network fine |
| Hypothesis 2 | Deploy script passes the wrong host | Ruled out: the script passes the right one |
| Probe | Print the runtime environment and the resolved config | The process is not reading the value we passed |
| Locate | Boot the bare image in a clean environment, print layer by layer | The base image bakes ENV at build time |
| Confirm | Clear that variable, re-run | Service starts; root cause confirmed |
The decisive probe is one command — trust no "should", print what the runtime actually sees:
# Same image as production, clean environment: what does it actually see?
docker run --rm <image> sh -c 'printenv | grep -iE "db|host|postgres"'
The unfamiliar hostname in that output was the whole answer.
Root cause: a default bypassed by "always set"
The entrypoint held the bug. We (like most templates) habitually write:
# Before: default only if unset
DB_HOST="${DB_HOST:-saas-db}"
Correct in a normal environment. But the base image bakes a set of ENVs at build time (including an internal hostname); to the process, those variables are never unset — the :- default never applied, and our explicitly passed values sat behind the baked ones. Not one log line said "the value you passed was ignored"; everything walked quietly to the wrong host.
The fix is direct — drop the "default if unset" assumption; export what you need, explicitly:
# After: explicit, with a startup self-check
export DB_HOST="${DB_HOST:?DB_HOST is required}"
export DB_PORT="${DB_PORT:?DB_PORT is required}"
# Self-check: fail loudly when a required dependency is unreachable
Why it took time
Three factors stacked:
- The error pointed elsewhere: a failed connection looks like networking, so networking is where you look first;
- "It's fine locally": local development uses another entry path that happens to bypass the baked variable — local can never reproduce the deployment environment's assumptions;
- Silence by default: the entrypoint's pattern is blameless in a normal environment and silently wrong only when hijacked.
Two guardrails we kept
The fix is one-off; the guardrails are permanent:
- Startup self-check: on boot, verify key dependencies (database reachable, required config present) and fail loudly if not — configuration errors and service failures read differently in the logs from now on;
- Clean-environment regression: deploy acceptance includes one full boot from a clean clone with non-default credentials. A local dev environment can never cover a deployment environment's assumptions; only a clean one can.
Reusable conclusions
- The container era's most expensive lessons are often not about code: the environment you think you have may just be the environment the image author imagined;
- Every "default if unset" in an entrypoint deserves one question — what if it is always set?
- When debugging environment problems, print facts first (runtime environment, resolved config) and hypothesise second — reversed, the time goes into testing the wrong hypotheses.