Instrument your code before you ship it
"We set up logging, error tracking, and metrics dashboards for every service before it handles production traffic."
Why this exists
The worst outages are silent. I have seen background tasks fail for three days straight, and APIs fail for 12% of requests while health checks remained green. Without instrumentation, you only learn about outages when clients email you. We instrument everything to find failures before our users do.
Operational Flow
Structured logs: Write logs in JSON rather than plain text. Include a timestamp, log level, request identifier, and payload. Use Pino in Node and structlog in Python. Do not use `console.log` in production code.
Error tracking: Route unhandled exceptions to Sentry with user identifiers and stack traces. Filter out personal data before transmission.
Metrics: Track request latencies at the 50th, 95th, and 99th percentiles. Generate a Grafana dashboard before deploying any new service.
Uptime checks: Configure Checkly monitors to ping every public endpoint at one-minute intervals.
Alerting: Route urgent alerts to PagerDuty and non-urgent notices to Slack. Sending non-actionable alerts to PagerDuty causes alert fatigue.
Tracing: Pass trace identifiers across service boundaries using OpenTelemetry.
Alert hygiene: Assign every alert an owner. Delete or adjust any alert that fires for 30 days without requiring human intervention.
What good looks like
- You get paged at 2:14 AM, open Grafana, and correlate a latency spike with a deployment completed eight minutes prior.
- A database connection pool runs out of sockets, Sentry notifies you within 90 seconds, and you scale the pool before users experience errors.
- The on-call engineer reviews weekly alerts, adjusts three noisy thresholds, and links a postmortem to the on-call log.
What NOT to do
- Do not use stdout plain text logging. It cannot be queried or aggregated during an incident.
- Do not deploy an alert without verifying that it triggers correctly under test conditions.
- Do not route low-priority warnings to PagerDuty.
- Do not deploy services with the intention of adding dashboards later.
Do not treat observability as an afterthought. It is a prerequisite for shipping software. Instrumenting a service before deployment takes an hour. Debugging an uninstrumented service during an outage takes days of parsing raw server logs. In one incident, our webhook deliveries failed silently because the execution script exited with a zero status code under load. We had no queue metrics and only learned about the failure when a client asked why their system had stopped updating.
Every service must ship with JSON logging, Sentry tracking, a Grafana dashboard, and a Checkly uptime check before handling its first production request. Alert on actionable events and silence noise. If PagerDuty is quiet, verify whether your systems are running correctly or your alerts are broken.