10 Essential Healthchecks Every DevOps Team Needs
A DevOps pipeline is a complex system of interconnected tools, scripts, and infrastructure. Without rigorous healthchecks, silent failures, configuration drift, and performance degradation become inevitable. Below are ten critical healthchecks that prevent cascading failures and ensure operational excellence.
1. CI/CD Pipeline Duration & Failure Rate
The CI/CD pipeline is the central nervous system of DevOps. A failing pipeline halts delivery. Monitor two key metrics: pipeline duration and failure rate. A sudden spike in duration often indicates resource contention (e.g., slow build agents, exhausted cache). Track failure rate as a percentage of total runs; a rate above 5% warrants immediate investigation. Use tools like Jenkins’ Prometheus plugin or GitLab CI’s built-in analytics. Set alerts for deviations beyond two standard deviations from the baseline. Examine failed stages—are they consistently failing at testing, building, or deployment? Distinguishing between flaky tests and genuine code issues is critical. Implement synthetic healthchecks that trigger a dummy commit hourly to validate the entire pipeline path.
2. Secret Rotation Age & Vault Seal Status
Compromised credentials are a leading cause of breaches. Healthcheck your secret management by auditing rotation age. Every service account, API key, and database credential should have a maximum lifetime. A rolling healthcheck verifies that secrets generated over 90 days ago are flagged for rotation. If using HashiCorp Vault, monitor the seal status. An unsealed Vault is dead to the system—services cannot retrieve secrets, causing widespread auth failures. Automate a healthcheck script that calls the Vault status endpoint every minute. Also check for orphaned secrets; these are credentials no longer referenced by any pod or application but still valid. Implement a “least time to verification” policy: any secret not used in 30 days should be revoked automatically.
3. Log Aggregation Pipeline Latency & Completeness
Logs are the black box of DevOps. A healthcheck on your log pipeline (e.g., Fluentd, Logstash, Elasticsearch) ensures you can debug incidents. Measure ingestion latency: the time between a log event and its appearance in the search index. Aim for sub-second for critical systems, under 30 seconds for standard logs. Use a heartbeat generator—an agent that emits a timestamped log every 5 seconds. If the heartbeat is missing for two intervals, the pipeline is likely dropping logs. Healthcheck log completeness by comparing the volume of logs from production nodes against expected patterns. A sudden 20% drop may indicate a broken log shipper. Also verify log schema integrity; a malformed JSON log can break downstream parsers.
4. Infrastructure as Code Drift Detection
Manual changes to cloud resources create drift between your declared state (Terraform, Pulumi) and actual infrastructure. A healthcheck must detect drift automatically. Run a terraform plan or pulumi preview daily against your state files. Any difference between the code and real-world resources indicates drift. Categorize drift as critical (security group changes, IAM role modifications) or cosmetic (tag updates). Automate remediation rules: for non-critical drift, revert automatically via cron. For critical drift, notify the team and block further deployments. Integrate drift detection into your CI pipeline; a pipeline should fail if drift is detected before a deploy. Use tools like Terratest or Atlantis to enforce a “no drift” policy.
5. Container Image Vulnerability Age
Images pulled from public registries decay over time as new CVEs emerge. A critical healthcheck is the age of vulnerabilities in your currently running images. Use a scanner like Trivy, Clair, or Snyk to query all active container instances. Generate a freshness score: an image scanned within 24 hours has a score of 100; beyond 7 days, it drops to 0. Flag any image with a known critical or high-severity CVE that is unpatched for more than 48 hours. Also check base image freshness; if your base image (e.g., node:18) is three months old, rebuild it even if no CVE exists. Automate a healthcheck that triggers a rebuild and re-deploy if vulnerability age exceeds your SLA. Store scan results in a central database to track remediation times.
6. Service-Level Objective (SLO) Burn Rate
An SLO burn rate healthcheck prevents slow-motion outages. If you have a 99.9% uptime SLO for a service, a 1% error rate over 30 days is acceptable. But a healthcheck measures the burn rate—how fast you are consuming your error budget. Define error budgets (e.g., 43 minutes of downtime per month for 99.9% SLO). A healthcheck triggers a warning if the burn rate exceeds 2x the budget over a rolling 24-hour window. Use a multi-window approach: a fast burn (10x rate over 1 hour) requires immediate rollback; a slow burn (1.5x rate over 7 days) signals a gradual degradation. Tools like Prometheus and Google Cloud Monitoring support direct SLO burn rate alerts.
7. Database Connection Pool Exhaustion
A common source of production incidents is database connection pool exhaustion. Healthcheck your database connection pool metric (e.g., pg_stat_activity for PostgreSQL, connection_pool.size for Redis). The healthcheck should measure active connections as a percentage of the pool’s maximum. Raise a warning at 70% utilization. At 85%, automatically scale the pool or add a read replica. Also monitor idle connections; a high count often indicates a connection leak in the application code. Implement a healthcheck that connects to the database, runs a simple SELECT 1, and disconnects—measuring the round-trip time. A latency spike above 500ms may indicate a dying connection or DNS resolution failure.
8. Certificate Expiry & SSL/TLS Chain Validity
Expired certificates are a leading cause of preventable outages. A healthcheck must audit every certificate used by your services—including internal mTLS certificates, ingress certificates, and third-party API certificates. Write a script that extracts expiry dates from all configured secrets and TLS endpoints. Trigger a critical alert when any certificate is within 14 days of expiry. Daily healthchecks should verify the entire certificate chain: ensure intermediate certificates are not broken, and that the Subject Alternative Names (SANs) match the actual domain. For internal services, check that your own CA hasn’t expired. Use tools like openssl s_client or Cert-Manager’s status reporting to automate chain validation.
9. Heap & Thread Dump Health (Production JVM/Node)
Memory leaks and thread starvation are silent killers. A healthcheck on production runtimes measures heap utilization and thread pool health. For JVM-based services, monitor the old generation heap: if it grows above 80% and fails to shrink after a garbage collection, there is a likely leak. For Node.js, monitor heap usage via process.memoryUsage(). A healthcheck should automatically trigger a heap dump when usage exceeds a threshold for 10 minutes. Additionally, check blocked threads. A thread pool that consistently has 90%+ active threads indicates resource starvation. Automate a thread dump capture if blocked threads persist for more than 30 seconds. Analyze dumps for patterns: do they always block on database queries? This indicates a slow query issue, not a memory problem.
10. Dependency Caching Hit Rate (Build & Package)
Builds that repeatedly download the same dependencies waste time and bandwidth. Healthcheck your package cache hit rate (npm, pip, Maven, Docker layers). For Docker builds, measure the percent of layers that are reused from the cache. A cache hit rate below 70% indicates inefficient layering or overly frequent base image changes. For language-specific caches, check the percentage of packages resolved from local mirrors versus remote registries. A drop below 80% suggests a mirror is down or a lockfile is outdated. Automate a healthcheck that validates the cache is populated for critical packages. If a cache miss occurs on a frequently used library (e.g., requests in Python), investigate immediately. Also measure cache eviction rates; aggressive eviction policies reduce hit rates over time.
Implement these ten healthchecks as code—every check should be version-controlled, deployable, and run on a schedule. Monitor the healthchecks themselves; a failing healthcheck silences your early warning system. Use a centralized dashboard (Grafana, Datadog) to visualize all ten metrics in a single view, with color-coded severity levels. Finally, bias toward automated remediation: when a healthcheck detects a problem (e.g., certificate expiry, log pipeline drop), trigger a self-healing action before a human has time to respond. This transforms reactive DevOps into a proactive, resilient system.