CI/CD Best Practices
Continuous Integration and Continuous Delivery practices for reliable, fast software delivery.
Core Principles
| Principle | Description |
|---|---|
| Small, frequent changes | Reduce batch size to reduce risk. Each deployment = minimum viable change. |
| Automate everything repeatable | If a human does it more than twice, automate it. Manual steps = variability + error. |
| Fast feedback loops | Developers should know within minutes if their change broke something. |
| Immutable artifacts | Build once, deploy the same artifact everywhere. Never rebuild per environment. |
| Everything as code | Pipelines, infrastructure, config, policies — version-controlled and peer-reviewed. |
| Shift left | Move quality gates (testing, security, compliance) earlier where fixes are cheaper. |
Pipeline Design
Standard Pipeline Stages
Source → Build → Unit Test → Static Review → Security Scan → Integration Test → Staging Deploy → Acceptance Test → Production Deploy → Post-Deploy Validation
Each stage = quality gate. A change must pass all gates.
Pipeline Design Principles
- Every commit triggers the pipeline — No manual start buttons
- Fail fast — Order stages by speed and likelihood of failure (unit → integration → E2E)
- Artifacts built once — CI produces versioned, immutable artifact; promoted through all envs
- Configuration injected, not baked — Env-specific values injected at deploy time via env vars, parameter stores, config services
- Stages isolated — Each stage runs in its own environment; failure in test doesn't affect staging
- Pipeline state visible — Every team member sees current state, what's deployed where, what's queued
- Pipeline defined as code — Pipeline definition lives in same repo as app code, versioned and reviewed
Continuous Integration (CI)
Rapid vs Full Pipelines
| Type | Duration | Trigger | Scope |
|---|---|---|---|
| Rapid | ≤10 min | Feature branches | Lint, unit tests, fast checks |
| Full | >10 min | PR/merge to main | Comprehensive tests, security scans, build |
Goal: Commit-stage feedback loop under 10 minutes. Beyond that, developers context-switch and productivity falls.
CI Best Practices
- Lint and static analysis early — Catch syntax, style, simple bugs before tests run
- Run tests in parallel — Maximize core utilization
- Cache dependencies — Restore
node_modules,.m2,target/,vendor/between runs - Fail on warnings — Treat warnings as errors in CI (configurable per project)
- Report coverage — Enforce minimum thresholds; fail build if coverage drops
- Test the artifact — Run integration tests against the built artifact, not source
Continuous Delivery (CD)
GitOps Methodology
GitOps = Declarative infrastructure stored in Git + CI/CD tools to deploy that infrastructure.
Benefits: - All changes reviewed via PR/merge requests - Single source of truth for cluster/app state - Snapshots enable recovery from failures - Audit trail built-in
Tools: ArgoCD, Flux, Terraform + Atlantis, Config Connector
Deployment Strategies
| Strategy | Description | Rollback | Risk | Best For |
|---|---|---|---|---|
| Rolling | Replace instances gradually | Slow (re-deploy) | Low | Stateless services, default choice |
| Blue-Green | Two identical envs, switch traffic | Instant | Medium | Critical services, zero-downtime needs |
| Canary | Route small % traffic to new version | Fast (route back) | Low | High-risk changes, data-driven validation |
| Feature Flags | Deploy code, toggle exposure | Instant | Lowest | Decouple deploy from release |
Progressive Delivery
Combine canary releases with automated metric analysis: - Define objective health criteria in advance (error rate, latency, saturation) - System promotes or rolls back automatically based on signals - Removes human hesitation that turns small incidents into major ones
Security in Pipeline (DevSecOps)
Shift Left Security
| Stage | Security Activities |
|---|---|
| Code | SAST (static analysis), secret scanning, dependency scanning |
| Build | Container image vulnerability scanning, SBOM generation |
| Test | DAST (dynamic analysis), API security testing |
| Deploy | Policy enforcement (OPA/Gatekeeper), admission control |
| Runtime | Runtime security (Falco), drift detection |
Key Practices
- Require code review for all production changes
- Scan container images for vulnerabilities; prevent vulnerable images from deploying
- Use Binary Authorization — require attestations (vuln scan passed, QA passed, owner sign-off)
- Least privilege pipeline credentials — separate roles for build/test/deploy; short-lived tokens
- Audit pipeline permissions regularly
Observability in Pipeline
Deployment Observability
Pre-deployment: - Record baseline metrics (error rate, latency, throughput) - Verify monitoring/alerting functional - Confirm rollback alarms armed
During deployment: - Track progress (% instances updated, health checks) - Compare real-time metrics vs baseline - Monitor canary metrics if using canary
Post-deployment (bake period): - Run automated smoke tests + synthetic monitors - Compare post-deployment metrics vs baseline - Watch for delayed-onset issues (memory leaks, connection pool exhaustion, cache warming) - Monitor business KPIs (conversion, orders, API volume)
Key Metrics to Monitor Post-Deployment
| Category | Metrics | Alarm Threshold Example |
|---|---|---|
| Availability | Error rate (4xx, 5xx), health check failures | >1% 5xx for 5 min |
| Latency | p50, p95, p99 response time | p99 >2x baseline for 5 min |
| Throughput | RPS, transactions/min | <80% baseline for 10 min |
| Resources | CPU, memory, disk, connection pool | >85% for 10 min |
| Business | Conversion rate, revenue/min, order volume | >10% drop from baseline |
| Downstream | Dependency error rates, latency | Any dependency >5% error rate |
Pipeline Observability (Monitor the Pipeline Itself)
- Build success rate — % builds completing all stages
- Pipeline duration — Commit to production (track p50, p95)
- Queue time — Time waiting for runners/agents
- Flaky test rate — % tests intermittently failing without code changes
- Stage failure distribution — Which stages fail most (indicates where to invest)
Branching Strategy: Trunk-Based Development
Integrate all work into a single shared branch (main/trunk) with few/no long-lived branches.
Practices: - Short-lived feature branches (hours to 1-2 days max) - Feature flags to hide incomplete work - Small, frequent commits - Main branch always deployable
Avoid: Long-lived feature branches, release branches, git-flow for continuous delivery.
Release Management
Versioning
- Semantic Versioning (MAJOR.MINOR.PATCH) for libraries/services
- Git tags for releases:
v1.2.3 - Container tags:
myapp:v1.2.3,myapp:latest(avoid in prod)
Change Control (Regulated Environments)
- Lightweight but real: auto-capture who approved, what tests ran, what artifact deployed
- Standard pre-approved change types flow without ceremony
- Reserve change advisory board for genuinely high-risk changes
Rollback Strategy
- Automated rollback on metric anomaly (progressive delivery)
- One-click manual rollback for all deployments
- Database migrations — backward compatible; rollback script tested
Common Anti-Patterns
| Anti-Pattern | Fix |
|---|---|
| Manual deployment steps | Automate everything; pipeline as code |
| Rebuilding per environment | Build once, promote immutable artifact |
| Long-lived branches | Trunk-based development + feature flags |
| No pipeline observability | Instrument pipeline; alert on duration/failure trends |
| Security scans only at end | Shift left: SAST in IDE, dependency scan in CI, container scan in build |
| No rollback plan | Every deployment must have tested rollback |
Related Notes
- Infrastructure as Code — Terraform, Pulumi, GitOps
- Observability — Logging, metrics, tracing, alerting
- Git Workflow — Branching, commits, PR practices
- Release Management — Versioning, rollback, progressive delivery
- Test Pyramid — Test strategy in pipeline
- Supply Chain Security — SBOM, dependency scanning, signing
Sources: Google Cloud CI/CD Best Practices, AWS Builder Center CI/CD Patterns, CD Foundation Best Practices, Microsoft Azure DevOps, GitLab CI/CD Guide, Software Engineering Guide (Chapter 8), Harness GitOps Blog