# CI/CD Best Practices

Continuous Integration and Continuous Delivery practices for reliable, fast software delivery.

---

## Core Principles

| Principle | Description |
|---|---|
| **Small, frequent changes** | Reduce batch size to reduce risk. Each deployment = minimum viable change. |
| **Automate everything repeatable** | If a human does it more than twice, automate it. Manual steps = variability + error. |
| **Fast feedback loops** | Developers should know within minutes if their change broke something. |
| **Immutable artifacts** | Build once, deploy the same artifact everywhere. Never rebuild per environment. |
| **Everything as code** | Pipelines, infrastructure, config, policies — version-controlled and peer-reviewed. |
| **Shift left** | Move quality gates (testing, security, compliance) earlier where fixes are cheaper. |

---

## Pipeline Design

### Standard Pipeline Stages

```
Source → Build → Unit Test → Static Review → Security Scan → Integration Test → Staging Deploy → Acceptance Test → Production Deploy → Post-Deploy Validation
```

Each stage = quality gate. A change must pass all gates.

### Pipeline Design Principles

1. **Every commit triggers the pipeline** — No manual start buttons
2. **Fail fast** — Order stages by speed and likelihood of failure (unit → integration → E2E)
3. **Artifacts built once** — CI produces versioned, immutable artifact; promoted through all envs
4. **Configuration injected, not baked** — Env-specific values injected at deploy time via env vars, parameter stores, config services
5. **Stages isolated** — Each stage runs in its own environment; failure in test doesn't affect staging
6. **Pipeline state visible** — Every team member sees current state, what's deployed where, what's queued
7. **Pipeline defined as code** — Pipeline definition lives in same repo as app code, versioned and reviewed

---

## Continuous Integration (CI)

### Rapid vs Full Pipelines

| Type | Duration | Trigger | Scope |
|---|---|---|---|
| **Rapid** | ≤10 min | Feature branches | Lint, unit tests, fast checks |
| **Full** | >10 min | PR/merge to main | Comprehensive tests, security scans, build |

**Goal**: Commit-stage feedback loop under 10 minutes. Beyond that, developers context-switch and productivity falls.

### CI Best Practices

- **Lint and static analysis early** — Catch syntax, style, simple bugs before tests run
- **Run tests in parallel** — Maximize core utilization
- **Cache dependencies** — Restore `node_modules`, `.m2`, `target/`, `vendor/` between runs
- **Fail on warnings** — Treat warnings as errors in CI (configurable per project)
- **Report coverage** — Enforce minimum thresholds; fail build if coverage drops
- **Test the artifact** — Run integration tests against the *built artifact*, not source

---

## Continuous Delivery (CD)

### GitOps Methodology

**GitOps** = Declarative infrastructure stored in Git + CI/CD tools to deploy that infrastructure.

**Benefits**:
- All changes reviewed via PR/merge requests
- Single source of truth for cluster/app state
- Snapshots enable recovery from failures
- Audit trail built-in

**Tools**: ArgoCD, Flux, Terraform + Atlantis, Config Connector

### Deployment Strategies

| Strategy | Description | Rollback | Risk | Best For |
|---|---|---|---|---|
| **Rolling** | Replace instances gradually | Slow (re-deploy) | Low | Stateless services, default choice |
| **Blue-Green** | Two identical envs, switch traffic | Instant | Medium | Critical services, zero-downtime needs |
| **Canary** | Route small % traffic to new version | Fast (route back) | Low | High-risk changes, data-driven validation |
| **Feature Flags** | Deploy code, toggle exposure | Instant | Lowest | Decouple deploy from release |

### Progressive Delivery

Combine canary releases with **automated metric analysis**:
- Define objective health criteria *in advance* (error rate, latency, saturation)
- System promotes or rolls back automatically based on signals
- Removes human hesitation that turns small incidents into major ones

---

## Security in Pipeline (DevSecOps)

### Shift Left Security

| Stage | Security Activities |
|---|---|
| **Code** | SAST (static analysis), secret scanning, dependency scanning |
| **Build** | Container image vulnerability scanning, SBOM generation |
| **Test** | DAST (dynamic analysis), API security testing |
| **Deploy** | Policy enforcement (OPA/Gatekeeper), admission control |
| **Runtime** | Runtime security (Falco), drift detection |

### Key Practices

- **Require code review** for all production changes
- **Scan container images** for vulnerabilities; prevent vulnerable images from deploying
- **Use Binary Authorization** — require attestations (vuln scan passed, QA passed, owner sign-off)
- **Least privilege pipeline credentials** — separate roles for build/test/deploy; short-lived tokens
- **Audit pipeline permissions regularly**

---

## Observability in Pipeline

### Deployment Observability

**Pre-deployment**:
- Record baseline metrics (error rate, latency, throughput)
- Verify monitoring/alerting functional
- Confirm rollback alarms armed

**During deployment**:
- Track progress (% instances updated, health checks)
- Compare real-time metrics vs baseline
- Monitor canary metrics if using canary

**Post-deployment (bake period)**:
- Run automated smoke tests + synthetic monitors
- Compare post-deployment metrics vs baseline
- Watch for delayed-onset issues (memory leaks, connection pool exhaustion, cache warming)
- Monitor business KPIs (conversion, orders, API volume)

### Key Metrics to Monitor Post-Deployment

| Category | Metrics | Alarm Threshold Example |
|---|---|---|
| Availability | Error rate (4xx, 5xx), health check failures | >1% 5xx for 5 min |
| Latency | p50, p95, p99 response time | p99 >2x baseline for 5 min |
| Throughput | RPS, transactions/min | <80% baseline for 10 min |
| Resources | CPU, memory, disk, connection pool | >85% for 10 min |
| Business | Conversion rate, revenue/min, order volume | >10% drop from baseline |
| Downstream | Dependency error rates, latency | Any dependency >5% error rate |

### Pipeline Observability (Monitor the Pipeline Itself)

- **Build success rate** — % builds completing all stages
- **Pipeline duration** — Commit to production (track p50, p95)
- **Queue time** — Time waiting for runners/agents
- **Flaky test rate** — % tests intermittently failing without code changes
- **Stage failure distribution** — Which stages fail most (indicates where to invest)

---

## Branching Strategy: Trunk-Based Development

**Integrate all work into a single shared branch** (main/trunk) with few/no long-lived branches.

**Practices**:
- Short-lived feature branches (hours to 1-2 days max)
- Feature flags to hide incomplete work
- Small, frequent commits
- Main branch always deployable

**Avoid**: Long-lived feature branches, release branches, git-flow for continuous delivery.

---

## Release Management

### Versioning
- **Semantic Versioning** (MAJOR.MINOR.PATCH) for libraries/services
- **Git tags** for releases: `v1.2.3`
- **Container tags**: `myapp:v1.2.3`, `myapp:latest` (avoid in prod)

### Change Control (Regulated Environments)
- Lightweight but real: auto-capture who approved, what tests ran, what artifact deployed
- Standard pre-approved change types flow without ceremony
- Reserve change advisory board for genuinely high-risk changes

### Rollback Strategy
- **Automated rollback** on metric anomaly (progressive delivery)
- **One-click manual rollback** for all deployments
- **Database migrations** — backward compatible; rollback script tested

---

## Common Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| **Manual deployment steps** | Automate everything; pipeline as code |
| **Rebuilding per environment** | Build once, promote immutable artifact |
| **Long-lived branches** | Trunk-based development + feature flags |
| **No pipeline observability** | Instrument pipeline; alert on duration/failure trends |
| **Security scans only at end** | Shift left: SAST in IDE, dependency scan in CI, container scan in build |
| **No rollback plan** | Every deployment must have tested rollback |

---

## Related Notes

- [[DevOps/Infrastructure-as-Code]] — Terraform, Pulumi, GitOps
- [[DevOps/Observability]] — Logging, metrics, tracing, alerting
- [[DevOps/Git-Workflow]] — Branching, commits, PR practices
- [[DevOps/Release-Management]] — Versioning, rollback, progressive delivery
- [[Testing/Test-Pyramid]] — Test strategy in pipeline
- [[Security/Supply-Chain-Security]] — SBOM, dependency scanning, signing

---

*Sources: Google Cloud CI/CD Best Practices, AWS Builder Center CI/CD Patterns, CD Foundation Best Practices, Microsoft Azure DevOps, GitLab CI/CD Guide, Software Engineering Guide (Chapter 8), Harness GitOps Blog*