# Performance Engineering

A structured, repeatable approach to meeting performance objectives throughout the software lifecycle.

---

## Performance Engineering Lifecycle

```
┌─────────────────┐
│ 1. Objectives   │  ← Set measurable goals (latency, throughput, resources)
└────────┬────────┘
         ▼
┌─────────────────┐
│ 2. Budgeting    │  ← Allocate resources (CPU, memory, network, cost)
└────────┬────────┘
         ▼
┌─────────────────┐
│ 3. Modeling     │  ← Predict performance; identify bottlenecks early
└────────┬────────┘
         ▼
┌─────────────────┐
│ 4. Design       │  ← Apply patterns; validate architecture early
└────────┬────────┘
         ▼
┌─────────────────┐
│ 5. Code Inspect │  ← Review for inefficient patterns
└────────┬────────┘
         ▼
┌─────────────────┐
│ 6. Testing      │  ← Load/stress testing; verify under load
└────────┬────────┘
         ▼
┌─────────────────┐
│ 7. Tuning       │  ← Iterative: baseline → collect → analyze → change → retest
└────────┬────────┘
         ▼
┌─────────────────┐
│ 8. Monitoring   │  ← Production metrics; capacity planning
└─────────────────┘
```

---

## Setting Performance Objectives

**Avoid ambiguous goals**: "The app must run fast" → **Measurable**: "p99 latency < 200ms at 1000 RPS with 99.9% availability"

### Requirements to Define

| Dimension | Example Objectives |
|---|---|
| **Response Time** | p50 < 50ms, p95 < 150ms, p99 < 300ms |
| **Throughput** | 10,000 RPS sustained; 50,000 peak |
| **Resource Utilization** | CPU < 70%, Memory < 80%, Disk I/O < 60% |
| **Workload** | 100K concurrent users; 1M daily active |
| **Scalability** | Linear scale to 10x load with added nodes |
| **Availability** | 99.99% uptime; < 30s recovery |

### Key Questions
- How long should a particular request take?
- How many users must be supported?
- What is peak load?
- How many transactions/second?
- What are acceptable resource thresholds?

---

## Budgeting

**Performance Budget** = Constraints on resources (time, CPU, memory, network, cost) allocated to each component.

```
Total Budget (e.g., 200ms p99)
├── Network: 20ms
├── Auth: 10ms
├── Business Logic: 50ms
├── Database: 80ms
├── External APIs: 30ms
└── Buffer: 10ms
```

- Enables team autonomy within constraints
- Makes trade-offs explicit
- Prevents "death by a thousand cuts"

---

## Performance Modeling

**Model early** — validate architecture *before* building.

### Decisions to Validate Up Front
- Deployment topology (regions, zones, instance types)
- Load balancing strategy
- Network bandwidth/latency
- Auth/authorization strategy
- Exception handling overhead
- Instrumentation overhead
- Database design + access patterns
- Data access strategies (caching, indexing, sharding)
- State management (stateless vs stateful)
- Caching strategy

### Modeling Techniques
- **Queuing theory** — Little's Law: `L = λW` (avg items = arrival rate × avg wait)
- **Capacity planning** — Resource demand × throughput = required capacity
- **Bottleneck analysis** — Identify serial vs parallel paths
- **Simulation** — Discrete event simulation for complex systems

---

## Design Guidelines for Performance

### 1. Set Measurable Goals
No "fast" — use numbers with percentiles and conditions.

### 2. Validate Architecture Early
Prototype key design choices. Be prepared to cut features that don't meet goals.

### 3. Cut the Deadwood
Often greatest gains from removing **whole sections of unnecessary work**.
- Interim results from function A not used by function B? Eliminate the waste path.
- Composed functions often have unused intermediate results.

### 4. Tune End-to-End
Optimizing one subsystem can hurt overall performance.
- Single bottleneck affects everything regardless of other tuning
- Identify bottlenecks → tune specific parts → move to next bottleneck

### 5. Measure Throughout Lifecycle
- Unit test performance of specific code pieces
- Verify code meets objectives before integration
- Continue measuring in production (usage patterns, data volumes change)

---

## Code-Level Optimization

### Instrument Your Code
- Add metrics collection (latency, throughput, resource usage)
- **Earlier = cheaper** to fix
- Automate analysis (static + dynamic)
- Production provides clearest understanding

### Identify Hot Paths
1. Collect runtime data (CPU, memory, I/O)
2. Profile with tools (pprof, perf, async-profiler, VisualVM, py-spy)
3. Find frequently executed / high-resource sections
4. Consider business criticality — optimize what matters to users

### Optimize Code Logic (Priority Order)

1. **Remove unnecessary function calls** — Redundant validations, duplicate checks
2. **Minimize logging** — Excessive logging kills performance; evaluate necessity
3. **Optimize loops/conditionals** —
   - Move computations outside loops
   - Eliminate redundant calculations
   - Minimize function calls in loops
   - Consider loop unrolling
4. **Reduce unnecessary data processing** — Redundant transformations, copies
5. **Optimize data structures** — Right structure for the problem (hash map vs list, etc.)
6. **Minimize network requests** — Batch, avoid round trips, connection pooling
7. **Minimize allocations** — Reuse objects, object pools, reduce GC pressure
8. **Reduce data structure size** — Remove unused fields, pack efficiently
9. **Use performance-optimized SDKs/libraries** — Native SDKs, Math.NET, etc.
10. **Review cross-cutting concerns** — Middleware, token checks, interceptors

### Language-Specific
- Follow language-specific performance recommendations
- Understand GC behavior, JIT, runtime characteristics

---

## Caching Strategies

### Cache Patterns

| Pattern | Description | Use When |
|---|---|---|
| **Look-aside** | App checks cache → misses → loads from DB → populates cache | General purpose; read-heavy |
| **Read-through** | Cache sits in front of DB; handles miss transparently | Abstract cache from app |
| **Write-through** | Write to cache + DB synchronously | Strong consistency needed |
| **Write-behind** | Write to cache; async flush to DB | High write throughput; eventual consistency OK |
| **Refresh-ahead** | Proactively refresh before expiry | Predictable access patterns |

### Invalidation Strategies
- **TTL** — Simple; stale data possible
- **Event-based** — Invalidate on data change (pub/sub); more complex
- **Write-through** — Update cache on write; strong consistency
- **Versioned keys** — `key:v2` — no invalidation needed; cache grows

### Cache Considerations
- **Cache stampede** — Use probabilistic early expiration or distributed locks
- **Cold start** — Pre-warm critical caches
- **Memory pressure** — Eviction policies (LRU, LFU, TTL)
- **Consistency** — Accept staleness or pay for strong consistency

---

## Database Optimization

### Query Tuning
- **Index strategically** — Cover query patterns; avoid over-indexing (write penalty)
- **EXPLAIN ANALYZE** — Understand query plans
- **Avoid N+1** — Eager load / batch fetch
- **Pagination** — Keyset pagination (`WHERE id > last_id`) over OFFSET
- **Denormalize judiciously** — Materialized views, read replicas for reporting

### Connection Management
- **Connection pooling** — Reuse connections; tune pool size
- **Prepared statements** — Parse once, execute many
- **Batch operations** — Bulk insert/update vs individual

### Scaling
- **Read replicas** — Offload reads; async replication lag
- **Sharding** — Horizontal partition by tenant/id/hash; complex
- **Caching layer** — Redis/Memcached between app and DB

---

## Concurrency & Scaling

### Thread/Async Models
| Model | Pros | Cons |
|---|---|---|
| **Thread pool** | Simple, blocking OK | Memory per thread; context switch overhead |
| **Event loop (async/await)** | High concurrency, low memory | Non-blocking everywhere; callback complexity |
| **Actor model** | Isolation, fault tolerance | Learning curve; message passing overhead |
| **Coroutine/fiber** | Lightweight, structured concurrency | Runtime support needed |

### Horizontal Scaling
- **Stateless services** — Scale out trivially
- **Session affinity** — Avoid; use shared session store
- **Load balancing** — Round-robin, least connections, consistent hashing
- **Autoscaling** — Metric-based (CPU, RPS, queue depth); predictive for known patterns

---

## Performance Testing

### Test Types
| Type | Purpose | When |
|---|---|---|
| **Load** | Expected production load | Every release |
| **Stress** | Beyond expected (breaking point) | Periodic; pre-major release |
| **Soak** | Extended duration (hours/days) | Pre-production; find leaks |
| **Spike** | Sudden burst | Event-driven systems |
| **Chaos** | Failure injection | Continuous in staging |

### Metrics to Capture
- Latency (p50, p95, p99, p99.9)
- Throughput (RPS, TPS)
- Error rate (by type)
- Resource utilization (CPU, mem, disk, network)
- GC pauses, thread counts, queue depths

---

## Production Monitoring

### Golden Signals (Google SRE)
1. **Latency** — Time to service request
2. **Traffic** — Request rate (RPS)
3. **Errors** — Failure rate
4. **Saturation** — Resource utilization %

### USE Method (Brendan Gregg)
For every resource: **Utilization, Saturation, Errors**

### RED Method (Tom Wilkie)
For every service: **Rate, Errors, Duration**

---

## Common Anti-Patterns

| Anti-Pattern | Fix |
|---|---|
| **Premature optimization** | Measure first; optimize bottlenecks only |
| **Masking with brute force** | Add capacity instead of fixing code |
| **Optimizing non-bottlenecks** | Profile; focus on hot paths |
| **No performance budgets** | Set budgets; enforce in CI |
| **Ignoring GC/allocations** | Profile allocations; reduce pressure |
| **Single-threaded thinking** | Design for concurrency from start |
| **No production observability** | Instrument; alert on golden signals |

---

## Related Notes

- [[Performance/Profiling-Optimization]] — Profiling tools, techniques
- [[Performance/Caching-Strategies]] — Deep dive on caching
- [[Performance/Database-Optimization]] — Query tuning, indexing, scaling
- [[Performance/Concurrency-Scaling]] — Thread models, horizontal scaling
- [[DevOps/Observability]] — Logging, metrics, tracing, alerting
- [[Testing/Test-Pyramid]] — Performance testing in pipeline

---

*Sources: Microsoft Patterns & Practices Performance Engineering, Google Cloud Well-Architected (Performance), Azure Well-Architected (Performance Efficiency), AWS Well-Architected (Performance Efficiency), Intel Software Performance Optimization, SonarSource Performance Guide, Brendan Gregg (USE Method), Google SRE (Golden Signals)*