Performance Engineering

A structured, repeatable approach to meeting performance objectives throughout the software lifecycle.

Performance Engineering Lifecycle

┌─────────────────┐
│ 1. Objectives   │  ← Set measurable goals (latency, throughput, resources)
└────────┬────────┘
         ▼
┌─────────────────┐
│ 2. Budgeting    │  ← Allocate resources (CPU, memory, network, cost)
└────────┬────────┘
         ▼
┌─────────────────┐
│ 3. Modeling     │  ← Predict performance; identify bottlenecks early
└────────┬────────┘
         ▼
┌─────────────────┐
│ 4. Design       │  ← Apply patterns; validate architecture early
└────────┬────────┘
         ▼
┌─────────────────┐
│ 5. Code Inspect │  ← Review for inefficient patterns
└────────┬────────┘
         ▼
┌─────────────────┐
│ 6. Testing      │  ← Load/stress testing; verify under load
└────────┬────────┘
         ▼
┌─────────────────┐
│ 7. Tuning       │  ← Iterative: baseline → collect → analyze → change → retest
└────────┬────────┘
         ▼
┌─────────────────┐
│ 8. Monitoring   │  ← Production metrics; capacity planning
└─────────────────┘

Setting Performance Objectives

Avoid ambiguous goals: "The app must run fast" → Measurable: "p99 latency < 200ms at 1000 RPS with 99.9% availability"

Requirements to Define

Key Questions

• How long should a particular request take?
• How many users must be supported?
• What is peak load?
• How many transactions/second?
• What are acceptable resource thresholds?

Budgeting

Performance Budget = Constraints on resources (time, CPU, memory, network, cost) allocated to each component.

Total Budget (e.g., 200ms p99)
├── Network: 20ms
├── Auth: 10ms
├── Business Logic: 50ms
├── Database: 80ms
├── External APIs: 30ms
└── Buffer: 10ms

• Enables team autonomy within constraints
• Makes trade-offs explicit
• Prevents "death by a thousand cuts"

Performance Modeling

Model early — validate architecture before building.

Decisions to Validate Up Front

• Deployment topology (regions, zones, instance types)
• Load balancing strategy
• Network bandwidth/latency
• Auth/authorization strategy
• Exception handling overhead
• Instrumentation overhead
• Database design + access patterns
• Data access strategies (caching, indexing, sharding)
• State management (stateless vs stateful)
• Caching strategy

Modeling Techniques

• Queuing theory — Little's Law: L = λW (avg items = arrival rate × avg wait)
• Capacity planning — Resource demand × throughput = required capacity
• Bottleneck analysis — Identify serial vs parallel paths
• Simulation — Discrete event simulation for complex systems

Design Guidelines for Performance

1. Set Measurable Goals

No "fast" — use numbers with percentiles and conditions.

2. Validate Architecture Early

Prototype key design choices. Be prepared to cut features that don't meet goals.

3. Cut the Deadwood

Often greatest gains from removing whole sections of unnecessary work. - Interim results from function A not used by function B? Eliminate the waste path. - Composed functions often have unused intermediate results.

4. Tune End-to-End

Optimizing one subsystem can hurt overall performance. - Single bottleneck affects everything regardless of other tuning - Identify bottlenecks → tune specific parts → move to next bottleneck

5. Measure Throughout Lifecycle

• Unit test performance of specific code pieces
• Verify code meets objectives before integration
• Continue measuring in production (usage patterns, data volumes change)

Code-Level Optimization

Instrument Your Code

• Add metrics collection (latency, throughput, resource usage)
• Earlier = cheaper to fix
• Automate analysis (static + dynamic)
• Production provides clearest understanding

Identify Hot Paths

1. Collect runtime data (CPU, memory, I/O)
2. Profile with tools (pprof, perf, async-profiler, VisualVM, py-spy)
3. Find frequently executed / high-resource sections
4. Consider business criticality — optimize what matters to users

Optimize Code Logic (Priority Order)

1. Remove unnecessary function calls — Redundant validations, duplicate checks
2. Minimize logging — Excessive logging kills performance; evaluate necessity
3. Optimize loops/conditionals — - Move computations outside loops - Eliminate redundant calculations - Minimize function calls in loops - Consider loop unrolling
4. Reduce unnecessary data processing — Redundant transformations, copies
5. Optimize data structures — Right structure for the problem (hash map vs list, etc.)
6. Minimize network requests — Batch, avoid round trips, connection pooling
7. Minimize allocations — Reuse objects, object pools, reduce GC pressure
8. Reduce data structure size — Remove unused fields, pack efficiently
9. Use performance-optimized SDKs/libraries — Native SDKs, Math.NET, etc.
10. Review cross-cutting concerns — Middleware, token checks, interceptors

Language-Specific

• Follow language-specific performance recommendations
• Understand GC behavior, JIT, runtime characteristics

Caching Strategies

Cache Patterns

Invalidation Strategies

• TTL — Simple; stale data possible
• Event-based — Invalidate on data change (pub/sub); more complex
• Write-through — Update cache on write; strong consistency
• Versioned keys — key:v2 — no invalidation needed; cache grows

Cache Considerations

• Cache stampede — Use probabilistic early expiration or distributed locks
• Cold start — Pre-warm critical caches
• Memory pressure — Eviction policies (LRU, LFU, TTL)
• Consistency — Accept staleness or pay for strong consistency

Database Optimization

Query Tuning

• Index strategically — Cover query patterns; avoid over-indexing (write penalty)
• EXPLAIN ANALYZE — Understand query plans
• Avoid N+1 — Eager load / batch fetch
• Pagination — Keyset pagination (WHERE id > last_id) over OFFSET
• Denormalize judiciously — Materialized views, read replicas for reporting

Connection Management

• Connection pooling — Reuse connections; tune pool size
• Prepared statements — Parse once, execute many
• Batch operations — Bulk insert/update vs individual

Scaling

• Read replicas — Offload reads; async replication lag
• Sharding — Horizontal partition by tenant/id/hash; complex
• Caching layer — Redis/Memcached between app and DB

Concurrency & Scaling

Thread/Async Models

Horizontal Scaling

• Stateless services — Scale out trivially
• Session affinity — Avoid; use shared session store
• Load balancing — Round-robin, least connections, consistent hashing
• Autoscaling — Metric-based (CPU, RPS, queue depth); predictive for known patterns

Performance Testing

Test Types

Metrics to Capture

• Latency (p50, p95, p99, p99.9)
• Throughput (RPS, TPS)
• Error rate (by type)
• Resource utilization (CPU, mem, disk, network)
• GC pauses, thread counts, queue depths

Production Monitoring

Golden Signals (Google SRE)

1. Latency — Time to service request
2. Traffic — Request rate (RPS)
3. Errors — Failure rate
4. Saturation — Resource utilization %

USE Method (Brendan Gregg)

For every resource: Utilization, Saturation, Errors

RED Method (Tom Wilkie)

For every service: Rate, Errors, Duration

Common Anti-Patterns

Related Notes

• Profiling Optimization — Profiling tools, techniques
• Caching Strategies — Deep dive on caching
• Database Optimization — Query tuning, indexing, scaling
• Concurrency Scaling — Thread models, horizontal scaling
• Observability — Logging, metrics, tracing, alerting
• Test Pyramid — Performance testing in pipeline

Sources: Microsoft Patterns & Practices Performance Engineering, Google Cloud Well-Architected (Performance), Azure Well-Architected (Performance Efficiency), AWS Well-Architected (Performance Efficiency), Intel Software Performance Optimization, SonarSource Performance Guide, Brendan Gregg (USE Method), Google SRE (Golden Signals)