Trade Study Methodology

A trade study is a structured, objective comparison of alternatives to support a decision. The goal isn't to "prove" your favorite choice - it's to make the tradeoffs explicit and document the rationale.

When to Do a Trade Study

Do a trade study when: - Multiple viable alternatives exist with no obvious best choice - Significant cost/schedule/performance impact - Irreversible or hard-to-change decisions (architecture, vendors, core tech stack) - Stakeholders disagree and need objective comparison - Need to justify decision to others or for future reference

Trade Study Process

graph TD
    A[1. Define Problem & Objectives] --> B[2. Identify Alternatives]
    B --> C[3. Define Evaluation Criteria]
    C --> D[4. Weight Criteria]
    D --> E[5. Evaluate Alternatives]
    E --> F[6. Analyze Results]
    F --> G[7. Document & Decide]
    G --> H[8. Review & Iterate]

    style A fill:#e1f5fe
    style G fill:#e8f5e9

Step by Step

1. Define Problem & Objectives

• State the decision clearly: "Should we use X or Y for Z?"
• Define objectives (what are we trying to achieve?)
• State constraints (budget, timeline, compliance)
• Identify stakeholders

2. Identify Alternatives

• Include at least 2-3 alternatives (including "do nothing" or baseline)
• Be creative but realistic
• Document why alternatives were considered or rejected

3. Define Evaluation Criteria

Good criteria are: - Relevant to objectives - Measurable where possible - Cover key concerns (technical, operational, cost) - Not redundant

Common criteria categories: Performance, Cost, Schedule, Risk, Scalability, Maintainability, Security, Usability, Vendor Lock-in

4. Weight Criteria

Not all criteria are equal. Use a weighting method: - Simple ranking: Rank criteria 1..N - Pairwise comparison: Compare each pair - Analytic Hierarchy Process (AHP): More rigorous for complex decisions - Consensus: Get stakeholder agreement on weights

5. Evaluate Alternatives

Score each alternative against each criterion. Use consistent scales: - 1-5 scale: 1=poor, 5=excellent - 0-3 scale: 0=doesn't meet, 3=exceeds - Satisfactory/Unsatisfactory: For binary criteria - Quantitative: Use actual numbers when possible (cost $, latency ms)

6. Analyze Results

• Compute weighted scores
• Do sensitivity analysis (what if weights change?)
• Look at risks and qualitative factors
• Don't just pick highest score blindly - consider if any alternative is disqualified

7. Document & Decide

Write a trade study report with all the above plus recommendation and rationale.

Pugh Matrix (Simple & Effective)

The Pugh matrix is the most common and practical approach. Use a baseline and rate +, 0, - relative to baseline.

Example: Choosing a Vector Database for RAG

Legend: ++ = much better, + = better, 0 = same, - = worse, -- = much worse

Example: Choosing Model Serving for Agents

Trade Study Document Template

# Trade Study: [Decision Title]
**Date:** YYYY-MM-DD  
**Author:** [Name]  
**Reviewers:** [Names]  
**Status:** [Draft/In Review/Approved]

## Problem Statement
[Clear statement of the decision to be made]

## Objectives
[What we want to achieve]

## Constraints
[Budget, timeline, compliance, etc.]

## Alternatives Considered
| Alternative | Description | Why considered/rejected |
|---|---|---|
| ... | ... | ... |

## Evaluation Criteria
| Criteria | Weight | Rationale | Scoring Method |
|---|---|---|---|
| ... | ... | ... | ... |

## Evaluation (Pugh Matrix or Scoring Table)
[Full matrix here]

## Analysis
- Sensitivity analysis
- Key risks per alternative
- Disqualifiers

## Recommendation
[Recommended alternative with rationale]

## Decision
[Final decision, approver, date]

## References
[Docs, benchmarks, POCs]

Best Practices

1. Define criteria before evaluating - Avoid bias by setting criteria first
2. Include a baseline - Makes comparison intuitive
3. Quantify when possible - Use real data from benchmarks/POCs
4. Do a POC - For high-risk decisions, test top 2 alternatives
5. Document everything - Future you will thank you
6. Consider total cost of ownership - Not just upfront cost (ops, maintenance, training)
7. Account for reversibility - If easily reversible, less need for perfect trade study
8. Involve right stakeholders - Include those who'll live with the decision
9. Challenge assumptions - Do sensitivity analysis on weights
10. Be honest about risks - Every choice has downsides

Related Notes

• Architecture Decision Records — Documenting final decisions
• Systems Thinking — Second-order effects
• Risk Analysis FMEA FTA — Risk in trade studies
• Architecture Decision Records — ADR format
• Framework Comparison — Example trade study

Sources

• INCOSE Systems Engineering Handbook
• NASA Systems Engineering Handbook
• MITRE Systems Engineering Guide - Trade Studies
• Blanchard & Fabrycky - Systems Engineering and Analysis
• Stuart Pugh - Total Design (concept origin)