# Trade Study Methodology

A trade study is a structured, objective comparison of alternatives to support a decision. The goal isn't to "prove" your favorite choice - it's to make the tradeoffs explicit and document the rationale.

## When to Do a Trade Study

Do a trade study when:
- **Multiple viable alternatives** exist with no obvious best choice
- **Significant cost/schedule/performance impact**
- **Irreversible or hard-to-change decisions** (architecture, vendors, core tech stack)
- **Stakeholders disagree** and need objective comparison
- **Need to justify decision** to others or for future reference

## Trade Study Process

```mermaid
graph TD
    A[1. Define Problem & Objectives] --> B[2. Identify Alternatives]
    B --> C[3. Define Evaluation Criteria]
    C --> D[4. Weight Criteria]
    D --> E[5. Evaluate Alternatives]
    E --> F[6. Analyze Results]
    F --> G[7. Document & Decide]
    G --> H[8. Review & Iterate]
    
    style A fill:#e1f5fe
    style G fill:#e8f5e9
```

## Step by Step

### 1. Define Problem & Objectives
- State the decision clearly: "Should we use X or Y for Z?"
- Define objectives (what are we trying to achieve?)
- State constraints (budget, timeline, compliance)
- Identify stakeholders

### 2. Identify Alternatives
- Include at least 2-3 alternatives (including "do nothing" or baseline)
- Be creative but realistic
- Document why alternatives were considered or rejected

### 3. Define Evaluation Criteria
Good criteria are:
- Relevant to objectives
- Measurable where possible
- Cover key concerns (technical, operational, cost)
- Not redundant

Common criteria categories: Performance, Cost, Schedule, Risk, Scalability, Maintainability, Security, Usability, Vendor Lock-in

### 4. Weight Criteria
Not all criteria are equal. Use a weighting method:
- **Simple ranking**: Rank criteria 1..N
- **Pairwise comparison**: Compare each pair
- **Analytic Hierarchy Process (AHP)**: More rigorous for complex decisions
- **Consensus**: Get stakeholder agreement on weights

### 5. Evaluate Alternatives
Score each alternative against each criterion. Use consistent scales:
- **1-5 scale**: 1=poor, 5=excellent
- **0-3 scale**: 0=doesn't meet, 3=exceeds
- **Satisfactory/Unsatisfactory**: For binary criteria
- **Quantitative**: Use actual numbers when possible (cost $, latency ms)

### 6. Analyze Results
- Compute weighted scores
- Do sensitivity analysis (what if weights change?)
- Look at risks and qualitative factors
- Don't just pick highest score blindly - consider if any alternative is disqualified

### 7. Document & Decide
Write a trade study report with all the above plus recommendation and rationale.

## Pugh Matrix (Simple & Effective)

The Pugh matrix is the most common and practical approach. Use a baseline and rate +, 0, - relative to baseline.

### Example: Choosing a Vector Database for RAG

| Criteria | Weight | Baseline (pgvector) | Weaviate | Pinecone | Qdrant |
|---|---|---|---|---|---|
| **Performance/Latency** | 25% | 0 | + | + | + |
| **Scalability** | 20% | 0 | + | ++ | + |
| **Operational Complexity** | 15% | 0 | - | ++ | 0 |
| **Cost** | 15% | 0 | - | -- | - |
| **Ecosystem/Integrations** | 10% | 0 | + | + | + |
| **Security/Compliance** | 10% | 0 | 0 | + | 0 |
| **Weighted Score** | 100% | 0 | +0.35 | +0.40 | +0.35 |

*Legend: ++ = much better, + = better, 0 = same, - = worse, -- = much worse*

### Example: Choosing Model Serving for Agents

| Criteria | Weight | Baseline (OpenAI API) | Self-hosted vLLM | Self-hosted SGLang | Together AI |
|---|---|---|---|---|---|
| **Latency/TTFT** | 20% | 0 | + | ++ | 0 |
| **Cost at Scale** | 20% | 0 | ++ | ++ | - |
| **Control/Flexibility** | 15% | 0 | ++ | ++ | - |
| **Ops Complexity** | 15% | 0 | -- | - | ++ |
| **Reliability/SLA** | 12% | 0 | - | - | ++ |
| **Data Privacy** | 10% | 0 | ++ | ++ | - |
| **Eval/Observability** | 8% | 0 | + | + | + |
| **Weighted Score** | 100% | 0 | +0.73 | +0.92 | +0.07 |

## Trade Study Document Template

```markdown
# Trade Study: [Decision Title]
**Date:** YYYY-MM-DD  
**Author:** [Name]  
**Reviewers:** [Names]  
**Status:** [Draft/In Review/Approved]

## Problem Statement
[Clear statement of the decision to be made]

## Objectives
[What we want to achieve]

## Constraints
[Budget, timeline, compliance, etc.]

## Alternatives Considered
| Alternative | Description | Why considered/rejected |
|---|---|---|
| ... | ... | ... |

## Evaluation Criteria
| Criteria | Weight | Rationale | Scoring Method |
|---|---|---|---|
| ... | ... | ... | ... |

## Evaluation (Pugh Matrix or Scoring Table)
[Full matrix here]

## Analysis
- Sensitivity analysis
- Key risks per alternative
- Disqualifiers

## Recommendation
[Recommended alternative with rationale]

## Decision
[Final decision, approver, date]

## References
[Docs, benchmarks, POCs]
```

## Best Practices

1. **Define criteria before evaluating** - Avoid bias by setting criteria first
2. **Include a baseline** - Makes comparison intuitive
3. **Quantify when possible** - Use real data from benchmarks/POCs
4. **Do a POC** - For high-risk decisions, test top 2 alternatives
5. **Document everything** - Future you will thank you
6. **Consider total cost of ownership** - Not just upfront cost (ops, maintenance, training)
7. **Account for reversibility** - If easily reversible, less need for perfect trade study
8. **Involve right stakeholders** - Include those who'll live with the decision
9. **Challenge assumptions** - Do sensitivity analysis on weights
10. **Be honest about risks** - Every choice has downsides

## Related Notes

- [[Architecture/Architecture-Decision-Records.md]] — Documenting final decisions
- [[Foundations/Systems-Thinking.md]] — Second-order effects
- [[Risk-Reliability/Risk-Analysis-FMEA-FTA.md]] — Risk in trade studies
- [[../SoftEngine/Architecture/Architecture-Decision-Records.md]] — ADR format
- [[../AgenticEngineering/Frameworks/Framework-Comparison.md]] — Example trade study

## Sources

* INCOSE Systems Engineering Handbook  
* NASA Systems Engineering Handbook  
* MITRE Systems Engineering Guide - Trade Studies  
* Blanchard & Fabrycky - *Systems Engineering and Analysis*  
* Stuart Pugh - *Total Design* (concept origin)