Module 056 min

Trade studies: choosing on purpose

A trade study doesn't find the "best" option. It makes the tradeoffs visible enough that a decision can be defended — and revisited.

After this module
  • You can run a weighted decision matrix that changes someone's mind with evidence instead of volume.
  • You know how to check whether your own conclusion survives a different weighting.

Some decisions are cheap to reverse: a library, a color, a query. Others are not: a vendor, a data model, a cloud region, a public API shape. The cost of changing your mind rises steeply with time — which is why the rigour belongs on the irreversible ones.

COST TO CHANGE YOUR MIND · SCHEMATIC reversible (config, library) irreversible (schema, vendor, public API) cost time → decision made put the trade study where the curve is steep
Illustrative, direction not magnitude. The point isn't the exact slope — it's that a two-hour study early is cheaper than a two-quarter migration later.

5.1The method, in eight steps

  1. Frame the decision as a question with a scope: "Which vector store for the support RAG service, for the next 18 months?"
  2. List alternatives, including the baseline (keep what you have) and, where honest, "do nothing".
  3. Name the criteria that actually matter to this decision: latency, cost, operational load, lock-in, compliance.
  4. Weight them. Weights are a claim about priorities — get them agreed before scoring, or you'll reverse-engineer them from the answer you wanted.
  5. Score each alternative per criterion, relative to the baseline: better (+), same (0), worse (−). Use evidence: benchmarks, quotes, a spike.
  6. Compute the weighted totals.
  7. Test sensitivity: change the weights, remove the weakest-scored criterion, see if the winner survives.
  8. Write it down: alternatives, criteria, weights, scores, results, decision, and the condition that reopens it.

Criteria deserve as much scepticism as scores. A criterion is only useful if it can distinguish between your alternatives — "security" as a criterion is meaningless if every option passes the same audit. Prefer criteria you can measure, and when you can't, name the evidence that would settle the score: a benchmark, a quote, a two-day spike, a reference customer. "I think it's faster" is a hypothesis, not a score.

5.2Run one now

This is a real-shaped decision: which vector store for a retrieval service. Move the weights and change the scores — the point is the argument, not the answer.

Try it · weighted Pugh matrixweights 1–5 · scores −2…+2
CriterionWeightpgvectorWeaviatePineconeQdrant
Latency 5
Scalability 4
Operational simplicity 3
Cost 4
Ecosystem fit 2
WEIGHTED TOTALS

Scores are relative to pgvector as the baseline. If the leader changes when you move one weight from 2 to 5, the honest conclusion is "this is close" — not "we found the answer".

The six-line method

A trade study can be one page. What matters is that it names the alternatives, the criteria, the weights, the evidence, the result, and the revisit condition. Anything longer is usually an appendix; anything shorter is usually a preference.

The trap · objectivity theatre

Weighted scores feel objective and are not. Four ways they mislead: weights chosen after scoring; scores with no evidence behind them; a criterion that is actually a constraint (a disqualifier, not a tradeoff — "must be HIPAA-eligible" is pass/fail); and averaging away a fatal weakness because nine other criteria were fine. Check disqualifiers first, then run the matrix, then test sensitivity.

Check yourselfQuestion 1 of 1

When is a trade study the wrong tool?

60-second recap

  • Spend the rigour where the curve is steep. Reversible decisions just need a decision.
  • Agree weights before scoring, or the weights become a retro-fit justification.
  • Constraints disqualify; criteria trade off. Don't average away a fatal flaw.
  • Test sensitivity. If one weight flips the winner, you have a close call, not a conclusion.