Challenges

01Deconstruct a product

Time: 15 minutes · Needs: a browser · You produce: a one-page teardown

Pick an AI product you've seen advertised — a coding assistant, a support bot, a "AI employee". Answer, in writing:

• What are its tools? List every action it can take in the world, not what it says it can do.
• What is its state? Does it remember across sessions? Where would that be stored?
• What autonomy level is it (L0–L5)? What does the human do at that level?
• Which single action would be most damaging if it went wrong, and what gate sits in front of it?
• What does its marketing call the model, and does that matter to the product's value?

• I listed the tools as actions, not features
• I said where the state lives and what happens without it
• I assigned an autonomy level and defended it
• I named the worst action and the gate in front of it

02Trace an agent by hand

Time: 20 minutes · Needs: paper · You produce: a numbered trace with one injected failure

Choose a task you know well: booking a meeting, filing an expense, ordering stock. Write the trace step by step — goal, context, decision, tool, result, state update — as if you were the harness.

Then inject a failure at step four: the tool returns "sold out", or the API times out, or the file is missing. Continue the trace. What does the agent do next? What would make it loop? What makes it stop and ask you?

• I wrote at least six steps with the tool call named at each one
• I injected one realistic failure and continued honestly
• I wrote down the exact condition that ends the loop

03Write a tool and its permission policy

Time: 20 minutes · Needs: a text editor · You produce: one JSON tool definition + a policy line

Pick something small from your own work — lookup_order, read_invoice, check_stock. Write the definition with name, a when-to-use description, a schema with types and an enum, and a description of what a failure returns.

Then write the policy: risk tier, whether it needs approval, what credential it uses, and what the worst outcome of a wrong call is. If you can't finish that sentence, split the tool.

• My schema rejects bad input before it reaches the model's logic
• The description says when not to use it
• I assigned a risk tier and said what credential it gets
• I can state the worst outcome in one sentence

04Build a ten-case evaluation set

Time: 25 minutes · Needs: a spreadsheet · You produce: ten rows you can run repeatedly

For the task you traced in challenge 02, write ten cases: six typical, two edge, one messy, one adversarial. Each row gets the input, the setup, and the end state that must be true — a field that changed, a row that exists, a draft that was created.

• Every case comes from a real input, not an imagined one
• Every check is about end state, never about wording
• At least one case tries to make the agent do something it shouldn't
• I could run all ten in under five minutes

05Capstone: the smallest working agent

Time: one weekend · Needs: a model API key · You produce: a program you'd let a friend run

Build the inbox-triage or document-Q&A agent from module 10. Constraints, and they are the point:

• One model provider, two tools maximum, eight turns maximum.
• Read-only plus drafts. Nothing is sent, deleted, or paid for.
• A hard token budget per task, enforced in code.
• Every run writes a trace file you can read afterwards.
• A ten-case eval set, run before and after your last change.
• Someone who isn't you uses it for ten minutes while you watch and say nothing.

• It runs end to end on ten real inputs
• The budget cap actually stopped a runaway run when I tested it
• I have a before/after comparison on my ten cases
• A real person used it, and I wrote down the first thing they got wrong
• My failure log has at least five entries