EP8 Let the system pick the tool, and measure whether it picks right (finale)

Over seven episodes we kept giving the AI more: a handbook, skills, subagents, workflows, several models, MCP and other providers. At some point that creates a new problem: which one should this job use? This final episode builds a matcher, called a router, and more importantly, a way to measure whether it picks correctly.

The problem: too much to find anything

The system we use ourselves holds 211 entries: 166 skills, 25 MCP tools, 17 agents and 3 document sources. Nobody remembers what suits which job, and neither does the AI. Feeding it the whole list every time is both costly and error-prone.

Large companies solve this with a front desk that knows which department handles what.

Principle: let the system match jobs to tools, and keep a test set to prove it matches correctly.

How a router works

1. Keep a register. Every entry has a name, a type and a description of when to use it (a good description matters, as in EP2)
2. Turn descriptions into numbers with embeddings: texts with similar meaning get similar numbers. Store them in a search database
3. When a job comes in, turn it into numbers the same way and find the ten nearest entries. This runs locally, fast and nearly free
4. Let an LLM rank just those ten, with reasons. This step costs more, so it only sees the shortlist
5. Check confidence. If even the best match is far from the job, say plainly that no tool fits, rather than guessing wrong

Steps 3 and 4 are EP5 in action: the cheap step filters, the expensive step decides.

The test set: build it before tuning

You cannot judge a router by feel. You need test jobs with the answers you would accept:

{ "task": "review the code in a pull request for bugs and improvements",
  "expect": ["code-reviewer", "code-review"] }

Then count how often a correct answer lands near the top:

def hit_at(k, cases, route):
    hits = sum(any(name in route(c["task"])[:k] for name in c["expect"]) for c in cases)
    return hits / len(cases)

Every time you edit descriptions, change the embedding model or adjust the ranking, run the set. The number going up means a real improvement; going down means roll back.

Real results

Our router has a 30-job test set written from requests we actually made. Running only the search-by-meaning step, with no LLM help:

Measure	Result
Right at rank 1 (hit@1)	22/30 (73%)
Right in the top 3 (hit@3)	28/30 (93%)
Right in the top 5 (hit@5)	29/30 (97%)

So the cheap, fast search alone gets the right tool into the top five for almost every job. Making the first pick more accurate is the LLM ranking step’s job.

The miss teaches something too. “Write a professional report as a Word document” should find the skill for Word files, yet it did not make the top five, even though that skill’s description already mentions Word. The test set is what showed us this case; without it, we would never know it was failing.

Finding new tools itself, with a person letting them in

This router also has a daily job: it searches GitHub for recently updated skills and has an LLM judge them on quality and freshness, not star count, then proposes them to a queue. Nothing enters the register on its own; a person reviews first. Same reason as EP6: anything an AI will pick up and use needs a person to judge it trustworthy.

Try it in 10 minutes

1. List 10 of your tools or skills, each with a description of when to use it
2. Write 10 test jobs from things you actually asked for last week, with the right answer for each
3. Have Claude read the list and choose for each job, and count how many it gets right. That is your first test set, before you even build a router

Where a person is needed

A person writes the test set from real work and approves what enters the register. Those two things keep a system with many AIs trustworthy even as it grows every day.

The whole series in one table

EP	Principle
1	What the AI should know lives in a file
2	Keep knowledge, procedures and enforced rules apart
3	Reading-heavy work with short answers goes to a helper in its own room
4	Scripts do fixed steps, AI does the thinking, a person is the last checkpoint
5	Capable models for thinking, small ones for heavy reading
6	Write a tool once with MCP, use it from many AIs
7	Route every model call through one layer; do not tie yourself to one provider
8	Let the system pick the tool, and measure whether it picks right

All eight share one idea underneath. AI can do more and more of the work, but a system is only good when a person designs who does what, who checks it, and who decides.