How Nova picks the right tool out of thousands
At scale, search finds the tools. Judgment picks one.
With thousands of tools, getting the right one into a list of 100 is cheap. Getting it into the 5 the model sees, and knowing when none fits, is the hard part. A score that tracks the outcome is what lets the control layer act instead of guess.
Nova is our AI platform engineer. It does Day 1 work, building and shipping infrastructure, and Day 2 work, running it: scaling, upgrades and incidents, where it acts as an AI SRE. Either way it works through skills: its own, plus every MCP server an organization connects. Connect a few observability tools, source control, a cloud account and some internal servers, and the catalog is thousands of tools. On every turn Nova has to put the handful that fits in front of the model.
This post is the engineering behind that: the pipeline, what each step was worth on a public benchmark, what did not work, and how to reproduce it.
Why the obvious answers stop scaling
Give the model every tool. At a few dozen tools this works. At thousands, the definitions crowd out the incident. The RAG-MCP authors measured tool selection at 13.62% with every tool in context, and 43.13% when retrieval chose first.
Use a bigger model. LiveMCPBench traces about 45% of agent failures to retrieval, the step before the model reasons at all, and ToolRet finds strong retrievers under 52% recall at ten. A bigger model reasons better over the tools it is given. It does not change which tools it is given.
Use the provider's tool search. This is the strongest answer, and we load tools on demand too. But Nova runs across several model providers and has to choose the same way on each, the choice has to respect what the organization's policy allows, and a search returns its best matches even when none fits.
The pipeline
- Eligible skills. Organization, plan, policy and connection health decide which skills are in play. This step filters and never ranks, and it uses no model.
- Need. The model writes one line on what the tool must do, such as "list shard allocation for an index". It writes it while asking for tools, so it costs no extra call.
- Retrieve 100. Hybrid search: embeddings plus BM25 keyword search, fused by reciprocal rank. Each tool is indexed with its server's description and five example requests, written once at index time.
- Score. For each of the 100 candidates, a decision model answers one yes/no question with a probability: would this tool, called with suitable arguments, itself do what the user asked?
- Decide. A high best score proceeds, and 5 to 10 tools go to the model. A low score searches once more with a new need. Low twice most likely means the organization has not connected the skill that would answer, so Nova says so instead of guessing.
- Execute with policy. Retrieval grants nothing. Policy and approval are checked again when the tool is called.
What each step was worth
We indexed the public MCP-Zero catalog through Nova's real indexing path: 292 servers and 2,643 tools. A model wrote 1,440 questions, each with one labelled correct tool, in the five phrasing styles from HumanMCP, from questions that name the tool to ones that only describe a symptom. We ran each with all 292 servers eligible, and again in an organization of 30 servers drawn at random around the right one. Then we added one step at a time.
The finding that matters for scale: with search tuned, the right tool was among the 100 candidates for 96.7% of questions, but in the top 5 for only 74.5%. Search had found it. Ranking was the gap.
- Across all 292 servers, the right tool in the top 5 went from 58.2% to 83.7%, and at rank 1 from 39.7% to 63.1%.
- In the 30-server organization, top 5 went from 75.6% to 96.2%, and top 20 reached 99.3%.
- The biggest single step was the scorer: 10.3 points in the top 5 and 15.8 at rank 1. A wider pile of candidates only pays off because something can rank it.
- The five example requests per tool were the cheapest win: search alone went from 64.4% to 74.5% in the top 5.
Symptoms gain the most
Engineers do not name tools, least of all on call. They describe symptoms: "my query is slow", "the pod keeps restarting". That is the style plain search handles worst, and the one this pipeline improves most.
One example. An engineer writes: "I'm trying to figure out why my search performance has degraded and I need to understand how my data is distributed across my cluster." The labelled tool is Elasticsearch's get_shards. Plain search latched on to the word "search" and returned three search tools. The example request "Are any shards unassigned or unhealthy?" brought get_shards to rank 15. The scorer moved it to rank 1, with a cautious 0.46: the right pick, on a question vague enough that the score said so.
The score is a control signal
A ranking orders tools. A probability is something a policy can set a threshold on, and it tracked the outcome:
That is what step 5 runs on. Where to draw the line depends on the organization, so it is a policy choice. Scoring 100 candidates cost about $0.0004 and took 185 ms at the median, less than the 250 ms it takes to embed the query, so it runs on every search.
What did not work
- Ranking skills first, then tools inside the best skills. Top 20 fell from 73.8% to 68.1%. Skills should filter what an organization can use, not rank.
- Searching with the raw user message. It was the weakest query at every size and in every style except questions that name the tool. Fusing it with the model's need added under one point of recall for a second search, so we dropped it.
What we have not shown
Models wrote the questions and the example requests, which can flatter the result; we used different models and prompts for each. Random 30-server organizations are easier than real ones, which hold look-alikes: three databases where we drew one. And one labelled tool per question understates success: for 45% of the misses we checked, another tool in the top 5 would have done the job, often another vendor's equivalent. We are building a dataset from real platform engineering and SRE requests, with look-alike servers and every acceptable tool labelled, to check these numbers hold.
Try it yourself
Everything here uses public material, and the whole study cost about $4.20 in model calls.
- Index the MCP-Zero catalog. For each tool, store its description, its server's description and five example requests, in both an embedding index and a BM25 index.
- Have a model write questions in the five HumanMCP styles, each with one labelled tool. Use a different model or prompt from the one that wrote the example requests.
- For each question, have a model write the one-line need, retrieve 100 candidates with both indexes, and fuse them by reciprocal rank.
- Score every candidate with a model that returns a probability for a yes/no answer. A classifier works, or any model that exposes token log probabilities. Rank by that probability.
- Measure the labelled tool at rank 1, top 5 and top 20, adding one step at a time.
This is the record the scorer sees for each question:
User request: {the question}
The assistant is looking for: {the need}
Candidate tools:
1. {tool name}: {tool description}
2. ...
Question (one per candidate): Would candidate {n}, called with suitable arguments,
itself do what the user asked? A merely related tool, or one that needs another
tool to finish the job, is not enough.
The second sentence of the question matters. Without it, related tools score high and the ranking blurs.
What this means for Nova
Chat already uses the score to judge whether the tools it found are enough, and searches again when they are not. Investigations and the fix planner move to the same path, and so does the step that suggests a skill to connect.
The model was never the bottleneck. Search will keep getting cheaper. What scales is a judgment the control layer can act on: proceed, look again, or tell the engineer what is missing, with policy and approval still deciding what actually runs.
Nova is the AI platform engineer. Astro Platform is the runtime that keeps it safe.
References
- MCP-Zero, the catalog (code)
- HumanMCP, the five phrasing styles
- RAG-MCP, tool selection with and without retrieval
- LiveMCPBench, where agent failures come from
- ToolRet, how well retrievers find tools
- Tool search tool and tool search, provider documentation