How Nova picks the right tool out of thousands
At scale, search finds the tools. Judgment picks one.
With thousands of tools, getting the right one into a list of 100 is cheap. Getting it into the 5 the model sees, and knowing when none fits, is the hard part. A score that tracks the outcome is what lets the control layer act instead of guess.
Nova is our AI platform engineer. It does Day 1 work, building and shipping infrastructure, and Day 2 work, running it: scaling, upgrades and incidents, where it acts as an AI SRE. Either way it works through skills: its own, plus every MCP server an organization connects. Connect a few observability tools, source control, a cloud account and some internal servers, and the catalog is thousands of tools. On every turn Nova has to put the handful that fits in front of the model.
This post is the engineering behind that: the pipeline, what each step was worth on a public benchmark, what did not work, and how to reproduce it.