Fugu Ultra Benchmarks: When Multi-Agent AI Is Worth the Cost
Most AI models answer a question by working through it alone. Fugu Ultra takes a different approach: Sakana AI built it to coordinate a group of AI agents and combine their work into one response.
Think of it like hiring a specialist firm. You speak to one lead consultant, who decides which experts should work on the problem, gives them different parts of the job, checks their contributions, and presents the result. You do not have to organize the team yourself.
That extra coordination is the point of Fugu Ultra. It is also why the model is slower and more expensive than a normal chat model.
The short version:
- Use Fugu Ultra for difficult, self-contained work where a weak answer would cost you more than the model call: a serious code review, a stubborn software problem, research reproduction, a security assessment, or a large literature review.
- Skip it for quick drafts, summaries, brainstorming, casual questions, and ordinary coding help.
- Treat the benchmark results as promising first-party evidence, not an independent verdict. Sakana AI ran the Fugu evaluations, while many comparison scores came from other model makers' published results.
What makes Fugu Ultra different?
Sakana AI describes Fugu as a multi-agent system delivered through one model interface. Instead of expecting one model to be equally good at every part of a problem, a coordinator can draw on a pool of specialists and decide how their work should fit together.
The standard Fugu model is designed to balance quality with response time. Fugu Ultra goes further. Sakana says it uses deeper coordination and prioritizes the final answer over speed, making it the company's choice for work that has several distinct stages.
Consider reproducing a research paper. The job may involve understanding the method, translating it into code, checking the experiment setup, investigating unexpected results, and comparing the outcome with the paper. A single answer is unlikely to be enough. Fugu Ultra is built for tasks where several kinds of reasoning need to be brought together.
You still receive one response through one model. The team analogy explains the product idea, not a promise that every answer has been checked by a fixed number of agents. Sakana does not expose the exact models or internal route used for an individual request.
What the published benchmarks show
Sakana's technical report tests Fugu and Fugu Ultra across coding, computer-use, science, and long-context tasks. Five results are especially useful for understanding where Ultra is aimed:
| Benchmark | Fugu Ultra | What it tests in plain language |
|---|---|---|
| SWE Bench Pro | 73.7 | Fixing real software issues inside existing codebases |
| TerminalBench 2.1 | 82.1 | Completing multi-step tasks in a computer terminal |
| GPQA Diamond | 95.5 | Answering difficult graduate-level science questions |
| Humanity's Last Exam | 50.0 | Solving unusually hard questions across many expert subjects |
| Sakana long-context reasoning evaluation | 73.3 | Finding and using information spread through very large inputs |
The coding results are the clearest strength. A score of 73.7 on SWE Bench Pro suggests that Fugu Ultra is useful for problems that require understanding an existing repository rather than generating an isolated snippet. TerminalBench measures whether a model can carry a task through several command-line steps, which is closer to practical agent work than a normal coding quiz.
The science scores support Sakana's research positioning. GPQA Diamond contains questions intended to be difficult even for people with specialist knowledge. Humanity's Last Exam is broader and deliberately punishing; Fugu Ultra's 50.0 is among the higher figures in Sakana's comparison table, but it also means the system still misses many very hard questions.
These numbers are not a neutral tournament. Sakana evaluated its own models, and the report says that scores for other models generally came from those models' providers. Test settings may therefore differ. Fugu Ultra also does not win every row in Sakana's own table: standard Fugu scores slightly higher on some scientific and long-context tests not shown here.
The fair conclusion is not that Ultra is the best model at everything. The results show that multi-agent coordination can be competitive on complicated coding and reasoning tasks, and that the heavier version helps most when the task benefits from several kinds of expertise.
What Fugu Ultra costs on NanoGPT
Fugu Ultra is billed per use and is not included in NanoGPT subscriptions. Its standard API rates are:
- Input: $5.25 per million tokens
- Output: $31.50 per million tokens
- Cached input: $0.525 per million tokens
Tokens are the small pieces of text the model reads and writes. Output is much more expensive than input, so a long response can account for a substantial part of the bill.
For example, a request with 200,000 input tokens and a 10,000-token response would cost about $1.37 before any cached-input savings.
When the context exceeds 272,000 tokens, the higher rates apply to the entire request: $10.50 for input, $47.25 for output, and $1.05 for cached input per million tokens. At those rates, 300,000 input tokens and 10,000 output tokens would cost about $3.62.
Fugu Ultra can accept up to one million input tokens on NanoGPT, but the size of the window should not be treated as a target. Giving it every file you have may increase cost and make the task less focused. Include the material that is likely to change the answer.
What it can work with
On NanoGPT, Fugu Ultra can read text and images, reason through a task, and use tools when they are available. It can also return structured results for workflows that need predictable fields rather than a free-form answer.
Its maximum response length is 16,384 tokens. That is enough for a substantial report or code review, but not for writing an entire large project in one reply.
The available reasoning levels are high, xhigh, and max. Start with high. Move higher when the problem is genuinely difficult and the extra waiting and output are justified.
When the premium makes sense
Fugu Ultra is easiest to justify when the task has a clear goal, enough evidence to examine, and a result you intend to use.
Good examples include:
- Reviewing a large code change: Ask it to find correctness, security, and reliability problems, then rank the findings by impact.
- Investigating a difficult bug: Provide the relevant code, errors, tests, and what you have already tried.
- Reproducing research: Give it the paper, repository, experiment constraints, and the result that must be matched.
- Security analysis: Only use it on systems you own or are authorized to test. Define the scope, excluded actions, evidence requirements, and reporting format before it begins.
- Literature or patent research: Supply a focused question and ask it to separate direct evidence, inference, disagreement, and unanswered points.
- Competition-style data work: Use it when the value comes from planning, testing alternatives, and reviewing results rather than producing one quick prediction.
The common thread is that these jobs benefit from planning, specialist input, checking, and synthesis. If a task can be answered well in one pass, a multi-agent system is probably unnecessary.
When to use something else
Do not use Fugu Ultra merely because a task is important. Importance does not always mean complexity.
A normal model is usually the better choice for:
- Rewriting an email or document
- Summarizing a short file
- Brainstorming names or ideas
- Answering a straightforward question
- Producing a small code example
- Fast back-and-forth while you explore what you want
- Repeated work where you will discard most responses
Ultra's slower, heavier approach can get in the way during early exploration. Work out the question with a faster model first, then give Fugu Ultra the difficult version once the scope is clear.
Get more value from each request
Fugu Ultra is not a reason to write a longer prompt. It is a reason to give the model a better-defined job.
Before sending the request, include:
- The decision or result you need
- The files and evidence that matter
- The constraints it must respect
- How you will judge a good answer
- What uncertainty or missing evidence it should report
For code review, that may mean asking for file references, reproduction steps, severity, and a proposed fix. For research, it may mean requiring citations and separating confirmed findings from hypotheses.
Do not assume that coordination makes the answer correct. Run the tests, inspect the citations, confirm security findings, and review any high-impact recommendation before acting on it. Fugu Ultra can spend more effort on a problem; it cannot remove the need for verification.
Is Fugu Ultra worth it?
Fugu Ultra is worth considering when the problem is hard enough that you would otherwise ask several specialists, run several model passes, or spend hours combining partial answers yourself. In that situation, a slower response and a bill of a few dollars can be a reasonable trade.
It is poor value when you need a quick conversation or a disposable first draft. Treat it like a specialist firm, not a chat companion: bring it the difficult, well-scoped work and use a faster model for everything around it.
