Agent Routing Gets Smarter: Alibaba's Feedback Loop Cuts Token Burn by 99%

The Real Bottleneck in Agent Orchestration

Enterprise AI agents today face a surprisingly mundane problem: they drown in choice. When a system has access to hundreds or thousands of discrete tools, deciding which one to invoke for each subtask becomes a computational nightmare. Dump every available API description into a prompt, and you burn through context windows, rack up six-figure token bills, and still watch the model pick the wrong tool.

At DailyTechWire, we've tracked agent frameworks evolving from single-purpose scripts to sprawling orchestration layers that wire together cloud APIs, database connectors, and visualization engines. Most routing logic still treats tool selection as a one-shot lookup: match a query to a single function, execute, and move on. But real workflows are compositional. A request to pull data, clean it, and generate a dashboard isn't one action. It's a chain. And existing retrieval methods consistently fail when tasks require sequencing multiple specialized tools.

Researchers at Alibaba have published work on SkillWeaver, a framework that reframes the problem. Instead of asking an LLM to pick one tool per query, SkillWeaver decomposes a complex request into a graph of subtasks, retrieves candidate tools for each node, then composes them into an executable plan. The key innovation is a feedback mechanism called Skill-Aware Decomposition. SAD runs an initial decomposition, retrieves loosely matching tools from the library, then feeds those results back into the LLM so it can refine its plan to match the actual vocabulary and granularity of available tools.

The outcome, tested across 300 multi-step queries and a library of 2,209 real tools from the Model Context Protocol ecosystem, is stark. Token consumption drops from an estimated 884,000 per query to roughly 1,160 - a reduction exceeding 99%. Accuracy on hard tasks requiring four or five distinct tools improves by 50 percentage points. And perhaps most surprisingly, a lightweight 7-billion-parameter model with the feedback loop outperforms a massive frontier model that sees every tool at once.

Why Task Granularity Matters More Than Model Size

Traditional agent architectures expose the full tool library to the LLM upfront or rely on semantic search to pull a shortlist before the model ever sees the user query. Both approaches assume the model knows how to describe what it needs. In practice, that assumption breaks. An LLM asked to handle a business workflow will generate vague, high-level steps like "process the data" or "fetch the file." Those descriptions rarely align with the technical naming conventions of actual APIs. A tool called "etl-pipeline" won't surface if the model writes "clean and transform dataset."

Alibaba's experiments reveal that decomposition accuracy is the single biggest lever. In a baseline setup without the feedback loop, a 7-billion-parameter model correctly predicted the number of required subtasks only 51% of the time. Activate SAD, and that figure jumps to 67.7%. With a larger model in the loop, accuracy reaches 92%. On hard, multi-step queries, the gain is even steeper.

Counterintuitively, larger models can degrade performance when left unguided. A 14-billion-parameter variant tested in the same baseline setup performed worse than the smaller 7B model because it over-decomposed tasks into microscopic, redundant steps. The feedback loop acts as a constraint, anchoring the model's output to the actual structure of the tool library. This suggests that alignment with real-world tool schemas often matters more than raw parameter count or reasoning depth.

How the Feedback Loop Works

SkillWeaver operates in three stages: Decompose, Retrieve, and Compose. An LLM breaks the user query into atomic subtasks. An embedding model then compares each subtask against the skill library and returns a shortlist of top candidates. Finally, a planner evaluates compatibility across tools, checking whether the output of one can serve as input to the next, and assembles a directed acyclic graph that maps dependencies and allows parallel execution where possible.

The Skill-Aware Decomposition layer sits between the first and second stages. After the initial decomposition, the system runs a preliminary search to find tools that loosely match the draft plan. Those retrieved tools, with their names and descriptions, are fed back into the LLM as hints. The model then rewrites its decomposition so the vocabulary and step granularity align with the actual capabilities in the library.

Consider a query: "Download the dataset, transform it, and create visual reports." The decomposer might initially produce generic steps. The retrieval layer surfaces candidates like "api-client," "csv-parser," and "chart-gen." The planner evaluates inter-tool compatibility and wires them into a final workflow. Without the feedback loop, the decomposer might produce steps too abstract to match those specific tools. With SAD, the model sees the shortlist and adjusts its plan to fit.

Benchmark Results and Token Economics

To test the framework, Alibaba built CompSkillBench, a dataset of 300 multi-step queries spanning easy, medium, and hard difficulty tiers. The tool library consisted of 2,209 real skills from the public MCP ecosystem, covering 24 functional domains including cloud infrastructure, finance, and databases. The core engine used Qwen2.5-7B-Instruct for decomposition and MiniLM with a FAISS index for retrieval.

SkillWeaver was benchmarked against three baselines: an LLM-Direct method that stuffed all tool names into a large model's prompt, a vanilla decomposition pipeline without the feedback loop, and a ReAct-style agent loop. The LLM-Direct baseline, despite near-perfect task breakdown capability, retrieved the correct tool category only 21.1% of the time when flooded with options. The ReAct baseline collapsed entirely, achieving 0% decomposition accuracy because its iterative loop treats each action in isolation rather than planning a cohesive sequence.

SkillWeaver's retrieve-and-route approach not only outperformed both on accuracy but also slashed context window usage by 99.9%. For practitioners, this translates directly to lower API costs and faster response times. The token savings are not marginal - they represent the difference between economically viable agent deployment and runaway inference bills.

Practical Considerations for Implementation

Alibaba has not yet released the source code for SkillWeaver, but the architecture relies on off-the-shelf components. The core innovation, SAD, is a prompt-engineering and retrieval loop. The paper includes prompt templates, and developers can replicate the approach using LangChain, LlamaIndex, or custom Python orchestration scripts.

The retrieval component uses all-MiniLM-L6-v2, an open-source embedding model. Swapping in a stronger encoder like BGE-base-en-v1.5 immediately boosted accuracy without fine-tuning. However, bi-encoders alone struggle to consistently rank the optimal tool at position one, achieving that only 37% of the time. A secondary reranker, either a cross-encoder or an LLM-based scorer, is likely necessary to refine the top candidates.

Vectorizing the tool library and building a FAISS index is a one-time setup cost. In the benchmark, embedding and indexing 2,209 skills took 15 seconds. Retrieval adds less than 15 milliseconds of latency per query. For enterprise deployments, syncing the index as new tools are added is a trivial background job.

One current limitation is error recovery. SkillWeaver generates a compatible execution graph, but the pilot study revealed that real-world tool chains break when individual API calls fail. If step two times out, the entire sequence halts. The paper focuses on routing and planning, not runtime resilience. Production deployments will need custom fallback, retry, and error-handling logic layered on top of the compose stage.

What This Means for Agent Design

The SkillWeaver work reinforces a broader pattern emerging across agent architectures: retrieval and alignment with tool vocabularies often outweigh brute-force context packing or model scale. The 99% token reduction is not a side effect - it's the result of treating tool selection as a structured search problem rather than a prompt-stuffing exercise.

For teams building agents in environments like the Model Context Protocol, where tool libraries can number in the thousands, the lesson is clear. Decomposition granularity drives accuracy. Feedback loops that align model output with actual tool schemas unlock performance gains that larger models alone cannot deliver. And the economics of inference at scale demand retrieve-and-route strategies, not context window maximalism.

SkillWeaver is not yet production-ready out of the box, but its design is reproducible with standard tooling. The gap between research prototype and enterprise deployment narrows when the core techniques are prompt templates, embedding models, and graph planners - all components already in wide use. What remains is operationalizing error recovery and building monitoring layers that surface when tool chains break. Those are engineering problems, not research problems. And for agent builders looking to scale beyond toy demos, that distinction matters.