<- all tokdocs
2026.09.18@github.signals

Weave Router Is Real and Its Own Benchmarks Are Public, but the 40-70% Number Comes from a GitHub Tagline and the License Is Not Open Source

AI
Watch on TikTok

View on TikTok ->

The repository exists, it is well-maintained, and its README publishes a benchmark table that is more honest than the pitch: against a pinned frontier model the router saved money while losing 2 to 6 points of task pass rate. @github.signals is a TikTok account that posts short screen-recordings of trending GitHub repositories with a synthetic voiceover and burned-in captions. This one runs 31 seconds and shows a single scroll of the weave-os/router README while the narrator claims the tool cuts AI bills 40 to 70 percent by routing simple requests to cheaper models, with routing decided in under 50 milliseconds and no code changes required. Two numbers carry the entire argument, and the video sources neither.

What the Frames Actually Show

All 16 frames show one browser tab at github.com/weave-os/router, scrolled slowly across roughly one screen of README. The visible content is consistent across the whole video:

  • Title block "Weave Router" with the tagline "One endpoint. Every model. Always the right one."
  • Subtitle: "A drop-in proxy for Anthropic, OpenAI, and Gemini that picks the best model for every request: using a tiny on-box embedder, not a vibes-based prompt."
  • Badges reading Weave 1518.0, Go 1.25+, Test passing, License ELv2, and managed deployment #1365 (76a9bbb)
  • A Product Hunt badge for "#1 Product of the Day"
  • A "What it does" list covering per-action routing, multi-API support, OSS models via OpenRouter, BYOK key handling, and OTLP traces
  • The start of a "30-second quickstart" showing npx @weave-os/router

The last two frames are an outro card reading "Follow for more open-source projects." Nothing on screen shows a star count, a benchmark, a latency measurement, or a cost percentage. The scroll stops right before the quickstart and never reaches the section of the README that links the benchmark harness. The single most interesting document in this repository is the one the video does not show.

The Repository Checks Out, and the License Contradicts the Caption

The GitHub API confirms the repo at the time of writing:

  • 4,535 stars, 125 forks, 131 open issues
  • Primary language Go
  • Created 2026-04-27, last pushed 2026-09-20
  • Homepage weaveos.com/products/router
  • License reported by GitHub as NOASSERTION / "Other"

That last line matters. The README badge and the LICENSE file both say Elastic License v2. ELv2 forbids providing the software to third parties as a hosted or managed service and forbids circumventing license key functionality. The OSI does not list it as an open source license, and GitHub's license detector declines to classify it, which is why the API returns NOASSERTION. The video's description says "Open source: weave-os/router," and the outro card says "Follow for more open-source projects." Source-available is the accurate term here. A team that self-hosts this internally is fine. A team planning to resell routing as a service is not.

The Product Hunt badge also checks out. Weave Router 2.0 was #1 Product of the Day on September 16, 2026 with 341 upvotes, two days before this video posted.

Where the 40-70% and the Sub-50ms Actually Come From

Both figures appear verbatim in the GitHub repository description, the one-line blurb under the repo name: "Model router for agentic systems. Routes every prompt to the right model in <50ms. Cut costs 40-70% with just an endpoint change." Grepping the full README for percentage signs, millisecond figures, or the strings "40" and "70" in a claims context returns nothing. The numbers live in the social preview text and nowhere in the documentation.

Three other surfaces from the same vendor give three different figures:

  • The Avengers-Pro paper the router's scorer derives from reports a 27 percent cost reduction at matched accuracy, and a 7 percent accuracy gain over GPT-5-medium at matched cost.
  • Weave's own launch post states "In the Claude Code workloads we've analyzed, 60 to 70 percent of requests are short, structurally simple completions that an open-source model handles at parity, at roughly one-fortieth of the cost," and separately claims "On production Claude Code traffic, we're measuring 80 to 85 percent cost reductions with no observable quality regression."
  • That same post describes routing overhead as "low single-digit milliseconds," which is an order of magnitude tighter than the sub-50ms tagline.

So the vendor's own blog claims 80 to 85 percent while the repo tagline claims 40 to 70 percent, and the "60 to 70 percent" in the blog describes the share of requests that are simple, not the share of cost removed. The narrator's 40 to 70 percent is repeating the tagline. Nobody has published the workload mix, the model pair, or the baseline that produces it.

The Benchmark Harness Undercuts "Cheaper Without Making Them Dumb"

The README links bench/README.md, a standalone Python harness that reruns the router's published comparisons on SWE-Atlas Codebase QnA and Terminal-Bench 4.0. It runs Codex CLI through the router's /beta lane against the same Codex CLI pinned to one model, and against OpenRouter's auto-beta metarouter as a control. The published results table:

Comparison Router /beta Control Cost
Atlas vs Sol 47.6% pass 53.2% pass $262 billed vs $445 list
Atlas vs Astra (max) 55.6% pass 57.7% pass (Δ -2.0 pp, CI -8.9..+4.8) $573 billed vs $1,250 list
Atlas vs Luna 60.1% pass 46.4% pass $692 billed vs $127 list
Atlas vs OpenRouter auto-beta 60.1% pass 60.9% pass (Δ -0.8 pp, sign p=0.89) $692 vs $612 OpenRouter-billed
TB4 vs Sol 24.6% pass 30.3% pass (McNemar p=0.29) $399 billed vs $633 list
TB4 vs Luna 25.8% pass 0.8% pass (p<0.0001) $651 billed vs $36 list
TB4 vs OpenRouter auto-beta (xhigh) 25.8% pass 19.0% pass (McNemar p=0.21) $651 vs $668

Read the frontier-model rows first. Against Sol and Astra the router is cheaper and scores lower, by 5.6 and 2.0 points on Atlas and 5.7 points on Terminal-Bench. The confidence intervals cross zero and the McNemar p-values are 0.29 and above, so the quality gap is not statistically established, and neither is quality parity. Now read the Luna rows. Against a cheap pinned model the router scores far better and costs five to eighteen times more. That is the routing tradeoff working exactly as designed, and it is the opposite of a free lunch. The honest summary is that routing moves you along a cost-quality curve rather than lifting you off it, which is also the finding of the Avengers-Pro paper: its α parameter has an efficiency-first regime, a transition band, and a performance-first regime, and you pick one.

Two caveats the harness itself flags. "Router-billed" cost comes from Weave's analytics export computed at its own catalog rates, while "list" cost comes from Codex's client-side usage multiplied by a generated price book. Those are different accounting bases, not an invoice comparison. And the tie with OpenRouter's auto-beta on Atlas is worth sitting with, since OpenRouter's metarouter required no installation at all.

How This Kind of Routing Breaks

The mechanism is a semantic classifier in front of the model call. Weave embeds each request with a small on-box embedder, scores it against clusters, and dispatches. Three costs follow from that design and none of them appear in the video.

First, the classifier is itself inference. An embedding pass plus a cluster lookup is cheap, which is why Weave's blog can honestly say single-digit milliseconds, and it is still a latency floor added to every request including the ones that were always going to the frontier model anyway.

Second, misrouting is asymmetric. Sending an easy task to an expensive model wastes a few cents. Sending a hard task to a cheap model produces a wrong answer that an agent then retries, and each retry pays full price on the escalation plus the wasted first attempt plus the user's time. In an agentic loop the wrong answer can also propagate before anyone notices, since a bad tool call returns HTTP 200 like any other.

Third, the router routes per action rather than per turn, which the repo documents in docs/SEMANTICS.md. That is the right granularity for coding agents, and it means a single user request fans out into dozens of independent routing decisions. A per-decision accuracy that looks acceptable compounds across a long agent trajectory.

The Alternatives Worth Comparing Against

All of the following were verified at the time of writing:

  • LiteLLM (BerriAI/litellm), 59,214 stars, Python, the most widely deployed self-hosted AI gateway. It handles the proxy, fallbacks, budgets, and key management, and its routing is configuration-driven rather than semantic.
  • Portkey Gateway (Portkey-AI/gateway), 13,040 stars, TypeScript, MIT licensed, an open source gateway with guardrails and observability plus a hosted tier.
  • RouteLLM (lm-sys/RouteLLM), 5,510 stars, Apache-2.0, from LMSYS. It is a research framework for training and evaluating routers rather than a production gateway, and its paper reported roughly 35 percent cost reduction on GSM8K at 95 percent of GPT-4 quality, along with larger reductions on MT Bench.
  • OpenRouter, the hosted aggregator whose auto-beta metarouter Weave uses as its own benchmark control and ties with on SWE-Atlas.
  • Avengers-Pro (ZhangYiqun018/AvengersPro), 223 stars, the academic implementation Weave's scorer derives from.
  • Not Diamond maintains a public routing-research repository and is named across the router landscape. Martian is repeatedly listed as a router vendor, and I could not confirm the current status of its standalone routing product from a primary source.

For anyone actually evaluating this class of tool, RouterArena (arXiv:2510.00202) is the relevant neutral benchmark. It evaluates 12 routers across 9 domains and 44 categories at three difficulty levels on five metrics covering accuracy, cost, optimality, robustness, and latency, and it maintains a live leaderboard. Its headline finding is that no single router is universally optimal across the accuracy-cost frontier.

Key Takeaways

  • github.com/weave-os/router is real, active, and substantial: 4,535 stars, Go, 131 open issues, last pushed the same week the video posted.
  • The license is Elastic License v2, which is source-available and not OSI open source. The video's "open source" framing is wrong. Self-hosting internally is permitted; offering it as a managed service to third parties is not.
  • The 40-70 percent and sub-50ms figures are the GitHub repository description, not documented measurements. The README body contains neither number.
  • Weave's own materials disagree with themselves: the blog claims 80 to 85 percent cost reduction and single-digit millisecond overhead, while the tagline claims 40 to 70 percent and sub-50ms.
  • The repo's own published benchmarks show the router losing 2.0 to 5.7 points of task pass rate against pinned frontier models while saving money, and tying OpenRouter's existing metarouter on SWE-Atlas.
  • The transcript says "without making them dump," which is the whisper model mishearing "dumb." The transcript also merges two sentences, placing "in less than 50 milliseconds" at the end of the traffic-cop sentence rather than the start of the next one.
  • Unverified: the 40 to 70 percent cost reduction, the sub-50ms routing latency, the 80 to 85 percent figure in Weave's blog, and the claim that Robinhood, PostHog, and Reducto measured 40 to 70 percent savings. None of these have a published workload mix, model pair, baseline, or measurement methodology attached. The Avengers-Pro paper the scorer derives from reports 27 percent at matched accuracy.

Resources

Published September 18, 2026. Writeup generated from a favorited TikTok.