<- all tokdocs

Calibrain Harness: An Open-Source Crypto Bot Tester on Bybit Testnet

By , founder of Mavric and Apso

View on TikTok ->

Calibrain Harness is an open-source, MIT-licensed tool that tests whether a fast AI classification model has a real edge in crypto perpetual futures. It grades every forecast the model makes against what the market actually did, using live Bybit market data. Orders go only to the Bybit testnet with simulated funds, a limit a TikTok from @github.signals never states aloud when it describes the tool as running on "live crypto markets."

What is Calibrain Harness?

The project's README calls it "A forward-testing harness for trading with fast classification models: plug a model in, run it on live perpetual markets, and find out honestly whether it has an edge." Its badges read "status under development," "venue Bybit testnet only," "models Kairos | Jev," "python 3.12," "next.js 16" and "MIT."

A warning box says the project is under active development, trades on Bybit testnet only, and is not financial advice. The testnet host is hard-coded, with no switch for mainnet.

Does Calibrain trade real money?

No. The video says "It plugs fast classification models into live crypto markets," and the description repeats "live crypto markets." Most viewers would take that to mean a bot trading real money.

The repository says signals use Bybit mainnet market data, while orders are placed on the Bybit testnet with simulated funds. Observe-only is the default, and auto-trading has to be switched on by hand. "Live" is accurate for the price feed and misleading for execution. The description's hashtag #PaperTrading is closer to the truth than the narration.

How does it grade predictions?

On each closed price bar, the harness asks the model a small set of questions that can be checked later. The README's example is "which is touched first in the next 2 h: +1.0% or -0.6%?" Code turns the model's probabilities into expected value and places trades that clear its gates. It then "scores every prediction against what the market actually did, traded or not."

That last phrase matters. Forecasts that never became trades are still graded, which supports the video's line that it "grades every single decision." Grading covers the model's answers to the harness's own question set.

The video gives no results. Neither it nor the repository shows a hit rate, profit figure or evidence that any model has an edge. The tool tests for an edge without claiming one.

Does it stop trading when the model is guessing?

Broadly, yes. The video says "If the model is just guessing, the system learns and stops trading." "Learns" here means calibration rather than retraining.

The repository describes the mechanism. Model outputs are blended with the instrument's historical base rates, and the blend weight is fitted on labelled outcomes. A model that adds nothing stops driving trades. Weights are refitted every 15 minutes once a group has 30 independent labelled windows. Drift detection can also set a model's weight to zero or pause trading.

Two details the video skips:

  • In "strict" mode, a symbol must be statistically validated first. Until then the system logs decisions and trades only at small size, so small trades can happen before a model has beaten base rates.
  • 30 windows is a small sample for a noisy market, so a short run proves little.

Which models does it support?

The video does not name any. The README lists two: Kairos (DMind, served through Minara Cloud) and Jev (TypeSafe). The Kairos row shows model id "DMIND/Kairos" and key "KAIROS_API_KEY," so you bring your own model access and credentials. You can switch models while it runs, and results are tracked per model.

Safety features

The repository describes a KILL button and a KILL file that close bot positions, cancel orders and halt the bot. A daily loss limit applies, set to 3% by default. The narration mentions none of this.

Is it a benchmark?

The video calls it "the benchmark you need to know you are not fooling yourself." A benchmark usually has fixed tasks so results compare across models and over time. The README says interfaces, question sets and defaults change between commits, so scores from different versions may not line up. "Forward-testing harness," the README's own term, is the better description.

When fetched, the repository showed 236 stars, 51 forks, 3 watchers and 218 commits under an MIT license.

Bottom line

Calibrain Harness grades every forecast against live Bybit market data, including forecasts it did not trade, and it can turn a useless model's influence down to zero. It trades only on the Bybit testnet with simulated funds. It is an early, changing project with no published performance results.

Sources

Published November 14, 2026. Based on a TikTok by @github.signals. Full video notes (markdown)