<- all tokdocs
2026.09.18@howtowebdev

Google's ARTEMIS Drives Real Android Phones From a Prompt, and Its 99% Benchmark Score Is Self-Reported

AI
Watch on TikTok

View on TikTok ->

ARTEMIS is a real repository inside GitHub's verified google organization, released under Apache 2.0, and the "99%+ SOTA" badge on its README is a number the ARTEMIS team submitted about itself rather than one an outside evaluator measured. @howtowebdev posts short clips that scroll a GitHub README while narrating it. This one runs 27 seconds and covers a single project. The narration says ARTEMIS is an open-source autonomous testing and device automation framework developed by Google, that you can drive it with natural language or with AI coding agents like Antigravity, and that it navigates complex UIs on real Android phones and generates structured diagnostic reports. Those claims all match the README. The video ends before the details that decide whether the thing is usable on your desk.

What the repository actually is

The on-screen frames show the real google/artemis page, including the mcp_server, packages, playground, and tests directories in the file tree. Here is the state of the repo as of 20 September 2026, pulled from the GitHub API:

Field Value
Owner google, a GitHub-verified organization with roughly 2,900 public repos
Created 13 August 2026
Last push 12 September 2026
License Apache-2.0
Language Python, 3.12 or newer
Stars / forks 8,253 / 781
Open issues 98
Topics ai-agents, android, google, test-automation, testing

The video was recorded earlier, and its first frame shows 31 issues and 60 pull requests, so the project has roughly tripled its open issue count since then. Commit history is concentrated: one account, somew1nd, has 107 of the commits, with the next contributor at 4. There are no tagged releases yet, so git clone of main is the only install path.

The transcript says "Antigravity" correctly. The burned-in captions split it into "anti-gravity", which is a caption artifact rather than a second product. Antigravity is Google's agent-first development platform, launched alongside Gemini 3 in November 2025, and ARTEMIS connects to it through MCP.

The 99% number needs a footnote the video does not give

ARTEMIS claims 99%+ task completion on AndroidWorld, Google Research's benchmark of 100+ multi-step tasks across 20+ apps. The specific figure on the AndroidWorld leaderboard is 99.1%, and the README embeds a bar chart showing ARTEMIS on top with human performance at 80.9%.

The AndroidWorld leaderboard does not independently verify submissions. Every entry on it, ARTEMIS included, is reported by the team that produced it. That makes 99.1% a vendor claim on a public benchmark rather than an audited measurement. Published robustness work on AndroidWorld also shows that varying task phrasing and parameters moves agent scores by a meaningful margin, so a single leaderboard row does not tell you how the agent behaves on your app.

The README chart is also not a complete field. It lists DroidRun, MobileUse by MadeAgents, gbox.ai, Mobile-Agent-v3, Gemini 2.5 Computer Use, and UI-TARS, and it omits Minitap's mobile-use, which was on the same leaderboard at 91.4% and later reported higher.

What you need before the first test run

The video says you can use natural language to drive real Android phones. The README fills in the prerequisites it skips:

  • An Android device with USB Debugging enabled, or an emulator, physically connected.
  • Python 3.12 or newer, plus ADB, scrcpy, and FFmpeg. The ./start.sh and .\start.bat scripts auto-install these.
  • At least one LLM API key in .env. The .env.example file accepts GEMINI_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, OPEN_ROUTER_API_KEY, or XAI_API_KEY. The shipped default in config/artemis.jsonc is provider google, model gemini-3.8-flash, with gemini-3.7-flash as fallback, so a Gemini key is the path of least resistance.

ARTEMIS also installs software on the phone. The first task on a device pushes an Artemis Accessibility Helper, an accessibility service that reads the screen layout without claiming the UiAutomation connection. You get a persistent "Artemis test helper is running" notification and a new entry under Settings > Accessibility. The README says it listens only on the phone and sends nothing elsewhere. You can pre-install it with uv run artemis helper install, remove it with uv run artemis helper uninstall, or skip it entirely by setting ARTEMIS_HIERARCHY_BACKEND=uiautomator in .env. If you run a test lab with managed devices, that is a change worth approving before you clone.

The commands the video never shows:

git clone https://github.com/google/artemis.git && cd artemis
./start.sh                                    # installs toolchain, opens localhost:8000
uv run artemis run "Open Settings, find Battery and tell me current level" --profile flash
uv run artemis mcp --install antigravity      # or --install all

Flash and Pro behave like two different tools

ARTEMIS ships two execution profiles, and the difference is larger than the names suggest.

Flash Pro
Speed 3 to 5 seconds per step 15 to 40 seconds per step
Structure Single reactive observe-and-act loop, no graph orchestration Multi-agent graph with Planner, Operator, and read-only Checker
Task plan None Living Markdown plan with milestones and verify / assert items
Safety checks None before an action Pre-execution safety net on every action, XML first with pixel fallback
Final report None Optional written report, verification level set by --verification-level
ADB shell No Yes

The "structured diagnostic reports" the video promises come from Pro, not Flash. The four-step Antigravity workflow shown in the README frames, prompt to plan to execution to report, is the Pro path. If you try Flash and conclude the reporting is missing, you picked the wrong profile.

Both profiles share a history compaction layer that replaces older screenshots with visual summaries and chunks completed steps into searchable eras, which is what lets a long run stay inside a context window. The MCP server exposes five tools to the IDE: mobile_run_task, mobile_manage_task, mobile_get_device_state, mobile_inspect_trace, and mobile_diagnose.

There is an open attribution dispute

Minitap, which maintains the mobile-use Android agent, published a post saying it found its own code in ARTEMIS without attribution, naming device connection code, agent instructions matching word for word, and identical example tasks. The post also says an earlier ARTEMIS package file listed the three Minitap authors and that a force push replaced it with a different author. Minitap says it emailed four times about updated benchmark results and received no reply.

That is one side of a dispute, published by an affected party, and it is a licensing question rather than a technical one. Apache 2.0 requires attribution notices to be preserved. Anyone adopting ARTEMIS inside a company should know the claim exists. The repo currently has 98 open issues and no releases, which is normal for a project five weeks old and also a reason to pin a commit rather than track main.

Key Takeaways

  • github.com/google/artemis is genuine, sits in the verified google organization, and is Apache 2.0 licensed Python.
  • The 99.1% AndroidWorld score is self-reported to a leaderboard that does not verify submissions.
  • You need a connected Android device with USB Debugging, Python 3.12+, and an LLM API key before anything runs. The default configuration points at Gemini.
  • ARTEMIS installs an accessibility service on the test phone. Disable it with ARTEMIS_HIERARCHY_BACKEND=uiautomator if that is a problem.
  • Use --profile pro if you want test plans, verification, and diagnostic reports. --profile flash gives you speed and none of those.
  • Minitap has publicly alleged that ARTEMIS reused its code without attribution, which is worth reading before adopting the project at work.
  • Unverified: whether Google's Pixel Test Engineering Fusion team authored it, which several fork descriptions state but the upstream repo does not; which profile and model produced the 99.1% benchmark run, since the repo does not publish the benchmark configuration; and the merits of the Minitap attribution claim, which Google has not publicly answered.

Resources

Published September 18, 2026. Writeup generated from a favorited TikTok.