AndroidWorld Is Solved. MobileWorld Just Reset the Bar for Phone Agents

For about a year, if you wanted to prove a phone agent was any good, you pointed at one number: its score on AndroidWorld. In early 2026 that number quietly stopped meaning much. Several agents pushed past 90% on AndroidWorld's 116-task suite, and at least one framework reported a clean sweep. A benchmark that everyone can ace can no longer separate a good agent from a great one. So a team from Alibaba's Tongyi lab published the replacement — MobileWorld, accepted to ACL 2026 — and the leaderboard reset hard. The best agent they tested finished barely half the tasks. If you are building on mobile agents, that drop is the most useful thing to understand this quarter.

A benchmark that got too easy

AndroidWorld earned its place because it was reproducible: a deterministic Android emulator environment where the same task grades the same way every time. That rigor is exactly why it became the default citation on every phone-agent launch slide. But most of its tasks live inside a single app — create a calendar event, add a contact, toggle a setting — and once agents learned to do those reliably, the ceiling was hit. Saturation is not a scandal; it is what happens to every good benchmark that survives long enough.

MobileWorld is built around what AndroidWorld left out. It keeps the same reproducible, deterministic evaluation, but grows to 201 tasks across 20 applications, and deliberately adds the categories real users actually live in — e-commerce, enterprise communication — plus the two things that make real phone use messy: vague instructions and hybrid tool usage. The headline results tell you how much of a jump that is:

Why the scores collapse on cross-app work

The single biggest change is the share of tasks that span more than one app: 62.2% in MobileWorld versus 9.5% in AndroidWorld. Cross-app work is where phone automation actually earns its keep — copy an address out of an email, search it in a maps app, share the result into a chat — and it is exactly where naive agents fall apart. Every app switch is a chance to lose state, hit an auth wall, or land on a screen the agent has never seen.

The other number that matters is length. MobileWorld tasks take 27.8 steps on average, nearly double AndroidWorld's 14.3. Reliability compounds over a horizon like that, and the math is unforgiving. As a rough rule of thumb, an agent that is 98% reliable per step still only finishes about 57% of a 28-step task (0.98 to the 28th power). Push per-step reliability down to 95% and that same task succeeds barely a quarter of the time. This is why a demo that nails three taps tells you almost nothing about a workflow that needs thirty.

Concretely, longer cross-app tasks break in a few predictable ways:

• State loss — the agent forgets what it copied or why it opened the second app.
• Auth walls — the next app wants a login the agent has no plan for.
• Data hand-off — something read on one screen has to survive intact onto another.
• Drift — more steps means more chances to take one wrong turn that dooms the rest.

Two task types that look like real life

Beyond raw difficulty, MobileWorld adds two categories that barely existed in older benchmarks, and both map directly onto how people actually use phones. Roughly a fifth of the suite is agent-user interaction: tasks with deliberately ambiguous instructions where the correct move is to ask a clarifying question, not to guess. “Book me something reasonable for Friday” has no single ground-truth tap sequence; a good agent has to hold a short dialogue instead of charging ahead.

Another fifth is MCP-augmented tasks — workflows that require a mix of screen navigation and external tool calls made through the Model Context Protocol. The point is that not everything on a phone should be solved by looking at pixels and tapping. Where a structured tool or API exists, using it is faster and more reliable than driving the GUI, and the benchmark now rewards agents that know when to switch modes. We argued this exact point earlier this year in why the best phone agents do more than tap the screen.

What this means if you are building on phone agents

The practical takeaways are not subtle. First, stop trusting any single headline benchmark number — including MobileWorld's. Saturation caught AndroidWorld and it will eventually catch its successors. The only score that predicts your production reliability is a measurement on your flows, especially the cross-app ones.

Second, design for the fact that reliability compounds. A 40-step business workflow will not survive on per-step accuracy alone; it needs the agent to observe the screen after every action, notice when a tap did nothing, and change strategy instead of blindly repeating itself. We went deep on that gap between demo and dashboard in the mobile-agent reliability gap, and MobileWorld's step counts are the clearest evidence yet for why it matters.

Third, treat GUI and tools as complementary, not rival, strategies. The MCP-augmented results say plainly that the strongest agents reach for an API when one exists and fall back to the screen only when it does not. That is the same philosophy Auten is built on: check the screen after each action, fall back to full reasoning the moment a cached plan no longer fits, and reuse a proven plan the moment it does — so a long cross-app flow gets both more reliable and cheaper the more you run it.

FAQ

Is AndroidWorld useless now?

No. It is still a perfectly good smoke test for single-app basics, and its reproducible design is why MobileWorld inherited it. It just can no longer act as a ceiling — a 95% AndroidWorld score tells you an agent handles simple in-app tasks, and nothing about how it survives a thirty-step cross-app job.

Does MobileWorld test real physical devices?

It runs in a reproducible emulator environment, like AndroidWorld, so results are deterministic and comparable. Real hardware adds its own variance — slower screens, popups, network flakiness — which generally makes production numbers lower than any emulator benchmark, not higher.

What is an MCP-augmented task?

One that cannot be solved by GUI tapping alone. The agent has to combine on-screen actions with external tool calls made through the Model Context Protocol — for example, pulling a value from a service and then using it inside an app. It rewards agents that mix modes instead of forcing everything through the screen.

What score should I expect on my own workflows?

Measure it rather than guess. Expect single-app tasks to look strong and multi-app tasks to look far weaker — that spread is the whole message of MobileWorld. If you have not tested your real cross-app flows end to end, you do not yet know your reliability; you know your demo.