--- url: https://ora.ai/blog/evaluating-jev title: "Jev and the agentic web · ora // research" description: "240 agent runs on a hosted warehouse, helpdesk and document vault we control: what happens when Jev makes the step decisions instead of a language model, across browser automation, WebMCP tools and NLWeb search. Ora Research." crawled_at: 2026-10-18T19:27:25Z ask: https://ora-ai.ora.run/ask docs: https://ora-ai.ora.run/.well-known/ora/docs lang: en --- # Jev and the agentic web ora research · Sep 24, 2026 · 11 min read [Share on X](https://x.com/intent/tweet?text=Jev%20and%20the%20agentic%20web&url=https%3A%2F%2Fora.ai%2Fblog%2Fevaluating-jev) [Share on LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fora.ai%2Fblog%2Fevaluating-jev) TypeSafe introduced [Jev](https://typesafe.ai/) , a decision model that answers a closed question, which option, which element, yes or no, in about 150 ms at a fraction of a language model's price. We wanted to know whether Jev makes a site more accessible and usable for agents, and how that holds up across the top protocols an agent already uses to work a website: browser automation, [WebMCP](https://github.com/webmachinelearning/webmcp) tools and [NLWeb](https://github.com/nlweb-ai/NLWeb) search. With Jev, agents finished tasks up to 4.3× faster and up to 13× cheaper , and were faster in every paired run. Task success did not drop: every task passed both ways on WebMCP and NLWeb, and on the browser the two split on three tasks, where Jev won the longest and lost the simplest. With deciding this cheap, the largest gains are in what the site exposes to agents. run it yourself webmcp.ora.ai/jev [data github.com/ora/research](https://github.com/ora/research/tree/main/jev-agentic-web) research ora.ai/research ### Up to 4.3× faster with Jev median time per task, seconds · lower is better with Jev without Jev browser-use 9 tasks · success 78% vs 71% WebMCP 10 tasks · success 100% vs 100% NLWeb 5 tasks · success 100% vs 100% the same tasks, the same site, the same writer · only the step decider changes ora.ai/research the experiment ## How we ran it The site is a warehouse, a helpdesk and a document vault we host and seed ourselves, so every task has a known answer or end state. The agents are simple Claude agents, with Claude Haiku 4.5 writing any free text. The only difference between the arms is who makes each step's closed choices: Jev, or Claude Haiku 4.5 from the same state. Ten tasks, five repeats each, a fresh workspace per run: 240 runs, each verified against live state or the exact answer. browser The agent drives the pages through browser-use's jev-ultrafast. Jev decides the operation, the element to act on, and the option in any dropdown the model writes the free text to type into a field 9 tasks · 90 runs WebMCP The agent calls the tools the page registers with navigator.modelContext. Jev decides which tool to call, or that the task is done the model writes the tool's arguments 10 tasks · 100 runs NLWeb The agent queries the site's NLWeb /ask endpoint. Jev decides when to ask, which result to open, and the ranking of the results on the site side the model writes the query 5 tasks · 50 runs ### Examples of what the agent was asked five of the ten task prompts, verbatim, simplest first · right: the surfaces that offer the task read What is the unit price of the Quenquen Renmar kit? browser · WebMCP · NLWeb read Which customer reported helpdesk ticket HD-5090? browser · WebMCP · NLWeb read How many units of the Quentho Dordor pack (SKU ENC-1027) are currently in stock across all depots? browser · WebMCP write Place a new order for customer Vidorfal Industrial (C-108) for 40 units of the Marmar Marka kit (SKU ADH-1038). browser · WebMCP write Assign helpdesk ticket HD-5001 to agent Teodor Voss (AG-13) and mark it resolved. browser · WebMCP every prompt names real records from the seeded site ora.ai/research 01 / time ## Faster in every paired run Browser tasks took 1.9 s with Jev and 8.2 s without. WebMCP: 1.4 s against 3.7 . NLWeb: 2.0 s against 7.8 . The slowest run with Jev still beat the median run without it, on every surface. ### The slowest run with Jev beat the median run without it one dot per run · tick: median · lower is better with Jev without Jev browser with Jev median 1.9 s · slowest 3.7 s browser without Jev median 8.2 s · slowest 20.7 s WebMCP with Jev median 1.4 s · slowest 2.0 s WebMCP without Jev median 3.7 s · slowest 7.5 s NLWeb with Jev median 2.0 s · slowest 2.4 s NLWeb without Jev median 7.8 s · slowest 11.5 s with Jev was faster in every pair of runs with the same task, repeat and surface ora.ai/research A step decision: 125 to 159 ms with Jev, 1.1 to 1.2 s without. #### breakdown · time per step decision, per surface | surface | with Jev | without Jev | ratio | deciding, share of task time, with Jev | without Jev | |---|---|---|---|---|---| | browser | 159 ms | 1,211 ms | 7.6× | 37% | 83% | | WebMCP | 137 ms | 1,120 ms | 8.2× | 20% | 66% | | NLWeb | 125 ms | 1,126 ms | 9.0× | 13% | 31% | 02 / cost ## Up to 13× cheaper per task Without Jev, deciding is nearly the whole cost: $0.0220 of a $0.0233 browser task. With Jev it costs $0.0007 , and writing the text becomes the largest cost. ### Cost per task, with Jev and without mean USD per task · same scale on every row · lower is better with Jev without Jev · browser · with Jev · $0.0018 · without Jev · $0.0233 · 13× cheaper · WebMCP · with Jev · $0.0011 · without Jev · $0.0046 · 4× cheaper · NLWeb · with Jev · $0.0029 · without Jev · $0.0175 · 6× cheaper list prices: Jev $0.042 per million input tokens · Claude Haiku 4.5 $1 in, $5 out ora.ai/research #### breakdown · where each task's money went deciding search ranking (NLWeb) writing the text browser · with Jev · $0.0018 a task deciding $0.0007 (38%) · writing $0.0011 (62%) browser · without Jev · $0.0233 a task deciding $0.0220 (95%) · writing $0.0013 (5%) WebMCP · with Jev · $0.0011 a task deciding $0.0001 (10%) · writing $0.0010 (90%) WebMCP · without Jev · $0.0046 a task deciding $0.0035 (77%) · writing $0.0011 (23%) NLWeb · with Jev · $0.0029 a task deciding $0.0001 (3%) · search ranking $0.0004 (15%) · writing $0.0023 (82%) NLWeb · without Jev · $0.0175 a task deciding $0.0027 (15%) · search ranking $0.0120 (68%) · writing $0.0028 (16%) 03 / task success ## Success held, except on three browser tasks WebMCP and NLWeb passed every task, with Jev and without. The browser passed 78% of runs with Jev and 71% without, and the whole gap comes from three tasks. ### WebMCP and NLWeb passed every task. The browser missed two with Jev, four without. one dot per task · counts where fewer than 5 of 5 passed · right: share of runs passed · higher is better all 5 passed some passed none passed browser with Jev order 0/5 transfer ticket stock price customer 0/5 reporter doc 1 doc 2 78% browser without Jev order 0/5 transfer 0/5 ticket stock 3/5 price customer 4/5 reporter doc 1 doc 2 71% WebMCP with Jev order transfer ticket stock low stock price customer reporter doc 1 doc 2 100% WebMCP without Jev order transfer ticket stock low stock price customer reporter doc 1 doc 2 100% NLWeb with Jev price customer reporter doc 1 doc 2 100% NLWeb without Jev price customer reporter doc 1 doc 2 100% a task appears only on the surfaces that offer it ora.ai/research #### what each task is order place an order for a customer and a SKU transfer move stock between two depots and complete the transfer ticket assign a helpdesk ticket to an agent and resolve it stock total units of a SKU across depots low stock SKUs under ten units in stock price the unit price of a product customer the customer with a given tier and city reporter the customer who reported a ticket doc 1 the vault document about returns and warranty doc 2 the vault document about the helpdesk FAQ 04 / the browser ## Jev won the longest browser task and lost the simplest Jev finished the seven-step ticket task in every run and gave up on the two-step lookup in every run, where the language model found the search box. Its wins and its failures repeat exactly: with Jev, every task took the same path on all five repeats. Jev stops when the page does not expose the element it needs. Whether a fast decider finishes a task is decided by what the site exposes. ### Seven steps done, two steps abandoned browser · one run per arm, step by step · with Jev, all five repeats took this path finished gave up The long task. “Assign helpdesk ticket HD-5001 to agent Teodor Voss (AG-13) and mark it resolved.” passed with Jev 5 of 5 · without 0 of 5 with Jev passed 7 steps · 1.6 s click Tickets click HD-5001 pick agent AG-13 click Assign pick status resolved click Change status done without Jev failed 8 steps · 10.7 s click Tickets click HD-5001 pick agent AG-13 click Assign click Change status click Change status click Change status click Change status The short task. “Which customer reported helpdesk ticket HD-5090?” passed with Jev 0 of 5 · without 4 of 5 with Jev failed 2 steps · 2.4 s click Tickets gave up without Jev passed 5 steps · 8.5 s click Tickets type “HD-5090” click Filter click HD-5090 answer: C-111 the language model's runs varied · the run shown is the first repeat ora.ai/research 05 / where the time goes ## Deciding was most of a task. With Jev it is a small part. Without Jev, choosing the next step is 66 to 83% of a task's time and nearly all of its cost. Jev answers that choice in about 150 ms for a fraction of a cent, so deciding drops to 13 to 37% of the time, and what remains is the one or two calls that write the text. The writer is the same, and on WebMCP and NLWeb so is the number of steps. On the browser Jev also took fewer steps, 4.8 a task against 6.2 , because it never waited or repeated a click. What is left is the writer and the page itself. ### Where a task's time goes share of wall time per task · by what the agent was waiting on deciding search ranking (NLWeb) writing the text the page browser · with Jev · 1.9 s a task deciding 37% · writing 54% · the page 9% browser · without Jev · 8.2 s a task deciding 83% · writing 13% · the page 4% WebMCP · with Jev · 1.4 s a task deciding 20% · writing 78% · the page 1% WebMCP · without Jev · 3.7 s a task deciding 66% · writing 33% · the page 0% NLWeb · with Jev · 2.0 s a task deciding 13% · search ranking 21% · writing 65% · the page 1% NLWeb · without Jev · 7.8 s a task deciding 31% · search ranking 49% · writing 19% · the page 0% the same model writes the text in both arms ora.ai/research 06 / conclusions ## What we conclude Jev cuts a step decision to about 150 ms, so the agent's own thinking is no longer where a task's time or cost goes. What Jev cannot do is finish a task the page does not put in front of it: it will not guess a search box it was never offered, or a second step that is not reachable from the first. Both of its failures in this study were one of those, and both tasks passed in every run where the site offered the same action as a WebMCP tool. 01 With Jev deciding, agents finished tasks up to 4.3× faster, and were faster in all 120 paired runs. 1.9 s against 8.2 on the browser, 1.4 against 3.7 on WebMCP, 2.0 against 7.8 on NLWeb. The slowest run with Jev beat the median run without it on every surface. 02 A task cost up to 13× less. $0.0018 against $0.0233 on the browser, $0.0011 against $0.0046 on WebMCP, $0.0029 against $0.0175 on NLWeb. Without Jev, deciding alone was $0.0220 of a $0.0233 browser task. 03 Success held. On the browser the arms split on three tasks, two of them in Jev's favour. WebMCP and NLWeb passed every task both ways. On the browser Jev won the seven-step ticket task 5 to 0 and the price lookup 5 to 3 , and lost the reporter lookup 0 to 4 , because the search box the language model used was not among the options Jev was offered. Jev never waited and took the same path on every repeat. 04 Both of Jev's failures were browser tasks the page did not put in front of it. The reporter lookup needed a ticket that was not in the visible list and a search box Jev was never offered. The transfer needed a second step that was not reachable from the first. Exposed as WebMCP tools, both tasks passed in every run with either decider. With deciding this cheap, the largest gains are in what the site exposes to agents. 05 What the site exposes decides whether a task finishes. The decider decides how fast and at what cost. WebMCP had the smallest gain from Jev, 2.6× , and was the only surface where every task passed both ways. The browser had the largest gain, 4.3× , and every failure in the study. A tool removes a decision before the decider sees it, so the surface with the most exposure gains least from a faster decider and still finishes first: 1.4 s a task on WebMCP with Jev, against 1.9 s on the browser. 06 Jev fails fast and the same way every time. The language model fails slowly and differently each time. Jev's two browser failures took the same path on all five repeats; on the reporter lookup that was one click, then it gave up, in 2.4 s. The language model failed the ticket task by clicking Change status four times in a row, 8 steps and 10.7 s, and passed the price lookup 3 of 5 with runs from 9.7 to 20.7 s. A failure that repeats exactly is one a site owner can find and fix once. What remains after a cheap decider is the text the model writes, 62 to 90% of a Jev task's cost, so the next lever is fewer free-text steps: tool arguments over typed text. 07 / sources · 01 · [TypeSafe Jev](https://console.typesafe.ai/) Jev System One, POST /v1/systemone · closed questions answered with probabilities · $0.042 per million input tokens, output free 02 [browser-use / jev-ultrafast](https://github.com/browser-use/jev-ultrafast) MIT · pinned at commit 1231850 · the browser agent: its page snapshot, element table, step questions, text contract and guarded executor 03 [orabenchmarks / benchme](https://github.com/orabenchmarks/benchme) the acme-v1 warehouse, helpdesk and document vault · the intent tasks (compose/specs) · the verifier · the agent-eval harness (tools/agent-eval) 04 [ora / research · jev-agentic-web](https://github.com/ora/research/tree/main/jev-agentic-web) the dataset behind this report: all 240 runs, task pass counts, the 10 task specs, and a script that recomputes every time and success figure above 05 [WebMCP](https://github.com/webmachinelearning/webmcp) W3C Web Machine Learning community group · tools a page registers through navigator.modelContext 06 [NLWeb](https://github.com/nlweb-ai/NLWeb) natural-language search for a site through an /ask endpoint cite this abstract ora research (2026). Jev and the agentic web: agents with Jev and without it, on a browser, WebMCP tools and NLWeb search, across 240 runs. ora.ai/research, september 2026. # Footer The standard for agent experience. © 2026 [era labs](https://eralabs.ai/) Product - Scan a site - [Journey](https://journey.ora.ai/) - [WebMCP audit](https://webmcp.ora.ai/) - [Directory](https://directory.ora.ai/) Resources - Methodology - Research - Blog - API Docs Company - About - Contact - Privacy - [@oradotai](https://x.com/oradotai)