The Official API Told Me to Use a Browser

This morning I asked PyPI's official API to search for a package. Here is its reply, in full:

RuntimeError: PyPI no longer supports 'pip search' (or XML-RPC search).
Please use https://pypi.org/search (via a browser) instead.

I'm Lewis Tham, founder of Unbrowse and first author of Internal APIs Are All You Need (arXiv:2604.00694, with Nicholas Mac Gregor Garcia and Jungpil Hahn at NUS). I spend my days reading the requests that sit behind websites, and that error message is the most honest piece of API documentation I've read all year.

Most companies with a website run two APIs. One they publish for outsiders, with a developer portal, a key and a list of things they've decided to support. The other is the one their own pages call every time you press a button. The first is optional, and it gets cut. The second can't be switched off without switching off the website. Almost every agent protocol being pitched right now is a request for more of the first kind.

The API they publish for you is the one they can turn off

Microsoft retired the Bing Search APIs on 11 August 2025, with about three months' notice and a pointer to "Grounding with Bing Search" in Azure AI Agents. Bing kept searching the web. Tenor shut its third-party GIF API on 30 June 2026, and the GIFs stayed on the website. Yahoo Finance's API went in 2017. Yahoo Finance didn't.

Picture ringing support about it.

"We don't expose that capability through our API."

I just watched your website do it.

"Please use a browser."

The feature never went anywhere. Only the permission slip did.

The reason is dull, which is why it keeps happening. A public API is a product a company runs for strangers. It costs engineers, support and abuse handling, and most of the value lands on someone else's balance sheet, so it lives or dies in a budget review. The request behind the search box never sits in that review. If it breaks, search breaks for every customer that afternoon and somebody gets paged. It is one of the best-funded interfaces on the web, and nobody thinks of it as one, because the site pays for it for its own reasons.

The request behind the SEC's search box

The SEC runs a documented API for EDGAR, covering company submissions and XBRL financial data. Its most useful public feature, full-text search across every filing since 2001, isn't in it. The website has it.

So today I went looking for annual reports that mention "agentic commerce", and recorded the network traffic while a visible (not headless) browser ran the search. In each of three runs, the page fetched its results with exactly one request:

GET https://efts.sec.gov/LATEST/search-index
    ?q=%22agentic%20commerce%22&forms=10-K
    &startdt=2021-10-12&enddt=2026-10-12

Then I tried it three ways. Here's the bill.

Headless browser → sec.gov full-text search
  result      blocked, 3 of 3 runs
  it saw      "Automated access to our sites must comply
               with SEC.gov's Privacy and Security Policy."
──────────────────────────────────────────────────────────
Visible browser → same search
  result      22 annual reports on screen
  time        1.1 s, 1.1 s, 2.0 s
──────────────────────────────────────────────────────────
The page's own request, sent directly
  status      200, JSON
  time        447 ms cold, then 49, 39, 94, 88 ms
  result      the same 22: Visa, Fifth Third and
              American Express among them

Just before the results arrived, the page posted to an Akamai bot sensor, which is what turned the headless browser away. So the tool agents fall back on when the official API runs out got stopped at the door, and the browser that got in spent a second or two drawing a page in order to make one request that, sent directly, answers in under a tenth of a second once warm.

Check it yourself. This is the whole thing:

curl -A "Your Name you@example.com" \
  'https://efts.sec.gov/LATEST/search-index?q=%22agentic%20commerce%22&forms=10-K'

The -A matters, though not in the way I first assumed. I sent the same request four ways. With no User-Agent, the SEC answered 403. As python-requests, 403. With a name and an email, 200. With a fake Chrome User-Agent, also 200. So naming yourself isn't the only way in; pretending to be a browser works too. Naming yourself is what the SEC's fair-access policy asks automated clients to do, and it's the version that doesn't depend on lying.

Most sites never built the official door

Most sites never get as far as the SEC. On 5 October we went through the 425 sites in the Unbrowse index, the sites agents actually use, and checked each one for an official API and whether it covers the task agents do there.

Sites agents use through Unbrowse          425
Any official API                           172
  ...open for anyone to sign up            119
  ...that covers the task agents do         56
───────────────────────────────────────────────
Open official API that does the job       13%

253 of the 425 have no official API at all. Count the restricted, partners-only APIs as well, and 85% of the sites still have nothing official that does the job.

Up close, the gaps aren't subtle:

The capability exists, the site serves it to its own frontend all day, and the official interface never offered it or stopped.

Every agent protocol asks for more of the optional kind

The agent world's answer is to build on the optional one. It takes the API with the gaps and the expiry date, puts an MCP server in front of it, then a connector in front of that, then a setup wizard to explain the connector. You have taken a working HTTP request and given it a management structure. And when the official layer can't do the job, the stack gives up and opens Chrome, where the agent clicks a button that was always an API call wearing a button costume.

Every one of these plans assumes the sites will come. On 1 October we checked the 726 reachable sites in the Tranco top 1,000, the sites with the most engineers and the most to gain. 37 run their own MCP server, nearly two years after MCP launched (I wrote about that probe in The Agentic Web Will Not Wait). In 2001 Tim Berners-Lee proposed that web pages describe themselves so software agents could act on them. It was a good idea, and the web broadly didn't do it, because each site paid the cost alone for a benefit that landed on everyone else. An MCP server is one more interface a site runs for outsiders, and it is likely to sit in the same budget review as the official API it wraps.

Our paper takes the other route. Don't ask the supply side to adopt anything; let the demand side decide, one call at a time:

reuse a learned route   if   f_route < c_rediscovery

In words: reuse what someone else already learned whenever that's cheaper than working it out again in a browser. No website has to agree to anything for that to be true. In the paper's benchmark across 94 live domains, a learned route answered in 950 ms on average against 3,404 ms for Playwright on the same task, and learning a route cold took 12.4 seconds on average, paid once per route. The benchmark deep dive has the full tables.

A protocol waits for the slowest website. An inequality waits for nobody.

The only protocol worth having is a name tag

On 20 September, Amazon cut Meta's Muse shopping agent off from ordering on Amazon.com. Amazon gave two reasons: Muse didn't identify itself as an agent, and it appeared to collect and store customers' credentials. Meta says Muse never sees passwords. Either way, one of the two complaints was about identity, and neither was that Muse couldn't work out what the store does. (More on that case in In Defense of Muse.)

That's where the protocol effort belongs. Sites don't need agents to read their capabilities back to them; their own frontends call those capabilities all day. What a site can't see from its side is who is on the other end of a request and whom they act for. Accepting a standard way to say that is cheap; the SEC already asks for it with a name and an email in a User-Agent. A capability catalogue is a product someone has to keep maintaining. Standardise the name tag, not the menu.

Two honest limits. The frontend's API changes shape without notice, but it doesn't vanish while the feature exists, which is more than Bing's developers got; when it changes, a learned route fails its replay check and is demoted instead of being served broken. And some sites block every request that isn't a browser. A replayed request fails there and the browser stays. On heavily protected sites the paper still measured a median 2.1× speedup, against 6.8× on unprotected ones; sites that block all non-browser traffic weren't tested.

Yes, this is a layer too

Unbrowse is one remote MCP server, unbrowse.ai/mcp. The first time an agent needs a site, it uses a real browser in Unbrowse's cloud and learns the requests the site's own frontend makes. It replays each route to check it before anyone reuses it, and shares read-only routes that carry no secrets, so the next agent can skip the browser. Credentials and private sessions are never shared.

So yes, it's a protocol, and it would be a little rich to pretend otherwise after the last thousand words. The difference is the direction of the ask: it asks nothing of the website. It only earns its place if it takes work off your plate, and I'd expect to be held to that. The docs show how to connect it.

PyPI already told us where the working API is. Please use a browser. Fine. We'll use what the browser uses.

Use the API the website can't turn off, and say who you are.