网站开始对 AI Agent 关门
Websites are closing their doors to AI agents
The Verge 报道:OpenAI、Meta、Google 等把自主 Agent 定位成能替用户上网办事的助手,但网站正在变得越来越不欢迎 Agent,这直接削弱了 Agent 的实用性[1]。它要解决的是开放 Web 与自动化访问之间正在重新划线:过去十年爬虫与 API 的默认假设是「只要不违反 robots.txt 就行」,而 Agent 会登录、点击、下单、读私人数据,站点所有者开始用反爬、登录墙与条款明确拒绝。技术上这不是单一技术问题——指纹识别、行为检测、付费墙与 OAuth 授权各管一段,Agent 生态因此被迫走向「持牌访问」而非「模拟人类」。参考价值是把「目标站点是否允许 Agent 访问」当成产品设计约束,而不是发布后再补;限制是报道是行业观察、未给出被封锁站点的统计口径,实际比例与各家策略差异仍需自行测试。
The Verge reports that while OpenAI, Meta, Google and others position autonomous agents as assistants that can act online, the web is becoming increasingly unfriendly toward agents, undermining their usefulness[1]. It addresses the redrawing of boundaries between the open web and automated access: for a decade crawlers and APIs assumed that respecting robots.txt was enough, but agents log in, click, order and read private data, so site owners are turning to anti-bot measures, login walls and terms that explicitly refuse. Technically no single mechanism is at fault — fingerprinting, behaviour detection, paywalls and OAuth each cover a slice — so the agent ecosystem is pushed from 'simulate a human' toward licensed access. The reference is to treat 'does this site permit agent access?' as a product design constraint, not a post-launch patch; the limitation is that the piece is industry observation without a statistical basis, so the actual share of blocked sites and the differences between policies need your own testing.