le will explore alternative robots.txt files in light of emerging technologies such as generative AI.

Google is looking at alternative or supplemental methods of controlling crawling, indexing and search beyond the 30-year-old standard of robots.txt. Google wrote that “we believe it is time for the web community and AI to explore other machine-readable ways for web publishers to choose and control for emerging AI use cases and research.”

Engaging the community. Google has invited members of the web and AI community to participate in a discussion on a new protocol. Google announced that it was “starting a public conversation” with “a wide range of voices across web publishers and civil society as well as academia from around the globe.”

Timing. Google stated that these discussions will take place over “the next few months” and we would be inviting those who are interested to participate.

Paywalled Content Issue. Open AI recently disabled the Browse with Bing function in ChatGPT because it had been able to access paywalled material without publisher permission. It is possible that Google is searching for alternatives to robots.txt.

Why we care. We’ve all been accused of allowing bots to access our websites through robots.txt or other newer forms of structured data. In the future, we might be looking into new methods. The discussion about these methods and protocols is ongoing, but it is not known what they will look like.

The original Search Engine land article Google will explore alternatives to robots.txt as a result of emerging technologies such as generative AI and other emerging technology first appeared on .