ot is OpenAI’s latest web crawler.

OpenAI published information on its new web crawler, GPTBot. You can find the GPTBot documentation here.

What is GPTBot? GPTBot, OpenAI’s web-crawler, is used to search the internet, gather knowledge, and then provide AI-generated answers to your questions.

Useragent. GPTBot’s User agent token is “GPTBot” and its full user-agent string: is “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)”.

Robots.txt. Use your robots.txt file to prevent GPTBot accessing your entire website or certain parts. You can block GPTBot from accessing your website by adding the GPTBot robot.txt to your site.

User Agent: GPTBot Disallow : /

Add the GPTBot token in your robots.txt file to allow GPTBot access to only certain parts of your website.

User agent: GPTBot Allow /directory-1, Disallow /directory-2

GPTBot IP Ranges. OpenAI has also published here the IP ranges used by GPTBot. It currently lists just one but I suspect that they will be adding more in time.

Why do we care? You can prevent GPTBot from using your content or crawling your website for its own purposes if you don’t want it to. You can use the same procedure to block GoogleBot, BingBot and other web crawlers.

The post Search Engine Land – GPTBot, OpenAI’s New Web Crawler appeared first on Search Engine.