Forum Moderators: goodroi

Message Too Old, No Replies

GPTBot and ChatGPT-User Blocking and Control Using robots.txt

         

engine

11:46 am on Sep 29, 2023 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Here's information on controlling OpenAI's crawler GPTBot and ChatGPT-User in robots.txt

GPTBot is OpenAI’s web crawler and can be identified by the following user agent and string.
User agent token: GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

Disallowing GPTBot

To disallow GPTBot to access your site you can add the GPTBot to your site’s robots.txt:

User-agent: GPTBot
Disallow: /

[platform.openai.com...]

ChatGPT-User is used by plugins in ChatGPT. This user-agent will only be used to take direct actions on behalf of ChatGPT users and is not used for crawling the web in any automatic fashion. User agent token: ChatGPT-User
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

To allow plugins to access your site you can explicitly add the ChatGPT-User to your site’s robots.txt:

User-agent: ChatGPT-User
Disallow:

[platform.openai.com...]

superclown2

1:14 pm on Sep 29, 2023 (gmt 0)



I wonder if blocking it could be a dangerous step if Google used more AI to arrange the SERPs?