Google Crawler Control in Robots.txt For Bard Training

[engine]

Google has announced controls for publishers [webmasterworld.com] to choose whether their content is used to train Bard or Vertex AI using Google-Extended.

Google-Extended


A standalone product token that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.
User agent token Google-Extended
Full user agent string Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.

[developers.google.com...]