Welcome to WebmasterWorld Guest from 3.227.249.234

Forum Moderators: Robert Charlton & goodroi

Why is Google indexing our robots.txt file and showing it in search re

     
3:15 pm on Oct 6, 2019 (gmt 0)

New User

joined:May 19, 2015
posts: 5
votes: 0


For some reason, Google is indexing the robots.txt file for some of our sites and showing it in search results. See screenshots below.

Our robots.txt file is not linked from anywhere on the site and contains just the following:

User-agent: *
Crawl-delay: 5
This only happens for some sites. Why is this happening and how do we stop it?

[i.stack.imgur.com...]

Screenshot 1: Google Search console

[i.stack.imgur.com...]

Screenshot 2: Google search results
5:37 pm on Oct 6, 2019 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4509
votes: 348


Google has changed the way they deal with robots.txt files since last July, see: [webmasterworld.com...]

They may not see your file as a robot.txt file (despite its filename) because they ignore craw-delay directives and there are no instructions specifically for their bots.
6:27 pm on Oct 6, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15872
votes: 869


Screenshot 2: Google search results
Yes, but what search did you perform in order to get those results to come up? If I search for "cats", search results will not include every robots.txt in the world that mentions a directory called /cats/.

99 times out of a hundred, GSC messages about “indexed but not crawled” and the like are non-issues. Yes, in some abstract theoretical sense this content is indexed--but so what? If the idea really bugs you, add a rule like
<FilesMatch "\.(js|txt|xml)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
or equivalent if you're not on Apache. (This is an actual rule that I have, though I wasn't especially thinking of robots.txt when I made it.)

The fact that G ignores the Crawl-Delay directive is not your problem and does not require any action on your part.
12:59 am on Oct 7, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10457
votes: 1091


robots.txt is a nothingburger from the get go. I allow all and sundry to have it, even if they don't follow the "rules".

No USER you want searched for robots.txt. Period.