Forum Moderators: goodroi

Message Too Old, No Replies

mobile friendly checker finding phantom robots.txt

         

kvetina

3:33 pm on May 3, 2018 (gmt 0)

5+ Year Member



When I ran my site through Google's mobile friendly checker, it claimed that I had a robots.txt file that says:
User-Agent: *
Allow: /ads/preferences/
Allow: /gpt/
Disallow: /
Noindex: /

When I checked it myself, the file only says:
User-agent: *Disallow:

And my web host checked and I only have one robots.txt. Anybody know what might be happening?

engine

3:44 pm on May 3, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Hi kvetina, welcome to WebmasterWorld

That's odd.
Check your file is called robots.txt and is in the root, and that there are no other files masquerading as robots.txt.

On your own robots.txt file you should check the syntax as that may be causing problems.

If you allow all robots access it should be as follows
User-agent: *
Disallow:

With the above example you don't really need a robots.txt file and you could delete it.

lucy24

4:23 pm on May 3, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



my web host checked
This line worries me. Do you mean that it is not in your power to check for yourself, or do you mean that you asked the host to double-check in case you'd overlooked something?