Welcome to WebmasterWorld Guest from 3.92.92.168

Forum Moderators: phranque

Message Too Old, No Replies

Robots.txt files

     
8:31 am on Jun 21, 2018 (gmt 0)

New User

joined:Oct 16, 2017
posts: 22
votes: 1


I notice both my sites have a robots.txt file with this as the only entry:

User-agent: Mediapartners-Google
Disallow:

I don't remember having created this file on either site, and see that Adsense suggests this be removed.

[support.google.com...]


Is the robots.txt file actually hurting my revenue?
8:35 am on June 21, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


It is *not* hurting you site nor your Adsense account.

There is no value for Disallow so it is not doing anything.

[fix typo]

[edited by: keyplyr at 8:38 am (utc) on Jun 21, 2018]

8:36 am on June 21, 2018 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 9, 2000
posts:26468
votes: 1078


If that's all it has in the robots.txt file, you don't need it at all.
It shouldn't be hurting your revenue as all it does is specify Google media partners bot is ok to scan your site.
Take it away and it'll do that same.
8:40 am on June 21, 2018 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11874
votes: 245


this tells a Mediapartners-Google user agent it can crawl everything and puts no exclusions on this or any other user agents.
12:04 pm on June 21, 2018 (gmt 0)

New User

joined:Oct 16, 2017
posts: 22
votes: 1


Why does adsense say to remove it from the robots.txt file?
12:48 pm on June 21, 2018 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:11874
votes: 245


the support page you referenced in the OP starts with:
If you’ve modified your site’s robots.txt file to disallow the AdSense crawler from indexing your pages

this is not the case with your robots.txt file, which has been stated 3 different ways above.
1:20 pm on June 21, 2018 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 9, 2000
posts:26468
votes: 1078


Why does adsense say to remove it from the robots.txt file?

If that is all your robots.txt file says, just delete the file, you don't need it.
7:24 pm on June 21, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member tangor is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 29, 2005
posts:10590
votes: 1125


Delete the content of the file, keep the file itself. This avoids having 404s appear in your log files.

As for all the rest, learn more about what robots.txt actually does then make your own decision. As it is it merely says NOTHING IS HAPPENING AND THANKS FOR ALL THE FISH.