homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

What would make you block Google in robots.txt?

 4:31 pm on May 9, 2011 (gmt 0)

This has been mentioned countless times on this forum... i.e. 'If you don't like what Google are doing, just block their bots'...
But of course, it's not realistic for most people, and if someone can tell me how NOT to rely on Google, I'll follow it up by asking you what next week's lottery numbers are...

However, whatson's thread about Google being the biggest scraper site out there made me wonder... At what point would you say enough is enough, and block the Google (scraper) bot from your site altogether... and would it suddenly take off like wildfire in the websmaster community?... After all, this is what would bring Google down to a more manageable level... but I need the traffic because they're too strong, that's the reality for nearly everyone, I think?... but if......... if....



 5:05 pm on May 9, 2011 (gmt 0)

I guess my question would be, if you are going to block Google why wouldn't you also block Bing? Yahoo?

Are there specific things that Google does that Bing and Yahoo don't do? Or vice-versa?

What is different about Bing and Yahoo that has you not wanting to block them as well?

What are you saying "Enough is enough" to?


 5:19 pm on May 9, 2011 (gmt 0)

What's that old cliché?

Don't cut off your nose to spite your face.

At what point would you say enough is enough, and block the Google (scraper) bot from your site altogether.

Probably never based on past, current, and predictable future trends.

I think ALL of us here at WebmasterWorld rely on Google to some extent. Many like to state that they don't but, they do! If only 10% of your traffic is from Google, that's 10% you didn't have. For most of us, we're probably at 30%+ with some as high as 90%+, we are reliant on Google for organic and paid traffic and that isn't going to change anytime soon. Not during this generation anyway. ;)


 5:19 pm on May 9, 2011 (gmt 0)

I'm at 89%... my traffic went back up to pre-panda levels today, but the last week or so has been jarring, but that's another thread...

Indeed, this is something I struggle with. For example, if Google stopped passing referrer data altogether, or if the snippets pop-up they display suddenly had a zoom button to make all the text readable from within google... I make use of <meta name="robots" content="noarchive" /> because I don't like a cache of my site being available but it's a long way from blocking altogether...

What if the site preview verifiably started taking 50 percent of your traffic, but you were still heavily reliant on the other 50 percent... How would you react to that?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved