Forum Moderators: open

Message Too Old, No Replies

Robots.txt being ignored

following on from the "Is there an update going on"

         

TinkyWinky

10:08 am on Nov 15, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Having read through the thread [webmasterworld.com ] regarding the apparent Googlebot following of excluded folders and urls via the robots.txt file I thought I would check a domain that I recently put live with a GoogleBot exclusion.

Whatd'ya know 9 pages listed with no titles or descriptions etc.

So is Google now cheating on the old ©2004 Google - Searching 8,058,044,651 web pages listed?

Seems so.

But more worringly what about repurcussions of this in a more general sense. I thought that the robots.txt was supposed to stop a bot going in and following links and folders that you wanted to exclude from public view ... obviously not.

Just glad I don't have cgi or critical administration pages I am exclusing.