Forum Moderators: open

Message Too Old, No Replies

Google and Robots txt

Is googlebot obeying robots files?

         

Terrier

7:55 pm on Jan 10, 2003 (gmt 0)

10+ Year Member



One of the sites I have has a robots exclusion file for Gogglebot. I have checked it with the validator and it validates OK.
What concerns me is that the site is now showing a white bar on the google tool bar, it used to be grey, White I understand to mean Google is aware of the site. This concerns me as I do not want google to spider or list this site.

Is this a concern?
Is googlebot obeying robots files?

Mohamed_E

8:21 pm on Jan 10, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If another site has a link to yours then Google has every right to know that your site exists. All that the robots.txt file does is to tell it not to spider your site:

These incidents indicated the need for established mechanisms for WWW servers to indicate to robots which parts of their server should not be accessed. This standard addresses this need with an operational solution.

Terrier

9:25 pm on Jan 10, 2003 (gmt 0)

10+ Year Member



Thank you for your reply Mohamed E

Yes I am perfectly happy that it should know of it exists but not to spider or list it. So the white bar does not mean it has been spidered.

White bar with many meanings!

ciml

1:38 pm on Jan 11, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Terrier, a page can have plenty of PageRank yet be unspidered. This leads to the listings with no snippet, and the URL listed instead of the title.

If you use the robots "none" META tag instead of /robots.txt then I don't think that Google will list the URL either (whether the PageRank shows or not I don't know).