Doing a search I have found robots.txt in googles cache?
stinkfoot
3:55 pm on Aug 28, 2005 (gmt 0)
Hi
I just did a site:www.mysite.com search and robots.txt is in a cached readable page? Never seen this before anyone have any ideas whats happening there?
No. Not all. I believe that it only indexes the ones that have been linked to from an HTML page somewhere. It cannot distinguish a robots.txt file from a standard text file.
Brett_Tabke
3:45 pm on Aug 29, 2005 (gmt 0)
>only indexes the ones that have been linked to
Ok - that makes sense.
Which leads to linking to someones robots.txt file to figure out what content they have blocked or filtered ;-)
stinkfoot
3:53 pm on Aug 29, 2005 (gmt 0)
AH ha! I knew somebody was up to no good. Now all I have to do is figure out what wombat is playing about with my sites and sue them!
Thanks guys ... Ill use you both as expert witnesses if you dont mind (joke)