Google indexing robots.txt

Forum Moderators: open

Message Too Old, No Replies

Google indexing robots.txt

HitProf

11:06 pm on Jul 9, 2003 (gmt 0)

Searching for information on LinkWalker I stumbled over a couple of robots.txt files in Google's serps. Is this new?

The robots.txt files from both webmasterworld and salon.com are indexed, so it shouldn't be because of errors in the robots exclusion file.

rfgdxm1

1:25 am on Jul 10, 2003 (gmt 0)

Technically permissable if Googlebot isn't excluded.

GrinninGordon

1:27 am on Jul 10, 2003 (gmt 0)

Jeez

Can anyone tell me how to stuff URL's I want to get crawled into my robots.txt file!

Krapulator

1:53 am on Jul 10, 2003 (gmt 0)

Yeah I've come across this recently when working on my robots.txt.

The funniest part is the webmasterworld robots.txt actually has pagerank and banklinks!

TheDave

1:56 am on Jul 10, 2003 (gmt 0)

Maybe its because of the fact that it does have backlinks that google has indexed it, ie. a linked to robots.txt gets indexed

Krapulator

2:01 am on Jul 10, 2003 (gmt 0)

ya think?

thewebboy

2:01 am on Jul 10, 2003 (gmt 0)

If a page has links going to it (and not excluded by robots.txt), it will be indexed. It is a little strange though.

rfgdxm1

2:11 am on Jul 10, 2003 (gmt 0)

Hmm...I never thought about it, but my robots.txt page is an orphan page. Thus, gray toolbar. Does this mean we should all start linking to our robots.txt pages? ;)

GrinninGordon

2:29 am on Jul 10, 2003 (gmt 0)

I would like Brett to explain what certain of his robot.txt's deny's are for, they seem to be for his own harvesting bots! Does this mean Brett is the source of all the Viagra and Nigerian emails we all get? Can I report this to Google?

:-)

rfgdxm1

2:58 am on Jul 10, 2003 (gmt 0)

I think we've uncovered Brett's dark secret. Hmm...if he is responsible for all the spam, why does he think I need male organ enlargement products? ;)

mcavic

3:34 am on Jul 10, 2003 (gmt 0)

Wait, wait - do spambots really obey robots.txt?

GrinninGordon

4:02 am on Jul 10, 2003 (gmt 0)

mcavic

They do if you own them and tell them too!

Funny, whenever I post here, I seem to get another Viagra, another diet pill, another loan and another 6 Nigerian emails. I just have a mental picture of Brett right now that would definately get edited :-)

HitProf

10:40 am on Jul 10, 2003 (gmt 0)

Hmm I just discovered it is nothing new and has been discussed before.:
[webmasterworld.com...]
and
[webmasterworld.com...] .

I think linking to it may explain it, not sure but sounds logical.

Thanks for the replies!