Forum Moderators: open

Message Too Old, No Replies

Googlebot didn't seem to follow my robots.txt

it may have caused some problems for me

         

waynet

5:18 pm on Mar 7, 2003 (gmt 0)

10+ Year Member



I have one page with a link that points to what is probably a bad neighborhood. The site has a grey bar for a long time. I didn't want to get a penality for this, so I created a robots.txt for this page. It looks like this:

User-agent: googlebot
Disallow: thepage.html

But yet I see this page indexed now. Also I noticed that my site had dropped one on pagerank, even though my incoming links are about the same. Could it be the result of it indexing this page?

What could I have done wrong with the robots.txt? It seems to validate with a robots.txt tester.

jdMorgan

6:11 pm on Mar 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



waynet,

Add a leading slash to the pathname, and always follow the last Disallow line with a blank line:


User-agent: Googlebot
Disallow: /thepage.html

Jim