Forum Moderators: open

Message Too Old, No Replies

Googlebot not listening to robots.txt

         

hdpt00

6:16 pm on Jul 16, 2004 (gmt 0)



First, this is a two part question. Part one is that I have a robotx.txt that disallows every user agent in folder "/". Which should be everything. AWStats show that googlebot visited pages and did not listen to robots.txt, it actually crawled. Is google behaving badly or could this be some type of server error? On a side note, somehow I got a PR1 with no links, only two other people know of the site. If only everyone should have such problems of trying to get googlebot to stop crawling, lol.

And another issue, I use a custom in-house CMS to run all the content. Should I put it so googlebot isn't allowed to visit those directories? I am afraid that someone might look in the robots.txt and see the directory for the admin login and somehow try to hack it. I will never have links pointing to that directory so is it better to just assume no one can get to it then and neither will google? What would you guys recommend under this situation?

Thanks!

hdpt00

3:27 am on Jul 17, 2004 (gmt 0)



Can anyone help?

</bump>

jdMorgan

3:39 am on Jul 17, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Validate your robots.txt here [searchengineworld.com].

There is usually no need to include un-linked, password-protected directories in robots.txt, for the reasons you cite.

Jim