Forum Moderators: goodroi
I set up a new robots.txt file 2 days ago.
I put it under [mashoomashoo.com...]
The file looks like this:
User-agent: *
Disallow: /modules/AdsManager/
Disallow: /modules/Lists/
Disallow: /modules/NewsManager/
Disallow: /modules/Polls/
Disallow: /modules/Users/
Disallow: /modules/Forums/Forum.aspx
Disallow: /modules/Forums/MyProfile.aspx
Disallow: /modules/Forums/PostMessage.aspx
Disallow: /modules/Forums/Settings.aspx
Disallow: /modules/Forums/Topic.aspx
Disallow: /modules/Forums/ForumRules.aspx
The search engines still look at the disallowed directories. For example Googlebot was in:
/Modules/NewsManager/ShowNews.aspx today.
What is wrong?
Thanks in advance for the help
Nir
User-agent: *
Disallow: /modules/
now that said googlebot has been prying in places it would not normally go, so it could be a glitch with googlebot at this present time.