Welcome to WebmasterWorld Guest from 54.166.54.215

Forum Moderators: goodroi

Search Engines doesn't work with my robots.txt file

   
9:50 pm on Jan 18, 2005 (gmt 0)

10+ Year Member



Hello All,

I set up a new robots.txt file 2 days ago.
I put it under [mashoomashoo.com...]

The file looks like this:
User-agent: *
Disallow: /modules/AdsManager/
Disallow: /modules/Lists/
Disallow: /modules/NewsManager/
Disallow: /modules/Polls/
Disallow: /modules/Users/
Disallow: /modules/Forums/Forum.aspx
Disallow: /modules/Forums/MyProfile.aspx
Disallow: /modules/Forums/PostMessage.aspx
Disallow: /modules/Forums/Settings.aspx
Disallow: /modules/Forums/Topic.aspx
Disallow: /modules/Forums/ForumRules.aspx

The search engines still look at the disallowed directories. For example Googlebot was in:
/Modules/NewsManager/ShowNews.aspx today.

What is wrong?

Thanks in advance for the help
Nir

10:38 pm on Jan 18, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



This is all you need to disallow any spiders what obey the robots.txt file access to your modules directory and any other sub directories below modules

User-agent: *
Disallow: /modules/

now that said googlebot has been prying in places it would not normally go, so it could be a glitch with googlebot at this present time.

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month