homepage Welcome to WebmasterWorld Guest from 54.196.201.253
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Search Engines doesn't work with my robots.txt file
nirpet




msg:1529351
 9:50 pm on Jan 18, 2005 (gmt 0)

Hello All,

I set up a new robots.txt file 2 days ago.
I put it under [mashoomashoo.com...]

The file looks like this:
User-agent: *
Disallow: /modules/AdsManager/
Disallow: /modules/Lists/
Disallow: /modules/NewsManager/
Disallow: /modules/Polls/
Disallow: /modules/Users/
Disallow: /modules/Forums/Forum.aspx
Disallow: /modules/Forums/MyProfile.aspx
Disallow: /modules/Forums/PostMessage.aspx
Disallow: /modules/Forums/Settings.aspx
Disallow: /modules/Forums/Topic.aspx
Disallow: /modules/Forums/ForumRules.aspx

The search engines still look at the disallowed directories. For example Googlebot was in:
/Modules/NewsManager/ShowNews.aspx today.

What is wrong?

Thanks in advance for the help
Nir

 

ncw164x




msg:1529352
 10:38 pm on Jan 18, 2005 (gmt 0)

This is all you need to disallow any spiders what obey the robots.txt file access to your modules directory and any other sub directories below modules

User-agent: *
Disallow: /modules/

now that said googlebot has been prying in places it would not normally go, so it could be a glitch with googlebot at this present time.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved