homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Allowing only search engines to crawl forums
Only want search engines in my forums

 10:39 pm on Mar 20, 2004 (gmt 0)

I have a phpBB forum that I want to hack so it is search engine friendly. The only problem is that I only want search engines in there, and nothing else. Robots.txt doesn't have an allow directive so I'm not sure how to accomplish this. Would this work:

User-agent: *
Disallow: /phpBB2/

User-agent: googlebot, altavista, scooter, slurp

I'm not sure if I have the user-agents right...it's case sensitive right?



 8:09 am on Mar 21, 2004 (gmt 0)

no, there is no "allow" format. This is one of the reason many of us say that thet robots.txt format is simply not usable any longer.


 8:24 pm on Mar 21, 2004 (gmt 0)

So there is no way of doing it? If you leave a disallow blank doesn't that mean the user-agent isn't disallowed from anything...so it would override the first disallow?


 8:41 pm on Mar 21, 2004 (gmt 0)

According to the second to last example here [robotstxt.org], what you propose should do what you intend.

I'd watch my logs VERY carefully for a few days to be CERTAIN it's working as intended. ;)


 12:35 am on Mar 25, 2004 (gmt 0)

Does anyone know the exact user-agents for the popular search engines? Taking Google as an example, would just putting "Googlebot" work or do you need it more specific like "Googlebot/2.0".


 8:37 pm on Mar 27, 2004 (gmt 0)

Does anyone have a list of the major search engine user-agents?


 8:40 pm on Mar 27, 2004 (gmt 0)

Quite a few here:

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved