homepage Welcome to WebmasterWorld Guest from 54.237.38.30
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Allowing only search engines to crawl forums
Only want search engines in my forums
Psycho111

10+ Year Member



 
Msg#: 337 posted 10:39 pm on Mar 20, 2004 (gmt 0)

I have a phpBB forum that I want to hack so it is search engine friendly. The only problem is that I only want search engines in there, and nothing else. Robots.txt doesn't have an allow directive so I'm not sure how to accomplish this. Would this work:

User-agent: *
Disallow: /phpBB2/

User-agent: googlebot, altavista, scooter, slurp
Disallow:

I'm not sure if I have the user-agents right...it's case sensitive right?

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 337 posted 8:09 am on Mar 21, 2004 (gmt 0)

no, there is no "allow" format. This is one of the reason many of us say that thet robots.txt format is simply not usable any longer.

Psycho111

10+ Year Member



 
Msg#: 337 posted 8:24 pm on Mar 21, 2004 (gmt 0)

So there is no way of doing it? If you leave a disallow blank doesn't that mean the user-agent isn't disallowed from anything...so it would override the first disallow?

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 337 posted 8:41 pm on Mar 21, 2004 (gmt 0)

According to the second to last example here [robotstxt.org], what you propose should do what you intend.

I'd watch my logs VERY carefully for a few days to be CERTAIN it's working as intended. ;)

Psycho111

10+ Year Member



 
Msg#: 337 posted 12:35 am on Mar 25, 2004 (gmt 0)

Does anyone know the exact user-agents for the popular search engines? Taking Google as an example, would just putting "Googlebot" work or do you need it more specific like "Googlebot/2.0".

Psycho111

10+ Year Member



 
Msg#: 337 posted 8:37 pm on Mar 27, 2004 (gmt 0)

Does anyone have a list of the major search engine user-agents?

dvduval

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 337 posted 8:40 pm on Mar 27, 2004 (gmt 0)

Quite a few here:
[webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved