homepage Welcome to WebmasterWorld Guest from 184.72.72.182
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Allowing only search engines to crawl forums
Only want search engines in my forums
Psycho111




msg:1528107
 10:39 pm on Mar 20, 2004 (gmt 0)

I have a phpBB forum that I want to hack so it is search engine friendly. The only problem is that I only want search engines in there, and nothing else. Robots.txt doesn't have an allow directive so I'm not sure how to accomplish this. Would this work:

User-agent: *
Disallow: /phpBB2/

User-agent: googlebot, altavista, scooter, slurp
Disallow:

I'm not sure if I have the user-agents right...it's case sensitive right?

 

Brett_Tabke




msg:1528108
 8:09 am on Mar 21, 2004 (gmt 0)

no, there is no "allow" format. This is one of the reason many of us say that thet robots.txt format is simply not usable any longer.

Psycho111




msg:1528109
 8:24 pm on Mar 21, 2004 (gmt 0)

So there is no way of doing it? If you leave a disallow blank doesn't that mean the user-agent isn't disallowed from anything...so it would override the first disallow?

DaveAtIFG




msg:1528110
 8:41 pm on Mar 21, 2004 (gmt 0)

According to the second to last example here [robotstxt.org], what you propose should do what you intend.

I'd watch my logs VERY carefully for a few days to be CERTAIN it's working as intended. ;)

Psycho111




msg:1528111
 12:35 am on Mar 25, 2004 (gmt 0)

Does anyone know the exact user-agents for the popular search engines? Taking Google as an example, would just putting "Googlebot" work or do you need it more specific like "Googlebot/2.0".

Psycho111




msg:1528112
 8:37 pm on Mar 27, 2004 (gmt 0)

Does anyone have a list of the major search engine user-agents?

dvduval




msg:1528113
 8:40 pm on Mar 27, 2004 (gmt 0)

Quite a few here:
[webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved