Welcome to WebmasterWorld Guest from 54.166.197.230

Forum Moderators: goodroi

Message Too Old, No Replies

Allowing only search engines to crawl forums

Only want search engines in my forums

     
10:39 pm on Mar 20, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 29, 2003
posts:32
votes: 0


I have a phpBB forum that I want to hack so it is search engine friendly. The only problem is that I only want search engines in there, and nothing else. Robots.txt doesn't have an allow directive so I'm not sure how to accomplish this. Would this work:

User-agent: *
Disallow: /phpBB2/

User-agent: googlebot, altavista, scooter, slurp
Disallow:

I'm not sure if I have the user-agents right...it's case sensitive right?

8:09 am on Mar 21, 2004 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38070
votes: 16


no, there is no "allow" format. This is one of the reason many of us say that thet robots.txt format is simply not usable any longer.
8:24 pm on Mar 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 29, 2003
posts:32
votes: 0


So there is no way of doing it? If you leave a disallow blank doesn't that mean the user-agent isn't disallowed from anything...so it would override the first disallow?
8:41 pm on Mar 21, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


According to the second to last example here [robotstxt.org], what you propose should do what you intend.

I'd watch my logs VERY carefully for a few days to be CERTAIN it's working as intended. ;)

12:35 am on Mar 25, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 29, 2003
posts:32
votes: 0


Does anyone know the exact user-agents for the popular search engines? Taking Google as an example, would just putting "Googlebot" work or do you need it more specific like "Googlebot/2.0".
8:37 pm on Mar 27, 2004 (gmt 0)

New User

10+ Year Member

joined:Dec 29, 2003
posts:32
votes: 0


Does anyone have a list of the major search engine user-agents?
8:40 pm on Mar 27, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 28, 2001
posts:1380
votes: 0


Quite a few here:
[webmasterworld.com...]