Welcome to WebmasterWorld Guest from 54.196.238.210

Forum Moderators: goodroi

Message Too Old, No Replies

Allowing only search engines to crawl forums

Only want search engines in my forums

     

Psycho111

10:39 pm on Mar 20, 2004 (gmt 0)

10+ Year Member



I have a phpBB forum that I want to hack so it is search engine friendly. The only problem is that I only want search engines in there, and nothing else. Robots.txt doesn't have an allow directive so I'm not sure how to accomplish this. Would this work:

User-agent: *
Disallow: /phpBB2/

User-agent: googlebot, altavista, scooter, slurp
Disallow:

I'm not sure if I have the user-agents right...it's case sensitive right?

Brett_Tabke

8:09 am on Mar 21, 2004 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



no, there is no "allow" format. This is one of the reason many of us say that thet robots.txt format is simply not usable any longer.

Psycho111

8:24 pm on Mar 21, 2004 (gmt 0)

10+ Year Member



So there is no way of doing it? If you leave a disallow blank doesn't that mean the user-agent isn't disallowed from anything...so it would override the first disallow?

DaveAtIFG

8:41 pm on Mar 21, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



According to the second to last example here [robotstxt.org], what you propose should do what you intend.

I'd watch my logs VERY carefully for a few days to be CERTAIN it's working as intended. ;)

Psycho111

12:35 am on Mar 25, 2004 (gmt 0)

10+ Year Member



Does anyone know the exact user-agents for the popular search engines? Taking Google as an example, would just putting "Googlebot" work or do you need it more specific like "Googlebot/2.0".

Psycho111

8:37 pm on Mar 27, 2004 (gmt 0)

10+ Year Member



Does anyone have a list of the major search engine user-agents?

dvduval

8:40 pm on Mar 27, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Quite a few here:
[webmasterworld.com...]
 

Featured Threads

Hot Threads This Week

Hot Threads This Month