Welcome to WebmasterWorld Guest from 188.8.131.52 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Allowing only search engines to crawl forums Only want search engines in my forums Psycho111 msg:1528107 10:39 pm on Mar 20, 2004 (gmt 0) I have a phpBB forum that I want to hack so it is search engine friendly. The only problem is that I only want search engines in there, and nothing else. Robots.txt doesn't have an allow directive so I'm not sure how to accomplish this. Would this work:
User-agent: googlebot, altavista, scooter, slurp
I'm not sure if I have the user-agents right...it's case sensitive right?