Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- robots.txt


Kenny B - 6:13 am on Apr 14, 2000 (gmt 0)


I am a litle confused on how to address spiders in my robots.txt file. Some are straightforward like ArchitextSpider, some are not. The spider Spotting Chart on this site lists the Alta Vista spider as:
Scooter/2.0 G.R.A.B. X2.0
Scooter/1.0
scooter@pa.dec.com

Can I address this spider as Scooter only? Do I need the version (2.0, 1.0)? Should I make a robots.txt file for each of these?

Also, I read some advice from a professional at the Web Position Gold forum that said to make a doorway page for each keyword, but I don't observe this in any page when looking at the source code. In fact, pages have come up on a search for my keywords that don't even list them as keywords. Would this be beneficial if I have only a half dozen or so keywords?

Any advice would be appreciated. I have been researching for months and I am ready to dig in and start submitting.

Thanks.


Thread source:: http://www.webmasterworld.com/robots_txt/81.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com