Page is a not externally linkable
Kenny B - 6:13 am on Apr 14, 2000 (gmt 0)
Can I address this spider as Scooter only? Do I need the version (2.0, 1.0)? Should I make a robots.txt file for each of these? Also, I read some advice from a professional at the Web Position Gold forum that said to make a doorway page for each keyword, but I don't observe this in any page when looking at the source code. In fact, pages have come up on a search for my keywords that don't even list them as keywords. Would this be beneficial if I have only a half dozen or so keywords? Any advice would be appreciated. I have been researching for months and I am ready to dig in and start submitting. Thanks.
I am a litle confused on how to address spiders in my robots.txt file. Some are straightforward like ArchitextSpider, some are not. The spider Spotting Chart on this site lists the Alta Vista spider as:
Scooter/2.0 G.R.A.B. X2.0
Scooter/1.0
scooter@pa.dec.com