| Welcome to WebmasterWorld Guest from 184.108.40.206 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|robot names - please help|
can anyone help me? I have a UK website and I am going to have to divide it into 2 sections british and international
I need to write a robots.text file but i do not have access to server logs, i need to know the names of these spiders, in particular the altavistaUK and miragoUK bots
does anyone know of an extensive and upto date list of spider names
Anyone on that side of the pond care to jump in?
Abe, you stumped littleman on a spider question! I would have this thread printed and framed, it is very rare that this happens.:)
The "Scooters" are the crawlers, I do not submit to AVuk.
Henry is the only mirago bot I know of.
littleman can still have a grab because I need a list of as many spider names as poss, not just UK ones ;)
is google's spider called googlebot or do they have multiple spiders with different names now?
I have a list of robots at [jafsoft.com...]
The names tend to exclude version numbers, and
I don't go back and check they're still active once added to the page.
I also have a page on search engines at
This includes a list of around a dozen UK
search engines, but doesn't include their spider
I seem to be posting this URL a few times :-)
Is there a resource page on this site I should
be submitting these links to?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved