Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL

robot names - please help



7:45 pm on Jun 10, 2001 (gmt 0)

10+ Year Member

can anyone help me? I have a UK website and I am going to have to divide it into 2 sections british and international

I need to write a robots.text file but i do not have access to server logs, i need to know the names of these spiders, in particular the altavistaUK and miragoUK bots

does anyone know of an extensive and upto date list of spider names


9:44 pm on Jun 10, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

Anyone on that side of the pond care to jump in?
1:03 am on Jun 11, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Abe, you stumped littleman on a spider question! I would have this thread printed and framed, it is very rare that this happens.:)


The "Scooters" are the crawlers, I do not submit to AVuk.
Henry is the only mirago bot I know of.


6:30 pm on Jun 11, 2001 (gmt 0)

10+ Year Member

thanks startup,

littleman can still have a grab because I need a list of as many spider names as poss, not just UK ones ;)

is google's spider called googlebot or do they have multiple spiders with different names now?


6:37 pm on Jun 11, 2001 (gmt 0)

10+ Year Member

I have a list of robots at [jafsoft.com...]

The names tend to exclude version numbers, and
I don't go back and check they're still active once added to the page.

I also have a page on search engines at
This includes a list of around a dozen UK
search engines, but doesn't include their spider

I seem to be posting this URL a few times :-)
Is there a resource page on this site I should
be submitting these links to?


Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved