Welcome to WebmasterWorld Guest from

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

robot names - please help



7:45 pm on Jun 10, 2001 (gmt 0)

New User

10+ Year Member

joined:Apr 25, 2001
votes: 0

can anyone help me? I have a UK website and I am going to have to divide it into 2 sections british and international

I need to write a robots.text file but i do not have access to server logs, i need to know the names of these spiders, in particular the altavistaUK and miragoUK bots

does anyone know of an extensive and upto date list of spider names


9:44 pm on June 10, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:June 17, 2000
votes: 0

Anyone on that side of the pond care to jump in?
1:03 am on June 11, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 8, 2000
votes: 0

Abe, you stumped littleman on a spider question! I would have this thread printed and framed, it is very rare that this happens.:)


The "Scooters" are the crawlers, I do not submit to AVuk.
Henry is the only mirago bot I know of.


6:30 pm on June 11, 2001 (gmt 0)

New User

10+ Year Member

joined:Apr 25, 2001
votes: 0

thanks startup,

littleman can still have a grab because I need a list of as many spider names as poss, not just UK ones ;)

is google's spider called googlebot or do they have multiple spiders with different names now?


6:37 pm on June 11, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:May 9, 2001
votes: 0

I have a list of robots at [jafsoft.com...]

The names tend to exclude version numbers, and
I don't go back and check they're still active once added to the page.

I also have a page on search engines at
This includes a list of around a dozen UK
search engines, but doesn't include their spider

I seem to be posting this URL a few times :-)
Is there a resource page on this site I should
be submitting these links to?


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members