Welcome to WebmasterWorld Guest from 50.16.24.12

Forum Moderators: Ocean10000 & incrediBILL

robot names - please help

   

Ade

7:45 pm on Jun 10, 2001 (gmt 0)

10+ Year Member



can anyone help me? I have a UK website and I am going to have to divide it into 2 sections british and international

I need to write a robots.text file but i do not have access to server logs, i need to know the names of these spiders, in particular the altavistaUK and miragoUK bots

does anyone know of an extensive and upto date list of spider names

~thankyou

9:44 pm on Jun 10, 2001 (gmt 0)

WebmasterWorld Senior Member littleman is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Anyone on that side of the pond care to jump in?
1:03 am on Jun 11, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Abe, you stumped littleman on a spider question! I would have this thread printed and framed, it is very rare that this happens.:)

Scooter-3.0.g84-95,209.73.164.129
Scooter-3.0.gmkd48-63,209.73.164.129
Scooter-3.0.gmkd64-79,209.73.164.130
Scooter-3.0.c1,209.73.164.126
Scooter-1.0,209.73.174.250
Scooter-3.0.FS,209.73.164.40:209.73.164.132
HenryTheMiragoRobot,217.33.60.249

The "Scooters" are the crawlers, I do not submit to AVuk.
Henry is the only mirago bot I know of.



Ade

6:30 pm on Jun 11, 2001 (gmt 0)

10+ Year Member



thanks startup,

littleman can still have a grab because I need a list of as many spider names as poss, not just UK ones ;)

is google's spider called googlebot or do they have multiple spiders with different names now?

Jaf

6:37 pm on Jun 11, 2001 (gmt 0)

10+ Year Member



I have a list of robots at [jafsoft.com...]

The names tend to exclude version numbers, and
I don't go back and check they're still active once added to the page.

I also have a page on search engines at
[jafsoft.com...]
This includes a list of around a dozen UK
search engines, but doesn't include their spider
names.

I seem to be posting this URL a few times :-)
Is there a resource page on this site I should
be submitting these links to?

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved