homepage Welcome to WebmasterWorld Guest from 54.167.179.48
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
robot names - please help
Ade

10+ Year Member



 
Msg#: 757 posted 7:45 pm on Jun 10, 2001 (gmt 0)

can anyone help me? I have a UK website and I am going to have to divide it into 2 sections british and international

I need to write a robots.text file but i do not have access to server logs, i need to know the names of these spiders, in particular the altavistaUK and miragoUK bots

does anyone know of an extensive and upto date list of spider names

~thankyou

 

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 757 posted 9:44 pm on Jun 10, 2001 (gmt 0)

Anyone on that side of the pond care to jump in?

startup

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 757 posted 1:03 am on Jun 11, 2001 (gmt 0)

Abe, you stumped littleman on a spider question! I would have this thread printed and framed, it is very rare that this happens.:)

Scooter-3.0.g84-95,209.73.164.129
Scooter-3.0.gmkd48-63,209.73.164.129
Scooter-3.0.gmkd64-79,209.73.164.130
Scooter-3.0.c1,209.73.164.126
Scooter-1.0,209.73.174.250
Scooter-3.0.FS,209.73.164.40:209.73.164.132
HenryTheMiragoRobot,217.33.60.249

The "Scooters" are the crawlers, I do not submit to AVuk.
Henry is the only mirago bot I know of.




Ade

10+ Year Member



 
Msg#: 757 posted 6:30 pm on Jun 11, 2001 (gmt 0)

thanks startup,

littleman can still have a grab because I need a list of as many spider names as poss, not just UK ones ;)

is google's spider called googlebot or do they have multiple spiders with different names now?

Jaf

10+ Year Member



 
Msg#: 757 posted 6:37 pm on Jun 11, 2001 (gmt 0)

I have a list of robots at [jafsoft.com...]

The names tend to exclude version numbers, and
I don't go back and check they're still active once added to the page.

I also have a page on search engines at
[jafsoft.com...]
This includes a list of around a dozen UK
search engines, but doesn't include their spider
names.

I seem to be posting this URL a few times :-)
Is there a resource page on this site I should
be submitting these links to?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved