homepage Welcome to WebmasterWorld Guest from 54.237.151.188
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New Fast/Overture UA
Yet another new one from FAST
jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2250 posted 11:06 pm on Sep 8, 2003 (gmt 0)

66.77.73.128 - - [08/Sep/2003:14:51:39 -0400] "GET /robots.txt HTTP/1.0" 200 3670 "-" "Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; [fast.no...]

Just spotted today.

It seems to be looking for the same old "fast" robot name in robots.txt. I'm assuming this because it thought it was allowed to spider my sites. But having an unrecognized user-agent string, it wasn't, and got a 403. Yeah, I fixed it! :)

Jim

 

heini

WebmasterWorld Senior Member heini us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2250 posted 11:41 pm on Sep 8, 2003 (gmt 0)

Hmm, probably a matter of rebranding (in logs ;))

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2250 posted 12:23 am on Sep 9, 2003 (gmt 0)

Yeah, it's a good argument for NOT using your company name in your robot name, since it might change tomorrow.

Jim

percentages

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2250 posted 7:09 am on Sep 10, 2003 (gmt 0)

>Yeah, it's a good argument for NOT using your company name in your robot name, since it might change tomorrow.

LOL...definitely true if you happen to be an SE spider ;)

fr_cnx

10+ Year Member



 
Msg#: 2250 posted 2:11 pm on Sep 10, 2003 (gmt 0)

this robot has visited my site in the same time as Fast, could it be just visiting Overture customers?

sidyadav

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 2250 posted 4:37 am on Sep 11, 2003 (gmt 0)

I'm assuming when the acquiring of Overture is finally finished by Yahoo, the User-Agent should look like this:
Yahoo-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; [fast.no...]

LOL!

Sid

Marcia

WebmasterWorld Senior Member marcia us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 2250 posted 10:09 am on Sep 11, 2003 (gmt 0)

>>could it be just visiting Overture customers?

No, I've just caught it at one site that had only a temporary index page up, the rest was down for a couple of months. Tiny site, I put the normal page back up on the 4th of September and the other 4 pages back up, and that user agent plus the normal one got all 5 pages between then and yesterday. It's not in Overture.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved