homepage Welcome to WebmasterWorld Guest from 54.163.70.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
New Fast/Overture UA
Yet another new one from FAST
jdMorgan




msg:399394
 11:06 pm on Sep 8, 2003 (gmt 0)

66.77.73.128 - - [08/Sep/2003:14:51:39 -0400] "GET /robots.txt HTTP/1.0" 200 3670 "-" "Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; [fast.no...]

Just spotted today.

It seems to be looking for the same old "fast" robot name in robots.txt. I'm assuming this because it thought it was allowed to spider my sites. But having an unrecognized user-agent string, it wasn't, and got a 403. Yeah, I fixed it! :)

Jim

 

heini




msg:399395
 11:41 pm on Sep 8, 2003 (gmt 0)

Hmm, probably a matter of rebranding (in logs ;))

jdMorgan




msg:399396
 12:23 am on Sep 9, 2003 (gmt 0)

Yeah, it's a good argument for NOT using your company name in your robot name, since it might change tomorrow.

Jim

percentages




msg:399397
 7:09 am on Sep 10, 2003 (gmt 0)

>Yeah, it's a good argument for NOT using your company name in your robot name, since it might change tomorrow.

LOL...definitely true if you happen to be an SE spider ;)

fr_cnx




msg:399398
 2:11 pm on Sep 10, 2003 (gmt 0)

this robot has visited my site in the same time as Fast, could it be just visiting Overture customers?

sidyadav




msg:399399
 4:37 am on Sep 11, 2003 (gmt 0)

I'm assuming when the acquiring of Overture is finally finished by Yahoo, the User-Agent should look like this:
Yahoo-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; [fast.no...]

LOL!

Sid

Marcia




msg:399400
 10:09 am on Sep 11, 2003 (gmt 0)

>>could it be just visiting Overture customers?

No, I've just caught it at one site that had only a temporary index page up, the rest was down for a couple of months. Tiny site, I put the normal page back up on the 4th of September and the other 4 pages back up, and that user agent plus the normal one got all 5 pages between then and yesterday. It's not in Overture.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved