homepage Welcome to WebmasterWorld Guest from 54.234.126.92
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Visit PubCon.com
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library : Charter : Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
FAST crawler - very strange behaviour
Hundreds of hits for the same page in the past 4 days.
Vlas_L




msg:221194		 1:13 am on Feb 11, 2003 (gmt 0)

Has anyone seen FAST Crawler request the same page hundreds of times a day?

Here are the stats:

UA - FAST-WebCrawler/3.6 (atw-crawler at fast dot no; [fast.no...]

IP - 66.77.73.77

Number of requests for the same page
February 7, 2003 559
February 8, 2003 771
February 9, 2003 699
February 10, 2003 386
(there are still about 7 hrs left today - PST)

Has anyone seen this before?

How is FAST about feedback?

I've been a little hesitant about contacting them as they have been very good to me. Wouldn't want to change any of that - most of my content is targeted at Canadians.

Thanks for any insight.

 

Rumbas




msg:221195		 12:43 pm on Feb 11, 2003 (gmt 0)

Hi Vlas - Welcome to the board.

Hmm, that's certainly strange crawler behaviour and I have no idea why the crawler would hit the same page over and over again.

FASTs crawler usually behaves like a good boy (girl?) and imo their feedback is really good. I wouldn't hesitate to contact them on this matter.

Good luck and let us know if you get it solved.

Big_Dug




msg:221196		 2:18 pm on Feb 12, 2003 (gmt 0)

Do you issue session IDs at all?
These can create spider traps.
Vlas_L




msg:221197		 6:28 pm on Feb 13, 2003 (gmt 0)

The site runs on ASP and issues session ID's. But if that would be the case I don't understand why that would affect just one page. This is the first time I've seen this; the site has been around for almost 3 years and indexed by FAST for much of that.

Big_Dug: Can you explain little more what you mean by session IDs spider traps?

BTW - The crawler quit requesting that page about 1/2 a day after posting the topic.

Vlas_L




msg:221198		 10:00 pm on Feb 13, 2003 (gmt 0)

I've been in contact with FAST and problem was on my end. The page that got all the requests has many links pointing to it with variable info. My tracking didn't record the variable strings so it appeared to me that FAST was retrieveing identical url.

FAST feedback was excellent, next time I won't hesitate emailing them right away.

Thanks.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved