homepage Welcome to WebmasterWorld Guest from 50.17.177.99
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
ATW Updates
Are they rolling or monthly/bi-monthly.
Canary




msg:223481
 10:48 pm on May 2, 2003 (gmt 0)

Lots of new pages listed for my sites.

Are ATW updates rolling or monthly - like the big G or when they feel like it?

 

heini




msg:223482
 11:01 am on May 3, 2003 (gmt 0)

Canary, ATW updates in chunks and slices. Frankly I have not been able to establish a pattern. Looking over the refresh dates on large and small sites I can't say I have found a way to force a refreshed listing yet.

Tim




msg:223483
 6:53 pm on May 3, 2003 (gmt 0)

We roll a new index every 7-8 days. This refreshes a little over a quarter of the index. We also have a daily update which refreshes 5-10 million pages.

heini




msg:223484
 7:21 pm on May 3, 2003 (gmt 0)

Hey, Tim, thanks for the info!

The daily update, that's not the news crawling, I assume? (as those are respidered comtinuously)
What kind of sites do you refresh daily?

HitProf




msg:223485
 8:00 pm on May 3, 2003 (gmt 0)

Hello Tim,

It's very good to have a Fast rep over here.
Hope you will participate and shed some light on some of our our questions.

Canary




msg:223486
 9:06 pm on May 4, 2003 (gmt 0)

Good Stuff - Thanks for all the replies.

Impressive to refresh quarter of the index over 7-8 days.

Getting some good referrals from some of ATW search partners at the moment too :)

indigojo




msg:223487
 10:17 pm on May 4, 2003 (gmt 0)

Wow this is a pretty impressive update from where I sit

Rumbas




msg:223488
 9:10 am on May 5, 2003 (gmt 0)

Tim, thanks for clarifying that. Much apreciated.

>daily updates

Nice. I'd really like to know too, what you base the selection of sites that get updated on?

heini




msg:223489
 9:36 am on May 5, 2003 (gmt 0)

*Slaps Head*
I guess Tim is talking about the PFI listings here.

Rumbas




msg:223490
 10:02 am on May 5, 2003 (gmt 0)

Guess you're right heini.
Wasn't their a number of PFI pages in the db floating around somewhere? If I remember correctly it was in that ball park area. Not sure my memory serves me here though..

On the other hand I see some pages that I know are not PFI get respidered on a rather frequently basis.

Tim




msg:223491
 1:11 pm on May 5, 2003 (gmt 0)

We have a crawl of both PFI and free pages every 24-48 hours. The number I was referring to was free crawled pages not paid. Of course we have paid inclusion too.

heini




msg:223492
 1:15 pm on May 5, 2003 (gmt 0)

Tim, thanks for clarifying. In any case I'd love to know which free pages get refreshed every 24-48 hours?

Canary




msg:223493
 7:11 pm on May 5, 2003 (gmt 0)

Yep, I wonder if we are talking about the same quarter of the index being refreshed - much like Altavista and Google do with their fresh dates or a different quarter so the whole index is update once every 28-32 days.

Gonzalez




msg:223494
 9:40 am on May 9, 2003 (gmt 0)

Hi,

Like said before, FAST does not spider the site at once. Last month I renamed a few pages and ATW lists both old and new file names. I think this will corrected once the spider tries to index old pages. I believe also that they don't do a massive update like Google. As they get new pages indexed they add it almost "live".

jranes




msg:223495
 7:40 pm on May 21, 2003 (gmt 0)

Thanks for the info Tim. You guys are doing a great job.

h_b_k




msg:223496
 10:49 am on Jun 11, 2003 (gmt 0)

How often is FAST-Webcrawler deepcrawling sites?

using the URL Investigator I can read about "Last changed". (for my homepage 2003-06-09).
But this date is valid for the Homepage only, not for all other pages.

Last time FAST-Webcrawler visited parts of my site, was 2003-05-06.
After then FAST-Webcraler/3.7./Firstpage visited regularly (every 1-4 days) the homepage (index.shtml) only.

h_b_k

(date format: yyyy-mm-dd)

[edited by: h_b_k at 12:03 pm (utc) on June 11, 2003]

trillianjedi




msg:223497
 11:18 am on Jun 11, 2003 (gmt 0)

Tim,

Glad we finally got you to surface. Did you get a response to the BT email by the way?

TJ

h_b_k




msg:223498
 4:07 pm on Jun 24, 2003 (gmt 0)

today I have noticed the first deepcrawl since 2003-05-06 this means, they have a schedule for my site of about 1,5 months?

is this depending on the "importance" of a page?
or is 1,5 month today the regular behavior of fast?

time before I had a shorter cycle of deepcrawls.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved