Fast - Spidering/Update Pattern - (deprecated) Altavista, Alltheweb.com forum at WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

Fast - Spidering/Update Pattern

Help put some flesh on the bones....

NFFC

12:25 pm on Dec 23, 2000 (gmt 0)

littleman asked here [webmasterworld.com]

>As far as updates go, it doesn't seem to be predictable - anyone else seen an update pattern?

To start this off I have put a chart here [searchengineworld.co.uk] which shows the visits from Fast's spider on one domain [black columns]. The domain went live December 16th 1999 and has never been submitted to Fast.

It would be nice if we can flesh the chart out with approx dates that new domains were added. Please take a look at the chart and feel free to post your comments in this thread, I will then update the chart.

Marcia

4:32 pm on Dec 30, 2000 (gmt 0)

A new site was visited in November, picking up the domain name of the prior inhabitant of that IP number (error since corrected), and the temporary under construction page put up by the host for my brand new site mid-November was what appeared at Fast.

I made submissions for the new site exactly one week after Thanksgiving, yes, FAST has been by in December, and the same under construction page, with the incorrect URL, is still there this morning. No sign of my end-of-Nov submission of the new site yet.

Fast is apparently not as *fast* as we might like it to be. (forgive me, I couldn't resist that)

>has never been submitted to Fast.

By what you've related, can we assume that Fast is following links to find sites? How heavily do links weigh with them?

Air

8:09 pm on Dec 30, 2000 (gmt 0)

Submitted a brand new domain/site to Fast on October 22/2000. Just the index page was submitted.

The index page was spidered once on November 7 and then twice on November 8th. 4 hours later the spider returned and went through all the pages on the site.

All pages appeared in the Fast index on December 5th.

NFFC

8:48 pm on Dec 30, 2000 (gmt 0)

Thanks all, updated the chart [searchengineworld.co.uk] with Air's info.

Would be particularly interested in the period from February to October if anyone can help.

>can we assume that Fast is following links to find sites?

Aggressively in my experience.

>How heavily do links weigh with them?

At the moment very heavily but I expect them to tone this down somewhat over the next update[s] and/or improve the checks on link quality.

startup

8:58 pm on Dec 30, 2000 (gmt 0)

Air:
Did you use any robot tags, meta tags, or <h> tags? Were there any sites in Fast with a link to the site?
I am experimenting with a site using only <head> <title> no meta tags and <h> tags.

Air

11:24 pm on Dec 30, 2000 (gmt 0)

>Did you use any robot tags, meta tags, or <h> tags?
No robot tags. Yes to description and keywords tags, yes to one header tag (H1) and title.

Keyword phrase appears in title, keywords, description and H1.

222 words in body counting stop words.

Total page size 2k

One link from another site in Fast.

Keyword density in body 4.6%

Overll Keyword density ~6.7% competitive three word phrase.

Site debuted at #99 (index page), other pages appear scattered on result pages 101 to 500+.

Note that two pages did not have any meta tags, no <h> tags, but did have a title. These are the lowest ranking pages.

Brett_Tabke

12:15 pm on Jan 2, 2001 (gmt 0)

I don't think you can detect a real pattern becaus there isn't one. Just from my end, I can't see a pattern to Fast's spidering. I do see a 30-45 day turn around time (most of the time).

From what I've heard, they are just playing it by ear on when and what they can index. It all depends on system resources (storage space, system space).

I do know they have a core rewrite in the works for 2001. Their db is showing strains and their algo needs improving. Especially since they are on so many sites now. I think it will be a very good year for Fast this year.

roscoepico

12:22 am on Jan 4, 2001 (gmt 0)

I have noticed that ATW is currently crawling links on my sites that have been removed since the last update. I'm guessing they are keeping a list of the links in there database that used exist and comming back for a recrawl or deep crawl at a later date. Im wondering if this is such a great idea as they may be wasting valuable resources in hammering links that no longer exist....

rtfmnews

3:43 am on Jan 5, 2001 (gmt 0)

Does FAST ranking really hit Lycos at some point, or does Lycos just use FAST's database and their own algo? One of my sites landed at #5 on AllTheWeb at their last update for my primary keywords, but it's still waaay down on Lycos.

Speaking of the algo, it's acting an awful lot like Northern Light. 7 of the top 10 (and 13 of 20) were identical between NL & AllTheWeb - higher correlation than I've seen between any other engines.

Dan0 :-)

han solo

4:14 pm on Jan 5, 2001 (gmt 0)

Let's see, I'll try to join in and help.

Aggregate time between getting one domain, and then the rest of the links, seems to be between 30-45 days.

I have one domain, just the root was hit, on Jan 3rd, and I'll keep you posted when fastlwspider/1.0 comes back for the rest of the stuff.

Is that the spider that grabs new stuff for the rest of you? I remember Ink used to have the traditional new site spider, which made updates nice and predictable, any guesses on if fast has one, or if this is it?

Thanks all,

Cheers,
Han Solo