homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

How to get deep crawled by Fast
How do I get the 8 legged creep to go deep? Bot Enticement

WebmasterWorld Senior Member 10+ Year Member

Msg#: 552 posted 3:42 am on Jul 17, 2002 (gmt 0)

I've got a site that's been up at the current URL for a couple of months. The Fast spider keeps grabbing the robot and index.html file every few days for last several weeks - but never goes any further.
Any suggestions for how to entice it to explore the site a bit more?
(The sits is non-commercial, so paid spidering isn't an option.)



10+ Year Member

Msg#: 552 posted 4:44 pm on Sep 1, 2002 (gmt 0)

Same here, except replace months with years. :(

It comes almost daily to get / and/or robots.txt but I only see ONE other deep URL that it grabbed.... :(

I just can't get the spider to go down..


WebmasterWorld Senior Member heini us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 552 posted 6:59 pm on Sep 1, 2002 (gmt 0)

I'm a bit short on sugestions here, I never really had this problem. Sure, the phase where Fast only "noticed" the site per index page is sometimes a bit lengthy, but in a 2 months time on average the site s got deep crawled...
Do you have good links to inner pages? a sitemap or similar spider friendly navigation?


WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member

Msg#: 552 posted 1:48 am on Sep 2, 2002 (gmt 0)

I have had the same problem with Fast over the past month. Fast dropped all but my index page from their SERPs, and only Fast's "Firstpage" crwaler has been visiting, GETting robots.txt and / only. The home page of the site is PR5 in Google, most internal pages are PR5 and PR4, and a few are PR3. No spam, no tricks. I think Fast is adjusting their algorithm, their spidering schedule, or both. Perhaps there is now a link popularity cutoff in place, and this site doesn't rank highly enough for frequent spidering. I really don't know...

Last week, I submitted a few of the internal pages to Fast. A few days later, they were spidered. I am still waiting to see if they show up in the results pages, though.



10+ Year Member

Msg#: 552 posted 12:24 pm on Sep 2, 2002 (gmt 0)

Same problem here. And I have alot of sites... Before Fast was as good as Google but now...Something is fishy.


WebmasterWorld Administrator 10+ Year Member

Msg#: 552 posted 12:30 pm on Sep 2, 2002 (gmt 0)

>Something is fishy

Sure is.

BUT, I have to admit too that having Fast crawl a site has not been one of my problems lately. I know a lot of you have trouble with this, but again a neat clean site with good inbounds and a resonable linking structure, you should be able to get in fairly quickly.

I know it doesn't help you all much, but it is possible to make it work :)


10+ Year Member

Msg#: 552 posted 3:51 pm on Sep 3, 2002 (gmt 0)

I have at least 50 inbound links. My index and one deep page is PR 6 on Google. I use text links only. No JavaScript. Valid HTML. Average size approx. 10 Kb. Sitemap. And after half a year, still no deep crawl and only this month they started to list two pages (that are now out again).

Global Options:
 top home search open messages active posts  

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved