How to get deep crawled by Fast

How do I get the 8 legged creep to go deep? Bot Enticement

[dcheney]

Howdy,
I've got a site that's been up at the current URL for a couple of months. The Fast spider keeps grabbing the robot and index.html file every few days for last several weeks - but never goes any further.
Any suggestions for how to entice it to explore the site a bit more?
(The sits is non-commercial, so paid spidering isn't an option.)

[savvy1]

Same here, except replace months with years. :(

It comes almost daily to get / and/or robots.txt but I only see ONE other deep URL that it grabbed.... :(

I just can't get the spider to go down..

[heini]

I'm a bit short on sugestions here, I never really had this problem. Sure, the phase where Fast only "noticed" the site per index page is sometimes a bit lengthy, but in a 2 months time on average the site s got deep crawled...
Do you have good links to inner pages? a sitemap or similar spider friendly navigation?

[jdMorgan]

I have had the same problem with Fast over the past month. Fast dropped all but my index page from their SERPs, and only Fast's "Firstpage" crwaler has been visiting, GETting robots.txt and / only. The home page of the site is PR5 in Google, most internal pages are PR5 and PR4, and a few are PR3. No spam, no tricks. I think Fast is adjusting their algorithm, their spidering schedule, or both. Perhaps there is now a link popularity cutoff in place, and this site doesn't rank highly enough for frequent spidering. I really don't know...

Last week, I submitted a few of the internal pages to Fast. A few days later, they were spidered. I am still waiting to see if they show up in the results pages, though.

Jim

[esset2002]

Same problem here. And I have alot of sites... Before Fast was as good as Google but now...Something is fishy.

[Rumbas]

>Something is fishy

Sure is.

BUT, I have to admit too that having Fast crawl a site has not been one of my problems lately. I know a lot of you have trouble with this, but again a neat clean site with good inbounds and a resonable linking structure, you should be able to get in fairly quickly.

I know it doesn't help you all much, but it is possible to make it work :)

[luma]

I have at least 50 inbound links. My index and one deep page is PR 6 on Google. I use text links only. No JavaScript. Valid HTML. Average size approx. 10 Kb. Sitemap. And after half a year, still no deep crawl and only this month they started to list two pages (that are now out again).