Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

How to get deep crawled by Fast

How do I get the 8 legged creep to go deep? Bot Enticement

3:42 am on Jul 17, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 25, 2002
votes: 0

I've got a site that's been up at the current URL for a couple of months. The Fast spider keeps grabbing the robot and index.html file every few days for last several weeks - but never goes any further.
Any suggestions for how to entice it to explore the site a bit more?
(The sits is non-commercial, so paid spidering isn't an option.)
4:44 pm on Sept 1, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 28, 2002
votes: 0

Same here, except replace months with years. :(

It comes almost daily to get / and/or robots.txt but I only see ONE other deep URL that it grabbed.... :(

I just can't get the spider to go down..

6:59 pm on Sept 1, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member heini is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Jan 31, 2001
votes: 0

I'm a bit short on sugestions here, I never really had this problem. Sure, the phase where Fast only "noticed" the site per index page is sometimes a bit lengthy, but in a 2 months time on average the site s got deep crawled...
Do you have good links to inner pages? a sitemap or similar spider friendly navigation?
1:48 am on Sept 2, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0

I have had the same problem with Fast over the past month. Fast dropped all but my index page from their SERPs, and only Fast's "Firstpage" crwaler has been visiting, GETting robots.txt and / only. The home page of the site is PR5 in Google, most internal pages are PR5 and PR4, and a few are PR3. No spam, no tricks. I think Fast is adjusting their algorithm, their spidering schedule, or both. Perhaps there is now a link popularity cutoff in place, and this site doesn't rank highly enough for frequent spidering. I really don't know...

Last week, I submitted a few of the internal pages to Fast. A few days later, they were spidered. I am still waiting to see if they show up in the results pages, though.


12:24 pm on Sept 2, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 18, 2002
votes: 0

Same problem here. And I have alot of sites... Before Fast was as good as Google but now...Something is fishy.
12:30 pm on Sept 2, 2002 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
votes: 9

>Something is fishy

Sure is.

BUT, I have to admit too that having Fast crawl a site has not been one of my problems lately. I know a lot of you have trouble with this, but again a neat clean site with good inbounds and a resonable linking structure, you should be able to get in fairly quickly.

I know it doesn't help you all much, but it is possible to make it work :)

3:51 pm on Sept 3, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 20, 2002
votes: 0

I have at least 50 inbound links. My index and one deep page is PR 6 on Google. I use text links only. No JavaScript. Valid HTML. Average size approx. 10 Kb. Sitemap. And after half a year, still no deep crawl and only this month they started to list two pages (that are now out again).

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members