INK and Fast free crawl

INK and Fast come every day do a few pages then leave?

[Catnip]

1. Website is CSE HTML validated and so is my stylesheet ( 0 errors).

2. Website has been up for about two years and has over 40,000 pages in Google and has a PR of 6 with over 500 incoming links.

3. Our partner website, has the same stats as above and uses the exact same URL re-writer. The last 2-3 weeks INK and Fast indexes thousands of pages a day on our partner website. And for our website only grabs a max of 60 a day. Any idea what would cause this?

4. Our robots.txt

User-agent: *
Disallow: /cgi-bin/

User-agent: TurnitinBot
Disallow: /

Here is a sample from WebTrends (02/05/03):

2. scooter 76 4.24%

3. Googlebot 61 3.4%

4. Mozilla/5.0 (Slurp/cat; [e-mail]; [inktomi.com...] 20 1.11%

5. FAST-WebCrawler 11 0.61%

6. Mozilla/5.0 (Slurp/si; [e-mail]; [inktomi.com...] 9 0.5%

7. Fast PartnerSite Crawler 7 0.39%

8. Szukacz 2 0.11%

9. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler) 2 0.11%

10. hogcrawler (.2a;http: 1 0.05%

Also scooter just is finishing on our website, and crawled over 50k pages.

The only difference I can see is that on our site we paid for FAST and INK on positiontech. But only for 6 URL's on both Fast and INK about 5 months ago.

Ideas?

CatNip
:)

[lazerzubb]

Sometimes they pick the most popular pages, and only spider the whole site every now and then.