Welcome to WebmasterWorld Guest from 54.226.246.160

Forum Moderators: open

Message Too Old, No Replies

INK and Fast free crawl

INK and Fast come every day do a few pages then leave?

     

Catnip

9:44 am on Feb 6, 2003 (gmt 0)

10+ Year Member



1. Website is CSE HTML validated and so is my stylesheet ( 0 errors).

2. Website has been up for about two years and has over 40,000 pages in Google and has a PR of 6 with over 500 incoming links.

3. Our partner website, has the same stats as above and uses the exact same URL re-writer. The last 2-3 weeks INK and Fast indexes thousands of pages a day on our partner website. And for our website only grabs a max of 60 a day. Any idea what would cause this?

4. Our robots.txt

User-agent: *
Disallow: /cgi-bin/

User-agent: TurnitinBot
Disallow: /

Here is a sample from WebTrends (02/05/03):

2. scooter 76 4.24%

3. Googlebot 61 3.4%

4. Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...] 20 1.11%

5. FAST-WebCrawler 11 0.61%

6. Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...] 9 0.5%

7. Fast PartnerSite Crawler 7 0.39%

8. Szukacz 2 0.11%

9. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler) 2 0.11%

10. hogcrawler (.2a;http: 1 0.05%

Also scooter just is finishing on our website, and crawled over 50k pages.

The only difference I can see is that on our site we paid for FAST and INK on positiontech. But only for 6 URL's on both Fast and INK about 5 months ago.

Ideas?

CatNip
:)

lazerzubb

1:32 pm on Feb 7, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sometimes they pick the most popular pages, and only spider the whole site every now and then.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month