Welcome to WebmasterWorld Guest from 54.146.1.178

Forum Moderators: open

Message Too Old, No Replies

INK and Fast free crawl

INK and Fast come every day do a few pages then leave?

     
9:44 am on Feb 6, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 1, 2002
posts:86
votes: 0


1. Website is CSE HTML validated and so is my stylesheet ( 0 errors).

2. Website has been up for about two years and has over 40,000 pages in Google and has a PR of 6 with over 500 incoming links.

3. Our partner website, has the same stats as above and uses the exact same URL re-writer. The last 2-3 weeks INK and Fast indexes thousands of pages a day on our partner website. And for our website only grabs a max of 60 a day. Any idea what would cause this?

4. Our robots.txt

User-agent: *
Disallow: /cgi-bin/

User-agent: TurnitinBot
Disallow: /

Here is a sample from WebTrends (02/05/03):

2. scooter 76 4.24%

3. Googlebot 61 3.4%

4. Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...] 20 1.11%

5. FAST-WebCrawler 11 0.61%

6. Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...] 9 0.5%

7. Fast PartnerSite Crawler 7 0.39%

8. Szukacz 2 0.11%

9. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler) 2 0.11%

10. hogcrawler (.2a;http: 1 0.05%

Also scooter just is finishing on our website, and crawled over 50k pages.

The only difference I can see is that on our site we paid for FAST and INK on positiontech. But only for 6 URL's on both Fast and INK about 5 months ago.

Ideas?

CatNip
:)

1:32 pm on Feb 7, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 16, 2001
posts:2059
votes: 0


Sometimes they pick the most popular pages, and only spider the whole site every now and then.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members