homepage Welcome to WebmasterWorld Guest from 50.17.66.61
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
INK and Fast free crawl
INK and Fast come every day do a few pages then leave?
Catnip

10+ Year Member



 
Msg#: 1934 posted 9:44 am on Feb 6, 2003 (gmt 0)

1. Website is CSE HTML validated and so is my stylesheet ( 0 errors).

2. Website has been up for about two years and has over 40,000 pages in Google and has a PR of 6 with over 500 incoming links.

3. Our partner website, has the same stats as above and uses the exact same URL re-writer. The last 2-3 weeks INK and Fast indexes thousands of pages a day on our partner website. And for our website only grabs a max of 60 a day. Any idea what would cause this?

4. Our robots.txt

User-agent: *
Disallow: /cgi-bin/

User-agent: TurnitinBot
Disallow: /

Here is a sample from WebTrends (02/05/03):

2. scooter 76 4.24%

3. Googlebot 61 3.4%

4. Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...] 20 1.11%

5. FAST-WebCrawler 11 0.61%

6. Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...] 9 0.5%

7. Fast PartnerSite Crawler 7 0.39%

8. Szukacz 2 0.11%

9. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; MSIECrawler) 2 0.11%

10. hogcrawler (.2a;http: 1 0.05%

Also scooter just is finishing on our website, and crawled over 50k pages.

The only difference I can see is that on our site we paid for FAST and INK on positiontech. But only for 6 URL's on both Fast and INK about 5 months ago.

Ideas?

CatNip
:)

 

lazerzubb

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 1934 posted 1:32 pm on Feb 7, 2003 (gmt 0)

Sometimes they pick the most popular pages, and only spider the whole site every now and then.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved