homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderators: Brett Tabke

Paid Inclusion Engines and Topics Forum

  posting off  
Deep crawling of AV.com?

 2:18 pm on Mar 23, 2001 (gmt 0)

I remember that someone metioned the av-spider doesnīt crawl deep into a site. Also a reason for submitting more than the root page.
Am I right? Is it past? I know that last year the av-spider did it, but I really donīt know if itīs still a current issue.



 8:35 pm on Mar 23, 2001 (gmt 0)

AV hasn't been very good at deep spidering recently, but with all their changes lately, its too early to see if this will be long term. I've been submitting some subpages (which I didn't used to do) and then leaving the rest for AV to find. I figure this way I'll be covered both shot term and long term.


 10:29 pm on Mar 23, 2001 (gmt 0)

Scooter has been at our site the last two days doing a deep crawl. So far, Scooter has grabbed 69 pages which is a little over half our site. Only submitted the root page and that was on 5-24-00

www.oursite.com scooter3.sv.av.com - - [21/Mar/2001:19:23:03 -0500] "GET /robots.txt HTTP/1.0" 200 1028 "-" "Scooter-3.0"

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved