Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Deep crawling of AV.com?



2:18 pm on Mar 23, 2001 (gmt 0)

10+ Year Member

I remember that someone metioned the av-spider doesnīt crawl deep into a site. Also a reason for submitting more than the root page.
Am I right? Is it past? I know that last year the av-spider did it, but I really donīt know if itīs still a current issue.


8:35 pm on Mar 23, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

AV hasn't been very good at deep spidering recently, but with all their changes lately, its too early to see if this will be long term. I've been submitting some subpages (which I didn't used to do) and then leaving the rest for AV to find. I figure this way I'll be covered both shot term and long term.


10:29 pm on Mar 23, 2001 (gmt 0)

10+ Year Member

Scooter has been at our site the last two days doing a deep crawl. So far, Scooter has grabbed 69 pages which is a little over half our site. Only submitted the root page and that was on 5-24-00

www.oursite.com scooter3.sv.av.com - - [21/Mar/2001:19:23:03 -0500] "GET /robots.txt HTTP/1.0" 200 1028 "-" "Scooter-3.0"


Featured Threads

Hot Threads This Week

Hot Threads This Month