Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Deep crawling of AV.com?

2:18 pm on Mar 23, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 15, 2000
votes: 0

I remember that someone metioned the av-spider doesnīt crawl deep into a site. Also a reason for submitting more than the root page.
Am I right? Is it past? I know that last year the av-spider did it, but I really donīt know if itīs still a current issue.
8:35 pm on Mar 23, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 20, 2000
votes: 0

AV hasn't been very good at deep spidering recently, but with all their changes lately, its too early to see if this will be long term. I've been submitting some subpages (which I didn't used to do) and then leaving the rest for AV to find. I figure this way I'll be covered both shot term and long term.
10:29 pm on Mar 23, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 1, 2001
votes: 0

Scooter has been at our site the last two days doing a deep crawl. So far, Scooter has grabbed 69 pages which is a little over half our site. Only submitted the root page and that was on 5-24-00

www.oursite.com scooter3.sv.av.com - - [21/Mar/2001:19:23:03 -0500] "GET /robots.txt HTTP/1.0" 200 1028 "-" "Scooter-3.0"


Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members