Welcome to WebmasterWorld Guest from 50.19.47.197

Forum Moderators: open

Message Too Old, No Replies

Deep crawling of AV.com?

     
2:18 pm on Mar 23, 2001 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 15, 2000
posts:482
votes: 0


I remember that someone metioned the av-spider doesnīt crawl deep into a site. Also a reason for submitting more than the root page.
Am I right? Is it past? I know that last year the av-spider did it, but I really donīt know if itīs still a current issue.
8:35 pm on Mar 23, 2001 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 20, 2000
posts:1702
votes: 0


AV hasn't been very good at deep spidering recently, but with all their changes lately, its too early to see if this will be long term. I've been submitting some subpages (which I didn't used to do) and then leaving the rest for AV to find. I figure this way I'll be covered both shot term and long term.
10:29 pm on Mar 23, 2001 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 1, 2001
posts:183
votes: 0


Scooter has been at our site the last two days doing a deep crawl. So far, Scooter has grabbed 69 pages which is a little over half our site. Only submitted the root page and that was on 5-24-00

www.oursite.com scooter3.sv.av.com - - [21/Mar/2001:19:23:03 -0500] "GET /robots.txt HTTP/1.0" 200 1028 "-" "Scooter-3.0"