homepage Welcome to WebmasterWorld Guest from 54.145.183.169
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
Deep crawling of AV.com?
oLeon

10+ Year Member



 
Msg#: 502 posted 2:18 pm on Mar 23, 2001 (gmt 0)

I remember that someone metioned the av-spider doesnīt crawl deep into a site. Also a reason for submitting more than the root page.
Am I right? Is it past? I know that last year the av-spider did it, but I really donīt know if itīs still a current issue.

 

seth_wilde

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 502 posted 8:35 pm on Mar 23, 2001 (gmt 0)

AV hasn't been very good at deep spidering recently, but with all their changes lately, its too early to see if this will be long term. I've been submitting some subpages (which I didn't used to do) and then leaving the rest for AV to find. I figure this way I'll be covered both shot term and long term.

Son_House

10+ Year Member



 
Msg#: 502 posted 10:29 pm on Mar 23, 2001 (gmt 0)

Scooter has been at our site the last two days doing a deep crawl. So far, Scooter has grabbed 69 pages which is a little over half our site. Only submitted the root page and that was on 5-24-00

www.oursite.com scooter3.sv.av.com - - [21/Mar/2001:19:23:03 -0500] "GET /robots.txt HTTP/1.0" 200 1028 "-" "Scooter-3.0"

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved