New Spidering pattern?

[jlara]

I have just noticed something in my logs that is attracting my attention... Many spiders recently stopped going after the submitted URLs and instead go for the index.html. To make this post short and sweet I'll concentrate on AV.

You submit [somesub.somesite.com...]
Then that site gets listed.
Two weeks later AV sends a different spider and goes to
[somesub.somesite.com...]

This is a little surprising. If the index.html was optimized for another engine, or if there is a meta refresh, then this could be a problem.

Has this always been an AV pattern? I will watch my rankings and see if it causes a drop. This index.html spidering trend throughs another wrench into SEO efforts, because no matter how well you make a page -- it is very difficult for it to match each search engine's algorithum.

[rogerd]

Jlara, I'd say it is pretty common SE behavior to go back to grab the index page of any site for which it has pages indexed. Although some are more aggressive than others, their objective is to capture the entire site (page and depth limits may exist, of course), as well as find links to other sites. AltaVista is employing some level of theme analysis, and for that to work it has to sample the entire site.

Many people submit only their home page or a hallway page and wait for the spider to find the rest of the pages on his own - "found" pages sometimes rank better than submitted pages. (This approach takes a lot of patience.)

If you have pages you don't want a particular SE to see, you should exclude them in your robots.txt file.

[Brett_Tabke]

Ya, if the index isn't in the db, Alta will grab it too. It usually takes longer than that though.