Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

New Spidering pattern?

New Spidering pattern?

1:03 pm on Oct 24, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 1, 2000
votes: 0

I have just noticed something in my logs that is attracting my attention... Many spiders recently stopped going after the submitted URLs and instead go for the index.html. To make this post short and sweet I'll concentrate on AV.

You submit [somesub.somesite.com...]
Then that site gets listed.
Two weeks later AV sends a different spider and goes to

This is a little surprising. If the index.html was optimized for another engine, or if there is a meta refresh, then this could be a problem.

Has this always been an AV pattern? I will watch my rankings and see if it causes a drop. This index.html spidering trend throughs another wrench into SEO efforts, because no matter how well you make a page -- it is very difficult for it to match each search engine's algorithum.

2:22 pm on Oct 24, 2000 (gmt 0)


WebmasterWorld Administrator rogerd is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 2, 2000
votes: 1

Jlara, I'd say it is pretty common SE behavior to go back to grab the index page of any site for which it has pages indexed. Although some are more aggressive than others, their objective is to capture the entire site (page and depth limits may exist, of course), as well as find links to other sites. AltaVista is employing some level of theme analysis, and for that to work it has to sample the entire site.

Many people submit only their home page or a hallway page and wait for the spider to find the rest of the pages on his own - "found" pages sometimes rank better than submitted pages. (This approach takes a lot of patience.)

If you have pages you don't want a particular SE to see, you should exclude them in your robots.txt file.

8:36 am on Oct 25, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
votes: 62

Ya, if the index isn't in the db, Alta will grab it too. It usually takes longer than that though.