Forum Moderators: open

Message Too Old, No Replies

Spiders

Which page is search first.

         

fashezee

1:01 am on Feb 19, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Do spiders by default look for index.* for every site as the first page to index?

Or for instance, if I have my server setup so that it redirects all users to:
www.widget.com/keyword/keyword.html; which page will by index first? Assuming
I have files at the root level?

wilderness

1:18 am on Feb 19, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



fashe
I'm not trying to be a smart alec here so please don't understand it that way?

Spiders work like a spider web crawling the different directions.

A robot comes into a site on a referral whether that is from another site or a submission.

In that regard the spider can NOT have an extension preference.
The same applies as it travels your site of the links it picks up from the page it comes in on.
If my link specified "mypage.html" why on earth would a spider look for any other extension. htm, asp, cgi whatever.