homepage Welcome to WebmasterWorld Guest from 54.205.119.163
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
Forum Library, Charter, Moderator: open

Deprecated - Altavista, Alltheweb.com Forum

    
Dynamicly served pages
does the alltheweb spider capture them?
OddDog




msg:222507
 11:59 am on Oct 30, 2003 (gmt 0)

I have a clietn with a page that is only served from a db, and alltheweb indexed it!

How?'

 

Rumbas




msg:222508
 6:03 pm on Oct 30, 2003 (gmt 0)

>How?

By following the links ;)

Most se's have played with indexing dynamic url's and most of them have lots of pages in their indexes.

OddDog




msg:222509
 6:18 pm on Oct 30, 2003 (gmt 0)

but the page that is generated does not exist as a page, but in multiple databases it does.

as such not link exists to the page listed.

so how is this done?

Rumbas




msg:222510
 12:26 am on Oct 31, 2003 (gmt 0)

>but the page that is generated does not exist as a page

Well, it did when the crawler visited it. It got "something".

jatar_k




msg:222511
 12:31 am on Oct 31, 2003 (gmt 0)

the particular string for the page exists as a singular entity, therefore, the page exists as much as any other page exists. The spider grabs the browser output that is uniqely identified by that url.

I always found that alltheweb was actually one of the better spiders with dynamic urls.

OddDog




msg:222512
 9:30 pm on Oct 31, 2003 (gmt 0)

ok

the particular string for the page exists as a singular entity, therefore, the page exists as much as any other page exists. The spider grabs the browser output that is uniqely identified by that url.

I always found that alltheweb was actually one of the better spiders with dynamic urls.

ok

the variables to create the page string is big. we are talking about over 500k cells in various dbs.

so what are the tricks to get these variable strings listed in alltheweb?

jatar_k




msg:222513
 10:01 pm on Oct 31, 2003 (gmt 0)

how many vars in the string?

less is always more when talking about query strings in general. Have you ever looked into tools like mod_rewrite? Take a look at this thread.

An Introduction to Redirecting URLs on an Apache Server [webmasterworld.com]

That thread is specific to apache but iis has mod_rewrite as well though I am not as familiar with it. They key is to either keep the query strings to minimum and when unavoidable or there are indexing problems a lot of people then use mod_rewrite.

Looking at good interanl linking structures always helps too.

None of this is really specific to ATW but is very good practice for all spiders.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Yahoo / Deprecated - Altavista, Alltheweb.com
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved