Welcome to WebmasterWorld Guest from 54.145.166.96

Forum Moderators: open

Message Too Old, No Replies

Dynamicly served pages

does the alltheweb spider capture them?

   
11:59 am on Oct 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I have a clietn with a page that is only served from a db, and alltheweb indexed it!

How?'

6:03 pm on Oct 30, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



>How?

By following the links ;)

Most se's have played with indexing dynamic url's and most of them have lots of pages in their indexes.

6:18 pm on Oct 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



but the page that is generated does not exist as a page, but in multiple databases it does.

as such not link exists to the page listed.

so how is this done?

12:26 am on Oct 31, 2003 (gmt 0)

WebmasterWorld Administrator 10+ Year Member



>but the page that is generated does not exist as a page

Well, it did when the crawler visited it. It got "something".

12:31 am on Oct 31, 2003 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member



the particular string for the page exists as a singular entity, therefore, the page exists as much as any other page exists. The spider grabs the browser output that is uniqely identified by that url.

I always found that alltheweb was actually one of the better spiders with dynamic urls.

9:30 pm on Oct 31, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



ok

the particular string for the page exists as a singular entity, therefore, the page exists as much as any other page exists. The spider grabs the browser output that is uniqely identified by that url.

I always found that alltheweb was actually one of the better spiders with dynamic urls.

ok

the variables to create the page string is big. we are talking about over 500k cells in various dbs.

so what are the tricks to get these variable strings listed in alltheweb?

10:01 pm on Oct 31, 2003 (gmt 0)

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member



how many vars in the string?

less is always more when talking about query strings in general. Have you ever looked into tools like mod_rewrite? Take a look at this thread.

An Introduction to Redirecting URLs on an Apache Server [webmasterworld.com]

That thread is specific to apache but iis has mod_rewrite as well though I am not as familiar with it. They key is to either keep the query strings to minimum and when unavoidable or there are indexing problems a lot of people then use mod_rewrite.

Looking at good interanl linking structures always helps too.

None of this is really specific to ATW but is very good practice for all spiders.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month