Welcome to WebmasterWorld Guest from 54.226.22.192

Forum Moderators: open

Message Too Old, No Replies

Dynamicly served pages

does the alltheweb spider capture them?

     
11:59 am on Oct 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 6, 2002
posts:702
votes: 0


I have a clietn with a page that is only served from a db, and alltheweb indexed it!

How?'

6:03 pm on Oct 30, 2003 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2533
votes: 2


>How?

By following the links ;)

Most se's have played with indexing dynamic url's and most of them have lots of pages in their indexes.

6:18 pm on Oct 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 6, 2002
posts:702
votes: 0


but the page that is generated does not exist as a page, but in multiple databases it does.

as such not link exists to the page listed.

so how is this done?

12:26 am on Oct 31, 2003 (gmt 0)

Moderator from DK 

WebmasterWorld Administrator 10+ Year Member

joined:Oct 23, 2000
posts:2533
votes: 2


>but the page that is generated does not exist as a page

Well, it did when the crawler visited it. It got "something".

12:31 am on Oct 31, 2003 (gmt 0)

Administrator

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 24, 2001
posts:15755
votes: 0


the particular string for the page exists as a singular entity, therefore, the page exists as much as any other page exists. The spider grabs the browser output that is uniqely identified by that url.

I always found that alltheweb was actually one of the better spiders with dynamic urls.

9:30 pm on Oct 31, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 6, 2002
posts:702
votes: 0


ok

the particular string for the page exists as a singular entity, therefore, the page exists as much as any other page exists. The spider grabs the browser output that is uniqely identified by that url.

I always found that alltheweb was actually one of the better spiders with dynamic urls.

ok

the variables to create the page string is big. we are talking about over 500k cells in various dbs.

so what are the tricks to get these variable strings listed in alltheweb?

10:01 pm on Oct 31, 2003 (gmt 0)

Administrator

WebmasterWorld Administrator jatar_k is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:July 24, 2001
posts:15755
votes: 0


how many vars in the string?

less is always more when talking about query strings in general. Have you ever looked into tools like mod_rewrite? Take a look at this thread.

An Introduction to Redirecting URLs on an Apache Server [webmasterworld.com]

That thread is specific to apache but iis has mod_rewrite as well though I am not as familiar with it. They key is to either keep the query strings to minimum and when unavoidable or there are indexing problems a lot of people then use mod_rewrite.

Looking at good interanl linking structures always helps too.

None of this is really specific to ATW but is very good practice for all spiders.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members