Static generated pages

[littledip]

Hi guys:

Great forum here.

Are any of you aware of engines discounting pages whose content dramatically changes from one visit to the next?

What I mean is I've noticed on some of my cloaked pages that a spider will visit, and then, a few minutes later, another spider will visit (from a different IP address). I can only assume this is to compare content.

We've been selling cloaking software for about a year, and haven't had any customers complain about this, but I'm wondering if SEs are getting more tricky with detecting pages.

Thoughts?

Thanks,
Peter

(edited by: rcjordan at 1:03 am (utc) on April 11, 2002)

[volatilegx]

Hi littledip and welcome to WebmasterWorld.

You have a valid concern. It does happen. If you are gathering a list of the IP addresses, User Agents, etc. of the second visits, I for one would very much appreciate it if you would post them in the Search Engine Spider Identification forum [webmasterworld.com].

[Air]

If a system is being used where pages are dynamically generated to feed spiders, most build in a consistency timer to go along with such a system that can be user set. That way, a page will be saved and the same saved page is served to all spiders until the timer pops, then a new page becomes the served page until the timer pops.

If the content is changing because it really is that dynamic, and therefore spiders may encounter different pages when successive spiders request the same page, you may want to feed spiders static pages, or employ a method similar to the one described above. Where a dynamic page is saved and any other spider request for the same page receives the saved copy until the timer expires.

Is it really necessary? IMO no, will it help rankings? IMO yes.

[littledip]

yah...
thanks...
i am actually in process of developing an item which tracks the dynamically generated content for the first 5 spider accesses to that page, and then resets. that should work well.

incidentally, is there a good spot for a central spider ip list? i want to compare mine to others. i have about 600 ip addrs.

[volatilegx]

You can see mine over at http*://www.iplists.com/

(edited by: Air at 11:21 pm (utc) on April 11, 2002)