freshbot - multivariable dynamic pages

freshbot is spidering dynamic pages differently

[Rick_M]

I posted a brief message about this in the deepbot thread, but didn't get any replies, and I think this is actually a very big deal.

I have never seen this before, but this morning I have records of freshbot spidering dynamic pages with up to 5 variables. If google is going to start indexing these sites, the index is going to grow tremendously and it likely will change the way google sees the structure of the web.

An example URL that got spidered today from my forums:

[example.com...]

When I look through last months logs, no pages with more than 2 variables got indexed. If they increase the number to 5 variables, that's a whole lot of pages getting added.

[edited by: heini at 8:27 pm (utc) on April 27, 2003]
[edit reason] fixed link [/edit]

[BGumble]

Good question, Rick_M. Maybe this coincides with the reason Freshbot is a little slower in the day-to-day updates lately.

OT-point, you might want to exclude robots from reading certain files on your BB like reply.php and newthread.php to save some bandwidth.