Page is a not externally linkable
bakedjake - 10:44 pm on Jul 28, 2003 (gmt 0)
For example, 1338 page requests in the last 24 hours, and 5770 page requests in the last week on one of our e-commerce sites. One thing I've noticed is that it seems to be going deeper into the site than ever before... requesting almost every page on the site, including just about every form of valid dynamic parameter that is linked from somewhere else. About 20% of the requests over the last week are duplicate. No signs of additional pages to the index. I'll continue watching this very closely.
Yes, slurp is going nuts over here.