Interestingly Wisenut is the only search engine I have seen that spidered and indexed the next level of pages beyond the home page on our dynamically generated site. How would that happen by meta crawling?
kmarcus
3:07 am on Oct 1, 2002 (gmt 0)
errr, wisenut isn't a meta search engine. many search engines out there will whack a url at the "?" in it to avoid getting stuck (or sucked into, for that matter) some sort of crazy huge batch of dynamically generated pages, which could implicitly become spam-like. I always thought this was weird because when people talk about the "invisible web", i always thought it was more of the "content hidden in databases that you only see when a form is filled out", although the likes of fast and google seem to have people believing it's really the stuff in .DOC and .PDF files. Ho hum?
