Spider Software

Software that acts like googlebot

[duelism]

Is there any spidering software out there that acts like the googlebot that can be run offline or online to test, for example, how far into the site the spider reaches?

[Dreamquick]

Not really for that you'd require the actual google spider and its algorythms - any "simulator" wouldn't be able to replicate how googlebot would react in terms of crawl depth etc.

From a purely technical perspective there are a few rules you can apply to work out what a spider couldn't/wouldn't crawl, most of these take the form of "if you can see the URL in the page source then it will probably be crawled unless you use excessive querystring data, or drop similar hints that you're trying to feed it overly dynamic pages".

- Tony

[duelism]

Is there any kind of spider simulator available?

[edit_g]

There's lots:
[google.com...]