Spider Testing

[cabowabo]

Does anyone have a test spider, script, or such that is easy to use that can mimic a search engine to test a site for spiderability?

CaboWabo

[DaveAtIFG]

Try Brett's Sim Spider at [searchengineworld.com...]

Is that easy enough? ;)

[brotherhood of LAN]

WGET [gnu.org] is fairly simple too, a command line program....it will follow links on a page and is robots.txt friendly.

[DaveAtIFG]

Look at Xenu too. It's a very handy (and free) site link checker.

[everlast]

I clicked on [searchengineworld.com...]

when i try to get to the sim spider, it goes to the login page. I log in and try again and it just repeats. Do you have to be a paid memeber for this to work?

[DaveAtIFG]

No, but Brett has access to it restricted and others have reported similar difficulties. Try [searchengineworld.com...] and click on SE Tools in the right side menu.