Welcome to WebmasterWorld Guest from 54.224.57.95

Forum Moderators: goodroi

Message Too Old, No Replies

Spider Testing

     
2:39 pm on May 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does anyone have a test spider, script, or such that is easy to use that can mimic a search engine to test a site for spiderability?

CaboWabo

3:34 pm on May 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Try Brett's Sim Spider at [searchengineworld.com...]

Is that easy enough? ;)

3:37 pm on May 3, 2004 (gmt 0)

WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



WGET [gnu.org] is fairly simple too, a command line program....it will follow links on a page and is robots.txt friendly.
4:14 pm on May 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Look at Xenu too. It's a very handy (and free) site link checker.
2:37 am on May 13, 2004 (gmt 0)

10+ Year Member



I clicked on [searchengineworld.com...]

when i try to get to the sim spider, it goes to the login page. I log in and try again and it just repeats. Do you have to be a paid memeber for this to work?

3:44 am on May 13, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No, but Brett has access to it restricted and others have reported similar difficulties. Try [searchengineworld.com...] and click on SE Tools in the right side menu.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month