Welcome to WebmasterWorld Guest from 54.196.232.162

Forum Moderators: goodroi

Message Too Old, No Replies

Spider Testing

     
2:39 pm on May 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
posts:723
votes: 0


Does anyone have a test spider, script, or such that is easy to use that can mimic a search engine to test a site for spiderability?

CaboWabo

3:34 pm on May 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


Try Brett's Sim Spider at [searchengineworld.com...]

Is that easy enough? ;)

3:37 pm on May 3, 2004 (gmt 0)

Moderator from GB 

WebmasterWorld Administrator brotherhood_of_lan is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 30, 2002
posts:4842
votes: 1


WGET [gnu.org] is fairly simple too, a command line program....it will follow links on a page and is robots.txt friendly.
4:14 pm on May 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


Look at Xenu too. It's a very handy (and free) site link checker.
2:37 am on May 13, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 16, 2003
posts:21
votes: 0


I clicked on [searchengineworld.com...]

when i try to get to the sim spider, it goes to the login page. I log in and try again and it just repeats. Do you have to be a paid memeber for this to work?

3:44 am on May 13, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 21, 1999
posts:2141
votes: 0


No, but Brett has access to it restricted and others have reported similar difficulties. Try [searchengineworld.com...] and click on SE Tools in the right side menu.