homepage Welcome to WebmasterWorld Guest from 54.196.62.132
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Spider Testing
cabowabo




msg:1527634
 2:39 pm on May 3, 2004 (gmt 0)

Does anyone have a test spider, script, or such that is easy to use that can mimic a search engine to test a site for spiderability?

CaboWabo

 

DaveAtIFG




msg:1527635
 3:34 pm on May 3, 2004 (gmt 0)

Try Brett's Sim Spider at [searchengineworld.com...]

Is that easy enough? ;)

brotherhood of LAN




msg:1527636
 3:37 pm on May 3, 2004 (gmt 0)

WGET [gnu.org] is fairly simple too, a command line program....it will follow links on a page and is robots.txt friendly.

DaveAtIFG




msg:1527637
 4:14 pm on May 3, 2004 (gmt 0)

Look at Xenu too. It's a very handy (and free) site link checker.

everlast




msg:1527638
 2:37 am on May 13, 2004 (gmt 0)

I clicked on [searchengineworld.com...]

when i try to get to the sim spider, it goes to the login page. I log in and try again and it just repeats. Do you have to be a paid memeber for this to work?

DaveAtIFG




msg:1527639
 3:44 am on May 13, 2004 (gmt 0)

No, but Brett has access to it restricted and others have reported similar difficulties. Try [searchengineworld.com...] and click on SE Tools in the right side menu.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved