homepage Welcome to WebmasterWorld Guest from 54.146.175.204
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
site setup for spider?
Gemelli_no1

10+ Year Member



 
Msg#: 396 posted 11:30 am on Jun 3, 2004 (gmt 0)

hello all,
i have just tried to spider my site using the sim-spider....and it does the home page fine...however all the other 27,500 pages are located evenly in 12 folders of assorted names...and it seems that it doesnt spider within those folders?
i am a total newbie to this and would love to post the url so someone can check on my behalf with the sim-spider......but i know the rules..LOLOL :-)
do i have to scrap the folders..and just have a long long long list of pages?

thanks for some advice
Gemelli

 

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 396 posted 2:45 pm on Jun 3, 2004 (gmt 0)

If your running Windoze, try using the Xenu link checker. It IS a spider and will crawl your entire site, identify broken links, etc.

Brett's sim spider only crawls the page you request. You could submit all 27,500 URLs to the sim spider of course... ;)

Welcome to WebmasterWorld! :)

Gemelli_no1

10+ Year Member



 
Msg#: 396 posted 8:20 am on Jun 4, 2004 (gmt 0)

if i am using the sim....or entering my site to search engines ,,,should i give the url as:

www.*mysite*.com

or

www.*mysite*.com/

does the / help?


Gemelli :-)

DaveAtIFG

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 396 posted 3:48 am on Jun 6, 2004 (gmt 0)

A properly formed hypertext URL is [example.com...] but most well written code will add the "http://" prefix or the trailing slash if you don't include them. It can't hurt to include them in case the programmer overlooked something.

webcertain

10+ Year Member



 
Msg#: 396 posted 12:51 pm on Jun 23, 2004 (gmt 0)

Hi all,
Xenu is normally very good, but for one site it just doesn't feel right: 99% of the address are canceled.. Do you know any other tool that could help me through?
Cheers,
Johann

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved