homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

How do I create a robot.txt?

 4:45 pm on Nov 28, 2002 (gmt 0)

OK Newbie here!

I've read over some of the posts about this topic. I would like to set up a robots.txt to stop and indexing of archived pages and also indexing of my images.

I have contacted my web host and know where to put the file once I create it. I also understand there is a validator and that the file needs to be in "unix."

How is all this accomplished through Frontpage?




creative craig

 4:57 pm on Nov 28, 2002 (gmt 0)

It would be better to have a read through:


There are a few brilliant tutorials there that will help.



 1:00 am on Nov 29, 2002 (gmt 0)

More specifically, look at this page. [searchengineworld.com] ;)

You can make your robots.txt file in Notepad. When you're done name the file 'robots.txt' and save it in the root directory of your web.


 2:25 am on Nov 29, 2002 (gmt 0)

<snip>How is all this accomplished through Frontpage?>

This may require some additional precautions in FP depending upon if you "publish?"
You may want to consult the FP documentation or the FP Newsgroup.


 2:43 am on Nov 29, 2002 (gmt 0)

Robots.txt files are independant of whatever software you use to create the site.

Just use the advice and links given above and then upload to your root directory.



 10:49 am on Dec 11, 2002 (gmt 0)

Another question I have...
I have successfully created a robots.txt file, great - and it works (have tested it with the bot sim), etc ...
but i want to know how I can check the log files on my domain - is there some sofware I can download? - at the moment I am unsure whether my site is being spidered?,


 2:56 am on Dec 12, 2002 (gmt 0)


This depends on your hosting service. If your site is small, you can usually just use your browser to view the raw log files in a secure subdirectory of your site, or download them using ftp. No special software is required. The internet explorer browser supports ftp access as well as normal browsing.

I'd suggest looking at your host's help pages, or looking in your site's "control panel" if it is set up with one.



 9:32 am on Dec 15, 2002 (gmt 0)

Have found that unfortunately, my hosting service does not log the traffic (due to it being free hosting) - I will have to pay to get this service!


 1:35 pm on Dec 15, 2002 (gmt 0)

Time to look for a new host, huh?

I spent many months without knowing what my logs were...when I moved hosts and finally got a look..WOW. It was well worth the money.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved