homepage Welcome to WebmasterWorld Guest from 54.163.139.36
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How do I create a robot.txt?
palmpal

10+ Year Member



 
Msg#: 169 posted 4:45 pm on Nov 28, 2002 (gmt 0)

OK Newbie here!

I've read over some of the posts about this topic. I would like to set up a robots.txt to stop and indexing of archived pages and also indexing of my images.

I have contacted my web host and know where to put the file once I create it. I also understand there is a validator and that the file needs to be in "unix."

How is all this accomplished through Frontpage?

Thanks!

Mona

 

creative craig

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 169 posted 4:57 pm on Nov 28, 2002 (gmt 0)

It would be better to have a read through:

[searchengineworld.com...]

There are a few brilliant tutorials there that will help.

Craig

bill

WebmasterWorld Administrator bill us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 169 posted 1:00 am on Nov 29, 2002 (gmt 0)

More specifically, look at this page. [searchengineworld.com] ;)

You can make your robots.txt file in Notepad. When you're done name the file 'robots.txt' and save it in the root directory of your web.

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 169 posted 2:25 am on Nov 29, 2002 (gmt 0)

<snip>How is all this accomplished through Frontpage?>

This may require some additional precautions in FP depending upon if you "publish?"
You may want to consult the FP documentation or the FP Newsgroup.

Woz

WebmasterWorld Senior Member woz us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 169 posted 2:43 am on Nov 29, 2002 (gmt 0)

Robots.txt files are independant of whatever software you use to create the site.

Just use the advice and links given above and then upload to your root directory.

Onya
Woz

milkman

10+ Year Member



 
Msg#: 169 posted 10:49 am on Dec 11, 2002 (gmt 0)

Another question I have...
I have successfully created a robots.txt file, great - and it works (have tested it with the bot sim), etc ...
but i want to know how I can check the log files on my domain - is there some sofware I can download? - at the moment I am unsure whether my site is being spidered?,
thx

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 169 posted 2:56 am on Dec 12, 2002 (gmt 0)

milkman,

This depends on your hosting service. If your site is small, you can usually just use your browser to view the raw log files in a secure subdirectory of your site, or download them using ftp. No special software is required. The internet explorer browser supports ftp access as well as normal browsing.

I'd suggest looking at your host's help pages, or looking in your site's "control panel" if it is set up with one.

Jim

milkman

10+ Year Member



 
Msg#: 169 posted 9:32 am on Dec 15, 2002 (gmt 0)

Have found that unfortunately, my hosting service does not log the traffic (due to it being free hosting) - I will have to pay to get this service!

ScottM

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 169 posted 1:35 pm on Dec 15, 2002 (gmt 0)

Time to look for a new host, huh?

I spent many months without knowing what my logs were...when I moved hosts and finally got a look..WOW. It was well worth the money.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved