Welcome to WebmasterWorld Guest from 54.163.40.152

Forum Moderators: goodroi

Message Too Old, No Replies

How do I create a robot.txt?

     

palmpal

4:45 pm on Nov 28, 2002 (gmt 0)

10+ Year Member



OK Newbie here!

I've read over some of the posts about this topic. I would like to set up a robots.txt to stop and indexing of archived pages and also indexing of my images.

I have contacted my web host and know where to put the file once I create it. I also understand there is a validator and that the file needs to be in "unix."

How is all this accomplished through Frontpage?

Thanks!

Mona

creative craig

4:57 pm on Nov 28, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It would be better to have a read through:

[searchengineworld.com...]

There are a few brilliant tutorials there that will help.

Craig

bill

1:00 am on Nov 29, 2002 (gmt 0)

WebmasterWorld Administrator bill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



More specifically, look at this page. [searchengineworld.com] ;)

You can make your robots.txt file in Notepad. When you're done name the file 'robots.txt' and save it in the root directory of your web.

wilderness

2:25 am on Nov 29, 2002 (gmt 0)

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



<snip>How is all this accomplished through Frontpage?>

This may require some additional precautions in FP depending upon if you "publish?"
You may want to consult the FP documentation or the FP Newsgroup.

Woz

2:43 am on Nov 29, 2002 (gmt 0)

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Robots.txt files are independant of whatever software you use to create the site.

Just use the advice and links given above and then upload to your root directory.

Onya
Woz

milkman

10:49 am on Dec 11, 2002 (gmt 0)

10+ Year Member



Another question I have...
I have successfully created a robots.txt file, great - and it works (have tested it with the bot sim), etc ...
but i want to know how I can check the log files on my domain - is there some sofware I can download? - at the moment I am unsure whether my site is being spidered?,
thx

jdMorgan

2:56 am on Dec 12, 2002 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member



milkman,

This depends on your hosting service. If your site is small, you can usually just use your browser to view the raw log files in a secure subdirectory of your site, or download them using ftp. No special software is required. The internet explorer browser supports ftp access as well as normal browsing.

I'd suggest looking at your host's help pages, or looking in your site's "control panel" if it is set up with one.

Jim

milkman

9:32 am on Dec 15, 2002 (gmt 0)

10+ Year Member



Have found that unfortunately, my hosting service does not log the traffic (due to it being free hosting) - I will have to pay to get this service!

ScottM

1:35 pm on Dec 15, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Time to look for a new host, huh?

I spent many months without knowing what my logs were...when I moved hosts and finally got a look..WOW. It was well worth the money.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month