Welcome to WebmasterWorld Guest from 54.161.21.157

Forum Moderators: goodroi

Message Too Old, No Replies

How do I create a robot.txt?

     
4:45 pm on Nov 28, 2002 (gmt 0)

Full Member

10+ Year Member

joined:Sept 19, 2002
posts:257
votes: 0


OK Newbie here!

I've read over some of the posts about this topic. I would like to set up a robots.txt to stop and indexing of archived pages and also indexing of my images.

I have contacted my web host and know where to put the file once I create it. I also understand there is a validator and that the file needs to be in "unix."

How is all this accomplished through Frontpage?

Thanks!

Mona

4:57 pm on Nov 28, 2002 (gmt 0)

Senior Member from ZA 

WebmasterWorld Senior Member 10+ Year Member

joined:July 15, 2002
posts:1721
votes: 4


It would be better to have a read through:

[searchengineworld.com...]

There are a few brilliant tutorials there that will help.

Craig

1:00 am on Nov 29, 2002 (gmt 0)

Administrator from JP 

WebmasterWorld Administrator bill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Oct 12, 2000
posts:14978
votes: 131


More specifically, look at this page. [searchengineworld.com] ;)

You can make your robots.txt file in Notepad. When you're done name the file 'robots.txt' and save it in the root directory of your web.

2:25 am on Nov 29, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5460
votes: 3


<snip>How is all this accomplished through Frontpage?>

This may require some additional precautions in FP depending upon if you "publish?"
You may want to consult the FP documentation or the FP Newsgroup.

Woz

2:43 am on Nov 29, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member woz is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 13, 2000
posts:4823
votes: 0


Robots.txt files are independant of whatever software you use to create the site.

Just use the advice and links given above and then upload to your root directory.

Onya
Woz

10:49 am on Dec 11, 2002 (gmt 0)

New User

10+ Year Member

joined:Dec 3, 2002
posts:16
votes: 0


Another question I have...
I have successfully created a robots.txt file, great - and it works (have tested it with the bot sim), etc ...
but i want to know how I can check the log files on my domain - is there some sofware I can download? - at the moment I am unsure whether my site is being spidered?,
thx
2:56 am on Dec 12, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


milkman,

This depends on your hosting service. If your site is small, you can usually just use your browser to view the raw log files in a secure subdirectory of your site, or download them using ftp. No special software is required. The internet explorer browser supports ftp access as well as normal browsing.

I'd suggest looking at your host's help pages, or looking in your site's "control panel" if it is set up with one.

Jim

9:32 am on Dec 15, 2002 (gmt 0)

New User

10+ Year Member

joined:Dec 3, 2002
posts:16
votes: 0


Have found that unfortunately, my hosting service does not log the traffic (due to it being free hosting) - I will have to pay to get this service!
1:35 pm on Dec 15, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 1, 2002
posts:1018
votes: 2


Time to look for a new host, huh?

I spent many months without knowing what my logs were...when I moved hosts and finally got a look..WOW. It was well worth the money.