homepage Welcome to WebmasterWorld Guest from 54.166.113.249
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
logs and robots.txt
tyrojds

10+ Year Member



 
Msg#: 137 posted 3:01 pm on Mar 4, 2003 (gmt 0)

i have searched for an answer to this, however, perhaps because the discussions of robots.txt are always too technical for me, no luck.

in my logs "/robots.txt" comes up under "documents not found." my site is indexed so i'm wondering what this means and if i should worry about this. thanks, and any comments must be at the "info for idiots" level.

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 137 posted 3:45 pm on Mar 4, 2003 (gmt 0)

tyrojds,
Are you aware if your website contains a robots. txt in the main/root folder?

It is not absolutely necessary to have a robots. The good spiders look for it out of protocol and and courtesy.
The majority of the bad guys aren't going to read it anyway.

Webmaster World encourages copying of their own robots should you desire something basic to get you rolling.
[webmasterworld.com...]

weesnich

10+ Year Member



 
Msg#: 137 posted 3:53 pm on Mar 4, 2003 (gmt 0)

In robots.txt you can give directives, which areas of your site wellbehaving bots should not visit. You can as well suggest, that certain robots sould not read and index your pages.
Bots from respected owners i.E. googlebot will read this file and follow your restrictions, others will not.

You do not need to have such a file if you have no guidelines for the robots visiting your site. Not having this file does not exclude you from searchengines.

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 137 posted 4:00 pm on Mar 4, 2003 (gmt 0)

tyrojds,

If you do not wish to place any restrictions on robots, you can still prevent the 404-Not Found errors by uploading a blank text file named robots.txt to your web root directory.

Jim

tyrojds

10+ Year Member



 
Msg#: 137 posted 6:05 pm on Mar 4, 2003 (gmt 0)

i knew i could count on great info, thanks.

wilderness - would i just paste that stuff at the link you sent somewhere onto my index's html page?

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 137 posted 6:55 pm on Mar 4, 2003 (gmt 0)

tyrojds,

No. You must create a text file on your server in the same directory as your index page called "robots.txt"! It can be blank (just to stop the 404's you see), or it can contain directives such as the ones you see in the page using wilderness' link.

You might want to read this robots.txt tutorial [robotstxt.org] for more info.

Jim

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 137 posted 7:42 pm on Mar 4, 2003 (gmt 0)

Hey Jim,
How goes it?

tyrojds, the link I provided to the Webmaster World robots previosuly you can open and select SAVE AS with your browser and then save it in your wesbite root folder.

Then later open with any text editor and remove the Top portion which belongs to Webamster World.

BTW you might also want to take a look at changing the closing lines for folder exclusions. Your site structure likely don't match Webamster World's.
Add in any of your own folders your desire excluded from robot traffic.
Save the file and then upload into your websites root folder.

You should also read the link Jim provided to understand the procedures involved in creating your own robots for future use.

Don

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved