homepage Welcome to WebmasterWorld Guest from 54.197.108.124
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robot.txt
<meta name="Robots" content="index,follow">
Trojanfx




msg:1526859
 8:58 pm on Feb 3, 2005 (gmt 0)

On all my websites I have this meta tag and I am told it is correct.(<meta name="Robots" content="index,follow">) My logs show that last month on 1 site I had 19 failed request for /robots.txt, another had 47, I do not know what this means. I did some research and it seems I am missing the robot.txt folder on my server. I was not aware I had to upload and create this new folder on server. My question is:
A. do I create this folder and name it robot.txt on the server?
B. What do I put in the folder?

Advise appreciated. Thanks

 

Lord Majestic




msg:1526860
 9:07 pm on Feb 3, 2005 (gmt 0)

> A. do I create this folder and name it robot.txt on the server?

No - robots.txt is only valid for root of your site, and note that each subdomain must have its own, ie: www.example.com is different site than example.com from robots.txt perspective (normally they point to the same directory, which will not be the case for normal subdomains).

> B. What do I put in the folder?

robots.txt format of which can be found by searching for "robots txt" on G. :)

Trojanfx




msg:1526861
 11:27 pm on Feb 3, 2005 (gmt 0)

I did searching on google as before and found this page
[robotstxt.org...]

Bottom line says, what I have in tag is fine and nothing else is needed. If I want them to spider entire site, then what I have is fine. But, question B. asked what do I put in folder and you direct me to google for that text but they say no text needed. Not understanding.

Same goes for failed requests. Total failed requests for all sites is 119 for /robot.txt. What does that mean? I've found no reading on that.

Time is very precious. I do not want to waste yours. Am I overly concerned about these failed requests for /robot.txt?

Frederick

PatrickDeese




msg:1526862
 11:37 pm on Feb 3, 2005 (gmt 0)

<meta name="Robots" content="index,follow">

There is no reason to put this in your meta tags - its just clutter.

If you want to eliminate the failed robots.txt requests, create a txt file with a program like notepad and put in this:

User-Agent: *
Disallow:

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved