homepage Welcome to WebmasterWorld Guest from 54.211.95.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots Meta and .txt
Where?
admin amoroustoys




msg:1526291
 11:55 am on Feb 3, 2004 (gmt 0)

I just have 3 small questions.
1) is it wrong to have a robots.txt in two dif locations?
2) is this proper syntax
"Disallow /catalog/product_info.php"
3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?

 

Dreamquick




msg:1526292
 12:20 pm on Feb 3, 2004 (gmt 0)

1) is it wrong to have a robots.txt in two dif locations?

Robots.txt is expected to live off the root of the website, you can try putting it in other places but it wont get used.

2) is this proper syntax
Disallow /catalog/product_info.php

Depends what you hope to achieve - the snippet you supplied is the disallow portion of robots.txt, but you also need a user-agent filter as well - a very simple example that applies to all robots would be;

User-agent: *
Disallow: /catalog/product_info.php

3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?

You can place robots metatags on every page if you want to, although since most commercial spiders default to "index follow" you don't need it in order to get crawled. Like all meta tag data they apply only to the current page so you can't just place it only on the front page and expect it to propagate through the site.

- Tony

tomda




msg:1526293
 12:54 pm on Feb 3, 2004 (gmt 0)

Sorry, I was not aware that they were a robots.txt forum.

I just copy the question I post this morning as it is related to this topic :

1/ I am working on a beta version of my website and will upload it on a subdomain (beta.DNS.com) just to try it.
I do not want robots to crawl the site. If I put a robots.txt file in the root of the directory (with User-agent: * and Disallow: /), will robots ignore the site despite I have on each file a meta tag CONTENT="all".

Tommy

pageoneresults




msg:1526294
 1:34 pm on Feb 3, 2004 (gmt 0)

I do not want robots to crawl the site.

Then you might want to put the site behind password protection. Both methods of robots directives are not foolproof. The robots.txt file at the root level is more reliable than the META robots tag.

Keep in mind that not all robots follow the protocol. Many will skip the robots.txt file and go right through your site.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved