Welcome to WebmasterWorld Guest from 107.21.175.43

Forum Moderators: goodroi

Message Too Old, No Replies

Robots Meta and .txt

Where?

     

admin amoroustoys

11:55 am on Feb 3, 2004 (gmt 0)



I just have 3 small questions.
1) is it wrong to have a robots.txt in two dif locations?
2) is this proper syntax
"Disallow /catalog/product_info.php"
3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?

Dreamquick

12:20 pm on Feb 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



1) is it wrong to have a robots.txt in two dif locations?

Robots.txt is expected to live off the root of the website, you can try putting it in other places but it wont get used.

2) is this proper syntax
Disallow /catalog/product_info.php

Depends what you hope to achieve - the snippet you supplied is the disallow portion of robots.txt, but you also need a user-agent filter as well - a very simple example that applies to all robots would be;

User-agent: *
Disallow: /catalog/product_info.php

3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?

You can place robots metatags on every page if you want to, although since most commercial spiders default to "index follow" you don't need it in order to get crawled. Like all meta tag data they apply only to the current page so you can't just place it only on the front page and expect it to propagate through the site.

- Tony

tomda

12:54 pm on Feb 3, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sorry, I was not aware that they were a robots.txt forum.

I just copy the question I post this morning as it is related to this topic :

1/ I am working on a beta version of my website and will upload it on a subdomain (beta.DNS.com) just to try it.
I do not want robots to crawl the site. If I put a robots.txt file in the root of the directory (with User-agent: * and Disallow: /), will robots ignore the site despite I have on each file a meta tag CONTENT="all".

Tommy

pageoneresults

1:34 pm on Feb 3, 2004 (gmt 0)

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



I do not want robots to crawl the site.

Then you might want to put the site behind password protection. Both methods of robots directives are not foolproof. The robots.txt file at the root level is more reliable than the META robots tag.

Keep in mind that not all robots follow the protocol. Many will skip the robots.txt file and go right through your site.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month