Welcome to WebmasterWorld Guest from 23.22.140.143

Forum Moderators: goodroi

Message Too Old, No Replies

Robots Meta and .txt

Where?

     
11:55 am on Feb 3, 2004 (gmt 0)

New User

joined:Feb 3, 2004
posts:2
votes: 0


I just have 3 small questions.
1) is it wrong to have a robots.txt in two dif locations?
2) is this proper syntax
"Disallow /catalog/product_info.php"
3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?
12:20 pm on Feb 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 25, 2002
posts:872
votes: 0


1) is it wrong to have a robots.txt in two dif locations?

Robots.txt is expected to live off the root of the website, you can try putting it in other places but it wont get used.

2) is this proper syntax
Disallow /catalog/product_info.php

Depends what you hope to achieve - the snippet you supplied is the disallow portion of robots.txt, but you also need a user-agent filter as well - a very simple example that applies to all robots would be;

User-agent: *
Disallow: /catalog/product_info.php

3) I know most Spiders have a default of index follow, but Is it ok if I place that meta in every page or should I only place it once in the home page?

You can place robots metatags on every page if you want to, although since most commercial spiders default to "index follow" you don't need it in order to get crawled. Like all meta tag data they apply only to the current page so you can't just place it only on the front page and expect it to propagate through the site.

- Tony

12:54 pm on Feb 3, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Jan 7, 2004
posts:929
votes: 0


Sorry, I was not aware that they were a robots.txt forum.

I just copy the question I post this morning as it is related to this topic :

1/ I am working on a beta version of my website and will upload it on a subdomain (beta.DNS.com) just to try it.
I do not want robots to crawl the site. If I put a robots.txt file in the root of the directory (with User-agent: * and Disallow: /), will robots ignore the site despite I have on each file a meta tag CONTENT="all".

Tommy

1:34 pm on Feb 3, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member pageoneresults is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Apr 27, 2001
posts:12166
votes: 51


I do not want robots to crawl the site.

Then you might want to put the site behind password protection. Both methods of robots directives are not foolproof. The robots.txt file at the root level is more reliable than the META robots tag.

Keep in mind that not all robots follow the protocol. Many will skip the robots.txt file and go right through your site.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members