not2easy - 3:19 pm on Aug 24, 2012 (gmt 0)

<meta name="robots" content="all" /> <meta name="robots" content="index, follow" /> <meta name="googlebot" content="index, follow" /> <meta name="msnbot" content="index, follow" /> Now I am setting following rule in .htaccess file <files admin.php> Header set X-Robots-Tag "noindex, nofollow" </files> And my admin.php is NOT restricted in robots.txt Question: How bot will behave in above mentioned case?

The bot will crawl but not index or follow the URL: xyz.com/admin.php

Why do you have so many separate robots meta tags? This one does the same as all of them together: <meta name="robots" content="all" />

You should think about password protecting the files you want to keep private and make sure they are not in your sitemaps.