| 2:26 pm on Jul 23, 2013 (gmt 0)|
The filename must be called robots.txt to be crawled by Google and other search engines. Copy the content of the other file to robots.txt and delete the other one.
| 2:46 pm on Jul 23, 2013 (gmt 0)|
according to The Robots Exclusion Protocol:
the url is:
you can also see "Examples of valid robots.txt URLs" in the Google Developers Robots.txt Specifications:
| 10:20 pm on Jul 23, 2013 (gmt 0)|
You can silently rewrite from robots.txt to anyname.anyextension. Some people hereabouts do this as part of sneakily serving up a UA-specific robots.txt. But the URL has to be hostname/robots.txt.
| 11:49 pm on Jul 23, 2013 (gmt 0)|
|sneakily serving up a UA-specific robots.txt |
It is not sneaky at all.
It is a simple server configuration.
No bots are misled or otherwise harmed in the process.
| 1:24 am on Jul 24, 2013 (gmt 0)|
We have differing definitions of the word "sneaky". In my lexicon, anything involving a rewrite is sneaky. Not necessarily bad, but undeniably hidden and behind-the-scenes.
| 2:14 am on Jul 24, 2013 (gmt 0)|
|undeniably hidden and behind-the-scenes |
What, like the Apache configuration, the PHP code, and so much else?
Serving different responses to different user-agents is perfectly normal and reasonable.
But nobody likes a sneak.
| 5:25 pm on Dec 17, 2013 (gmt 0)|
thanks guys for helping me out.. thanks for you help.. God bless you all.. Sorry for late reply, actually got ill for long. Thanks again dudes