Welcome to WebmasterWorld Guest from 23.20.79.227

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

Can we give it another name

     
2:12 pm on Jul 23, 2013 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts: 303
votes: 0


Hello
One of my dear fellow has given robots.txt file another name like robetistic.txt. Could you please explain me whether it is good or a bad practice to name a robots.txt file with another name? Or, if it is valid (giving another name), what about Google then? will google crawl the new name of robots.txt.
thanks,
Bilal
2:26 pm on July 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 18, 2002
posts:3065
votes: 1


The filename must be called robots.txt to be crawled by Google and other search engines. Copy the content of the other file to robots.txt and delete the other one.
2:46 pm on July 23, 2013 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10551
votes: 10


according to The Robots Exclusion Protocol:
http://www.robotstxt.org/robotstxt.html [robotstxt.org]
the url is:
http://www.example.com/robots.txt


you can also see "Examples of valid robots.txt URLs" in the Google Developers Robots.txt Specifications:
http://developers.google.com/webmasters/control-crawl-index/docs/robots_txt [developers.google.com]
10:20 pm on July 23, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13201
votes: 345


You can silently rewrite from robots.txt to anyname.anyextension. Some people hereabouts do this as part of sneakily serving up a UA-specific robots.txt. But the URL has to be hostname/robots.txt.
11:49 pm on July 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


sneakily serving up a UA-specific robots.txt

It is not sneaky at all.

It is a simple server configuration.

No bots are misled or otherwise harmed in the process.

...
1:24 am on July 24, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:13201
votes: 345


We have differing definitions of the word "sneaky". In my lexicon, anything involving a rewrite is sneaky. Not necessarily bad, but undeniably hidden and behind-the-scenes.
2:14 am on July 24, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


undeniably hidden and behind-the-scenes

What, like the Apache configuration, the PHP code, and so much else?

Serving different responses to different user-agents is perfectly normal and reasonable.

But nobody likes a sneak.

...
5:25 pm on Dec 17, 2013 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts: 303
votes: 0


thanks guys for helping me out.. thanks for you help.. God bless you all.. Sorry for late reply, actually got ill for long. Thanks again dudes