Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt

Can we give it another name

     
2:12 pm on Jul 23, 2013 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts: 303
votes: 0


Hello
One of my dear fellow has given robots.txt file another name like robetistic.txt. Could you please explain me whether it is good or a bad practice to name a robots.txt file with another name? Or, if it is valid (giving another name), what about Google then? will google crawl the new name of robots.txt.
thanks,
Bilal
2:26 pm on July 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 18, 2002
posts:3061
votes: 0


The filename must be called robots.txt to be crawled by Google and other search engines. Copy the content of the other file to robots.txt and delete the other one.
2:46 pm on July 23, 2013 (gmt 0)

Administrator

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 10, 2004
posts:10542
votes: 8


according to The Robots Exclusion Protocol:
http://www.robotstxt.org/robotstxt.html [robotstxt.org]
the url is:
http://www.example.com/robots.txt


you can also see "Examples of valid robots.txt URLs" in the Google Developers Robots.txt Specifications:
http://developers.google.com/webmasters/control-crawl-index/docs/robots_txt [developers.google.com]
10:20 pm on July 23, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12704
votes: 244


You can silently rewrite from robots.txt to anyname.anyextension. Some people hereabouts do this as part of sneakily serving up a UA-specific robots.txt. But the URL has to be hostname/robots.txt.
11:49 pm on July 23, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


sneakily serving up a UA-specific robots.txt

It is not sneaky at all.

It is a simple server configuration.

No bots are misled or otherwise harmed in the process.

...
1:24 am on July 24, 2013 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:Apr 9, 2011
posts:12704
votes: 244


We have differing definitions of the word "sneaky". In my lexicon, anything involving a rewrite is sneaky. Not necessarily bad, but undeniably hidden and behind-the-scenes.
2:14 am on July 24, 2013 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


undeniably hidden and behind-the-scenes

What, like the Apache configuration, the PHP code, and so much else?

Serving different responses to different user-agents is perfectly normal and reasonable.

But nobody likes a sneak.

...
5:25 pm on Dec 17, 2013 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts: 303
votes: 0


thanks guys for helping me out.. thanks for you help.. God bless you all.. Sorry for late reply, actually got ill for long. Thanks again dudes
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members