homepage Welcome to WebmasterWorld Guest from 54.197.110.151
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt
Can we give it another name
bilalseo

5+ Year Member



 
Msg#: 4595957 posted 2:12 pm on Jul 23, 2013 (gmt 0)

Hello
One of my dear fellow has given robots.txt file another name like robetistic.txt. Could you please explain me whether it is good or a bad practice to name a robots.txt file with another name? Or, if it is valid (giving another name), what about Google then? will google crawl the new name of robots.txt.
thanks,
Bilal

 

sem4u

WebmasterWorld Senior Member sem4u us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4595957 posted 2:26 pm on Jul 23, 2013 (gmt 0)

The filename must be called robots.txt to be crawled by Google and other search engines. Copy the content of the other file to robots.txt and delete the other one.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4595957 posted 2:46 pm on Jul 23, 2013 (gmt 0)

according to The Robots Exclusion Protocol:
http://www.robotstxt.org/robotstxt.html [robotstxt.org]
the url is:
http://www.example.com/robots.txt

you can also see "Examples of valid robots.txt URLs" in the Google Developers Robots.txt Specifications:
http://developers.google.com/webmasters/control-crawl-index/docs/robots_txt [developers.google.com]

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4595957 posted 10:20 pm on Jul 23, 2013 (gmt 0)

You can silently rewrite from robots.txt to anyname.anyextension. Some people hereabouts do this as part of sneakily serving up a UA-specific robots.txt. But the URL has to be hostname/robots.txt.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4595957 posted 11:49 pm on Jul 23, 2013 (gmt 0)

sneakily serving up a UA-specific robots.txt

It is not sneaky at all.

It is a simple server configuration.

No bots are misled or otherwise harmed in the process.

...

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4595957 posted 1:24 am on Jul 24, 2013 (gmt 0)

We have differing definitions of the word "sneaky". In my lexicon, anything involving a rewrite is sneaky. Not necessarily bad, but undeniably hidden and behind-the-scenes.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 4595957 posted 2:14 am on Jul 24, 2013 (gmt 0)

undeniably hidden and behind-the-scenes

What, like the Apache configuration, the PHP code, and so much else?

Serving different responses to different user-agents is perfectly normal and reasonable.

But nobody likes a sneak.

...

bilalseo

5+ Year Member



 
Msg#: 4595957 posted 5:25 pm on Dec 17, 2013 (gmt 0)

thanks guys for helping me out.. thanks for you help.. God bless you all.. Sorry for late reply, actually got ill for long. Thanks again dudes

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved