Forum Moderators: open

Message Too Old, No Replies

robots.txt file

Do robots pick up on html in the robots.text file?

         

IanTurner

12:53 pm on Jul 24, 2001 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



An interesting thought: robots read the robots .txt file for directives. If I placed a HTML hyperlink in the comments of the robots.txt file, would this be picked up as a link by the search engines?

All the best
Ian

agerhart

1:00 pm on Jul 24, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Welcome to WebmasterWorld Ian,

The robots.txt does not follow the HTML standards that most web pages follow, but instead has to follow its own standards.

Here is a thread that we were discussing about the benefits and pitfalls of a robots.txt : http://www.webmasterworld.com/forum27/239.htm [webmasterworld.com]

Here is also the link to some helpful tools that will make sure your robots.txt is functioning properly: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi [searchengineworld.com]

Lastly, here is the link to standard that the robots.txt has to follow: http://www.robotstxt.org/wc/norobots.html [robotstxt.org]

I know I may have indirectly answered your question, but here is your answer:

No, this method of placing a link in the robots.txt will not work......sounds nice though.