Forum Moderators: goodroi

Message Too Old, No Replies

hacking robots.txt for RSS features?

         

phildog

3:53 pm on Oct 4, 2004 (gmt 0)




I run a site called dodgeit.com that offers "private" RSS feeds. I'd like to allow web-based RSS aggregators to use these feeds, but NOT allow them to list the feeds in any search engines that index RSS feeds. (Examples of web-based RSS aggregators: my.yahoo.com, bloglines.com)

basecamphq.com also provides "private" RSS feeds that anyone can theoretically access, but are meant for private consumption. There are probably other sites with this problem, and I expect it to become more common as RSS gets more pervasive.

Now my first gut reaction to this problem is to hack robots.txt to support a new designation. For example:

User-agent: *
Noindex: /rss

"Noindex" meaning: it is ok to programmatically retrieve data from the /rss directory, but NOT ok for a search engine to display any of the RSS feeds to people who aren't already subscribed.

Yes, this is a hack. Can you think of a better way? using robots.txt or something else?

I'd love to get buy-in from the web-based aggregators out there to make this an unofficial standard of sorts.

What do you think?

Apologies if this is confusing, this is not (for me) an easy problem to describe.

Phil