Forum Moderators: open

Message Too Old, No Replies

UniversalFeedParser

         

keyplyr

4:50 am on Dec 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




UA: UniversalFeedParser/3.3 +http://feedparser.org/
Protocol: HTTP/1.1
Robots.txt: No
Host: eonix.net (serverhub.com)
104.140.0.0 - 104.140.255.255
104.140.0.0/16

URL in UA string for sale

lucy24

5:05 am on Dec 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



One of these days I'm going to dig up all the robots that have "Feed" in their name; I can't tell them all apart. Are yours hitting pages that have appeared on someone's (not necessarily your own) RSS feed?

Favorite UA to date (URL obfuscated):
Feedly/1.0 (+http://www.example.com/fetcher.html; like FeedFetcher-Google)
Is "like FeedFetcher-Google" meant to lull suspicion and inspire confidence, much like calling yourself Baiduspider?

keyplyr

5:19 am on Dec 23, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Are yours hitting pages that have appeared on someone's (not necessarily your own) RSS feed
No, I actually filter "feed", "fetch", "rss", "parse" and a couple other related UA attributes, allowing only a couple of these UAs access.

Rss feed retrievers, including Google's, are known to scrape content from a web site and "feed" it into iframes (or other container methods) at remote sites. It's not difficult to do and the original site has no breadcrump to alert them. [webmasterworld.com...]

One of the reasons I use a script that busts both frames & iframes bringing the user to the Parent source... my site.

I also use this header field:
X-FRAME-OPTIONS: deny