Forum Moderators: open

Message Too Old, No Replies

Flipboard redux

         

lucy24

8:29 pm on Jul 30, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



And now the good news...

This is Flipboard 4 years ago [webmasterworld.com]. Short version:
Asks for robots.txt some of the time BUT ignores it ALL of the time.

Is it possible they've now seen the error of their ways?
{ long string of robots.txt + 403 pairs from assorted AWS IPs, ending in }
54.158.59.24 - - [28/Jul/2015:13:38:38 -0700] "GET /robots.txt HTTP/1.1" 200 569 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
54.158.59.24 - - [28/Jul/2015:13:38:39 -0700] "GET /ebooks/abbey/ HTTP/1.1" 403 1716 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
184.72.191.180 - - [28/Jul/2015:15:06:53 -0700] "GET /robots.txt HTTP/1.1" 200 580 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)"
{ long string of robots.txt ONLY from the same assorted IPs}
What happened between 13:38 and 15:06? Answer: I decided it can't hurt to try, because the only thing better than a blocked request is no request at all, and so added a comprehensive FlipboardProxy Disallow in robots.txt. Astonishingly, it seems to work.

Backstory: I recently added one ebook to a respectable curated directory. My goodness, what a lot of robots follow this directory's RSS feed and/or "New Listings" page! It's been illuminating. In particular, more than half of the visits are Flipboard. (What the ### is Flipboard, anyway? I could swear I've never heard of it.)

wilderness

6:15 pm on Jul 31, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



What the ### is Flipboard, anyway?


Google is your friend.
There's a wiki page on same, which provides a description that might also be called a 'harvester'.

lucy24

6:52 pm on Jul 31, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Google is your friend.

:-P
Things can always be looked up. (A few minutes ago I was looking up a language name I'd never heard of.) But when I ask, it tends to be because I think I'll get more useful information from a human.

Now to see how long it takes before they stop asking ...