Starting around May 16th, 2013 I noticed a surge in Direct traffic on all of my sites. The pages that got this surge in traffic were just the topics pages (/topics/). I drilled down to see if I could get a country location but the main location that I see a huge increase in is (not set), so that wasn't helpful. I then took a look at the network properties and service providers and found one service provider that had a spike in visits - amazon.com inc. The browser which saw the highest increase was Internet Explorer, and the Operating System is Windows. There are two scenarios that I think could be possible:
1. Someone is scraping the content from the topic pages causing this influx of visits. The only thing that makes me doubt this, is that the engagement metrics (bounce rate, avg time on site, etc..) are not what a bot would usually show.
I'm currently analyzing the server log files, but not 100% sure of what I'm looking for. i've identified a couple of bots that are requesting hits for those specific pages (start with the directory /topics/). Can someone help with this situation?
While I don't consider myself the right person to help you with the underlying issue, let's see if we can't get someone in here with the right knowledge. Anyone?
Your initial assessment appears plausible. You're saying it appears to be real human traffic? I would say it likely is, then. I mean, you may have benefited from a recent SERP update. Have you changed your site ranking recently?
So I went through my server logs and found certain IP's that were only requesting the /topics directory and were only hitting that directory. I'm conflicted to block those IP's because I couldn't glean any other information from them (like user agent etc..). All I know is those IP's started requesting these pages around the 16th of May, they are all using IE 9 or 10 and for the user agent they have this: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)and Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0). I'm not sure if someone just wrote a script to get content from my topics directory and is not trying to do anything malicious. I know they are not stealing the content and trying to rank for it in Google cause I did a couple of operator searches using random pieces of content from these pages and only found my sites showing up. I guess the only thing left to do is block these IP's? I also think that the person doing this will just switch the IP once I block the one's they are using now. Any thoughts?