dblock - 7:33 pm on Aug 29, 2013 (gmt 0)
So I went through my server logs and found certain IP's that were only requesting the /topics directory and were only hitting that directory. I'm conflicted to block those IP's because I couldn't glean any other information from them (like user agent etc..). All I know is those IP's started requesting these pages around the 16th of May, they are all using IE 9 or 10 and for the user agent they have this: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)and
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0). I'm not sure if someone just wrote a script to get content from my topics directory and is not trying to do anything malicious. I know they are not stealing the content and trying to rank for it in Google cause I did a couple of operator searches using random pieces of content from these pages and only found my sites showing up. I guess the only thing left to do is block these IP's? I also think that the person doing this will just switch the IP once I block the one's they are using now. Any thoughts?