Forum Moderators: open
I have a page that profiles a penny stock for a diamond (speculator). The robot visited yesterday and grabbed only that page along with robots.txt.
Judging by the other "undercurrents" relating to this stock I would not put nafarious activity to originate from this source - see previous posts relating to gator et al.
TsF
Is there anything that can be done (on your site) to stymie these guys? idoc seems to indicate that there is.
Not that it is a problem for me (yet) but I am more than happy to pass on tips to those that it does.
TsF