enigma1 - 4:37 pm on Jul 14, 2009 (gmt 0)

After reading the comments here and doing a bit of testing it is possible to scrap content from sites and the site owner looking the server logs will believe that everything is a legitimate request.

So first of all, the referrer link from google is always valid regardless of geo location. If I take a referrer link I see in my logs and place it on my browser I get a redirect either via jscript or meta-refresh (if js is off).

Now a server could simply setup that link of the page he wants to scrap and use one of his clients to do the job. So the client can act as the man in the middle, assuming scripting is enabled on his browser, contact the target server retrieve the content and pass it over to the scrapper via js without even knowing it.

At the moment to get around it, I deployed a form, if the web application detects a redirection script on the referrer field. Then a visitor on my site needs to click a button to continue to the real page. This seem to work, but of course it may cause other side-effects I haven't yet considered.