Forum Moderators: DixonJones
My site is a small scale search engine and I apear to have recieved literaly thousands of referals from other search engines ad url form.
On an average day I will serve 3000 page views on the 1st of November I served 10000 page views loads of the referals seamed to be comming from se add url forms. Some I have never heard of and some that I know of.
open find, Aesop, ah-ha, Claymont to name but a few...
I have never seen these referals from any search engine in the past... now i am getting them from loads of them
Does anyone have any idea what could be going on?
For example the log for openfind looks like this...
POST /search/search.pl HTTP/1.1" 200 3840 "http://www.openfind.com/WriteReg.php" "Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)"
In the above entry from my log /search/search.pl is the location of my search script.
It had left referals from the add URL pages of about 20 se's!
This sounds very much like a rogue bot that just feeds you fake referer information.
None apear to be from the same IP address
My stats just apdated... figures for the 2nd whre just as bad as they where for the 1st.
What to do?
It's like an enemy that I cant see.
IP address doesnt resolve so I can bar it that way.
No useragent - useless for htaccess
What is the propper thing to do in a situation like this.
We've implemented three small searchable directories using Gossamer-Threads Links 2.0
One of these is a business directory which is open for public "suggestion" of URLs. This directory got put into one of these d**m autosubmitter services and we were getting literally hundreds of junk submits daily until I modified the submission script to reject automatic submissions.
The most obvious way of achieving this is for the script to check the referrer and ONLY allow posts that have as referer your own submission form. It's pretty easily worked around by an autosubmitter though - referer is the easiest header parameter to spoof!
Because of this, I added a couple of other traps as well. These are useable as long as the submission form is generated by a script.
One trap is to include the current date/time as a hidden field in the script generated submit form. Then when the data is posted, the script can compare that field with the current date/time, rejecting or ignoring the submission if the post date/time is in the future or more than, say, 10 minutes ago. This value can be tweaked as necessary to fit requirements.
The other trap ... Links 2.0 assigns a unique ID code each time it serves the submission form, assuming that it is going to be for a valid submission. The script then checks the post data and if the hidden ID field value is greater than the current counter value, or more than 5 less, the POST data can be rejected (or just plain ignored). That value may have to be tweaked if you can get a lot of valid submissions, to minimise the chances of ignoring a valid submissions.
Hope this helps :)
The search tool in question is fdse. I have been using it as a general web search tool but this morning I woke up to 54 meg of submited sites.
what I have done is simply delite the added sites db file. then went back to the fdes admin page and refresh until a new site is added. At this point i am grabbing their email address and asking them what submission service thay are using.
and when I find out who it is.. my htacess will be redirecting all the unwanted traffic to their homepage :)
It apears that my site has been added to a piece of web positioning software, not WPG but something similar.
My question is , do these products hold their own database or is the Database remotly hosted so the software company can edit it at a later date?
I am not sure that the user even gets to see any of my pages bacause of the post in the referer logs. Is it possible to direct all users to my actual website?
