Any help appreciaited.
i'll be nervously refreshing this topic for the next few hours, but i probably wont be here next month.
many thanks in advance.
I am positive that they can although I would say most don't.
It is a lot of work to get a crawler to run JS depending on how it was written and how it does the crawling.... using wget type methods or a hacked browser up browser running automatically.
Can it be done... yes.
thanks for the reply
its simple code, its the link here:
im convinced its happening as there are just so many complications with modern code. I have loads of sites and robots are a constant hassle, no matter what i do there are always robots doing stuff.
In this case they have a pretty straight forward link almost:
And when that link is clicked, its firing a hit. As far as im concerned thats suseptable to robot traffic, and as long as the robot conceals its headers and makes itself look like a browser or does something i cant quite put my finger on then its going to generate some degree of false hits.
its my belief that advanced data mining robots will trigger that link and like any indexing system might not follow through as all they want it the result of that link, ie whats at the other end, without actually going there, or something like that.
hhmm, maybe i should put the link in code.
i have nothing to do with that company by the way so dont think im spamming by linking etc.
in fact the more i look at this the more im thinking that this is truly open to robot traffic.
i cant see how its not.
what can the aspx do? check the headers? check a list of robot IP's?
im beginning to feel very confident that they dont have any magical bot proof formula in place like they say?
we know the likes of google can, but i'm assuming you mean more the nefarious types of bot that are up to no good.(or at least no good that benefits you)
but bots don't 'click' links, they store page data, parse the links and store them as pages they will visit in the future, they will then at some point visit the newly discovered page (this can happen very quickly)
... the directory should be taking steps to stop bots scraping their content.
however, that seems like a bit of an achilles heal because all you need is some pearl? or a web service? to fire that actual redirect page, again and again with various domains and you can cause no ends of havoc?
i think its obviously a reasonable method but i just cant see it as being foolproof?
thanks for the reply topr8
i think its obvious that the link i have posted above is going to be subject to robot activity which the question being to what extent. Its also clear that the directory is already doing quite a bit to stop it. However, im just saying to them at least that its just impossible to put a 100% gaurantee on that, especially today.
Not expecting any hard coded solutions to this, but thoughts and options very eagerly recieved as its really frustrating, especially when they want nearly £700 for the pleasure of this nebulous activity.
thanks for all your replies.
i have to close this browser down so i wont see any further replies.
That said, 99% of robots dun use it
Cheers for the reply.
i still cant figure it out but im sure there are bots that do that. personally i believe there is a massive cyber war going on, just like in the Matrix and i think the countries like China etc with a lot of technical competance have developed very sophisticated intelligence gather systems such as robots that leave no stone unturned, and i think they love business directories which are perfect for profiling a countries economic picture as they have all the companies in them. As such a business directory would possibly be a target for a clever robot.
i do agree its a bit far fetched, but truth is often stranger than fiction.
How are you tracking where hits come from? Are you checking the Referer header or do they pass in a parameter in the request?
referer header field can be filled by anything that are you wanted.