| 5:31 pm on Mar 10, 2011 (gmt 0)|
I am positive that they can although I would say most don't.
It is a lot of work to get a crawler to run JS depending on how it was written and how it does the crawling.... using wget type methods or a hacked browser up browser running automatically.
Can it be done... yes.
| 5:48 pm on Mar 10, 2011 (gmt 0)|
thanks for the reply
its simple code, its the link here:
im convinced its happening as there are just so many complications with modern code. I have loads of sites and robots are a constant hassle, no matter what i do there are always robots doing stuff.
In this case they have a pretty straight forward link almost:
And when that link is clicked, its firing a hit. As far as im concerned thats suseptable to robot traffic, and as long as the robot conceals its headers and makes itself look like a browser or does something i cant quite put my finger on then its going to generate some degree of false hits.
its my belief that advanced data mining robots will trigger that link and like any indexing system might not follow through as all they want it the result of that link, ie whats at the other end, without actually going there, or something like that.
| 5:50 pm on Mar 10, 2011 (gmt 0)|
hhmm, maybe i should put the link in code.
i have nothing to do with that company by the way so dont think im spamming by linking etc.
| 5:53 pm on Mar 10, 2011 (gmt 0)|
in fact the more i look at this the more im thinking that this is truly open to robot traffic.
i cant see how its not.
what can the aspx do? check the headers? check a list of robot IP's?
im beginning to feel very confident that they dont have any magical bot proof formula in place like they say?
| 6:00 pm on Mar 10, 2011 (gmt 0)|
we know the likes of google can, but i'm assuming you mean more the nefarious types of bot that are up to no good.(or at least no good that benefits you)
but bots don't 'click' links, they store page data, parse the links and store them as pages they will visit in the future, they will then at some point visit the newly discovered page (this can happen very quickly)
... the directory should be taking steps to stop bots scraping their content.
| 6:01 pm on Mar 10, 2011 (gmt 0)|
however, that seems like a bit of an achilles heal because all you need is some pearl? or a web service? to fire that actual redirect page, again and again with various domains and you can cause no ends of havoc?
i think its obviously a reasonable method but i just cant see it as being foolproof?
| 6:11 pm on Mar 10, 2011 (gmt 0)|
thanks for the reply topr8
i think its obvious that the link i have posted above is going to be subject to robot activity which the question being to what extent. Its also clear that the directory is already doing quite a bit to stop it. However, im just saying to them at least that its just impossible to put a 100% gaurantee on that, especially today.
Not expecting any hard coded solutions to this, but thoughts and options very eagerly recieved as its really frustrating, especially when they want nearly £700 for the pleasure of this nebulous activity.
| 12:23 pm on Mar 11, 2011 (gmt 0)|
thanks for all your replies.
i have to close this browser down so i wont see any further replies.
| 5:17 pm on Apr 6, 2011 (gmt 0)|
That said, 99% of robots dun use it
| 5:50 pm on Apr 6, 2011 (gmt 0)|
Cheers for the reply.
i still cant figure it out but im sure there are bots that do that. personally i believe there is a massive cyber war going on, just like in the Matrix and i think the countries like China etc with a lot of technical competance have developed very sophisticated intelligence gather systems such as robots that leave no stone unturned, and i think they love business directories which are perfect for profiling a countries economic picture as they have all the companies in them. As such a business directory would possibly be a target for a clever robot.
i do agree its a bit far fetched, but truth is often stranger than fiction.
| 8:01 am on Apr 11, 2011 (gmt 0)|
How are you tracking where hits come from? Are you checking the Referer header or do they pass in a parameter in the request?
| 8:17 am on Apr 19, 2011 (gmt 0)|
referer header field can be filled by anything that are you wanted.