Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
My business partner claims that since it is downloading the entire site it is in some form taking cookies.
I said it is not a browser and not displaying anything.
Therefore it is not accepting cookies but just taking information.
perhap an expert can answer this.
see this recent WebmasterWorld thread on the subject relating to how search engines work [webmasterworld.com].
That applies to robot visitors as well as user agent visitors.
Some robots will use it. Some won't.
Some will go to extreme lengths to get a cookie. Such as pairing with a roomful of underpaid spam assistant who will help the bot step past a catcha, site sign up, or othe such bot barrier.
let me be more specific and say that you can certainly design a robot to accept any cookies and store them, just as a typical browser would.
you could also design that bot to return any stored cookies with subsequent GET request headers which would make the usual server application act as if this was a user agent that accepts cookies.
the typical search engine robot will not be looking for cookie-dependent text.
if necessary, the server application can be designed to thwart access to cookie-dependent content by robots spoofing as cookie-accepting browsers.
and thus begins the arms race...