Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Do robots get cookies?

robots cookies possible dumb question


Michel Samuel

3:40 am on Oct 11, 2007 (gmt 0)

5+ Year Member

Yes it is stupide question.
But my business partner asked it and instead of just saying non.
I tried to search for the answer. And I have not found anything.

My business partner claims that since it is downloading the entire site it is in some form taking cookies.

I said it is not a browser and not displaying anything.
Therefore it is not accepting cookies but just taking information.

perhap an expert can answer this.


6:46 am on Oct 11, 2007 (gmt 0)


7:11 am on Oct 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

If you send a visitor a cookie, then that visitor can decide whether to use or discard the cookie.

That applies to robot visitors as well as user agent visitors.

Some robots will use it. Some won't.

Some will go to extreme lengths to get a cookie. Such as pairing with a roomful of underpaid spam assistant who will help the bot step past a catcha, site sign up, or othe such bot barrier.


10:55 am on Oct 11, 2007 (gmt 0)

WebmasterWorld Administrator phranque is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

a browser and a robot are simply two different types of user agents.

let me be more specific and say that you can certainly design a robot to accept any cookies and store them, just as a typical browser would.
you could also design that bot to return any stored cookies with subsequent GET request headers which would make the usual server application act as if this was a user agent that accepts cookies.

the typical search engine robot will not be looking for cookie-dependent text.

if necessary, the server application can be designed to thwart access to cookie-dependent content by robots spoofing as cookie-accepting browsers.
and thus begins the arms race...


Featured Threads

Hot Threads This Week

Hot Threads This Month