homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Do robots get cookies?
robots cookies possible dumb question
Michel Samuel

 3:40 am on Oct 11, 2007 (gmt 0)

Yes it is stupide question.
But my business partner asked it and instead of just saying non.
I tried to search for the answer. And I have not found anything.

My business partner claims that since it is downloading the entire site it is in some form taking cookies.

I said it is not a browser and not displaying anything.
Therefore it is not accepting cookies but just taking information.

perhap an expert can answer this.



 6:46 am on Oct 11, 2007 (gmt 0)


see this recent WebmasterWorld thread on the subject relating to how search engines work [webmasterworld.com].


 7:11 am on Oct 11, 2007 (gmt 0)

If you send a visitor a cookie, then that visitor can decide whether to use or discard the cookie.

That applies to robot visitors as well as user agent visitors.

Some robots will use it. Some won't.

Some will go to extreme lengths to get a cookie. Such as pairing with a roomful of underpaid spam assistant who will help the bot step past a catcha, site sign up, or othe such bot barrier.


 10:55 am on Oct 11, 2007 (gmt 0)

a browser and a robot are simply two different types of user agents.

let me be more specific and say that you can certainly design a robot to accept any cookies and store them, just as a typical browser would.
you could also design that bot to return any stored cookies with subsequent GET request headers which would make the usual server application act as if this was a user agent that accepts cookies.

the typical search engine robot will not be looking for cookie-dependent text.

if necessary, the server application can be designed to thwart access to cookie-dependent content by robots spoofing as cookie-accepting browsers.
and thus begins the arms race...

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved