Girafabot

does it disobey robots.txt? and a question...

[davelms]

64.210.196.198 - - [05/Aug/2003:04:13:18 +0100] "GET /robots.txt HTTP/1.0" 200 499 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)" mydomain.co.uk

I questioned their activity via the e-mail in the UA a few days ago - no response yet - but they've been back since now and took the same files from my disallowed images directory.

They did take robots.txt first.

Now to my question, just so that I'm not confused, but if a page is not disallowed in any way, the bot has legitimate access to it. If you deny the image directory (like I do) then because the bot took the page it doesn't have any sort of legitimacy to be able to take the images as seperate requests too? This is what Girafabot did, now it doesn't make sense that a bot *would* be allowed, but I thought I'd check! IMO Girafabot should not be doing this, but just checking first...

[wkitty42]

have you looked at their site to see what their stuff is all about? i just popped over there and went in thru the business entrance in the middle of the screen... i'd normally quote the text directly in other forums but will paraphrase it in here for safety's sake...

---

what they are doing is providing a search that provides a thumbnail of the web site/page they are possibly interested in... if the site/page appears similar towhat they are looking for, then they can visit it...

---

basically, i see it like users hopping from site to site and remembering that something they were interested in was on a site that had a visual appearance of blah... since they can't remember how they got there but they do remember the search terms, they can then browse the thumbnails of the sites/pages looking for that visual layout...

i hope that makes sense...

then i went in thru the webusers entrance... it appears to be a tool that they can download and run alongside their browser to provide those thumbnails of the sites they search for... actually, closer reading says that it is a toolbar that allows you to see the sites of the search you ran without having to go to the site first to see it...

personal opinion? i'm not going to download the toobar nor am i going to add them to my site on the business end... its just not something that i see users of my site using... they're either leeching files, playing games, learning about fidonet, or doing something in the space and amateur telescope making areas...

[added]
whoops! sorry, guess i didn't actually answer your question... i would think that they should not be getting the images, yes... however, to provide the thumbnail of the site, i don't see how they could not...

maybe its time to inquire of them directly?
[/added]

[jdMorgan]

IIRC, girafa used to (and may still) provide the thumbnails for the "Search Preview" function of MSN Search, avaialble if you search from the IE "Search" button.

Try this search to see the results of allowing/disallowing girafabot:

[search.msn.com...]

The top left Web site should be familiar to you: ;)

Jim

[wkitty42]

jd! its says "Preview not available" for *that* particular site... hehehehe...

good call, though... i had forgotten that they were the ones providing that (questionable?) "feature" to msn's search...

[davelms]

Thanks, yeah I've disallowed them. Yet to return, so I'll have to see if they obey that. Like I said originally, I contacted their support about it before coming here and they haven't, as yet, responded to me ...