Can Google tell what's in a picture?

[londrum]

I dont know if you've visited Google Image Search's homepage recently, but there are a few pictures underneath with a little sentence encouraging you to drag the pictures onto the search box, as a "new way to search".
one of the pictures is a painting by van gogh, and when you drag the picture onto the search box -- lo and behold! -- it "knows" what it is and brings up a load more.

I dont think any of us would be surprised by that, but i was bored so i did a little test where i dragged a thumbnail image of charles dickens into the box, straight from the hardrive on my computer (not a website), with a gibberish filename and all the metadata stripped out, and it still knew what it was! at the top of the page it said "Best guess for this image: Charles Dickens"

It was quite a famous picture, and some of the images it brought up where the same image at full size, but even so, i thought that was quite impressive seeing as my thumbnail was tiny (80px by 60px) -- and it was cropped.

Can google honestly tell what a picture is with no textual help?

[Propools]

yeah, I saw that a few weeks ago and want to say somewhere that I've heard of Google helping the Feds with some facial recognition stuff. Not sure though, but it makes sense.

[londrum]

i just did a harder one... i dragged a photo of a famous landmark straight from my digital camera onto the search bar, and it got that right too.

so this photo was taken by myself and has never existed before, with no meta-data on it, and a filename which is just a string of numbers, and it still got it right.

[netmeg]

The official announcement is down the page a bit in this one, from June or so.

[googleblog.blogspot.com...]

[Lame_Wolf]

londrum, have you tried reversing the image, or making it upside down ?

[londrum]

i flipped it horizontally and upside down too, and it still works.
then i coloured it so it was sepia tone all over, and it still got it

[danijelzi]

This is very important regarding general SEO for Google. If they can sort out what images are similar then they can figure out also if a site uses copied or their own original images.

[GifAnimator]

Did not realise this existed.
I was able to upload one of my hand made widgets and was able to see just how many websites were using it. One big advantage is that it also looks at the widget size.
Very good for tracking down big widget online collections using my stuff that are getting better Google search ranking than myself.
Spent half an hour using this tool and it even recognised a distorted image of Gordon Brown.
Thanks for bringing my attention to this.

[GifAnimator]

I hope you are right danijelzi, it would be a big advantage for me.

[Planet13]

So in theory I could test one of my copyrighted images to see who is also using it, right?

Also, isn't there an android app where you can take a photo of a local landmark and google will identify it and provide info on the landmark as well as nearby locations? (I am sure that google will monetize it with google palces).

[ken_b]

Don't get your hopes up.

I tried it with one of my photos of a 1925 classic car.

It return what it said were 4 "similar images".

1 was a telescope, 1 was a late model mustang, and 2 were motorcycles.

[Planet13]

1 was a telescope, 1 was a late model mustang, and 2 were motorcycles.

Out of curiosity, was there any kind of semantic relationship between the model of the classic car and the brand of telescope or the two motorcycles?

[ken_b]

Out of curiosity, ...

Not that I can see.

But I just did the same search again and the results were a bit different. Today there is a section at the top of the page that shows "maching images". That showed my image, on my site and a couple other sites as well as a nearly identical image on another site. That last image is almost impossible to tell from my image because the other photographer and I were standing shoulder to shoulder when we took the pictures (actually there were 5 - 6 of us in a row).

Below the "matching images" section were several images in a "visually similar images" section.

None of those were even of the same make of car, let alone thesame car. I did the search a few times and the results changed every time. Results included things a diverse as motorcycles and crowds with a few unrelated cars in between, none of the cars were even from the same era.

The really odd thing is that if you do a normal image search for this year and make of car there are 3 - 5 pictures of the same car on the results plus a couple other of a different car of the same make from the same era, but those images only occasionally show up in the new drag-n-drop type search.

Work in progress I guess.

[londrum]

i've just sliced up three black-and-white faces and merged them together. the hair/forehead was terry wogan, the middle bit was bob hope, and the chin was george harrison, and it spotted it as bob hope.

it brought up the complete picture of bob hope as one of the answers as well.

i suppose this means that if you copy a photo from another site, even if it's just a small segment of it, which you have then altered, then google will easily be able to tell where it came from.

i suppose it also means that stuffing alt tags and image filenames with keywords when the image isnt totally relevant might land you with a penalty? because google can presumably tell that you are trying to hoodwink it

[GifAnimator]

I love this tool, I have no copyright on any of my work but stipulate that none of my work should be included in other online collections. It shows all sites that are using my stuff, especially other "collection" websites and also gives me a big insight as to the most used images. Elvis is far more popular than I thought :0)

[lucy24]

Is there a pattern to which browsers it works in?

For me, drag-and-drop only works in GC and Firefox.

Safari (!) and Camino get as far as "drop your image here" but refuse to recognize the drop, even using one of the demo images. (Signed in/out makes no difference.) The "upload image" function works; didn't try URL.

And the new-style image search doesn't seem to exist at all in Opera.

That's an odd configuration of behaviors.

[Sgt_Kickaxe]

In my opinion ABSOLUTELY they can and they even use snapshots of our websites to measure content vs ads and such above the fold.

[tedster]

I remember back in 2002 or 2003 we already assumed Google was doing OCR on image-text. Now they've taken a giant leap, even though it's in its young days and has a long way to go. When you think about it, this kind of technology is pretty exciting. Will be fascinating to see where it goes.

[lucy24]

I got a lot of matching color schemes, which looks very pretty but is probably not useful.

Did my own experimenting with a group of line drawings of heraldic lions (public domain, from an e-text of a 1915 book). If you have ever looked up something like "lion rampant" you know that there are gazillions of them, all in very distinctive line-drawing form* and all sticking closely to the same canonical design. G### doesn't seem to have heard of them though; I got a lot of b/w anime instead.

In some cases, they didn't even recognize my original. (They're all on the same page. In fact it's my single biggest source of legitimate image searches.) But I did find one site that was scrupulous about crediting every single image; the one I was looking for was "courtesy Charles Boutell". This is pretty impressive, since Boutell died in 1877 and his re-editor in 1928.** Wonder how they obtained his permission?


* Also an awful lot in the color scheme "or, a lion rampant gules" and sometimes vice versa. But even omitting those, there are heaps of plain drawings.

** People in Life + 90 countries will have to skip the 1915 material for a few more years.

[aristotle]

I have a little site that has some blank images as well as some real images. The blank images are just small skinny rectangles used as spacers. Google has indexed all of the real images but none of the blank images.

[leadegroot]

In a slightly different usage, I happened to notice yesterday when I was reading a particular comic that the adsense on the side was on topic for the mentions of 'lounge' and 'bed' in the comic - and nowhere else on the page (I checked).
That was unexpected (although I suppose it shouldn't have been)

[HuskyPup]

I have had many images copied over the years, I didn't mind at first since I wanted my widget trade to use them rather than the crap ones many were using, however after using this and seeing just how many have watermarked my images etc, I've just spent all day watermarking my new site's images since it is plainly obvious no one gives a fig or recognition to the original, least of all Google.

Good luck to anyone who tries to manipulate the image and remove the watermark!

[networkliquidators]

I put in a picture of batman and got superman with a corrective suggestion.

[aristotle]

This discussion has caused me to wonder if the content of the images on a page can affect its rankings in Google's regular search. For example, an extreme case would be a page with wriiten content about electronic widgets but with images of dinosaurs (but meta-labeled as widgets, with no mention of dinosaurs anywhere on the visible page or in the source code). Would the presence of the unrelated images hurt the page's rankings in regular searches for "electronic widgets"?

[danijelzi]

On a pandalized site I have a couple of pages that rank normally, like the site has not been hit. On these pages I have images that are delivered by product manufacturers (but hosted on my domain) and often used on other sites. For these images from these unaffected pages Google can't find their copies via Image Search for some reason.

On other pages, I'm also using images widely used on other sites. Google can find their copies via the Image Search and these pages are de-ranked.

I've started using my original images on newer posts and have a couple of pages with these images that rank normally. Also, some pages are pandalized regardless of image originality.

This my little research uses a small sample, so I would like to hear your experiences regarding this if you have any.

Maybe we could make some conclusion regarding this.

Note: All texts on the site are originally written.

[SEOMike]

I saw that shortly after it went live. I took a picture of Dallas out the window of the plane when it was landing and uploaded it. Google knew right away what city it was even though there were no exact matches. It does a pretty good job of identifying cities by their skyline. It's gotten a lot more sophisticated since then because it was easy to stump it a month or so ago.

I think the technology they are using is based on the algorithm they acquired from Like which they use to match outfits and accessories in Google Boutiques that they launched last November.

[indyank]

danijelzi, This is interesting. I have always been suspecting this and am seeing something similar.But one of these sites is a review site and it has mostly screenshots of online apps and a few software products. But they are all original screenshots taken for the purpose of the review.When i drag, drop and do a search, google is finding similar images. I don't understand the logic of pushing down sites based on these screenshots, as they are still original screenshots taken by the reviewer.

[indyank]

I always considered it to be just a new tool to find similar images on the web. Do you think they will be using it to determine uniqueness or duplicates? I don't see a reason for them doing this but with whatever they are doing recently, you never know.

[Sgt_Kickaxe]

They're using it on snapshots of your pages indyank, you can test that by placing text inside an image. In "google previews" the page will (eventually, takes a few weeks) get pulled up with the image highlighted if you search for that text.

I've also seen previews catch what's inside an adsense unit and whats in a flash section which suggests they may be using OCR exclusively for previews since adsense is 3rd party file javascript based and flash is hard to parse.

Together with Google's latest push for people to get google profiles and use real names it's apparent Google is building a database of everyone. Imagine the day when one picture of a protest or shot of the fans behind home plate at a baseball game is enough to identify every single person present. That is likely already a reality.

Google watches from space, give them access to traffic cams and they have the tools to identify people and track them around their daily lives, our privacy may already be cooked.

It makes you wonder why the NSA or Google doesn't step in when someone is abducted. More than likely they fear the backlash from citizens who object to having their privacy invaded to that degree if details of HOW they tracked an individual are made public. Man this stuff sounds like sci-fi but then again how do they find the person who threw a cigarette butt out a window to start a major forest fire? They just rewind the tape to the start of the blaze... The creepy edge indeed!

I'm all for it if it's used to track criminal activity. I'm firmly against it if NON criminals can be readily identified. I don't see how Google could be making a distinction.

[MrSavage]

This is the other change I'm noticing and that is with the Google images pages. Perhaps not all searches just yet, but duplicates are getting weeded out. If you could convert some image traffic with some pretty common pictures, you might be in trouble! I'm expecting to lose a lot of image related traffic. I suppose it should have been expected that the "intelligence" behind the image search would improve to the point where Google wants one photo per page and not 10 of the same photo.