Excluding image urls from being indexed by robots.

[born2run]

So I did a google search for my website using site:example.com and in the google search index are my thousands of image files which have their own url alias (via drupal) as an example being:

www.example.com/examplejpg

The above is the exact url being indexed by google search bot (without the period)..

Should I just disallow searchbots by placing the following in my robots.txt :

Disallow: /*jpg

Is this a good practice or a good implementation? Please advise Thanks!

[born2run]

I was thinking this robots.txt directive:

User-agent: Googlebot
Disallow: /*jpg$

Would this help or is it accurate? Thanks!

[mack]

I can understand why some may wish to exclude image search, but in some cases, it can lead to real traffic "View image - Visit page".

I sometimes use image search for general searches because I will click on an image that is closest to my result without having to read result snippets. This doesn't work for all searches but does very well within certain niches.

One approach I have seen used is hosting all content you do not want to be indexed on a sub-domain (images.example.net) and using a disallow all condition for that sub-domain.

Mack.

[born2run]

Thanks Mack, can you please check my following robots.txt directive:

User-agent: Googlebot
Disallow: /*jpg$

Is it worth doing it btw?

[not2easy]

Those are good points, mack though I know a good number of legitimate reasons not to want your images to show up in an image search. If you really don't want your images in search results, yo can send a noindex header with the image.

You do not need to disallow robots to prevent indexing. It is a topic that comes up here so often you can use the search here for "X-robots" and find hundreds of "how-to" threads. Disallowing robots can cause problems with Google's determinations of mobile friendly checking as will other blocked resources. That won't prevent your images from showing up in search results from other sites that have copied your images.

[born2run]

So not2easy I did a google search and found this directive that can be added to .htaccess:

<IfModule mod_headers.c>
<FilesMatch "images/*\.jpg$">
Header append X-Robots-Tag "noindex"
</FilesMatch>
</IfModule>

Would this be a good method? Thanks!

[born2run]

Actually I don't want file match I just want the directive to match url like "www.example.com/namejpg"

This is the url example I wanna block Thanks!

[not2easy]

There is a very nice search right here on WebmasterWorld, it might return more results than you wanted, but at least you can see who is recommending it, If you use results you find on Google, good luck. Hint - you may not need <FilesMatch>

[lucy24]

For image files, Disallow and noindex have the same end result. That is, I've never seen an image SERP containing blank boxes that say "the site's robots.txt would not allow us to show this picture". If you have huge numbers of image files, a robots.txt Disallow is probably a better approach, as it means the files won't get crawled in the first place. Less load on your server, and maybe a reallocation of crawl budget.

[born2run]

So I added the following to my robots.txt:

User-agent: *
Disallow: /*jpg$

Do I have to specify googlebot or the above would work in general? Thanks!

[lucy24]

If you specify googlebot, the rule will only be heeded by googlebot. If you say * it will apply to everyone.

In robots.txt, each user-agent can only be named once. After the first match, they stop looking. Surely you've already got a block for "User-Agent: *" (the generic Disallow directives for directories you don't want anyone crawling). Any new Disallows go there.

Most robots will not understand the /*jpg$ syntax (with RegEx-style closing anchor) anyway.

[born2run]

Well Lucy24 Bing ang Google understand the syntax /*jpg$

[born2run]

Yep you guys are right.. googlebot (google search console) complained about blocked resources for thousands of pages in my site when I added following to robots.txt:

Disallow: /*jpg$

So I deleted the directive. Thanks guys!