homepage Welcome to WebmasterWorld Guest from 54.145.238.55
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google Sucking My Images Up
Why Why Why?
delboy1978uk

10+ Year Member



 
Msg#: 4538634 posted 12:26 pm on Jan 23, 2013 (gmt 0)

given the following:
https://example.com/robots.txt

User-agent: *
Disallow: /download/

User-agent: *
Disallow: /*.jpg$

User-agent: *
Disallow: /public/view-players/


why the hell wouldn't google pay attention and index anyway?
if you do an image search for POL0033 it appears

[edited by: incrediBILL at 12:54 am (utc) on Jan 24, 2013]
[edit reason] removed specifics, no specifics please [/edit]

 

not2easy

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



 
Msg#: 4538634 posted 1:16 pm on Jan 23, 2013 (gmt 0)

Is the image shown only from your site or might it have been indexed from another site?

Are you appending X-Robots-Tag: noindex to your images?

BTW, you should always use example.com here rather than your own URL.

lucy24

WebmasterWorld Senior Member lucy24 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4538634 posted 12:15 am on Jan 24, 2013 (gmt 0)

robots.txt = don't crawl

It says nothing about indexing.

Yes, this is counter-intuitive and will drive you berserk. But that's the way it is.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4538634 posted 8:21 pm on Jan 24, 2013 (gmt 0)

The robots.txt file is not checked during Web Preview hits on your site. Which is one reason I block Web Preview accesses. It will fetch anything and everything!

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4538634 posted 1:53 pm on Jan 27, 2013 (gmt 0)

assuming apache server...

http://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag [developers.google.com]:
You can use the X-Robots-Tag for non-HTML files like image files where the usage of robots meta tags is not possible.
Here's an example of adding a noindex X-Robots-Tag directive for images files (.png, .jpeg, .jpg, .gif) across an entire site:

<Files ~ "\.(png|jpe?g|gif)$">
Header set X-Robots-Tag "noindex"
</Files>

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4538634 posted 7:28 pm on Jan 27, 2013 (gmt 0)

Still doesn't apply to web preview, though: that ignores all robot directives.

phranque

WebmasterWorld Administrator phranque us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4538634 posted 10:50 pm on Jan 27, 2013 (gmt 0)

the OP was discussing indexing for image search.
web preview is a different problem.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved