Forum Moderators: open

Message Too Old, No Replies

Image Bot Names?

         

keyplyr

12:21 pm on Feb 27, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



To get Google's mobile rendering tools to return a comprehensive report, it is necessary to not block images via robots.txt. So I while ago I removed:

User-agent: *
Disallow: /images1/
Disallow: /images2/
Disallow: /images3/
...etc

Google seems to be the only one who fully supports the header X tags, where I use noarchive for image directories. Now Google requests these image files, reads the header & does not index them into it's image search.

However since image directories are no longer disallowed in robots.txt Bing, Yandex, Yahoo, et al are once again indexing these images. So I have switched to bot specific disallows:

User-agent: bingbot
Disallow: /images/
Disallow: /images/
Disallow: /images2/
Disallow: /images3/
Disallow: /images4/
Disallow: /images5/
Disallow: /quotes/

But it's been a while since I've logged image bot names. Are these still the correct names for these image bots?

Bing: bingbot
MSN: msnbot
Yahoo: Yahoo-MMCrawler
Yandex: YandexBot

Any other image bots that obey robots.txt?

tangor

5:23 pm on Feb 27, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Leave your master disallow in robots.txt then add a google named disallow section giving access:

User-agent: Google
Disallow:

User-agent: *
Disallow: /example

This way you don't have to know the names of the other image bots. At least those that actually respect robots.txt!

lucy24

8:52 pm on Feb 27, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Bing: bingbot
MSN: msnbot

The msnbot and msnbot-media still exist, but they are exceedingly rare [webmasterworld.com]. The mobile bingbot's name includes "bingbot" so you should be covered either way. Worth noting that the mobile bingbot gets images (the mobile Googlebots don't) so plan accordingly.

But why do you need all those separate sections? If you've got a separate Googlebot section-- one that omits the Disallow lines for image directories-- everyone else will still default to the generic * version.

keyplyr

10:46 pm on Feb 27, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks, however I could have done that from the start. There's more to it than that. :)

I guess I wasn't as clear as I thought. I only asked for info regarding various image bot names.

I do see msnbot but I agree it is less often.