Googlebot-Video/1.0"

audio & video files

[keyplyr]

66.249.73.223 - - [09/May/2015:02:55:42 -0700] "GET /file_name.mp3 HTTP/1.1" 304 200 "-" "Googlebot-Video/1.0"

First time I've seen this bot. Valid Google crawl range. Grabbed a couple mp3 audio & mp4 video files, but no request for robots.txt. The normal Googlebot had already read robots.txt several times earlier in the day, so I guess it's within reason.

crawl-66-249-73-223.googlebot.com
66.249.64.0/19
66.249.64.0 - 66.249.95.255

[Pfui]

Do you allow G access to those mp files? Or did it violate robots.txt?

[not2easy]

It is listed as one of their UAs: [support.google.com...]
I think if it is going to be getting large files I would not want it visiting. :(

[keyplyr]

@Pfui - I allow Googlebot access. If I didn't, it would not get 'em :) IMO SEs need access to all files a browsers uses nowadays, at least for the purpose of my sites.

@not2easy - I'm not on a bandwidth budget, but unlike Bing, Googlebot doesn't DL the same file 10x each and every day.

[lucy24]

Yup, I never saw it until some time last year when I added a few .wav files. But I looked up and found they've been around forever; it's just about file format.

:: detour to doublecheck ::

Oops, my bad, they're mp3. But I'm pretty sure it's the same UA for both.

All the different Googlebots seem to share robots.txt information. The only place I've seen where UA really makes a difference is in the If-Modified-Since header, where the time is based on the last visit by that same UA.