Forum Moderators: open

Message Too Old, No Replies

Googlebot only taking my images what gives?

         

BigDreweyStyle1

12:29 am on Feb 5, 2003 (gmt 0)

10+ Year Member



I noticed Googlebot coming around today. It seems to only crawl my images. Is it to early to start being concerned, has anyone gotten deep crawled, here is a piece of my log file

64.68.82.67 - - [03/Feb/2003:18:43:01 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.67 - - [03/Feb/2003:18:43:02 -0500] "GET / HTTP/1.0" 302 271 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.67 - - [03/Feb/2003:18:43:05 -0500] "GET /tools/index.html HTTP/1.0" 200 28913 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:08:09:59 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:08:09:59 -0500] "GET /tools_images/images/thumb/Sr356A.jpg HTTP/1.0" 200 19174 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.79 - - [04/Feb/2003:08:11:42 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.79 - - [04/Feb/2003:08:11:42 -0500] "GET /tools_images/images/thumb/9737pa HTTP/1.0" 200 1787 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.43 - - [04/Feb/2003:09:00:59 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.43 - - [04/Feb/2003:09:00:59 -0500] "GET /tools_images/images/thumb/d55580_1.gif HTTP/1.0" 200 1364 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.43 - - [04/Feb/2003:09:01:56 -0500] "GET /tools_images/images/thumb/dg2900_1.gif HTTP/1.0" 200 1999 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:09:21:21 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:09:21:21 -0500] "GET /tools_images/images/thumb/dg2900_1.gif HTTP/1.0" 200 1999 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:11:19:34 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:11:19:34 -0500] "GET /tools_images/images/thumb/0299-20a.gif HTTP/1.0" 200 2998 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.49 - - [04/Feb/2003:14:34:19 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.49 - - [04/Feb/2003:14:34:23 -0500] "GET / HTTP/1.0" 302 271 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.49 - - [04/Feb/2003:14:34:53 -0500] "GET /tools/index.html HTTP/1.0" 200 29006 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.31 - - [04/Feb/2003:14:35:17 -0500] "GET / HTTP/1.0" 302 271 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.82.31 - - [04/Feb/2003:14:35:27 -0500] "GET /tools/index.html HTTP/1.0" 200 29028 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.59 - - [04/Feb/2003:17:59:11 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.59 - - [04/Feb/2003:17:59:12 -0500] "GET /tools_images/images/foundation/login.jpg HTTP/1.0" 200 1867 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.59 - - [04/Feb/2003:18:07:36 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.59 - - [04/Feb/2003:18:07:36 -0500] "GET /tools_images/images/thumb/0299-20a.gif HTTP/1.0" 200 2998 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:18:07:43 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:18:07:43 -0500] "GET /tools_images/images/thumb/333vsa HTTP/1.0" 200 4453 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:18:15:05 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:18:15:05 -0500] "GET /tools_images/images/foundation/dewaltsander.gif HTTP/1.0" 200 5797 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:18:15:33 -0500] "GET /tools_images/images/thumb/dw362_1.gif HTTP/1.0" 200 1641 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:19:01:46 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:19:01:46 -0500] "GET /tools_images/images/thumb/6537-22a.gif HTTP/1.0" 200 3009 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:19:09:15 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.87.42 - - [04/Feb/2003:19:09:15 -0500] "GET /tools_images/images/foundation/telephone.jpg HTTP/1.0" 200 6338 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:19:14:57 -0500] "GET /robots.txt HTTP/1.0" 200 15 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" 64.68.86.54 - - [04/Feb/2003:19:14:57 -0500] "GET /tools_images/images/thumb/dw533k_k1.gif HTTP/1.0" 200 1486 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

jdMorgan

4:40 am on Feb 5, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



BigD,

Looks like the freshbot found a bunch of new images that it hadn't seen before. Either that, or you don't have your Last-Modified http server headers set up - I see that the 'bot got 200-OK responses most of the time. On my sites, when freshbot comes around, it gets a 304-Not Modified response on anything that it has already seen that hasn't changed. GoogleGuy posted several months ago that it's a good idea to include Last-Modified info in your headers, and I do.

64.68.xx.xx is the freshbot, and 216.239.xx.xx is the deepcrawler, whose presence is expected and keenly anticipated... Soon.

Jim

Googlebot Deepbot and Freshbot FAQ [webmasterworld.com]