Hi, I've been checking my logs recently. I want to make sure I understand these entries.
Throughout the day, I see several single-page requests by Googlebot for various pages on the site. I would expect a regular "scheduled crawl" to enter through the home page and request additional pages. Is it safe to assume that most single-page requests, like below, are Googlebot following backlinks from external pages?
Here, it looks like Googlebot is accessing the page as a mobile device. I wonder why it would do that? To test my site for mobile compatibility?
18.104.22.168 - - [28/Feb/2013:14:07:41 -0500] "GET /page2.htm HTTP/1.1" 200 20997 "-" "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)"
Also, I am seeing alot of 301 results on live pages that shouldn't be redirected. The 301's are only happening with Googlebot and various other bots. Even my robots.txt file is getting a 301 from bingbot (second example). It doesn't look like human users are encountering any 301's. Here are some examples:
Maybe the "MirrorDetector" on that last one is a clue. Is that normal for Googlebot and other bots to artificially trigger a 301 response? Or is there some other issue I need to look at? My error logs are about empty.