Annoyed by 404s from mobile devices

[salzano]

i am sick of 404 errors like this showing up in my error logging table for some sites.

mysites.com/pda/
mysites.com/iphone/
mysites.com/mobi/
mysites.com/mobile/
mysites.com/m/
mysites.com/pda/
mysites.com/iphone/
mysites.com/mobi/
mysites.com/mobile/
mysites.com/m/
mysites.com/apple-touch-icon.png

how do i tell these mobile devices that i don't care they are mobile devices, and they should browse the sites the best they can as is.

would they obey robots.txt, or should i deliver a 403 to requests for these locations?

[g1smd]

If you send 404 then you are saying there is no content to be had. Maybe 410 would see them off sooner?

If you send 403, then you are saying you are not allowed to look at the content - which to me suggests that the URL would be valid if I had the right credentials.

The easiest way might be simply to disallow via robots.txt then the URLs shouldn't be requested again in the future.

[Gibble]

There are really only 5 or 6, you duplicated your list.

I would setup a 30x redirect for those locations?

[salzano]

g1, i think the 404 is not good enough to stop the requests. this is the result of doing nothing...no file or directory exists so they get a 404. over and over again.

i know what you mean about the 403. i am always puzzled by a 403, but we're seasoned nerds. i just want to say, "no, stop that."

gib, i know there are dupes above. that's how they hit the server--consecutive duplicate requests.

a 301 sounds interesting, but its a little more work. what would that mean, though? does the phone hit the /iphone/ directory before it hits the root of the domain? if so, then maybe a 301 is the ticket.

[jdMorgan]

I'd be looking at these requests more carefully; Be sure whether they are actual mobile phone requests or whether they are robots, and if they are robots, whether they're legitimate or not.

I don't see any of the subdirectory-type requests you list on any of the sites I manage, whether those sites have mobile content or not.

As for "/apple-touch-icon.png", you might consider creating and adding this file -- basically the "Apple iPhone" version of a Favicon. Doing so would be wise from a marketing standpoint, as it provides an opportunity to attractively "brand" you site when displayed on iPhones, as well as getting rid of at least one of your 404-URLs. It should be a 57x57-pixel, PNG format image file.

Jim

[salzano]

here are some more details on the requests i saw this morning. i have seen these type of requests for about a year now from various devices. usually a nokia, but this morning it looks like an iphone

2009-06-04 14:35:57 GET /apple-touch-icon.png - 80 - 212.44.45.230 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-GB;+rv:1.8.1.6;+taptu-downloader)+Gecko/20070725+Firefox/2.0.0.6 - 404 0 2
2009-06-04 14:35:58 GET /m/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:35:58 GET /mobile/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:35:58 GET /mobi/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:35:59 GET /iphone/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:35:59 GET /pda/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:35:59 GET /m/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:36:00 GET /mobile/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:36:00 GET /mobi/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:36:00 GET /iphone/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-04 14:36:00 GET /pda/ - 80 - ***.44.45.230 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2

sorry about the ugly log file paste here. these requests were sequential as you can see by the time stamps.

jd, the requests are usually from phone-looking agents. sometimes a mozilla looks for these directories, but i am blocking most "default name" bots like wget, libwww-perl, java etc across the board.

i have no interest in the apple icon. i have made some for our own brands, but the bulk of the sites we host are for a few hundred small businesses that barely have their own logos. i would make favicons before i make some device specific icon.

[edited by: tedster at 12:31 am (utc) on June 6, 2009]
[edit reason] obscured the IP address [/edit]

[jdMorgan]

I assumed that you already had favicons. I recommend having both favicons and the iPhone icon, since iPhones are used to generate the majority (by a slight margin) of all mobile traffic -- at least in the U.S. where my sites are centered. Graphic icons are a great site-branding tool, so don't miss out.

Thinking/talking to myself, here:

Since some (but not all) of these requests could be legitimate, the question becomes, "Where did they find links to these non-existent subdirectories?" Spend some time looking for incorrect backlinks. If you don't find any, then these user-agents are likely spoofed, and you may want to block that IP addresses or range temporarily.

That "taptu downloader" should probably go straight into your user-agent filter.

However, it's not clear from your log listing above whether that is your IP address, whether the IP address shown is the requestor's IP address, or whether you edited the data for posting here. If that's the IP address of the requestor's ISP, and that ISP does not provide a mobile gateway (or mobile proxy), then it appears that the user-agent string is changing -- Not definitive, but another red flag that these may be spoofed user-agents. The fact that the user-agent is changing so quickly is also a red flag. Finally, the speed of the requests to different subdirectories is too fast for a human user, and especially a mobile user.

If I saw this sequence of requests, I'd block that IP address temporarily, based on the subdirectory names in the URL and the IP address range. Then I'd watch to see if 'they' clicked on the 'help' link on my custom 403-Forbidden error page, and behaved like a human after that...

Jim

[salzano]

here's today's annoying mobile errors hitting a different site from different IPs in the same fashion:

2009-06-05 08:38:01 GET /apple-touch-icon.png - 80 - 212.44.45.253 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-GB;+rv:1.8.1.6;+taptu-downloader)+Gecko/20070725+Firefox/2.0.0.6 - 404 0 2
2009-06-05 08:38:02 GET /m/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:02 GET /mobile/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:03 GET /mobi/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:03 GET /iphone/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:03 GET /pda/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:04 GET /m/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:04 GET /mobile/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:04 GET /mobi/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:04 GET /iphone/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2
2009-06-05 08:38:04 GET /pda/ - 80 - ***.44.45.253 Mozilla/5.0+(iPhone;+U;+CPU+like+Mac+OS+X;+en)+AppleWebKit/420++(KHTML,+like+Gecko)+Version/3.0+Mobile/1A542a+Safari/419.3 - 404 0 2

i edited out the IIS service number and my internal server IP only.

[edited by: tedster at 12:32 am (utc) on June 6, 2009]
[edit reason] obscured the IP address [/edit]