Error: Huge 404 issue

Hundreds of 404s, as if it though it was crawling someone else's site?

[JAB Creations]

Ask keeps crawling pages like ...
/forums/inspirations/posts/24432.html

MSN is also doing some really goofy stuff. It's plain obnoxious and wastes my time when I'm trying to fix real 404s visitors may be encountering.

I've contacted Ask via email about two weeks ago about this and didn't get even a "we're letting you know we got your mail and someone might contact you three days after you've died" email.

- John

[Robert Charlton]

Ask is the only engine that still indexes an old site that was 301ed to a new domain two years ago. For some searches, the old domain outranks the current site. For others, the new site is winning. ;)

I used to attribute this simply to stale results. After this much time, though, I've got to say that there are some things that Ask clearly hasn't figured out.

[JAB Creations]

The 404s are URLs that have never existed on my website so while your post is interesting it does not apply to my situation specifically.

I still haven't received any emails and I actually waited for about twenty minutes on a silent line waiting for someone at Ask to pick up. No one's home?

I suppose the easiest way to deal with it is to use robots.txt and just deny the path it keeps generating 404s on. After all Ask crawls maybe 10% of what Google does tops so I'd prefer they crawl real and important pages first and foremost. ;)

- John

[vpathak]

John,

Please share the following information about the offending crawls:

- Site being crawled
- Lines from the web server log showing the 404 crawls by our crawler

Sorry for the inconvenience, and thanks for reporting the problem.

Best,

Vivek Pathak
Search Product Manager
Ask.com

[JAB Creations]

Hi Vivek,

I've copied 85 lines and counted in my stats script 85 lines (though I just kept finding them anyway)...

jabcreations.com is my domain. I've slimmed down the lines to not kill the thread too much.

I hope this helps! :)

The three parts of each line I've removed...

65.214.44.42 - -

+0000] "GET

HTTP/1.0" 404 12858 "-" "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)"

Here is the slimmed down log...

30/Jun/2006:11:41:42 /forums/inspirations/posts/15378.html
30/Jun/2006:11:48:16 /forums/inspirations/posts/17556.html
30/Jun/2006:11:54:47 /forums/inspirations/posts/18241.html
30/Jun/2006:12:01:17 /forums/inspirations/posts/21634.html
30/Jun/2006:12:07:45 /forums/inspirations/posts/17709.html
30/Jun/2006:12:14:16 /forums/inspirations/posts/18100.html
30/Jun/2006:12:20:45 /forums/inspirations/posts/17830.html
30/Jun/2006:12:27:15 /forums/inspirations/posts/17683.html
30/Jun/2006:12:33:46 /forums/inspirations/posts/18494.html
30/Jun/2006:12:40:17 /forums/inspirations/posts/18675.html
30/Jun/2006:12:46:50 /forums/inspirations/posts/17184.html
30/Jun/2006:12:53:23 /forums/inspirations/posts/17623.html
30/Jun/2006:12:59:53 /forums/inspirations/posts/17680.html
30/Jun/2006:13:06:23 /forums/inspirations/posts/19805.html
30/Jun/2006:13:12:54 /forums/inspirations/posts/17253.html
30/Jun/2006:13:19:24 /forums/inspirations/posts/21356.html
30/Jun/2006:13:25:55 /forums/inspirations/posts/24406.html
30/Jun/2006:13:32:27 /forums/inspirations/posts/23655.html
30/Jun/2006:13:38:58 /forums/inspirations/posts/18577.html
30/Jun/2006:13:45:29 /forums/inspirations/posts/18424.html
30/Jun/2006:13:52:00 /forums/inspirations/posts/17198.html
30/Jun/2006:13:58:32 /forums/inspirations/posts/17986.html
30/Jun/2006:14:05:02 /forums/inspirations/posts/18805.html
30/Jun/2006:14:11:34 /forums/inspirations/posts/17789.html
30/Jun/2006:14:18:06 /forums/inspirations/posts/17976.html
30/Jun/2006:14:24:39 /forums/inspirations/posts/14479.html
30/Jun/2006:14:31:11 /forums/inspirations/posts/16418.html
30/Jun/2006:14:37:43 /forums/inspirations/posts/23705.html
30/Jun/2006:14:44:15 /forums/inspirations/posts/10131.html
30/Jun/2006:14:44:15 /forums/inspirations/posts/10131.html
30/Jun/2006:14:50:49 /forums/inspirations/posts/17360.html
30/Jun/2006:14:57:20 /forums/inspirations/posts/21183.html
30/Jun/2006:15:03:54 /forums/inspirations/posts/24921.html
30/Jun/2006:15:30:07 /forums/inspirations/posts/2110.html
30/Jun/2006:15:50:06 /forums/inspirations/posts/1072.html
30/Jun/2006:16:03:29 /forums/inspirations/posts/18757.html
30/Jun/2006:16:16:37 /forums/inspirations/posts/23491.html
30/Jun/2006:16:23:11 /forums/inspirations/posts/24137.html
30/Jun/2006:16:29:46 /forums/inspirations/posts/18345.html
30/Jun/2006:16:36:20 /forums/inspirations/posts/18868.html
30/Jun/2006:16:42:54 /forums/inspirations/posts/18720.html
30/Jun/2006:16:49:29 /forums/inspirations/posts/17921.html
30/Jun/2006:16:56:04 /forums/inspirations/posts/21887.html
30/Jun/2006:17:02:40 /forums/inspirations/posts/18099.html
30/Jun/2006:17:09:22 /forums/inspirations/posts/19500.html
30/Jun/2006:17:15:58 /forums/inspirations/posts/18416.html
30/Jun/2006:17:22:34 /forums/inspirations/posts/18303.html
30/Jun/2006:17:29:10 /forums/inspirations/posts/18664.html
30/Jun/2006:17:35:46 /forums/inspirations/posts/19751.html
30/Jun/2006:17:42:23 /forums/inspirations/posts/17962.html
30/Jun/2006:17:55:37 /forums/inspirations/posts/24090.html
30/Jun/2006:18:03:25 /forums/inspirations/posts/17382.html
30/Jun/2006:18:10:08 /forums/inspirations/posts/24123.html
30/Jun/2006:18:16:48 /forums/inspirations/posts/20719.html
30/Jun/2006:18:23:24 /forums/inspirations/posts/15328.html
30/Jun/2006:18:30:00 /forums/inspirations/posts/22438.html
30/Jun/2006:18:36:36 /forums/inspirations/posts/17824.html
30/Jun/2006:18:49:55 /forums/inspirations/posts/18840.html
30/Jun/2006:18:56:33 /forums/inspirations/posts/18700.html
30/Jun/2006:19:03:19 /forums/inspirations/posts/18455.html
30/Jun/2006:19:09:58 /forums/inspirations/posts/18328.html
30/Jun/2006:19:16:36 /forums/inspirations/posts/17233.html
30/Jun/2006:19:23:23 /forums/inspirations/posts/18838.html
30/Jun/2006:19:48:59 /forums/inspirations/posts/18838.html
30/Jun/2006:20:14:38 /forums/inspirations/posts/18698.html
30/Jun/2006:20:27:13 /forums/inspirations/posts/24382.html
30/Jun/2006:20:39:49 /forums/inspirations/posts/18246.html
30/Jun/2006:20:52:39 /forums/inspirations/posts/17736.html
30/Jun/2006:21:05:14 /forums/inspirations/posts/24079.html
30/Jun/2006:21:17:51 /forums/inspirations/posts/19528.html
30/Jun/2006:21:30:26 /forums/inspirations/posts/19548.html
30/Jun/2006:21:43:04 /forums/inspirations/posts/21114.html
30/Jun/2006:21:55:41 /forums/inspirations/posts/23421.html
30/Jun/2006:22:08:18 /forums/inspirations/posts/19921.html
30/Jun/2006:22:20:55 /forums/inspirations/posts/21891.html
30/Jun/2006:22:33:33 /forums/inspirations/posts/22897.html
30/Jun/2006:22:46:13 /forums/inspirations/posts/19466.html
30/Jun/2006:22:59:17 /forums/inspirations/posts/19148.html
30/Jun/2006:23:11:56 /forums/inspirations/posts/23518.html
30/Jun/2006:23:24:36 /forums/inspirations/posts/21267.html
30/Jun/2006:23:37:15 /forums/inspirations/posts/20853.html
30/Jun/2006:23:49:55 /forums/inspirations/posts/21913.html
01/Jul/2006:00:02:35 /forums/inspirations/posts/20220.html
01/Jul/2006:00:15:16 /forums/inspirations/posts/19951.html
01/Jul/2006:00:27:59 /forums/inspirations/posts/18630.html
01/Jul/2006:00:40:42 /forums/inspirations/posts/19827.html
01/Jul/2006:00:53:24 /forums/inspirations/posts/19062.html
01/Jul/2006:01:06:08 /forums/inspirations/posts/24432.html
01/Jul/2006:01:18:51 /forums/inspirations/posts/20765.html
01/Jul/2006:01:31:34 /forums/inspirations/posts/20611.html

- John