File does not exist: /public_html/undefined error in log file?

Forum Moderators: phranque

Message Too Old, No Replies

File does not exist: /public_html/undefined error in log file?

joyryde

7:11 am on Sep 15, 2005 (gmt 0)

Can anyone help me figure out why all of a sudden, after months with VERY few erros, I get this in my log, hundreds of times daily?

I have no reference to the word "undefined" anywhere on my site...yet there are hundreds of "file does not exist" errors?

[Mon Sep 12 12:07:53 2005] [error] [client 69.138.186.***] File does not exist: /home/deals/public_html/undefined
[Mon Sep 12 12:09:09 2005] [error] [client 24.174.145.*] File does not exist: /home/deals/public_html/undefined
[Mon Sep 12 12:09:17 2005] [error] [client 81.245.51.***] File does not exist: /home/deals/public_html/undefined

[edited by: jdMorgan at 1:38 pm (utc) on Sep. 15, 2005]
[edit reason] Obscured specifics [/edit]

jdMorgan

1:48 pm on Sep 15, 2005 (gmt 0)

What appears in your raw access log at the times corresponding to these error log entries?

If you see "GET /undefined HTTP/1.1" in your access log, then that would indicate some vulnerability scan being run from DSL clients in the U.S. and Belgium (from the sample IP addresses you provided)... maybe some new worm outbreak.

If you don't see requests for "GET /undefined" in your access log, then the error is being caused by some process inside your server -- An incorrect URL rewrite or an error in a script, for example.

Jim

joyryde

4:32 pm on Sep 15, 2005 (gmt 0)

All that I see on the error file is exactly as shown:

Repeated for hundreds of different IP's. You think it could be a problem in my .htaccess file?

jdMorgan

9:42 pm on Sep 15, 2005 (gmt 0)

> All that I see on the error file is exactly as shown:

Yes, now check your access logs and correlate the entries by time.

I can't give an opinion on your .htaccess file without seeing the access requests.

Jim

joyryde

5:43 pm on Sep 17, 2005 (gmt 0)

What do you need, I'm not sure wha tyou mean by seeing the "access requests". You want to see the access log not the error log? The "undefined" errors continue today, hundreds of them...

[Fri Sep 16 21:02:18 2005] [error] [client 69.47.178.*** File does not exist: /home/deals/public_html/undefined
[Fri Sep 16 21:05:04 2005] [error] [client 68.40.34.***] File does not exist: /home/deals/public_html/undefined
[Fri Sep 16 21:10:17 2005] [error] [client 64.81.65.**] File does not exist: /home/deals/public_html/undefined
[Fri Sep 16 21:14:40 2005] [error] [client 68.50.206.**] File does not exist: /home/deals/public_html/undefined
[Fri Sep 16 21:16:04 2005] [error] [client 68.50.206.**] File does not exist: /home/deals/public_html/undefined
[Fri Sep 16 21:20:08 2005] [error] [client 68.49.134.**] File does not exist: /home/deals/public_html/undefined

[edited by: jdMorgan at 2:38 pm (utc) on Sep. 19, 2005]
[edit reason] Obscured specifics. [/edit]

joyryde

5:59 pm on Sep 17, 2005 (gmt 0)

OK, so assuming you wanted some info from my access log (I have no experience with this) here is one example of an IP that created the "undefined" error in the error log. THese are that IP's entries in the access log:

"GET /images/equipment/korgESX1small.jpg HTTP/1.1" 304 - "http://www.example.com/production.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; iebar; acc=jocker; (R1 1.5); .NET CLR 1.1.4322)" 201.239.128.*** - - [17/Sep/2005:10:50:47 -0700]

"GET /images/equipment/herculesXP2016small.jpg HTTP/1.1" 304 - "http://www.example.com/production.htm" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; iebar; acc=jocker; (R1 1.5); .NET CLR 1.1.4322)" 201.239.128.***

There are dozens of pages he requested, but any page he requests creates the "undefined" error in the log.

[edited by: jdMorgan at 2:39 pm (utc) on Sep. 19, 2005]
[edit reason] Obscured specifics per TOS. [/edit]

jdMorgan

8:10 pm on Sep 17, 2005 (gmt 0)

The first line in your posted access log shows the request sent by the browser:


GET /images/equipment/korgESX1small.jpg HTTP/1.1

It also shows that the request was for an image, korgESX1small.jpg. The browser asked for the image, and indicated that it had a cached copy by including the If-Modified-Since date/time in the request. The server responded that the browser's cached copy was still valid (server code 304) and that no data transfer was necessary (with "-" indicating no data was sent).

So this request was handled properly.

1) Are there any requests in the same 'session' (same time, same IP, same user-agent) in the access log marked with a 500-Server Error response, or any code other than 200, 301, 302, or 304?

2) Are all requests for images, or do you see some requests in the access log for pages, for example, with .html, .php, or other extensions?

Jim

joyryde

8:18 pm on Sep 17, 2005 (gmt 0)

Hi Jim -

I would be more than happy to email you both the error log and the access log if it makes it easier on you to figure out why there are so many errors in the error log...

Would that work?

I've never seen anything odd in the access log that would generate this:

File does not exist: /home/deals/public_html/undefined

joyryde

5:24 pm on Sep 18, 2005 (gmt 0)

OK, so after searching both logs for about 20 minutes, I found some "undefined" errors that directly match accesses (based on time of access).

ERROR:

[Sun Sep 18 10:13:58 2005] [error] [client 24.186.85.**] File does not exist: /home/deals/public_html/undefined

ACCESS:

[18/Sep/2005:10:13:58 -0700] "GET /files/error.htm HTTP/1.1" 200 8018 "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 24.186.85.** - -

[18/Sep/2005:10:13:58 -0700] "GET /images/equipment/alesisSR16small.jpg HTTP/1.1" 304 - "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 24.186.85.** - -

[18/Sep/2005:10:13:58 -0700] "GET /images/equipment/vocoproVHF48small.jpg HTTP/1.1" 200 5025 "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 24.186.85.** - -

[18/Sep/2005:10:13:58 -0700] "GET /undefined HTTP/1.1" 302 298 "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 66.85.48.** - -

[18/Sep/2005:10:13:58 -0700] "GET /files/scriptBREAKFRAMES.js HTTP/1.1" 200 48 "http://example.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322)" 24.186.85.** - -

[18/Sep/2005:10:13:58 -0700] "GET /images/equipment/skb106DJsmall.jpg HTTP/1.1" 200 6738 "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 24.186.85.** - -

[18/Sep/2005:10:13:58 -0700] "GET /images/equipment/ortofonGTsmall.jpg HTTP/1.1" 200 6926 "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 24.186.85.** - -

[18/Sep/2005:10:13:58 -0700] "GET /files/scriptEMAIL.js HTTP/1.1" 304 - "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 24.186.85.**

[edited by: jdMorgan at 2:43 pm (utc) on Sep. 19, 2005]
[edit reason] Example.com [/edit]

jdMorgan

6:02 pm on Sep 18, 2005 (gmt 0)

[18/Sep/2005:10:13:58 -0700] "GET /undefined HTTP/1.1" 302 298 "http://www.example.com/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)" 66.85.48.171 - -

OK, so there is a direct client request for the non-existent /undefined.html file, which means that this is *not* some problem with your server. It could be that you have a link on your home page somewhere that refers to this /undefined.html page, but it does not appear to be a server or .htaccess error.

However, the response to that request is a 302-Found response, which is a *big* problem, because it tells client browsers --and more importantly, spiders-- that the page does exist, but has moved.

If you are using custom error documents, defined in .htaccess or httpd.conf, make sure that they are defined as local URL-paths only, and not as canonical URLs. That is:

Wrong:


ErrorDocument 404 http://www.example.com/error404.html

Right:


ErrorDocument 404 /error404.html

If a canoical URL is given, the server produces a 302 redirect -- as clearly described in the ErrorDocument documentation [httpd.apache.org]; This can lead to nasty problems with your search engine listings.

Also, check your server's error responses using the Server Header Checker [webmasterworld.com] after making any change which could affect error responses.

Jim

[edited by: jdMorgan at 2:44 pm (utc) on Sep. 19, 2005]

joyryde

6:33 pm on Sep 18, 2005 (gmt 0)

Before we even posted this topic, we searched every file on the site for the word "undefined" and it exists nowhere on the site at all, thus there is no link to it.

jd01

6:50 pm on Sep 18, 2005 (gmt 0)

If there is no link, it must be a direct request... Which means someone is either checking your 404 handling (usually a SE), looking for a hole in your system, testing a new spider/robot/scraping script, or is really stupid and keeps typing undefined as a URL for your site (my guess is it is not this).

The IP in your logs (most recent post) is registered to WhiteHorse Communications. You can find more information at dnsstuff.

None of this will correct the 302 status. See jdMorgan's previous post RE:error documents -- My guess is if you have not 'hand set' one in your .htaccess, you are using the control panel to set it, and it is setting the location as a canonical location http://yoursite.com/your404doc.html rather than a relative location /your404doc.html. If this is the case, you will need to turn the custom error pages off and set the location in the .htaccess yourself if you wish to use a custom page.

Justin

joyryde

1:54 am on Sep 19, 2005 (gmt 0)

Thanks for all of your help so far :)

The undefined error is coming from hundreds of different IP's.

I guess I should just use a redirect in .htaccess to send those requests to the home page, as we can't figure out why it's doing that.

As far as the error pages go, here's what I have in .htaccess:

ErrorDocument 404 /files/error.htm
ErrorDocument 403 http://joyryde.example.com/banned.htm
ErrorDocument 401 /v-web/errdocs/401.html
ErrorDocument 500 /v-web/errdocs/500.html
ErrorDocument 400 /files/error.htm

[edited by: jdMorgan at 2:45 pm (utc) on Sep. 19, 2005]
[edit reason] Example.com [/edit]

joyryde

2:26 am on Sep 19, 2005 (gmt 0)

not to continue to annoy you, but here's another example, out of about 60 from the last hour, showing that it is almost every IP that uses the site...

ACCESS:

[18/Sep/2005:19:14:25 -0700] "GET /undefined HTTP/1.1" 404 8018 "http://www.example.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" 24.10.247.** - -

ERROR:

[Sun Sep 18 19:14:25 2005] [error] [client 24.10.247.**] File does not exist: /home/deals/public_html/undefined

I've looked everywhere on the site, no mention of the word undefined in any file, and nothing major has been changed on the site lately ecept .htaccess when we re-direct pages or ban IP addresses.

[edited by: jdMorgan at 2:46 pm (utc) on Sep. 19, 2005]
[edit reason] Obscured specifics per TOS. [/edit]

joyryde

2:36 am on Sep 19, 2005 (gmt 0)

in addition, I don't seem to be the only site having this issue.

[I've found threads on two others forums discussing this problem.]

Unfortunately for me, nobody on those forums ever helped those people with an answer...

[edited by: jdMorgan at 2:47 pm (utc) on Sep. 19, 2005]
[edit reason] No URLs, please. See TOS. [/edit]

jdMorgan

3:32 am on Sep 19, 2005 (gmt 0)

This sounds like a malicious user-agent making bad requests.

Please review my comments on correct ErrorDocument syntax above, as your 403 error page will return a 302-Found status, rather than the intended 403-Forbidden status with the code you have posted above.

Also if you are using php, verify that there is no possibility that a malformed request might cause your scripts to produce a dynamic page with a link to /undefined.html on it. However, I suspect this is a malicious and badly-coded user-agent as noted above.

Jim

joyryde

3:52 am on Sep 19, 2005 (gmt 0)

I host the banned page on an external website because it gets so much traffic that I don't want to use the bandwidth.

As far as being a malicious agent, why would there be so many different IP's from all over the world giving the same error? (I don't have any php pages so it's not that).

Here's just a small portion from a small portion of 1 day, see all of the different people?

[Sat Sep 17 20:39:14 2005] [error] [client 216.**.232.213] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 20:42:43 2005] [error] [client 69.***.104.168] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 20:44:04 2005] [error] [client 70.***.93.249] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 20:44:08 2005] [error] [client 216.**.232.213] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 20:50:53 2005] [error] [client 24.0.***.15] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:20:15 2005] [error] [client 216.***.250.16] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:27:54 2005] [error] [client 69.***.246.168] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:31:53 2005] [error] [client 38.***.239.3] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:35:07 2005] [error] [client 201.***.245.172] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:36:17 2005] [error] [client 24.***.53.163] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:48:48 2005] [error] [client 24.**.148.220] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:49:38 2005] [error] [client 24.**.148.220] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:50:05 2005] [error] [client 70.**.245.242] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:50:16 2005] [error] [client 24.**.148.220] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:50:55 2005] [error] [client 68.***.148.191] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 21:51:27 2005] [error] [client 24.***.37.173] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:04:43 2005] [error] [client 71.***.70.22] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:05:24 2005] [error] [client 209.**.110.156] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:05:41 2005] [error] [client 69.***.57.13] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:09:35 2005] [error] [client 70.***.156.78] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:18:21 2005] [error] [client 64.***.27.228] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:19:32 2005] [error] [client 201.***.0.211] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:21:59 2005] [error] [client 24.**.105.159] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:38:06 2005] [error] [client 216.***.250.17] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:38:06 2005] [error] [client 64.***.27.228] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:38:19 2005] [error] [client 64.***.27.228] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 22:41:22 2005] [error] [client 71.***.193.11] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 23:04:08 2005] [error] [client 167.***.147.135] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 23:31:50 2005] [error] [client 207.***.217.138] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 23:38:38 2005] [error] [client 207.***.116.132] File does not exist: /home/deals/public_html/undefined
[Sat Sep 17 23:46:43 2005] [error] [client 222.***.39.51] File does not exist: /home/deals/public_html/undefined

...again, THANK YOU for your help!

[edited by: jdMorgan at 2:51 pm (utc) on Sep. 19, 2005]
[edit reason] Obscured specifics per TOS. [/edit]

jdMorgan

3:13 pm on Sep 19, 2005 (gmt 0)

> Here's just a small portion from a small portion of 1 day, see all of the different people?

These are probably due to one computer (or a few computers) infected by a worm or running a robot, and connecting to your site through various open proxies. Don't assume that they are 'people'.

Avoid external ErrorDocuments at all costs. The simplest way to reduce bandwidth is to make the error page *very* short and place both a plain-text link and a 5-to-10-second meta-refresh on it that lead to a longer explanatory page. Human users will click the link or be meta-refreshed to the 'long' 403 info page, while robots and malicious user-agents will see only the 403 response and ignore the meta-refresh.

Again, if you use a canonical URL to get to an ErrorDocument, your server will not return the proper error code, it will return a 302 redirect. This may, in fact, be why these pests keep coming back to your site, because you are telling them (with the 302 response) that the /undefined document exists! Also, since a 302 redirect itself consumes bandwidth, you'll be much better off just doing this locally.

Example short 403 error page:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"><meta http-equiv="Content-Language" content="en-US"><meta name="robots" content="noindex,nofollow"><meta http-equiv="Refresh" content="5;URL=/403info.html"><title>Access Forbidden</title></head>
<body text="#000000" bgcolor="#F7FFFF" link="#0000CC"><center><font face="Arial,Helvetica"><p><h1>Access Forbidden</h1><p>Click <a href="/403i.html">here</a> for more information if you are not redirected automatically.</font></center></body></html>

Also, I suggest that you do not tell anyone that they are 'banned' or why on your 403 information page. Your 403 error page only needs to say that the access is forbidden, and suggest (to humans) that it might be an error on your server, or that they might want to try again by visiting your home page. If you tell malicious persons that they are banned, some of them will take it as a challenge to keep hacking, or get mad and set you up for a DOS attack. Simple solution: Do not give information to the enemy.

Jim

joyryde

1:54 pm on Sep 20, 2005 (gmt 0)

I realized just now that this error began happening when we upgraded our hosting account to get more bandwidth and space. Could the host have caused this?

It is not slowing down at all:

[Mon Sep 19 21:20:14 2005] [error] [client 68.**.26.10] File does not exist: /home/dealsc/public_html/undefined
[Mon Sep 19 21:21:36 2005] [error] [client 69.***.52.99] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:30:13 2005] [error] [client 69.***.52.99] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:31:05 2005] [error] [client 24.***.220.189] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:32:32 2005] [error] [client 70.***.52.136] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:34:22 2005] [error] [client 211.***.236.11] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:36:17 2005] [error] [client 64.**.130.171] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:39:01 2005] [error] [client 64.**.130.171] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:39:42 2005] [error] [client 69.***.52.99] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:41:16 2005] [error] [client 24.**.175.218] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:42:04 2005] [error] [client 24.***.194.117] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:42:23 2005] [error] [client 68.***.182.189] File does not exist: /home/deals/public_html/undefined
[Mon Sep 19 21:48:11 2005] [error] [client 128.**.240.32] File does not exist: /home/deals/public_html/undefined

[edited by: jdMorgan at 1:45 am (utc) on Sep. 21, 2005]
[edit reason] Removed specifics per TOS. [/edit]