500.13 - ASP 0147 Errors -- Crazy!

500.13 Iis Asp_0147

[ihmatt]

For the past month, we have had some serious problems with IIS and can't seem to crack this one.

We run a very high traffic site and recently, we installed a new server (monster Dell) with Windows 2000 AS on it. Shortly after, the problems started.

Problem: Sporadically, IIS just dies. When it does, it stops responding to all requests and shows a 500.13 Server Too Busy error. Server utilization falls to 0/1%.

Symptoms:
-We have noted no consistency in the problem.
-Server utilization is always fine -- in the 30% range
-Once it does crash, it will still serve HTML or JPG, and other static pages.
-I wanted to think it was traffic-related (we are busiest between 8am and 6pm), but it crashed on a Sunday night recently with hardly any traffic on it.
-A reboot will "correct" the problem temporarily.
-Around the time of the crash (after), we see ASP_1047 and ASP_1048 errors in the logs.

Does anybody have ANY ideas?

We are completely dumbfounded and have exhausted everything we know and can't find anything on the 'Net to help us.

[IanTurner]

Is the server up to the latest service pack?

[ihmatt]

Ian,

Yes, all patches have been applied.

M

[Dreamquick]

If you can serve static pages then IIS is fine, instead it sounds like the ASP engine is what's causing you the problems.

Whenever I've seen the ASP engine fail before it's nearly always because of memory leaks - sometimes it can be from code that doesn't clear objects when done with them, but most of the time it's the result of using a component (DLL etc.) that has a memory leak. I've also seen a site exhibit the same behavior when using MS Access for their back-end db, but that's unlikely to be relevant to you!

If it was a memory leak causing the problem it would also explain why a reboot fixes it temporarily.

Out of curiousity I refined a search for those error numbers and spotted this;
msdn.microsoft.com/library/default.asp?url=/library/en-us/script56/html/vsmscsyntaxerrors.asp

Type: Syntax
Code: 1048
Hex: 800A0418
Description: Must be defined inside a Class.

I have to admit I had quite a bit of trouble working out the 0147 aspect - the only place I could find it listed was on a related page which listed the Hex codes in addition to the decimal codes...

Type: Syntax
Code: 1047
Hex: 800A0417
Description: Expected 'Class'

Stupid question I know, but did you copy the contents of the old server or did you copy the material from somewhere else (maybe an internal test server)? If it worked before and doesn't work now then my immediate thought would be that the files you copied aren't the same.

If you're using your own DLL to house the business logic then those would be my prime candidate for what's different.

- Tony

[webdevsf]

What isolation model are you running under? You might try putting high isolation (as a temporary solution) so that you don't have to reboot when it dies.

Do you have CPU throttling on? If so, turn it off. :)

[ihmatt]

Webdevsf:

We have tried it on high isolation for 2 weeks and it crashes still. We've also had it on low and medium isolation. Neither helps.

Throttling has never been turned on.

[ihmatt]

>>the ASP engine is what's causing you the problems.

Agree.

>>Whenever I've seen the ASP engine fail before it's nearly always because of memory leaks - sometimes it can be from code that doesn't clear objects when done with them, but most of the time it's the result of using a component (DLL etc.) that has a memory leak. I've also seen a site exhibit the same behavior when using MS Access for their back-end db, but that's unlikely to be relevant to you!

We don't use any custom DLL's to run the site. Just whatever is in IIS/Win2k. However, I've noted a few places talking about memory leaks. Where or how would I check for these?

The only ASP errors we get are 500.13 errors. That's it. In the logs, it shows the ASP_0148/_0148.

Thanks for your suggestions so far.

[bytor]

I'm Matt's partner and the geek responsible for keeping the site running and doing all the programming.

Isolation is currently set to Low to minimize CPU overhead (we're running about 30% on a Dell 2650 with a pair of 3.06Ghz processors). I'd previously tried High isolation (as well as Medium), but the same error condition happened in both of those scenarios, too.

Though my understanding is that connections and recordsets get closed at the end of an ASP whether it's done manually or not, I've at least made sure they're closed manually at the end of the busiest pages.

Since we can use a LOT of different recordsets within any given page, it's often tough to find all of them and make sure we're explicitly closing them. I'm looking for a way to step through open connections in a FOR EACH type setup, but am finding no variables to use for this. If I can come up with something, I figure adding something like this to our bottom include would at least make sure it's not a too-many-connections problem.

for each item in recordsets
item.close
next

I uninstalled/reinstalled IIS last week. No change.

I also noticed this morning that we do keep a LOT of session variables open, some of which are very large. I will try explicitly destroying these as I'm done with them, but am not optimistic it'll help. The server has 2 gig of physical memory in it and typically has 1.6 gig of it available.

We've been considering upgrading to Server 2003/IIS6, as a possible solution, but I'm pretty hesitant now after reading of someone else encountering this same problem after doing the upgrade.

Although if IIS6 can be setup to restart ASP automatically every time it dies, at this point I'd consider it a "good enough" solution.

I should point out, if Matt hasn't, that we used to equate this problem with server workload (it gets a little over 1 million page views per day), but we had the same crash happen on a Sunday, when traffic is about 1/4th its weekday volume.

Other relevant info:

We're running exactly the same code we were running on our previous webserver. The previous machine was spending too much time near 100% CPU utilization (but still running fine) so we replaced it with something with a lot more oomph as the first step in a future project of setting up the site to run on multiple webservers behind a Coyote Point E350.

We're running no DLL's other than what came with W2K.

This machine does nothing but run IIS and Primalscript.

[bytor]

Dreamquick, you guessed correctly on the db. Our backend is SQL Server 2000 (all patches applied) running on a separate machine. ASP talks to it exclusively through DSNless connections. We've noticed nothing out of the ordinary at all on the db server (typically runs at 20% or lower utilization).

Matt says he once observed a massive spike in CPU utilization on the webserver right before a crash, but I've watched it crash with no spike.

I'm also running Performance Monitor. 4600 current sessions, 1200 current connections (session timeout is 20 minutes, connection timeout is 900 seconds, unlimited connections and http keep-alives allowed) and no queued requests, until ASP dies.

Nothing appears in the Event Log.

[bytor]

Quick related question.

would my_conn.close also close all of the recordsets opened with my_conn?

[webdevsf]

Set all your objects to nothing after you are done with them. No exceptions. This is an relatively simple fix and you can probably do it in a few hours.

Asp does not always close objects when you expect it to.

[bytor]

We'll go through the asp and make sure all recordsets are explicitly closed then set to nothing. Actually, this'll take days, as we're dealing with tens of thousands of lines of code. Might even be worthwhile to use PERL to do the job.

Will also make sure session variables are dumped when they're no longer needed.

Thanks for the feedback everyone.

I wish ADO had a Recordset collection, but just confirmed on MSDN that it doesn't. That's too bad. Would've made it really easy to just close every open recordset without having to find and hardcode their names.

[webdevsf]

You are not storing connection objects in the session or application are you? If so, don't. Connection pooling will take care of this.

BTW, make sure all objects (not just recordsets) are closed and set to nothing. This includes connections, scripting dictionaries, etc.

[plumsauce]

a real shot in the dark, but ..

i noticed that you mentioned moving servers.

the new server has cpu's that are capable of
having hyperthreading turned on. and i would
bet that it is turned on as delivered out
of the box.

have you tried turning off the hyperthreading?

it's not really helping you because you are
already running smp at the physical level.

[makeupalley]

are you using the built in session object?