Forum Moderators: open
I am running asp on a Win 2000 Server machine.
After 1 year without any troubles now my ASP pages suddenly are not working due to an " Out of process application" and/or some other vague event-messages.
Restarting solves the problem for 1 or 2 days.
I am using Access 2000 but only a very few visitors will query the dBase, although it is rather great (more than 100.000 records).
Less than 50 visitors daily.
Is there any solution?
Could it be of any help to make "blank" queries impossible?
(I mean searching inside my database leaving all the text- and dropdown boxes empty, so giving in fact all results which are just made impossible by using a "TOP"-select set on "40"?
Of course someone may suggest to use mySQL but I am rather unfamiliair with it. Do I have to change the ASP-scripts when changing to this database?
(which is very hard as I use more than 100 ASP pages, not made by myself).
Thanks for the help!
Peter.
You will need to be more specific with your error messages if you want help here. Please post the event log errors.
Try that, if it still fails then read this;
[webmasterworld.com...]
Bear in mind that even with perfect code your problems could still be down to Access - it was designed as a personal desktop style database, not a shared multi-user database and so it will sometimes cause you great hassles when integrated into a website. Personally I wouldn't touch access-on-the-web with a very large stick.
Of course someone may suggest to use mySQL but I am rather unfamiliair with it. Do I have to change the ASP-scripts when changing to this database?
Off the top of my head you'd have to change at least the initial connection configuration so you talk to a SQL engine rather than Access, secondly you'd have to rewrite any of the queries which use Access's proprietory structures rather than industry standard SQL.
- Tony
Application Log:
The COM+ Event System detected a bad return code during its internal processing. HRESULT was 800706BF from line 42 of .\eventsystemobj.cpp. Please contact Microsoft ...
And some same alike.
System Log:
The Remote Procedure Call (RPC) service terminated unexpectedly. It has done this 1 time(s).
Out of process application '/LM/W3SVC/3/Root' terminated unexpectedly.
The server stop serving requests for application '/LM/W3SVC/3/Root' because the number of Out of Process component crashes exceed the limit. For additional ....visit ....
The RDP protocol component X.224 detected an error in the protocol stream and has disconnected the client.
.........................................
I make a connection with Terminal Services and the strange thing is that logging the first time I not able to see any desktop items an I am not able to read the Logs.
I have to log a second time and then I am able to do so.
...W3SVC... is a log directory.
Bad code as one suggested ... why did it work one year without troubles (and no code changes!)?
Can it be due to a hack? It also seems to happen the same time of the day more or less.
Thanks for help, Peter.
Can it be due to a hack? It also seems to happen the same time of the day more or less.
It could be some kind of attack as you suggest, but if this was the case I'd expect patching the box to fix any problems that your average cracker would use.
Alternately it might just be that someone is using the site in a way that you didn't anticipate for the first time - maybe they just login at a certain time once a day? If this was the case you would be able to tell from the logs what's happening around that time.
- Tony
1. MSAccess ... is NOT suitable (and no self-respecting tech from Microsoft will dispute this) for the work you are making it do. 100K+ pages is really, REALLY pushing it. Combine Access tables of the size you mention with multiple simultaneous queries and I'd expect to see smoke coming out of your server.
2. From what I can tell, all your code is in ASP. Your ASP pages really need to be little more than wrappers for compiled COM/COM+ components written in VB6, VC++6 or Delphi 4/5/6. The ASP collects information about the request, database connection strings, whatever, works out which component to pass the required task to and then leaves it to the selected component.
3. Your weird RPC errors could indicate an intrusion, or they could indicate a usage scenario, as already noted.
I once made quite a lot of money fixing a site that would crash if one user tried to submit a credit card payment while another user was executing a (naively implemented) description search on the site database. This site used Access and all code was in ASP (sound familiar?). At the end of the job, the site was running SQL Server and compiled components.
Because the ASP is just sitting on your box in clear text, it's awfully tempting for a successful hacker to mess around with it (no decompilation/analysis necessary).
Whatever the problem is, your approach is the same (again, already noted by previous posts in broad terms):
3.1 Determine from your logs the candidate page/pages that seem to be causing the problem.
3.2 Audit them thoroughly (all original code?), noting any database accesses these pages perform. If the code looks weird or if two or more of the pages are hitting your DB, I think you've found your problem.
Symptoms we were experiencing had to do with very slow response times from the server due to processes running and hogging up system resources. The RPC Network Server is busy error messages when accessing a database. Server unexpectedly quitting or closing a connection while editing sites. Usually occurred in the early morning hours (PST).
Go to Symantec and read the following...
[securityresponse.symantec.com...]
The necessary patch from Micro$oft can be found at:
[microsoft.com...]
The problem persisted once per day since July 30. I installed the patch Aug 4. Hopefully by this time tomorrow all will be well with my machine. I will post a follow-up by Aug 6.
You would have laughed at us running around the server room like headless chooks after I yelled out your post.
Our boxes already had the patch (thanks to mindless manual execution of Windows Update once a day), but we explicitly shut down DCOM on all of them and retested the applications just the same (thank you SQL Server for not using it except for replication).
We closed the NETBIOS ports (137-139) long ago, but 135 was still open. Even with the patch, the disabling of DCOM and removing all DCOM protocol support (go to [grc.com...] for a quick, benign probe) the damn thing remains open on all machines.
(i.e. Just begging the world's seemingly infinite multitude of snot-nosed, girlfriendless script kiddies to choke our bandwidth with pointless intrusion attempts.)
I discovered in 01.08. that asp pages on my server dont work and that html do work?!
I restarted the server and everything was fine.
Until next morning - same story.
And next morning after that again!
I've noticed the pattern ("morning") and i was thinking that is some kind of an overload...
But when i opened event viewer i saw same thing you described here.
Now, i have installed patch (M$ did not included it in service pack that normally is enough for normal protection?) and now we'll wait and see.
Please, Peter and Krobra, would you be so kind to post here if patch worked?
Thanks.
You saved my life guys :)
A very solid comment, however ...
Compiled components are useful, but IMO they are the next step once you realise that the bottleneck is actually ASP and not your choice of database engine and/or server.
Typically you don't reach that stage until you have several hundred thousand pounds worth of hardware sitting in your server room, because after poor choice of database the next bottleneck is your server hardware / platform and either it's processors or it's I/O.
- Tony
I think I've been in a server room with >= "several hundred thousand pounds" worth of servers twice (for those of you who are currency-challenged, we're talking millions of US dollars, zillions of OZ dollars, all the NZ dollars ever minted or enough Thai baht to send our planet spiralling into the sun) - one a telecoms data facility and the other a banking data facility, i.e. we're talking BIG business.
With respect, the performance, scalability, security and low-level system control/functionality advantages of compiled components (especially on MS Windows/IIS) kick in way before that.
How do you connect to a payment gateway in script? You use a COM/COM+ component or ActiveX control (two shades of the same thing - both compiled).
How do you dynamically block hostile remote IPs (by definition your firewall has to let ports 80 and 443 through) on the cheap before IIS gets an opportunity to choke on it? You write an ISAPI filter - another compiled component.
How do you obtain IUnknown references to ADO and MSXML2 objects in script? You don't - you're stuck with IDispatch and Variants; ugly, slow things that they are (and if you don't think there's a _noticeable_ performance difference, you haven't compared implementations).
Basically, compiled components allow you to act like you've got Euros coming out of your ears on gear that costs much less. Granted, it takes greater discipline and planning to design a site based on compiled components than one that uses script exclusively. In my experience the result is a better (faster, more scalable, etc., blah) product, every time.
IMHO =)
Rock on,
Andrew
I seem to have been a little too imprecise & aggressive (I'm blaming it on being a hot day and posting just before lunch) - what I should have probably said was;
While converting *your own code* (not 3rd party add-ins) to a compiled format instead of leaving it as a script does have great benefits there may be simpler routes which achieves some of the benefits of the complex solution.
For instance if you are looking to speed up an operation the first thing you should always do is examine where your bottlenecks are occuring - otherwise your solution might not be addressing the problem and so would have negligible effect.
- Tony
I found it to be a combination of lax MS security (patch required) and an IRC hack/trojen that caused the inital errors and crash of ASP and Http servers....
We run monitoring software sao I was aware imeadiatly after the trogen flooded the applications
It is a variation of the DDoS-Kaiten.gen virus...
You can see if you have it by the presence of a file called system.exe in your winroot/system32 folder To verify you can save a copy and rename it with the .txt extension and see all the nice flood DOS programming included....
I have checked our other machines and there is no presence of the system.exe file so I know this was not a mod to an existing....
When we found it our virsion of netshield which was patched 3 days prior would not detect it... But when We updated our dat to august 1 dat file it say it right away and removed it....
It also modifies your registry key hklm/software/microsoft/windows/currentversion/run and adds the location of the system file to load on windows start....
After a few hours it has flooded several services and can not only cause a denial of service but also gives the hacker access to run commands via a chat window.....
This can be prevented by updating to the most current ms patches.....
Patching and removing this vaiant does work but we have found there are other issues no handled by the patch.... We have noticed that some services such as event viewer(details view) and multi pane display of some folders is effected....
That is all the information we have at this time...
Hope it helps.
Brett V.
Is there any way to close/secure port 135 besides firewall, shutting it down or putting a NAT?
Any suggestions?
And do you think M$ patch is enough?
Thanks.
1.1 Start->Run->C:\winnt\system32\Dcomcnfg.exe
1.2 IGNORE the Applications tab (most are red herrings)
1.3 click on "Default Properties" tab
1.4 uncheck "Enable Distributed COM on this Computer"
1.5 click on "Default Protocols" tab
1.6 delete ALL bindings with the "Remove" button
1.7 reboot
1.8 test EVERY PAGE of your site
Please note that our sites are composed entirely of HTML/XML-generating COM components using ADO and MSXML2 with extensive use of XSLT scripts, and we were _completely unaffected_ by this reconfiguration. Our databases are on separate machines, so we were a little worried that maybe ADO over the network uses DCOM. Happily, this is not the case: it turns out that ADO uses the target DB netowrk transport protocol - in our case (SQL Server), it uses a Microsoft variant of the old Sybase TDS protocol, operating on ports around 1434, 1435.
I've already complained in this thread that the port is still open, which means you'll be immune to any attempted compromise but you're likely to be the recipient of a whole lot of exploit traffic nonetheless. Whether this has an appreciable effect on your (and our) bandwidth, only time will tell.
The problem is, without staring at a packet sniffer (not the most edifying of experiences, but sometimes necessary), you'll never know how much/often you're being probed because the web logs only care about TCP ports 80 and 443, and this attack is coming in on 135.
You can close the port using a software firewall (Zone Alarm comes highly recommended), but you'll have to spend the night configuring it not to block everything else as well.
2. Does the patch work?
The exploit is one of the oldest in the book, a "buffer overflow". A buffer overflow is what you get when you pass a parameter to a method that is too large for the buffer provided to cache and process it. You usually find them in C code, because C requires you to explicitly manage all memory, and you usually find C code in very critical places (OS kernels, etc.), owing to its compactness and speed of execution.
Your C programmer thinks something like this to herself (yours truly is guilty of this too): "No idiot would be stupid enough to pass more than [x] bytes into this method - [x] bytes is the most you'd need to properly represent the information it requires! Therefore I don't need to test the length of the input data before processing it."
No idiot, of course, except for someone who is deliberately being "stupid" in order to crash your code.
Microsoft programmers (and to be fair, LINUX, Solaris, MaxOS, etc. also - although arguably to a lesser extent) think like this a fair bit. Sometimes an unchecked buffer overflow occurs in a critical place that leaves the system in an indeterminate state susceptible to code insertion, and there's your exploit.
Fixing it is simply a matter of going back to the code and adding an input data length sanity check, so yes, I think the patch will work.
[edited by: Woz at 1:47 am (utc) on Aug. 17, 2003]
[edit reason] spelling [/edit]
anyone?
thanks
I am having the same problem. Have you managed to fix it?
Best regards,
Zel
