Drupal Warns – Most Drupal 7 Sites Compromised Unless Patched on Oct 15

via Sucuri

[lorax]

[blog.sucuri.net...]

You just never know what vulnerabilities lie in your CMS. It's part of the risk. A small one - IMO - for the flexibility and time savings we gain by using a CMS.

[edited by: lorax at 6:10 pm (utc) on Oct 30, 2014]

[engine]

Oh dear, this is going to leave many webmasters dazed and confused, but worse still, any that are not aware may have a compromised site and not realise it.

Simply updating to Drupal 7.32 will not remove backdoors.

If you have not updated or applied this patch, do so immediately, then continue reading this announcement; updating to version 7.32 or applying the patch fixes the vulnerability but does not fix an already compromised website. If you find that your site is already patched but you didn’t do it, that can be a symptom that the site was compromised - some attacks have applied the patch as a way to guarantee they are the only attacker in control of the site. Drupal Core - Highly Critical - Public Service announcement [drupal.org]

[rish3]

It's a pretty bad situation...it's not a typical SQL injection problem where you can pass arbritrary data to a SELECT call.

The hole is wide enough that you can easily craft INSERT or UPDATE statements in a single url, and because of how drupal works, immediately run arbitrary php code as whatever uid the webserver is running under.

It is a one-line fix, provided you caught it before you were exploited.

[JesterMagic]

It is a one-line fix, provided you caught it before you were exploited.

The problem is you can't be 100% sure you caught it in time.

[incrediBILL]

WordPress finally has some competition! :)

[lorax]

Oh iBill.... when will you ever learn - there is no competition for WordPress. :)

[ergophobe]

Yeah, it's more like Heartbleed than your average vulnerability. Oy veh!

[ergophobe]

People are calling this one Drupageddon...

Some resources from respected sources

[acquia.com...]

[modulesunraveled.com...]

[jebernier]

Totally not true. Not every Drupal 7 site was impacted.

[jebernier]

More info on how to tell if your site was hacked and how to fix:
[modulesunraveled.com...]

[ergophobe]

Not every site was impacted, but the impact was high. I was just back from vacation and digging myself out and didn't get to this right away and almost all my Drupal 7 sites were hit.

It's not immediately obvious. The sites run. They tend to be updated to the newest version. But there is malicious code buried in various files. The server needs to be torn down, rebuilt and all sites rebuilt from known backups.

This was exploited broadly in the wild very quickly.

[ergophobe]

Tweaked the title though I have yet to find an unpatched site that isn't compromised

[LifeinAsia]

So far, I have not been able to find any evidence of being compromised. I checked all the git commits and all of the files are ones that I personally added/modified.

I tried applying the Hotfix described at [drupalize.me...] but now I get a WSOD. (Luckily, did this on a DEV server instead of Production...) Compared the old/new database.inc files, and the modified one looks as expected from the patch file.

Apache error message: PHP Fatal error: require_once(): Failed opening required '[Drupal root]/includes/database/database.inc'
The file permissions are the same as before applying the patch. If I delete the patched file and restore the unpatched file, all is well. (Other than still being unpatched...)

[LifeinAsia]

Bizarre- if I update the file manually with the change, everything works fine...

[ergophobe]

That is bizarre - that's the only file that really changed with the hot fix

If all the git commits are files you changed and git status doesn't show any modified or new files, you're in the clear. Lucky you. given the nature of this, they say you have to just kill the server in question and start over with known good copies and fresh passwords. Major PITA

[ergophobe]

By the way LifeinAsia... are you now glad that I made you "git" with the program?

[LifeinAsia]

are you now glad that I made you "git" with the program?

Definitely! That was one of my first thoughts when I was reading one of the articles that mentioned this issue is one of the best arguments for using version control.

Still need to bite the bullet soon and do the core upgrade. But at least the Hotfix well help me sleep a little better for a while.

[jamie]

we run two sites on drupal. a complete rebuild is something i am trying to avoid.

so far every test i have run has returned negative. these include:

1. several security modules, including site_audit, drupalgeddon, hacked, security_review.

2. i have the code base in git and nothing was changed.

3. only the files directory is writeable by the webserver and there are no suspicious files in there (and i have now disabled php in that dir). (a friend was hit because his modules dir was writeable by the webserver).

4. there are no suspicious users/roles in the database.

5. i am now monitoring port 25 and the mailqueue.

6. i have updated all pw (drupal and mysql).

i am 95% certain my server is ok.

a complete rebuild would be a nightmare; i have pre-oct 15th backups; but we produce so much content that i'd have to write some pretty awsm scripts to migrate all updated data and check it for validity. plus how could i be sure they haven't saved something to the server which doesn't get copied over to the new one. (it's like changing to a new pc - you just sync your docs over - most of which haven't been looked at for 10 years).

one comfort is that we don't run any ecommerce through drupal nor have a user base apart from staff; so the only thing they could do is steal content (which is easy enough to scrape anyway) and/or use site for malicious software / email. we have regular pci scans which help protect against this.

nonetheless.... i've aged in the last week. i am now going through the server access logs to see whether anyone tried to get in.

i think i want someone to pat me on the back and say reassuringly "there, there jamie, you don't need to reinstall" - but every drupal expert on the web says otherwise :(

[jamie]

just to update, i am restoring anyway for peace of mind. ouch.

i've been through every web-writeable dir on the server and checked them for malicious content - all negative. so now it is just a question of migrating/updating the db from the backup.

wish me luck :)

[ergophobe]

Been out for a few days. I would say if there are no new files in git, you're okay. There's a pretty distinct fingerprint on the infected sites I've seen.

That said, you would want to restore from a database backup even so, because in some cases they try to inject stuff in the DB

[jamie]

i am in the middle of it right now. the trickiest thing was getting all the new photos from updated content to sync across to the older version of the db (we are constantly publishing new content and updating old). drupal's file management system is quite complex and involves a lot of tables ;)

however, i have the import scripts now, they work fine on local. i have uploaded to the staging server and am running them right now. if that goes well then i'm going to bring down apache, sync the new db to the live server and bring everything up again later today.

it's been a very hectic 3 days, but it was worth it for peace of mind. we have a lot of ecommerce (even though it is not run through drupal) and i need to know the server is ok, because even though we don't store cc data, we still capture and pass through to the gateway. i think i'm being paranoid, but btbsts :)

it's also nice (if you like that sort of thing lol) to see what is involved in a complete restore. all the scripts are documentation are now in place, which will really help the next time.

[ergophobe]

I don't think you're being paranoid. If credit card numbers are coming through the system, I'd say it's absolutely necessary. You might want to expire all passwords too.

You should be able to just trnasfer files within the same file structure. Everything is root relative within Drupal and the file tables have data of a known format - you could vet that pretty quickly and then just transfer over the files, limiting the transfer to common image types.

[jamie]

tks ergophobe, doh! me. i was trying to do it all through node hooks and file_copy, file_save_data, etc; when actually all i needed to do was sync over the file_* tables themselves because the file system hasn't changed. anyway, a bit of hacking later and all files are synced across beautifully. tks :)

[ergophobe]

Excellent - like I say, though, make sure you're only bringing over media files - the attack can put php and javascript files in your files directory and since you aren't typically tracking those in git, you might not notice them.

So I would do a "find" on the server to get a report of all files that are not .jpg/.png/.gif

And maybe use find to see if any files have had the executable bit set.

[jamie]

nice tip again, tks. to be safe i've just chmod all files in there og-x and checked for extra file types. one of my earlier checks involved scanning files just for *.php, but doing it the other way against known and safe file types is better.

just as an aside, here is a classic bit of self-induced and unnecessary panic. i ran drush up drupal as root instead of my normal ssh user. the root file partition is tiny, so drush broke in the middle saying "not enough file space" - shiiiiiiiii...

so i did a quick ls of the htdocs dir and it was empty apart from just the drupal files - normally we have lots of directories with ecommerce apps, client folders, etc, etc - i completely panicked - the whole site was offline - rang the data center and initiated a restore of that dir from last night's backup. i'd forgotten that drush copies the entire webroot into a backup folder during it's update process and then copies it back out again. 2 hours of intense anguish whilst the data centre restored 300,000 files. i can laugh about it now - all apart of a webmaster's job - but i got a few extra grey hairs yesterday that's for sure.

all these stories make me sound rather sloppy lol! - i'm not usually, but the last few days (after discovering the drupal issue) have been very intense :)