Automated Testing with your CMS

[ergophobe]

I've lately started using Selenium IDE to do automated testing. It's one of those things I've started a few times and then gotten impatient and dropped before I really got any benefit.

On the current project I'm working on, however, every time I add a feature to the site, I add a suite of automated tests. So let's say I add a gallery feature, my tests might go like this

-Login as site admin
- go to the edit page and verify that elements x, y, and z are present
- go to the view page and verify that elements x, y and z are present

-Login as site editor
- go to the edit page and verify that elements x and y are present, but z is not
- go to the view page and verify that elements x, y and z are present

-Logout (so anonymous visitor)
- go to edit page. Verify that it's a 403
- go to the view page. Verify that elements x and y are present.

Now let's say I've been doing this right from the get go, so every feature on the site has viable test data and a set of tests.

Now I add a new feature or I change something that impacts the galleries. Did I remember to update everything to take the change into account?

Old method: click around until I'm satisfied and hope I didn't miss something.

New method: open the Firefox Selenium IDE plugin, load the test suite for the site, start the tests, go have a cup of tea, come back and see if anything failed.

[edited by: ergophobe at 5:25 am (utc) on May 6, 2014]

[ergophobe]

So is anyone else using automated testing?

If so, what tools are you using and why?

If not, why not?

[coopster]

I'm not personally but I have worked with Quality Assurance teams that run automated testing on code after I've implemented. I couldn't even tell you at this time what tools were being used though.

Why am I not using them? Just haven't had the time to try some out and implement a plan! It's one of those things that will save you time in the long run but cost you time in the present -- someday I'll get there!

[ergophobe]

>>save you time in the long run but cost you time in the present

The cost in the present is lower than expected. I don't know why, but my first couple of efforts didn't go anywhere. I just couldn't get started. Once I did, I was surprised how little time it takes, especially to create new variations of an existing test.

Next time I see you, I'll have to give you a demo... it's like version control. Once you've worked with it, it feels "unsafe" to work without it. Unlike version control, though, the ongoing time investment is not negligible.

[LifeinAsia]

Some pushy guy keeps sending me e-mails to try and get me to start using that Selenium IDE thing. He's pretty trustworthy, though, so I guess I'm going to have to bite the bullet and try it out soon.

[ergophobe]

I think he's dodgy. I wouldn't trust him.

[ergophobe]

I wish we could all sit down in cafe once a month for demos and whatnot.

[lorax]

I like the demo idea as well. Hangouts or other ways of getting some time to interact with my peers is becoming more and more important. I started a Meetup to help with this but I don't get to see you yahoos except every few years...

Oh, and to answer the question, no I don't. Seems like an awful lot of time invested for something that seems like it needs to be constantly tweaked/updated?

[ergophobe]

lorax, quite the opposite.

I would say that setting up a suite of Selenium IDE tests takes maybe three times as long as testing it manually. So if it takes you 6 minutes to log in as three different users and scan two pages as each of them, you can set up the Selenium test in 15-20 mins.

Then once it's set up, you only tweak it when the test fails, and in that case, you surely want to know... otherwise why are you testing for it?

Now instead of 6 minutes, it takes me 30 seconds and runs completely unattended while you stretch your legs.

More importantly, have you ever worked out feature 1 and then moved on to feature 2, 3 and 4 only to cycle back to feature 1 and see it broken? You have no idea when you broke it. But your tests would have failed and since they are so easy to run now, you actually do and you caught it right off, saving hours of debugging.

So I tend to test for "element present" rather than "text present" because you might change the text fairly often, but I want to know that a site member sees a "Quick Reply" box, but an anonymous user does not. I want to know that a site admin sees a button with a given name/class/id

[ergophobe]

I just spent three hours creating a patch for a Drupal site that lets admin users arrange TinyMCE toolbar buttons and rows with a drag and drop interface.

I spent about 10 mins creating a test that would verify that the patch was installed and working, so when I upgrade and forget to apply the patch, my test will fail. It's not comprehensive, it's just enough to verify that the default behavior is absent.

[lorax]

Good points ergo. It seems that it would be prudent to create a clearly define SOP so the tool doesn't become a victim of poor QA practices. I can see how it would be very useful. Damn... another weekend lost to research and playing. ;)

[ergophobe]

That said, your initial reaction is not wrong. I think the way I see it is this: if things start to get so dizzyingly complex that you wouldn't really trust yourself to catch when something goes wrong, you need automated QA testing.

I can't imagine bothering on say a Wordpress blog with a small handful of exceedingly popular plugins. In that situation you are right - the risk isn't high enough, the reward not great enough.

When you have five different levels of user privilege, you've got custom code that hooks into a possibly fluid API and depends on the correct order of operations, etc, etc, then it just simply becomes too onerous to test manually unless you actually have a QA team... who will no doubt start by building automated tests.

BTW, I tried a few tools. Some seem like they might be more efficient once you learn them, but Selenium test files are actually just simple HTML, so easy to edit outside the IDE (so if you do a global change on the site, you could update your tests with search and replace). So I ended up defaulting there after wasting time trying "better" alternatives.

The title of this makes it sound Drupal-specific, but there's nothing in this video series that doesn't work on Wordpress or a static HTML page. It's quite old and Selenium has evolved, but the essentials are the same:

[nodeone.se...]