FlexAbility

There have been a couple of questions on the Google FlexUnit list recently about how to structure a project for unit testing. I’m putting my response here so that I can point others at this post in the future.

If you’re interested in doing testing as part of an automated build script, one of the best resources is Antennae. Antennae is an Allurent open source project and does much more than automated testing (compiling, etc.) but it provides a good example of one way to structure a project.

Also, I use a “modified antennae” approach for the Visual FlexUnit (VFU) project. Wheres Antennae tends to use Allurent-standard folder naming (“flex2″, etc.), I decided to use more standard folder names (“src”, etc.) for VFU, so that non-Allurent users would find it a bit easier to incorporate into their projects.

BTW, you might also want to take a look at my Testing, Debugging & Agile Methods page – you’ll find a lot of related links there, and I update it regularly.

HTH!

Here are a few UML (Unified Modeling Language) diagrams that show a bit about how Visual FlexUnit works.

All are simplified to enhance clarity.

This post is intended to complement my Highlights of Visual FlexUnit’s Code post.

1. TestCase Classes

When you use VFU to write visual tests you subclass VfuTestCase rather than TestCase. But VfuTestCase is a subclass of TestCase, so you still have access to all the standard TestCase functionality. This diagram shows an example test class, VfuExampleTests, which is included in the VFU download, and its two superclasses.

Also, note that VfuTestCase uses a TestSequenceManager instance. We’ll look more closely at this shortly.

2. TestSequenceManager & the TestSequenceCommand Classes

VFU uses the classic Command design pattern…

TestSequenceManager has an array holding zero to many TestSequenceCommand instances. All TestSequenceCommand instances also hold a reference to TestSequenceManager so that they can make calls back to it, including its commandFinished() method.

3. Sequence Diagram for TestSequenceManager & Some TestSequenceCommand Classes

Before we look at the sequence diagram, heres a sample visual test method from the VFU docs:

Note that we call multiple add…() methods in VfuTestCase.testSequence – its TestSequenceManager instance – then we call its start() method.

This sequence diagram is a simplified depiction of what happens when a very simple test method is run, on with just two add…() calls. Most add…() calls to TestSequenceManager create and add a singleTestSequenceCommand instance to its _testSequence array. The addAssertComponentMatchBaseline() call does this, as shown below. The addRefresh() method is unusual in that it creates and adds three. Once all TestSequenceCommands are created the manager starts to execute them.

I was recently asked to write up a summary pointing out some of the more interesting bits of Visual FlexUnit‘s code for a code review by a potential employer. As I wrote this I realized that I might as well dual-purpose these notes as a blog post. If you’re curious about how VFU works, read on.

Unless otherwise specified, all code is in VFU’s dev/vfu/visualflexunit/src/flex/ folder.

1. com.allurent.flexunit2.vfu.flexui.VfuTestRunner – This file includes comments that talk about VFU’s ‘tacked-onto-FlexUnit’ nature, resulting kludgishness, and potential next-step refactoring.

2. Take a quick look at VFU’s documention. I don’t suggest that you read the whole thing but, in particular, Step 2.B contains a sample test method that shows the sort of thing users would be writing.

3. com.allurent.flexunit2.vfu.framework.testsequence.
TestSequenceManager – One of the primary challenges of writing VFU was the combined result of these factors:

• We wanted test writers to be able to write clear, concise test methods like the sample method I just mentioned.
• Such methods inevitably result in multiple asynchronous processes, e.g. component rendering & file reading.
• FlexUnit has a mechanism for handling asynchronous processes, but can only handle one such process per test method.

A poor solution to this set of factors would be to ask test writers to write a chain of methods for each test, linked together with FlexUnit’s addAsync() mechanism. Ack!

Instead I created this manager class that creates and manages a queue of command objects, as requested by the user’s test code.

4. You might also want to take a look at some of the command classes in com.allurent.flexunit2.vfu.framework.testsequence.commands, especially AssertComponentMatchBaseline.

5. AssertComponentMatchBaseline includes a _judgmentStrategy:IBitmapMatchJudge private variable. The strategy object stored here is responsible for deciding whether the rendered component’s bitmap is close enough to a stored baseline bitmap to be visually indistinguishable. By default this variable holds an instance of com.allurent.flexunit2.vfu.framework.strategies.DefaultBitmapMatchJudge. Test writers have the option to write their own alternative strategies and pass them in. A simple example of using composition to allow flexibility.

6. /dev/vfu/vfuexampleTest/build.xml – Shows how we simply override a single Ant property in order to run automated visual tests. The build process is identical to the process used when running standard FlexUnit tests, except that we tell it to use a different MXML app template. Marty Frenzel showed me this elegantly simple approach.

7. /dev/tools/build-common-targets.xml – For the most part this is slightly-modified Antennae code. I added the build-visualflexunit and test-visualflexunit targets.

8. /dev/vfu/visualflexunit/template/testprojecttemplate/build.xml – VFU includes a bootstrap mechanism which makes it easy for users to create the framework for visual test projects. This file’s bootstrap-visual-test-project target copies files and uses Ant’s token replacement mechanism to insert appropriate project, file and folder names.

Visual FlexUnit (VFU) is a FlexUnit extension that allows you to do automated testing of components’ visual appearance using “visual assertions”. In a nutshell, a visual assertion asserts that a component’s appearance is identical to a stored baseline image file. These visual tests can be run in either a GUI or an Ant-based build process.

I created VFU during my recent internship at Allurent. We’re releasing it as open source on Google Code under the MIT license. Its home page is here.

Why should you be interested in VFU? Here’s an extract from its main documentation page:

Why Is This Useful?

Regression Testing

By adding visual tests to your build process you can be assured that any changes to your codebase, or to the Flex/Flash framework, that effect the appearance of your components will be noticed immediately.

Testing For Cross-Platform Rendering Differences

While Adobe has done a great job of making Flash & Flex a write-once / deploy-anywhere platform, there may be differences in how images render:

• Between the Flash Player and the AIR runtime
• Between Windows, Mac, and Linux
• And, of course, between the many combinations of these variables

To date our limited testing has only detected a) differences that aren’t visually noticeable, and b) differences in very unusual cases that you would never use in the real world. Still, it doesn’t hurt to check. By running regression tests on multiple platforms you can be assured that any unseemly differences in component rendering will be brought to your attention quickly.

(end of extract)

We invite you to play with VFU and would love to get your feedback. You can either enter issues into Google Code or add a comment to this post. While it looks as though this could be a very useful tool, at this point we haven’t worked with it enough to assert that visual assertions belong in your toolkit. Let us know what you think!

The following people contributed greatly to this project, and made my work on VFU both easier and extremely educational. Thank you!:

• Joe Berkovitz
• Marty Frenzel
• Daniel Rinehart
• Tim Walling

Finally, I should mention that Allurent is an extremely smart and competent company, at the forefront of Flex development, and a really nice group of people. I learned an incredible amount working there, including a great deal about using Ant to implement automated build processes. (I’ll be blogging shortly about the Ant build process that I’ve included with VFU.) If you get an opportunity to work with Allurent, I suggest that you grab it.

The challenge: I want to use FlexUnit to test whether a method returns an error under certain conditions. Specifically, I want to be able to confirm that an error was returned.

I assumed that FlexUnit would provide some means to do this but couldn’t find anything. I googled “FlexUnit assertError” and got zero results. I then posted to FlexCoders and Angus Johnson was kind enough to respond with this code snippet:

try
{
   ... // code that should throw exception
   fail('Exception was not thrown');
}
catch
{
   // test exception type / message etc is what is expected
}

When I tried it out, I found out that fail() also throws an error, so we end up in the catch block in either case. I assume that this is, at least in part, what is meant by Angus’s “test exception type” comment. We need to test and see what kind of error we’ve caught. If its an AssertionFailedError this indicates that fail() has been called and we should rethrow the error. If not, this indicates that our code did indeed throw an error, which is what we wanted to confirm, so we should stop the error there.

This can be done in three lines of code but I didn’t want that repetitive code – I plan to do this a lot! Instead, I encapsulated it by adding an assertError() method to flexunit.framework.Assert, which is a superclass of all TestCases. The comments in the body of the method below show how it’s used:

    public static function assertError(error:Error):void {
        //
        // Use by inserting code like this in a TestCase:
        //
        //  try {
        //      // some code that -should- throw an error
        //      fail("someErrorMessage");
        //  } catch (error:Error) {
        //      assertError(error);
        //  }
        //
        if (error is AssertionFailedError) {
            throw error;
        }
    }

This seems to work fairly well. It still takes a lot more code than the single line needed for assertTrue() etc, but I don’t see a better solution. Suggestions?

It’s also worth noting that one can do further testing in the catch block after the call to assertError() to confirm that one’s code returned the type of error expected.

The current version of FlexUnit (.85) has some glitches in its documentation. There’s very little to start with – only two classes have significant ASDoc comments, with code examples, etc: TestCase and TestSuite.

Unfortunately, even this documentation doesn’t appear in the ASDocs because the ASDoc comments have been implemented incorrectly.

While there are a couple of articles explaining how to use FlexUnit they are pretty basic and nothing I’ve found explains how to use FlexUnit’s new asynchronous testing functionality, for example. The TestCase docs touch on this so, if you’re trying to figure this out, they’re important.

The solution? You can download my corrected version of the ASDocs here. I’ve included the entire docs set, but have only modified the formatting of the ASDoc comments in TestCase.as and TestSuite.as.

FlexUnit_0-85_ASDocs_Fixed.zip

Enjoy.