Spring 2013: High Volume Automated Testing
This is a programming course. We will create very useful, advanced software test tools using the Ruby programming language. This is state-of-the-art work in this field.
UNDERGRADUATES: Suitable for undergraduates who did well in CSE 2410 (data structures) and software testing (3411 or 4415). Possibly suitable for undergrads who know a lot about embedded systems and are adequate programmers.
GRADUATES: For grad credit, the course requires a little more work and leadership. The course qualifies as an "Advanced" course. You must be able to program and you must be knowledgeable about testing.
Many of the most maddening software errors cause intermittent failures. Do something—the program fails. Do what you think is the same thing again—no failure. These are hard to diagnose and almost impossible to provide customer support for, but they can be serious irritants or critically dangerous. For example, at least some of the flaws underlying automotive fuel / braking problems have been traced to software.
Bugs like these are invisible to traditional software testing techniques. There ARE techniques for hunting these bugs. Most of the ones we're aware of were developed for telephone systems and embedded software in consumer electronics. What the techniques have in common is massive sampling—automated creation, execution and evaluation of very large sets of automated tests. Traditional "automated" test suites often have a few thousand tests. HiVAT suites might have a few million tests, tests that are individually not as optimized as the traditional, handcrafted tests, but that are designed to overwhelm the software under test with their variety.
Many companies are interested in applying these techniques but held back by a lack of qualified staff. People find it difficult to teach themselves the HiVAT techniques and they are covered in very few courses. Most software testing texts and courses (university courses and industrial courses) ignore the HiVAT techniques completely or mention only the simplest and least powerful member of the family, an approach developed in the 1960's called "fuzzing". The problem is that it is easy to describe the theory of the other techniques (Kaner has been doing that in courses and conference talks since 1998) but it takes a lot of time to work through the nuts and bolts of any of these techniques well enough for someone to actually do it successfully.
WHAT WE WILL ACTUALLY DO:
We will create and publish open-source examples of several techniques and apply them to open source software. This is a "project" course and students will be welcome to collaborate, but individual students will be given individual tasks and held accountable for their own productivity.
I think this course allows enough time for the work, and provides enough focus, that it should be about average in CS-course difficulty. Probably easier than CSE 4415 (Testing 2) and less time consuming than CSE 3411 (Testing 1) – if you have OK programming skills, can pick up a reasonably-easy-to-learn programming language (Ruby) largely on your own, and an OK background with testing techniques. I will make a personal effort to accommodate students who know a lot about embedded systems but are less skilled at programming/testing. You will still work (including writing code) on projects but you might serve more as a consultant and less as an individual programmer.
Cem Kaner is the lead instructor. For more about me, see http://www.softwaretestpro.com/Item/4829/Cem-Kaner---3rd-Annual-Luminary-Award-2012/. For more about HiVAT, see http://wtst.org/ and http://kaner.com/pdfs/KanerVISTACONkeynote.pdf and http://www.kaner.com/pdfs/testarch.pdf
Carol Oliver is doing her Ph.D. in this area and leading a team of students doing research on it. Kaner and Oliver are co-teaching the course.
Cem Kaner, JD., Ph.D.
Professor of Software Engineering, Florida Tech