Software Testing at inviCRO
At inviCRO we strive to deliver software in accordance to highest scientific and quality standards. As a consequence, testing our software has become an integral part of the development process. In particular we follow a continuous integration process that allows us to keep our software working at most times and to detect and fix bugs very quickly. Additionally, with test-driven development we are continuously expanding our testing library and improving the quality of our software one test at a time.
Continuous Integration Testing with Jenkins
One of the tools we use in our pursuit for continuous integration is Jenkins. It watches our source code repositories for changes and then triggers test runs on various build machines. For instance, we currently run test runs for iPACS 1.07 as well as the upcoming 1.08 every time a change is made to the source code. Jenkins also keeps track of what has changed between the last good run and a potential failure, to then notify all affected developers and report on any failed tests. This allows a developer to immediately review and fix issues, while he is still immersed in the topic and long before a piece of software reaches the customer.
The history of all tests is kept, and developers get a kind of weather report with all sunshine for tests that have succeeded five or more times in a row, while the weather icon becomes partly clouded or heavy rain if tests are currently failing or have failed in the recent past.
Functional Requirements Management with Testlink
To be able to comprehensively test our software we have to define which functions the software is supposed to provide. These so called functional requirements define the scope of the software, while design specifications tell a developer how a specific requirement should be implemented. Based on this information, which we manage in a software called Testlink, we derive test suites. Testlink allows us to make sure all requirements are sufficiently tested, as well as to record tests and document outcomes. Tests can either be automated tests and/or manual test cases. These manual test cases then include instructions on how to perform tests and define what the desired outcome should be.
Issue Tracking with Redmine
New features, along with their design specifications, as well as bugs are recorded in our issue tracking and project management software Redmine. Here, issues can be classified, prioritized, assigned for development, documentation or testing. Redmine also tightly integrates into our version control system Git.
VivoQuant Testing
To automatically Test VivoQuant, we make use of its integrated scripting engine VivoScript. VivoScript is a full implementation of ECMA Script (also known as JavaScript) with VivoQuant-specific extensions that allow you to fully control the program, from using any menu entries or clicking any button down to executing internal functions. This allows the user to automate repetitive tasks and implement complex processing algorithms by combining multiple steps.
For testing, VivoScript has proven to be invaluable, since it allowed us to simulate operations such as loading of data and comparing the results to a defined reference. Currently, VivoQuant supports (depending on definition) 20-30 pre-clinical and clinical file formats from DICOM over Siemens CT, Bruker MR, GE PET data, … down to image files such as JPEG, PNG, TIFF. A large part of the current testing library covers loading all these different formats and checking the results. This way a developer can be sure that a change does not accidentally break any file format. Additionally, we test a wide array of functions, such as the pre-processing, filtering, registering, … as well as interaction with the iPACS, etc…
In the movie at the top, you see a run through all tests, which all follow the protocol to start VivoQuant, execute a VivoScript and compare results. Currently 254 tests have been defined in 43 categories, and these tests are automatically run on 4 different platforms (Windows 32-bit, Windows 64-bit, Mac OS X and Linux).
iPACS Testing
iPACS is a perl-based web application and as such it is tested based on Perl’s TAP (Test Anything Protocol), which allows us to test small units such as individual functions up to simulating complete HTTP web requests and checking their output against defined references. Currently, 2,161 tests in 65 categories are defined, covering everything from the login, over interaction with the image browser, webdisk, public links, the database models, the APIs, … back to the logout of a user.
Every time a developer makes a change to the iPACS source code and checks it into the version control system, a test build is triggered. This way a developer is immediately notified if a problem has come up and can fix the issue right away.