Cantata and Jenkins in Concert: Continuous Integration Testing

Challenges of maintaining a healthy code base.

In many business segments, be it transportation, industrial automation or in medical devices, industry leaders have developed finely tuned software code bases over the years. This code tends to consist of many interlocking parts with common dependencies. Small changes in one part of a project can have wide-ranging impacts on other parts of a project.

Maintaining consistent software quality across generations of production code releases is vital to maximizing reliability, functional safety and certification compliance while minimizing liability exposure in the process.

Identifying why regressions occur and fixing changes in unanticipated runtime behavior can be challenging, especially in collaborative projects involving multiple software development teams. As a result, a lot of productive development time can be lost and time-to-market for major product updates can suffer.

Continuously fine-tuned software integration symphony

For smaller projects we can rely on individual skill and code reviews to be in sync across the project, similar to a player in a jazz band picking up on the riffs and themes from a fellow artist.

This cannot however not be sustained for more intricately interwoven software designs requiring high levels of exact reproducibility across software generations. To avoid frustration and resolve problems efficiently in all phases of software development, verification and certification, it is best to have the orchestra of software professionals work from the same musical arrangement. This way the symphony stays in-tune, departures from ideal melodic flow are quickly caught and we are always ready for a great concert performance, always ready for the next release.

Continuous Integration principles apply to both software development and verification to help achieve this goal of a beautifully orchestrated software development symphony.

Thus, it comes naturally that companies would look at continuous integration solutions and take advantage of their existing development pipeline framework. Software product unit and integration testing, code coverage and certification can be part of a standardized development flow. Thus, testing becomes but another stage in a tightly integrated continuous software update cycle as expressed by the design flow of your CI solution of choice.

If Jenkins is the continuous integration platform of choice, Cantata can be fully integrated and become the software unit testing and integration section of the software design symphony orchestra. The two solutions will form a perfectly in-tune ensemble of continuous development, integration, and testing.

Unit and integration testing in quality assurance

Unit testing and integration testing whether required for functional safety certification or assuring business critical applications are an integral part of any successful software development effort. This is especially true for the complex and distributed modules that form the backbone of critical transportation infrastructure and industrial automation (Industry 4.0) these days.

Distributed edge computing has only accelerated the need for a common orchestration framework to be used for both software production and software testing.

Within Jenkins a Cantata testing stage can be added in a very straightforward fashion, regardless of whether it is implemented as a freestyle, maven or pipeline project, or even a multi-configuration combination of these. For the purposes of this blog we will focus on freestyle projects and pipeline projects.

Cantata and Jenkins using a Freestyle Project

For use with Jenkins, Cantata provides a ready-to-use plugin, which allows for the definition of Cantata specific build environment settings and unit test specific build and execution settings.

Below we are using this cantata-plugin.hpi file available on GitHub to configure a Jenkins Freestyle project that can be used for regularly scheduled tests and software health update reports.

 After installing the plugin via the Jenkins plugin manager, build environment settings are accessible as part of the freestyle project configuration:

This kind of Freestyle project can then be used for regularly scheduled tests and software health update reports.

The Cantata Technical Note – Integration with Jenkins: Using the Cantata Plugin explores the easy steps to add Cantata awareness to your Jenkins automation server.

Cantata and Jenkins using a Pipeline Project

For tighter integration with a multi-stage continuous integration flow, it is common in Jenkins to define a pipeline or multi-branch pipeline project. Cantata has scriptable and transparent Java as well as a command line interface making it intuitive to orchestrate and arrange your test automation in such a way that it organically fits in with the other movements of your software production automation symphony.

As Jenkins pipelines are designed around the Groovy scripting language and are thus highly script driven, you may decide to not rely on the Cantata plugin so much, but rather use command line scripting.

In the Jenkins Stage View below, we depict a pipeline project taking advantage of the Cantata command line capabilities. The Cantata Technical Note – Command Line Use, provides details on how the command line helps with this.

In this pipeline uses GIT to check out the software under test (SUT) from a repository. We then use the command line to generate a Cantata static instrumentation file. We use that in turn to generate a Cantata AutoTest test script for the SUT for the required level of code coverage. Once the test script is generated, it is moved to a local sandbox for the test run itself.

Launching the Cantata Team Reporting server in the next steps allows for advanced build health monitoring over multiple project build generations.

For each Jenkins pipeline test run, a link to a Cantata Team Reporting view is available to provide details on unit tests and code coverage obtained for a particular run. The Cantata Technical Note – Integration with Jenkins, details writing the Groovy script synchronizing Team Reporting containers with the Jenkins build history.

The test execution itself is written as its own pipeline stage. The command taking advantage of the plugin for Jenkins is commented out, as we are taking advantage of the unit test execution being nothing more than a Cantata makefile project.

Full HTML reports generation – example

After the test is built and executed we then print the test results *.ctr into the Cantata and Jenkins console log for tracking. Finally, a full HTML report is generated so it can be accessible as part of the Jenkins workspace view history. How to do this is shown below:

cppctr2html converts the CTR results file and formats the test results into a single HTML file, which can be submitted as certification evidence of test results in compliance with all the software safety standards supported by Cantata.

From within Jenkins, we can review the results and workspace a given unit test was run on.

In addition to inspecting the test build, execution log and trendline of software health based on failed and successful test runs within Jenkins, Cantata Team Reporting will then provide detailed unit and integration test statistics for continuous monitoring.

The final stages of our pipeline will then remove build and test artifacts and check the project and test results back into the repository, so the whole cycle can begin again whenever desired.

Let the symphony commence

Taking advantage of continuous software integration and automation servers like Jenkins not only for software production, but also for software verification enables us to consistently maintain the highest level of software quality.

Code changes resulting in test regressions are instantly flagged and can be addressed as part of offline release readiness reviews without delay.

Seamless organic integration into scriptable automation frameworks, enables us to keep the meticulously arranged symphony maintaining software quality and health in tune regardless of the amount of code base changes without missing a beat. It also makes it very straightforward adding a testing and certification stage to an already established continuous integration framework augmenting the process that takes us all the way from product conceptualization and design towards iterative product releases.

Fully integrated unit testing thus becomes the keystone to ensuring software health and ensuring readiness for the next leap in ongoing product improvement.

When will you start to perform your own continuous integration and testing symphony?