AdaCAST

This white paper was written to introduce a new software tool called AdaCAST. This tool focuses on unit and regression testing for Ada language programs. Some background information on software test; more specifically unit and regression test is also added for completeness.

A Missing Component to the Ada Toolset

AdaCAST represents a significant advance to the state of the art by automating the development of application specific Test Harnesses required to unit and regression test Ada language programs. We regard a tool of this type as the missing component in the expanding market of Ada language productivity tools. AdaCAST will help transform the software development process in three distinct ways. First, it will help contain the spiraling costs of developing Ada language systems by reducing the man-hours spent on unit and regression testing; a process that historically represents the largest single expense in the development of large software systems. Second, it will ensure higher quality Ada software products by exposing flaws earlier in the software life-cycle when these flaws are more easily and inexpensively resolved. Third, it will automate the labor intensive and repetitive exercise of developing test harness software.

Because it is not possible to access the data structures and subprograms contained in a package body, many programmers will use a test technique called instrumentation to test Ada software. Instrumentation is an obtrusive software technique which has often been used to test software components written in earlier generation software languages. It involves the insertion of additional lines of code into the Unit Under Test (UUT) to record data as the software is executed. The flaw with this intrusive approach is that the performance of the UUT is affected by the additional lines of code and the tested software cannot be delivered into a configuration control system without the instrumentation being removed. Additionally, regression testing is not easily accomplished because the software would have to be re-instrumented each time to recreate the test cases. For embedded and real-time systems, timing constraints must be verified. To assess time-critical software , it is essential to collect data in as unobtrusive manner as possible [SEI 92].

Specification-Oriented Testing
Seeks to show that every requirement is addressed by the software. An unimplemented requirement may be reflected in a missing path or missing code in the software. Specification testing assumes a functional view of the software and is sometimes called functional or black-box testing [Howden 86].

Implementation-Oriented Testing is based on the fact that test data selection is derived from the implementation [Howden 75]. The goal is to ensure that various computational characteristicsof the software are adequately covered. It is hoped that the test data that satisfy these criteria have higher probability of discovering faults. Each execution of a program executes a particular path. Hence, implementation-oriented testing focuses on the following questions: What computational characteristics are desirable to achieve? What paths for this program achieve these characteristics? What test data will execute those paths? What are the computational characteristics of the set of paths executed by a given test?

Implementation-oriented testing addresses the fact that only the program text reveals the detailed decisions of the programmer. For the sake of efficiency, a programmer might choose to implement a special case that appears nowhere in the specification. The corresponding code will be tested only by chance using specification-oriented testing, whereas use of a structural coverage measure such as statement coverage should indicate the need for test data for this case.

Error-Oriented Testing - is necessitated by the potential presence or errors in the programming process. Techniques that focus on assessing the presence or absence of errors in the programming process are called error-oriented.

While our experience tells us that Specification-oriented testing and more specifically input domain testing which consists of extremal and midrange test techniques (those which cover the extremes and interiors of the input domains) [Myers 79] is most widely used; each of the Unit Testing methodologies described above have a single common denominator. They all require an application specific test environment from which all tests, regardless of the method, can be executed. AdaCAST provides this application specific environment for each test methodology. Whichever method of testing is used, AdaCAST provides the components necessary for the testing to be carried out.

Software Component Stub
A stub is a software component body fragment that is used during unit test to take the place of a component that the UUT depends on to perform its processing. Stubs are generally used when a tested version of the dependent component is not available or when the variability of adding another new software component to the test environment is undesirable. The processing coded into the stub may vary depending on the approach used by the test engineer. For example, the stub may record the value of parameters passed to it in the course of the execution of the test.

Test Bed
Also referred to as a test harness, a test bed is the combination of all the STUBS with the UUT and the DRIVER into a single executable program that allows for the stimulus of all UUT functionality.

This test bed generally provides the following capabilities:

Historically, unit testing is performed on a project at the end of the official coding phase. Subsequent to the completion of unit testing, the software goes through many iterations due to design or requirement refinement and as a result, the final product can bear little resemblance to the software components that underwent initial unit testing. Additionally, it is likely that the changes implemented to the software to solve one group of problems caused other problems that may remain hidden through integration and product release.

There are many reasons that unit testing of Ada software is not efficient. Some of the problems are applicable to all programming languages, and some are unique to the Ada language. The following is a list of these problems:

(1) Schedule and Staff Constraints

Rarely is time allocated within a program schedule to run unit test procedures as a means of regression testing software as the requirements and design evolve. Sufficient knowledgeable engineers are rarely available to perform initial unit testing or regression testing because of the schedule pressure that most large development programs are under. Building an environment with which to test software units is not a simple task. Knowledge of the entire surrounding environment is necessary before a test can be written. Inexperienced engineers can spend much time spinning their wheels building such environments.

(2) Test Procedure Maintenance Problems

It is a tedious process to update software test drivers and component stubs that are used to perform unit testing. Most often, the tests are developed months earlier by engineers other than those who are now tasked with re-unit testing or regression testing the updated software. Because they are often-times not deliverable software components, test drivers and component stubs are seldom well documented making reuse by anyone other that the original author virtually impossible. Most often, the old test environment is discarded and a completely new unit test environment is constructed to test each iteration of the software.

(3) Ada Specific Problems

Ada is inherently difficult to unit test using traditional techniques. Implementation details of a software component that must be verified during unit test are hidden from other program components, including test drivers.

(4) Lack of Detailed Testing Guidance

Unit testing methods are almost always left to the discretion of the engineer. This results in as many flavors of unit test approaches and output data formats as there are engineers on the project. Indeed, the GAO report for the A/N BSY-2 Combat System program noted this as a fundamental obstacle in the development of quality software components. (GAO/IMTEC-91-30 BSY-2 Development Risks and Production Schedule)

AdaCAST is the automation of a proven Vector Engineering developed unit testing methodology. This methodology has been used successfully on all company Ada language software development projects since 1990. AdaCAST will make the unit test process integral to the entire software development life-cycle. Each time a software component is modified it will be quickly tested (using previously developed test cases) with the test results compared against the test results for the previous version of the component. This regression testing will ensure that each change made to the software has only the intended effects on the functionality.

(1) Ease Schedule and Staff Pressures

We project that the time dedicated to unit testing will be reduced by at least fifty percent for large projects. This will allow for quicker off loading of personnel during the unit test phase and ensures more productive use of the limited staff engineers.

Automating the unit test process frees staff for writing the applications software that contributes to the profits of the company rather than writing the disposable test software that can drain a companies budget.

The AdaCAST tool will automate the building of the complex environment of stubs and drivers necessary to exercise a particular unit. This automation of the testing environment will save the inexperienced and experienced developer the time required researching how a particular unit fits into the system.

(2) Eliminates Test Maintenance

The entire test environment is generated automatically, no maintenance of test software is required. All of the environment components are automatically regenerated for each revision of the software component being tested. The test cases and results will automatically be retained for use in performing regression test and analysis.

(3) Ada is well Suited for Automated Testing

There are several attributes of the Ada language that make it well suited to a tool of this type. The strong data typing enforced by the language guarantees that the interfaces between components are well defined and the data values have finite ranges. Even numeric types not explicitly bounded by the user have an implementation defined range that is accessible using the syntax of the language (i.e. integer'last). The handling of erroneous results during program execution using the exception processing provided in the language makes it easy to develop software that is fault tolerant. In the case of unit test drivers this is important because the software components being tested are not mature and it is very likely that they contain catastrophic errors.

(4) Replace Detailed Testing Guidance

AdaCAST will reduce software unit test guidance to a single instruction: "use AdaCAST to test each version of the software prior to delivery into configuration control." AdaCAST will give each organization and individual developer the ability to perform unit-testing and generate unit test reports with a common format.

SEI 92

Morell, Larry J. ,and Lionel E. Deimel. "Unit Testing and Analysis." Software Engineering Institute Curriculum, Module SEI-CM-9-2.0, June 1992

Howden 75

Howden, William E. "Methodology for the Generation of Program Test Data" IEEE Trans. Computers C-24, 5 (May 1975), 554-560.

Howden 86

Howden, William E. "A Functional Approach to Program Testing and Analysis." IEEE Trans. Software Eng. SE-12, 10 (Oct. 1986), 997-1005

Myers 79

Myers, Glenford J. The Art of Software Testing. New York: John Wiley, 1979.