Table of Contents

• Executive Summary
• 1. Introduction
• 2. Glossary of Terms
• 3. Organizations and Responsibilities
• 4. The Ada Compiler Validation Capability
• 5. Validation
• 6. Significance Of Validation

• Appendix A. Declaration Of Conformance
• Appendix B. Implementer Dispute Format
• Appendix C. Test Challenge and Resolution Process
• Appendix D. Points Of Contact
• Appendix E. References
• Appendix F. Reference Configuration Registration Form
• Appendix G. VSR Supplement Form

EXECUTIVE SUMMARY

The Ada Compiler Validation Procedures define the certification body for the Ada programming language and set forth its rules of procedure for testing implementations of the Ada language for conformity to the language standard. The process of testing implementations is known as validation. This version (5.0) of the Procedures is a revision to Version 4.2 to reflect two important changes in Ada Validation: the end of the period of transition in validation from the use of the original language standard [Ada83] to the revised standard [Ada95] (January 1995 to July 1997); and the transferal of validation authority for the Ada language from the National Institute for Standards and Technology (NIST) to the United States Department of Defense (DoD).

This document supersedes all previous versions of the Ada Compiler Validation Procedures.

During the transition period, which began with the publication of the revised Ada language standard, [Ada95], in January 1995, validation was conducted according to either the original or revised standard. Testing for the revised standard was introductory in both the limited number and extent of the test programs. Furthermore, there was an allowance for implementations to fail tests for language specifications of [Ada95] that are not present in [Ada83]. This validation lenience was intended to encourage implementers to bring useable partial implementations of [Ada95] to the market quickly; validation would still give a common conformity check on these implementations.

With the completion of [Ada95] and near completion of the full validation test suite, the DoD transferred the certification authority to the NIST in October 1996; the NIST is the certification authority for other FIPS standards. However, the NIST changed its focus and announced that it would soon bring to a close all of its conformity-testing activities; the DoD then decided that it should continue to support Ada compiler validation. Moreover, the National Research Council issued a report on Ada ¹ recommending that the DoD sustain its investment in an Ada infrastructure, including support for validation. As a result, the Ada certification authority was assumed by the DoD in April 1997.

The Ada certification body comprises the DoD’s Ada Joint Program Office (AJPO), a principal technical advisor called the Ada Validation Organization (AVO), and the AJPO-recognized test laboratories called Ada Validation Facilities (AVFs) which conduct validation testing. The AJPO is responsible for the maintenance of the validation test suite, the issuance of validation certificates, and the establishment of these Ada Compiler Validation Procedures.

Ada Compiler Validation Capability (ACVC)

The ACVC is a conformity test suite (and supporting documents); it is designed to ensure that validated Ada compilers achieve a high degree of conformity to the Ada language standard. The ACVC is customized by an AVF for each Ada implementation that is tested; customization consists of adjusting the ACVC appropriately for various implementation characteristics. The ACVC is maintained by the AJPO; the ACVC is upgraded according to changes in interpretations of the Ada standard by the language maintenance body (ISO Working Group 9, Ada Rapporteur Group) and the discovery of deficiencies in the test programs. New versions of the ACVC are released according to a schedule set by the AJPO. ACVC versions that tested conformity to [Ada83] were numbered "1.1" through "1.11", the final version; versions that test conformity to [Ada95] are numbered "2.0", "2.0.1", and "2.1". ACVC versions 2.0 and 2.0.1 were used during the transition period; ACVC 2.0.1 was an intermediate release with corrections to many errors of version 2.0, and also was used with slightly stronger validation requirements. ACVC 2.1 is the first version that is considered to be a full test of [Ada95]. (The successors to this version will be numbered "2.2", "2.3", and so on, as needed.)

For the compiler of an Ada implementation to receive validated status under ACVC 2.1, the implementation must process each test program (for the "core" language) of a customized ACVC so that the result is graded passed, inapplicable, or unsupported by ACVC grading criteria. An AVF customizes the ACVC for a particular Ada implementation by appropriately setting test parameters, by removing "withdrawn" tests (tests ruled by the AVO to be in error) and certain inapplicable tests, by splitting as needed test files with multiple intended errors so to enable complete error detection, by making any other modifications to the tests as directed by the AVO, and by including the set of tests for each Specialized Needs Annex as requested by the validation customer.

In addition to the specification of a "core" language, [Ada95] contains several Specialized Needs Annexes; these specify language requirements designed to meet the particular needs of various general application domains, such as information-systems programming. An implementation of [Ada95] need not include implementation of any of these annexes, or it may implement only some of the features of these annexes. Whereas all ACVC test programs for the core language must be processed during validation, those for the Specialized Needs Annexes are processed only upon vendor request. A validation certificate is issued only if all tests for the core are correctly processed. It will additionally give credit for support of a Specialized Needs Annex to the extent that the relevant set of tests is correctly processed.

Validation

Validation involves interaction between the AVF customer and the Ada certification body. The testing process consists of well defined actions which, when completed successfully, result in the award of a validation certificate for the tested Ada compiler. The key actions in the validation of an Ada compiler are:

1. The customer and an AVF reach a formal agreement for validation, including the dates for the submittal of the results of customer-administered processing of the ACVC and for AVF witness testing.
2. The customer petitions for deviation from the requirements of each ACVC test program that is believed to be wrong for the candidate implementation(s).
3. The AVO rules on the customer petitions.
4. The customer processes the ACVC on the candidate implementation(s) and submits the results to the AVF.
5. The AVF analyzes the results of the customer’s independent processing of the ACVC. (If the results are not acceptable, the previous action must be repeated, and new results analyzed.)
<>
6. The AVF conducts witness testing of the candidate Ada implementation(s), documenting this testing in a Validation Summary Report (VSR) which the AVF submits to the AVO for review.
7. The customer signs a Declaration of Conformance for each candidate Ada implementation.
8. The AVO reviews the VSR, with comment to the AVF. The AVO recommends to the AJPO that a validation certificate be issued for the tested compiler(s), if the testing is successful.
9. The AJPO issues a validation certificate for each tested compiler upon the successful completion of the preceding actions.

1. INTRODUCTION

This document provides operating policy and procedures of the Ada certification system. The Ada certification system is managed by the Ada Joint Program Office (AJPO) of the United States Department of Defense (DoD). The two other principals of the Ada certification body are the Ada Validation Organization (AVO) for technical support and the Ada Validation Facilities (AVFs) for performing witness testing of Ada implementations.

These procedures address the validation of Ada compilers of implementations of [Ada95] and provide an operational definition of a validation Ada compiler. These procedures are effective September 1997 until they are superseded or revoked. Detailed procedures regarding the application of the Ada Compiler Validation Capability (ACVC) are given in the ACVC User’s Guide.

The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. The AJPO established a certification system to realize the benefits of standardization which include the ability to transfer software and programming expertise between computer systems that use a conforming Ada compiler.

It is important to note the scope and intent of validation. The purpose of validation is to ensure that Ada implementations achieve a high degree of conformity with the standard. Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada validation. Moreover, it is important to note that the ACVC conformity test is the processing of a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; validated implementations may fail to conform to the standard in ways peculiar to each, under particular circumstances.

Validation testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada validation is to identify Ada processors that may be procured and used to develop application programs that meet the [Ada95] goals of portability and interoperability.

The ACVC test suite is not designed to replace the vendor’s quality assurance testing or systematically to detect inconsistencies or "bugs." The technical goal of Ada validation testing is primarily to verify that the Ada product correctly supports all required features. Rather than exhaustive testing of permutations of features, the ACVC test suite contains a carefully-chosen set of test cases that cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard.

Neither is validation intended as a means of performance benchmarking. The VSR does not contain information about the speed, cost, or efficiency of executing the validation tests.

2. GLOSSARY OF TERMS

Ada programming language: The language defined by [Ada95].

Ada Compiler Validation Capability (ACVC): The means for testing conformity of Ada implementations, consisting of the test suite, the support programs, and the ACVC User’s Guide.

Ada implementation: An Ada compiler with its host and target computer and operating systems (kernels).

Ada Rapporteur Group (ARG): The Ada Rapporteur group (ARG) is a subgroup of ISO/IEC/JTC1/SC22/WG9, the International Organization for Standardization Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language.

Ada Validation Facility (AVF): The part of the certification body that performs validation testing of Ada compilers.

Base Compiler: The compiler that was subject to successful validation testing; this compiler is identified on the associated validation certificate.

Base Configuration: The host and target computer systems that were used for successful validation testing; these are identified on the associated validation certificate.

Certification body: An impartial body, governmental or non-governmental, possessing the necessary competence and reliability to operate a certification system, and in which the interests of all parties concerned with the functioning of the system are represented. [ISO/IEC 86]

Certification system: A system having its own rules of procedure and management for carrying out conformity certifications. [ISO/IEC 86]

Compiler: The software and any needed hardware that must be added to a given host and target machine to allow transformation of programs into executable form and execution thereof.

Computer system: A system containing one or more computers and associated software. [IEEE 90] In this document, a computer system comprises the hardware and software (operating systems, kernels) that are essential to the operation of the compiler or the compiled code (e.g., I/O devices are generally not included in this definition).

Configuration management: A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements. [IEEE 90]

Conformity: Fulfillment by a product, process or service of all requirements specified. [ISO/IEC 86] See also Subclause 1.1.3 of [Ada95].

Core language: The Sections 1-13 and Annexes A, B and J of [Ada95].

Customer: An individual or corporate entity who has an agreement with an AVF that specifies the terms and conditions for AVF services (of any kind) to be performed.

Customized test suite: The ACVC tests, adjusted as necessary, that must be used for validation testing of a given Ada implementation (see Section 4.5).

Declaration of Conformance: A formal statement from a customer assuring that conformity is realized on the Ada implementation for which validation status is requested. [See Appendix A for the format of a Declaration of Conformance.]

Host computer system: The computer system on which a compiler is installed and executes.

Instruction set: The complete set of instructions recognized by a given computer or provided by a given programming language. [IEEE 90]

Operating system: A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. [IEEE 90]

Perfective maintenance: Maintenance performed to improve performance or maintainability. [ANSI/IEEE 90]

Prevalidation testing: Process of supplying to an AVF the results of processing an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests.

Reference configuration: A host and target computer system that is registered with the certification body, and which will produce an approved results profile when used in conjunction with some validated compiler. Reference configurations are the computer systems used by vendors to ensure that compilers produce an approved results profile. (The base configuration is a reference configuration.)

Results profile: The aggregate of test results produced by processing the customized test suite according to given evaluation criteria (see Section 6.6).

Software maintenance: Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. [ANSI/IEEE 83]

Specialized Needs Annexes: Annexes C through H of [Ada95]. These Annexes define standards for additional functionality required by specific application areas. An Ada Implementation may support some or none of these annexes.

Target computer system: The computer system on which the executable code generated by a compiler is loaded and executes.

Target run-time system: The set of subprograms that may be invoked by linking, loading, and executing object code generated by an Ada compiler. If these subprograms use or depend upon the services of an operating system, then the target runtime system includes those portions of that operating system.

Test issue: Any problem arising during validation (see Section 5.2.3).

Validation: The process of checking the conformity of an Ada implementation to the Ada programming language and of issuing a validation certificate for the implementation.

Validation Certificate (VC): A certificate issued by authority of the AJPO for a successfully tested Ada compiler (see Section 5.6).

Validation Summary Report (VSR): A report produced by an AVF that documents the validation testing of an Ada implementation (see Section 5.5).

3. ORGANIZATIONS AND RESPONSIBILITIES

This section specifies the roles of organizations that form the certification body and of customers who receive service from them.

3.1 Ada Joint Program Office (AJPO)

The AJPO is the head of the certification body. It directs the certification system by:

• establishing the structure and roles of the certification body and designating its members;
• setting validation standards to be followed by the AVFs;
• controlling the use of the Ada certification mark and setting the conditions for the award of and issuing of validation certificates;
• contracting for the development of each ACVC version, overseeing ACVC quality control and configuration management, approving the content of the ACVC, and establishing the ACVC release schedule;
• resolving issues that may arise during validation when these issues cannot be resolved through the best efforts of the AVO and the AVF;
• publishing the official list of validated Ada compilers; and
• publishing these Ada Compiler Validation Procedures.

3.2 Ada Validation Organization (AVO)

The AVO provides the technical and administrative support required to operate the certification system by:

• advising the AJPO and the AVFs with regard to the validation procedures;
• resolving issues that may arise during the validation process;
• reviewing all validation summary reports (VSRs) prepared by AVFs:
• reviewing each validation summary report (VSR), assessing the fulfillment of validation requirements by a customer, and recommending to the AJPO that a validation certificate be issued for the successful completion of validation;
• participating in the ACVC quality control and configuration management process;
• receiving and ruling on challenges to ACVC test programs, including making the withdrawal of, or requiring modifications to, such test programs; and
• convening meetings of the members of the certification body at appropriate intervals to discuss the process and to evaluate practices.

An AVF is a test laboratory chartered by the AJPO to conduct validation testing by:

• entering into a contract for validation services with a customer;
• consulting pre-validation analysis of a candidate Ada implementation’s results from processing the ACVC test suite, and conducting validation testing of Ada implementations;
• forwarding unresolved test issues to the AVO for review
• documenting validation testing in a VSR, forwarding the VSR to the AVO for review and comment, and distributing the VSR.

Customers obtain services from the Ada certification body in matters concerning Ada validation. In requesting services of the Ada certification body, customers are to provide accurate and complete information to perform validation or to obtain other services.

4. THE Ada COMPILER VALIDATION CAPABILITY

The ACVC is designed to demonstrate the conformity of an Ada implementation with the standard. The ACVC is distributed as a collection of test programs, support programs which facilitate processing the tests, and an ACVC User’s Guide that explains the criteria for evaluating the results.

4.1 Versions

New ACVC versions are released periodically according to a schedule which is determined by the AJPO. Each new version incorporates changes to the ACVC as deemed necessary by the AJPO. These changes are made in order to reflect a revision of the standard, to incorporate ISO WG9 interpretations, or to address implementer or user comments. The test objectives and test programs for each new ACVC version are available for public review and comment before that version is issued for use in validation.

Questions concerning Ada validation or comments on ACVC test programs should be submitted by e-mail to acvc-comment@sw-eng.falls-church.va.us or by FAX or regular mail to the AVO (see Appendix D, Points of Contact).

4.2 Applicability of ACVC Test Programs

Each ACVC test program has one or more test objectives which are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada implementation (e.g., "check that the proper exception is raised when Float’Machine_Overflows is True"). These test programs generally contain an explicit indication of their applicability and the expected behavior of Ada implementations for which they do not apply. The determination of applicability is made according to the grading criteria in the ACVC or as a ruling by the AVO. All applicable test programs must be processed and passed according to the specified grading criteria.

The set of ACVC test programs for any of the Specialized Needs Annexes will be processed only upon customer request (to demonstrate full or partial support of the Annex).

4.3 Test Modifications

The certification body strives to apply the ACVC as uniformly as practical to all Ada compilers. In order to apply common test objectives that depend on implementation-dependent characteristics (e.g., line lengths and numeric types), some test programs must be adjusted to a given implementation following the procedures given in the ACVC User’s Guide. These adjustments consist of the insertion of implementation-dependent values in prescribed places in certain test programs. The ACVC User’s Guide is supplemented by the Test Modifications List, which is a list of test programs of a particular ACVC version for which there are required or permissible modifications. The AVO maintains and distributes this list, which it updates as necessary.

In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (e.g., the algorithm for recovering from syntax errors). The allowable changes for each Ada implementation are determined by the ACVC User’s Guide and the AVO, and may require AVF assistance — especially in the case of compiler error-recovery problems.

In order to meet the test objective, it may be required to modify the code, the processing method, or the grading of a test program. Only the AVO shall make the decision to use any of the following Test Modifications. Possible kinds of modification are:

• Code Modification: The source code of the test is changed. Examples of code modifications are the insertion of a pragma, the insertion of a representation clause, or the splitting of a B-test into several individual tests, if the compiler does not detect all intended errors in the original test. (This last example is the only exception to the rule that only the AVO shall make the decision to apply a Code Modification.)
• Processing Modification: The processing of the test by the Ada implementation for validation is changed. Examples of processing modification is the change of the compilation order for a test that consists of multiple compilations or the additional compilation of a specific support unit in the library.
• Grading Modification: The grading of a test result is changed. An example of a grading modification is the grading of a test other than what the output from Report.Result indicates. This may be required if the test makes assumptions about implementation features that are not supported by the implementation (e.g. the implementation of a file system on a bare target machine).

Despite quality control efforts, some ACVC test programs might contain unintended errors or otherwise impose checks on an implementation that the certification body deems unwarranted. If such a program cannot be simply corrected with a test modification (see above, Section 4.3), then the AVO will withdraw the test program from the ACVC. Withdrawal is effected by including the test on the Withdrawn Tests List, which the AVO maintains and distributes. Withdrawn test programs are not considered to be part of the ACVC for validation purposes (an AVF will usually delete them from the customized test suite).

4.5 Customization

The ACVC is customized by the AVF for each Ada implementation that is subject to validation testing. This customization always consists of removing withdrawn test programs and in making required modifications to test and support programs. It may also include removal of some inapplicable test programs as allowed by the ACVC User's Guide.

4.6 ACVC Grading

The result of processing an ACVC test program can be given only one of four possible grades: Passed, Inapplicable, Unsupported, and Failed. The first three grades are considered to constitute an acceptable result. ACVC test programs that contain illegalities (which an implementation must detect) generate diagnostic output which must be graded manually, matching system diagnostics to the intended errors. Executable ACVC test programs generate output using report procedures, which can be graded automatically. The ACVC report package, Report, contains specific output procedures for the two grades Failed and Inapplicable; if neither of these is invoked, the Report.Result procedure will report Passed. These three results are the only ones that are generated by the test code (if no result is reported, i.e., if the test completes abnormally, the result is graded Failed). The grade Unsupported is established as a means of grading tests that apply to the Specialized Needs Annexes, as explained below.

The ACVC test programs for the Specialized Needs Annexes posed two problems for using the three conventional grades of Passed, Inapplicable, and Failed. The broad problem is that full support of any such Annex is not required for conformity to [Ada95] — there may be no support, or merely partial support — but there is no way to discriminate between full and partial support if only those three grades are used, since the grades Inapplicable and Failed are not appropriate for this (an implementation is not allowed to provide deviant semantics for an unsupported Annex feature — that would be a validation failure). The second problem is that there are some test programs for Core features that are applicable also to a Specialized Needs Annex, in particular, the test programs for representation items. These programs constitute a test for features that are defined in the Core as optional, but are mandatory for full support of the Systems Programming Annex (which itself is mandatory for full support of the Real-Time Systems Annex).

Therefore, the certification body decided to grade the result of processing such an ACVC test program (i.e., one that uses a feature required by, or defined in, an Annex) as Unsupported, if the prima facie result is Failed but the implementation's processing of the test program is an acceptable form of non-support. For example, if a compiler doesn't support a particular form of a representation clause, it must reject any test program that uses it — such behavior on an executable test is usually graded Failed. (This decision was necessitated because the design of the ACVC assumed that validation would require full conformity to an Annex, and also did not fully appreciate the complexity of grading support for representation items.)

4.7 Availability

The ACVC is available to the general public from an AVF or from the AJPO on the Internet. AVFs may assist the customer in format conversion when providing the ACVC in a particular distribution medium. See APPENDIX D for points of contact.

5. VALIDATION

There are a number of steps that must be completed by a customer and the certification body so that the customer obtains a validation certificate and a VSR. The same ACVC version must be used to complete the steps described in this section. The AVF must begin validation testing of the Ada implementation at the customer’s site before the current ACVC version expires or else validation with that ACVC version will not be allowed. Anyone intending to obtain a validation certificate should contact an AVF without delay for advice on the handling of the ACVC, on interpretation of the test grading criteria, and on the operational details of that AVF’s management practices. The steps are:

1. Validation Agreement
2. Prevalidation
3. Validation Testing
4. Declaration of Conformance
5. Validation Summary Report
6. Validation Certificate

In order to obtain services from the certification body, an interested party must become a customer of an AVF by reaching a formal agreement. This agreement should address the following topics:

• identification of the Ada implementation(s) to be tested and the ACVC version to be used;
• a statement of work, including analysis of prevalidation testing, conducting validation testing, and preparation of the VSR;
• the format of data to be exchanged;
• a schedule of events and the site of validation;
• financial arrangements;
• retention of records;
• AVF liability; and
• confidentiality of validation information.

5.2 Prevalidation

Prevalidation entails a series of actions and is usually where the bulk of the validation effort is expended. These actions are described in the following subsections.

5.2.1 Customer Testing

After entering into a formal agreement, the customer either provides the necessary information for the AVF to prepare a customized test suite or, the customer may prepare a customized test suite according to instructions in the ACVC User’s Guide. The customer then processes all the tests in this customized test suite using the candidate Ada implementation or another Ada implementation that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of an option controlling the production of information output. Any other exception constitutes a test issue that must be resolved with the AVF (see Section 5.2.3). Test issues should be sent to the AVF for analysis as soon as they are known. The customer must provide (unless explicitly waived by the AVF) the prevalidation materials in a timely manner or by the agreement requirements between the customer and the AVF. Prevalidation materials include as a minimum the processing of an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests.

5.2.2 Submission of Results

Upon completion of testing, the customer delivers the complete set of results in the agreed format to the AVF. These results are accompanied by the following information:

• a list of test programs for which the implementation does not produce an acceptable result (see Section 4.6), together with a rationale for the implementation’s behavior;
• the necessary information for the AVF to prepare the customized test suite, if applicable;
• an annotated sample command script; and
• the complete set of option settings used for processing the customized test suite, including the default settings.

The AVF analyzes the customer's submitted results of prevalidation testing, checking that all test programs have produced an acceptable result according to the ACVC evaluation criteria. During this analysis period, any test issues are resolved.

A test issue is defined to be any of the following:

• a missing or incomplete result to a test program;
• a result presented in an inadequate form;
• a result that is not graded Passed, Inapplicable, or Unsupported by the ACVC evaluation criteria;
• a disagreement between the customer and AVF as to the interpretation of a result;
• a change in the choice of options to be used during testing; and
• any implementation characteristic that might affect the conformity of the implementation during testing.

5.2.4 Incomplete Prevalidations

The AVF and the customer may agree that, at the customer’s risk, parts of the customized test suite need not be processed during prevalidation. The customer must certify that the results from a previous prevalidation submitted to the AVF or validation results obtained by the AVF are identical to those that would have been obtained by the customer. The normal practice is to submit complete prevalidation results.

5.2.5 Successful Prevalidations

Prevalidation testing is successful if the analysis of results and the resolution of test issues show that all results are provided and are acceptable. Prevalidation is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until validation testing by agreement between the AVF and the customer.

5.3 Validation Testing

Upon successful completion of prevalidation, with or without caveats, the AVF witnesses testing of the Ada implementation at the site and time agreed by the AVF and customer. The AVF has prepared a customized test suite based upon customer information and any information collected during the resolution of test issues. The customized test suite is installed and processed in the presence of an AVF representative. If the AVF determines that the results agree with those obtained from prevalidation and are satisfactory with respect to the caveats, the conformity test has been successful; otherwise, the test is unsuccessful. If any result of testing with a set of test programs for a Specialized Needs Annex is unacceptable, validation will not recognize that the set was processed.

5.4 Declaration of Conformance

As part of validation testing, the customer will provide to the AVF a completed Declaration of Conformance. The customer should provide the information contained in the Declaration of Conformance to the AVF prior to the on-site testing as part of the prevalidation effort, and may provide the completed document then. The Declaration of Conformance states that the organization that is responsible for the production, maintenance or distribution of the Ada compiler is offering a product that is in conformity with [Ada95]. The customer will ensure that the information contained on the Declaration of Conformance does not infringe on the rights of a third party, and may be required to provide a written statement of consent from any third party involved. The Declaration of Conformance becomes part of the AVF records and is copied into the VSR. A validation certificate will not be issued unless a signed Declaration of Conformance has been provided to the AVO. (See Appendix A for an example of the Declaration of Conformance.)

5.5 Validation Summary Report (VSR)

A VSR is produced for each validation testing effort. A single VSR may cover validation testing of several Ada implementations, provided that they all have the same results profile. The VSR identifies the following validation information:

• the validation customer;
• the organization responsible for the production, maintenance, or distribution of the Ada compiler;
• the tested Ada implementation;
• the compiler options that were used during validation as well as the set of available options;
• the implementation-dependent values used to customize the ACVC;
• the test programs that were graded Inapplicable or Unsupported;
• the test programs that were modified for validation testing, with an explanation of each modification; and
• the test programs that were withdrawn at the time of validation testing.

The VSR is prepared by the AVF but includes material that is produced by the customer, such as the documented compiler and linker options used during the Ada validation process. Draft versions of the VSR are sent to the AVO for review. The final version of the VSR is signed by the AVF, the AVO, and the AJPO.

5.5.2 VSR Availability

The final version of the VSR is available to the general public from the customer and from the AVF that produced it. The AVF may require payment of a fee for VSR reproduction and mailing costs. (See Appendix D for points of contact.)

5.6 Validation Certificate

The AJPO issues a validation certificate for each compiler that was subject to successful validation testing. The information on the certificate is derived from the customer's Declaration of Conformance and the VSR. The validation certificate conveys the AJPO's award of validated status to the base compiler, and serves as the basis for extending validated status to associated compilers as defined in Chapter 6. An entry is made in the AJPO's Validated Compilers List for each validation certificate; this entry will be removed when the certificate expires.

A validation certificate identifies:

• the base compiler
• the base configuration
• the ACVC version used for validation testing
• the number of test programs graded Passed, Inapplicable, or Unsupported for Specialized Needs Annexes (for any such set that was processed)
• the AVF that performed the validation testing
• the VSR reference number;
• the certificate's reference number
• the certificate's dates of effect and expiration.

Validation certificates for validation testing done with ACVC 2.1 will expire on 31 March 2000.

5.7 Validation Testing with Expired ACVC Versions

For some special procurement requirements, a customer might wish to have validation testing done with an expired version of the ACVC. An AVF may perform such testing at the customer's request, but the AJPO will not issue a validation certificate for this validation testing. Thus, the result of such testing is the VSR. Additionally, the AJPO will issue a letter to the customer noting whether the tested compiler met the requirements for validation of that expired ACVC version. An Ada compiler so tested is not considered to be validated.

5.8 Retention of Validation Records

The certification body retains certain validation records for at least six months after the expiration of the associated validation certificate. These records include the VSR (which includes a copy of the Declaration of Conformance and the validation certificate), any VSR Supplements, and any reference-configuration registration requests. Each AVF will determine the length of time that its validation records are retained.

5.9 Advertising Validated Status

The customer will not advertise or make public claims that the Ada compiler is validated until after receiving a validation certificate or after receiving formal notification from the AVF that the AJPO has issued a validation certificate. A waiver of confidentiality must be signed by a customer who intends to advertise the completion of events that indicate progress toward completion of validation. If a waiver of confidentiality has been signed with the AVF, the AVF will respond to inquiries about the customer’s advertisements or public claims by acknowledging receipt of validation materials (i.e., a formal agreement, pre-validation results, or validation testing results) without judgment concerning the success of the validation testing.

6. SIGNIFICANCE OF VALIDATION

An Ada compiler is typically designed to be used in some computer-systems domain, i.e., on any member of a set of host and target computer-system pairs; furthermore, a compiler is usually provided with different modes of operation (also known as "options" or "switch" settings). In validation testing, a compiler is tested under one mode of operation on a particular member of a computer-systems domain. Thus, the significance of validation is that it establishes that the compiler has a mode of operation in which it passes the Ada certification body's conformity test for some member of the intended domain; it thereby establishes an approved results profile (see Section 6.6) of a conforming implementation for this domain.

A validation certificate identifies both the tested compiler and the host and target computer systems used in validation testing; these are known as, respectively, the "base compiler" and "base configuration". A base compiler is considered validated. Moreover, the validated status granted to this compiler may extend to maintained versions of it (i.e., which operate in the same domain) and also to re-hosted versions (i.e., which operate in another domain that differs only with regard to the host computer systems), if these versions produce an approved results profile that is associated with the certificate.

The AJPO publishes the Validated Compilers List, which reports the information contained on validation certificates. This list also identifies the registered reference configurations (see Section 6.7) and the intended computer-systems domain(s) (see Section 6.2) that are associated with each validation certificate.

6.1 Definition of a Validated Ada Compiler

A vendor that markets an Ada compiler as being "validated as per AJPO procedures" for some computer-systems domain, or that uses the Ada certification mark on products directly associated with a compiler, warrants that the compiler can process the ACVC and produce an approved results profile on some reference configuration belonging to that domain, where both this profile and configuration are associated with a current (i.e., not expired) validation certificate awarded to the vendor. Additionally, a vendor that markets an Ada compiler as being validated is held to be reiterating the same declaration made on the Declaration of Conformance that was signed and submitted for the base compiler.

6.2 Host/Target Computer Systems Domain

The AJPO will carry the vendor's description of the intended computer-systems domain on its Validated Compilers List. The vendor will submit this description to the AVO for approval. A computer-systems domain shall comprise only those computer-systems whose hardware implements compatible instruction sets, and whose operating-systems are versions of the same operating system. The vendor may update the description of the domain as necessary to accommodate changes introduced by compiler maintenance.

Common examples of compatible instruction sets and operating systems are two different computer models in a manufacturer's product line and an older and upgraded version of an operating system, or the computers produced by different manufacturers that use the same instruction set and operating systems, or any computer system and a simulation of it.

6.3 Maintenance and Validation

It is expected that vendors will perform various kinds of software maintenance on their compilers in order to continually improve their products. The AJPO allows the validated status of a base compiler to be extended to compilers that are derived from the base compiler by software maintenance. Maintenance may be corrective (to fix errors), perfective (to improve performance), or adaptive (to accommodate different operating environments).

6.4 Compiler Modes of Operation

An Ada compiler generally provides various modes of operation which are selected when the compiler is invoked through compiler switches/options. A validated compiler must provide at least one "standard" mode, as required by 1.1.5(11) of [Ada95], in which the Ada implementation conforms to [Ada95]. Similarly, a validated compiler has at least one mode in which it can process the ACVC to produce the approved results profile. A user should inquire of the vendor as to which modes of operation will produce the approved results profile.

6.5 Re-hosted Validated Compilers

Validation for one computer-systems domain may be used as a basis for granting validated status to compilers for other computer-systems domains that have the same target computer systems (but a different set of host computer systems). This is permitted in consideration that the target computer system is the principal determinant of the Ada implementation, and that re-hosting a compiler involves changes that are unlikely to introduce subtle non-conformities. If a vendor has a current validation certificate for an Ada compiler, then the vendor may re-host that compiler for any computer-systems domain and market it as being "validated as per AJPO procedures" (see Section 6.1) if the vendor registers some reference configuration for it.

The AJPO will carry the vendor's description of any re-hosted compiler's computer-systems domain on its Validated Compilers List, subject to AVO approval as described in Section 6.2.

6.6 Approved Results Profile

A results profile is the complete set of results of an implementation's processing a customized ACVC test suite. The results profile comprises:

• all compiler or linker diagnostic output that corresponds to code that causes the compiler or linker to fail to (successfully) compile or link a test program;
• all output from the executable tests' calls to the ACVC report procedures (of package Report);
• and the customization of the ACVC to enable it to be processed.

• in the exact placement or content of compiler or linker diagnostic output;
• in the output from procedure Report.Comment; or
• in the values of the NAME_SPECIFICATION macros (file-path names) used to customize the ACVC.

The results profile of the base compiler is documented in the VSR by the AVF upon the completion of validation testing. Hence, this results profile is associated with the validation certificate and is generally applicable to compilers that are derived from the base compiler by software maintenance and that target computer-systems that are compatible with that of the base configuration. However, in some cases the maintenance that is performed on a compiler will require that the customization of the ACVC be changed, or will result in changes to the output from processing the ACVC test suite, for the maintained compiler. Such differences in the results profile must be documented by the vendor and approved by the AVO in order for the different results profile to also be associated with the certificate. The form for a VSR Supplement is given in Appendix G. A vendor submits a completed form to the AVO for review; if it is acceptable, the VSR Supplement will be listed on the Validated Compilers List (and assigned an identification number that links it to the associated certificate).

6.7 Registration of Reference Configurations

As described in sections 6.1 and 6.5, all compilers that are "validated as per AJPO procedures" must process the ACVC and produce an approved results profile (see Section 6.6) on some reference configuration. A reference configuration is a computer system used by a vendor to confirm that a compiler produces an approved results profile, and which the vendor has registered with the certification body. (The base configuration is a reference configuration.) Reference configurations are to be representative of the computer-systems domain for which the compiler is marketed. Users can assess the applicability of compiler validation to their own computer systems by way of reference to those that a vendor uses for in-house testing.

In order for some host and target computer-systems configuration to be considered a "reference configuration", a vendor must register it with the certification body. The registration form for this is given in Appendix F. A vendor submits a completed form to the AVO for review; if it is acceptable, the configuration will be listed on the Validated Compilers List (and assigned an identification number that links it to the associated certificate).

Declaration of Conformance

Customer: <customer name>

Certificate Awardee: <name>

Ada Validation Facility: <name of Ada Validation Facility> ACVC Version: <version number of ACVC>

Ada Implementation:

Compiler: <name and version number of Ada compiler>

Host Computer System: <host hardware and operating system>

Target Computer System: <target hardware and operating system>

Declaration:

I, the undersigned, representing the Customer, declare that the Customer has made no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:1995) in the Ada Implementation above.

Declaration:

I, the undersigned, representing the Certificate Awardee, declare that the Certificate Awardee has made no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:1995) in the Ada Implementation above.

Configuration:<host / target hardware and operating systems>

ACVC Version:<ACVC version number>

Pre-Validation Submittal Date:<due date for in-house results>

Part A will be completed once by each implementer; part B will be completed for each dispute. It is not necessary for a pre-validation date to have been established. Part A information is treated as confidential.

Summary:<brief description of the dispute>

Discussion:<detailed description of the dispute>

In this Discussion, arguments should be specified using test line numbers and references to pertinent sections of the Ada standard or Commentaries (AI-xxxx). The implementer must describe the behavior of the implementation for the test or tests that are disputed, stating the particular test messages that are produced. It is sufficient for the detailed description to be limited to the particular segment of test code that is disputed. Relevant source code with compiler messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.)

If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the AVO to assess the reasonableness of these limitations.

Failure to fully specify the points pertinent to a dispute might result in an adverse decision being made, with the disputer having to further argue the case with a second submittal to the AVO. Yet it is possible that the Summary will suffice to adequately present a dispute.

A "deviation" is defined by the ACVC User’s Guide as any result from processing an ACVC test program that is not a passed, inapplicable, or unsupported result according to the established grading criteria. This intentionally broad definition of a "deviation" is intended to ensure that compiler implementers bring all deviant test results to the attention of the AVO or AVF, without assuming that such results are acceptable. In petitioning for acceptance of a deviation, the petitioner provides a rationale for each challenge made against a test program. Petitions are sent to the AVO, usually electronically, by the petitioner or by an AVF on behalf of their validation customer. For each deviation that is accepted (i.e., when the AVO rules in favor of the petition), generally some correction is indicated for the cited tests. The AVO may withdraw a test program or require that it be processed with some modification. The AVO will withdraw any test that is found to be incorrect to a degree that makes it unsuitable for validation. The withdrawal of a test consists of including it on a list of tests that are ignored for validations and applies only to a particular ACVC version. Additionally, withdrawn tests are usually not a part of the customized test suite used in validation.

C.2 Expedited Resolution Process

The AVO resolves challenges by any of three methods:

1. a resolution that was made previously is applied to the current petition (e.g., the same petition might be submitted at different times by different petitioners);
2. the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or
3. the resolution is based on the deliberations of the consulted Ada experts.

On receipt of a petition, the AVO checks whether the issue matches any that had been previously resolved. If the challenge is new, it is given an initial AVO analysis which involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous dispute deliberations. Often the AVO consults Ada experts in order to resolve any petition. The identity of the petitioner is not disclosed. Resolution of a petition made by the AVO.

C.3 Types of Resolutions

The resolution of a petition is either an acceptance or rejection of the petitioner’s arguments. Acceptance can result in either withdrawal of the test program from the ACVC, or in a "Code, Processing, or Grading" modification for validation. A dispute may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. Withdrawn test programs have no effect on validation (they are generally not processed). If the challenge shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the AVO will direct that the test(s) be processed with a test modification.

There are three types of test modification: Code, Processing, and Grading modifications.

1. A Code Modification is an actual change to the code of the test (e.g., adding a choice to an exception handler).
2. A Processing Modification is a change to the way in which the test is processed (e.g., re-ordering the compilation of component files of a multiple compilation test).
3. A Grading Modification is simply the grading of the observed results by other than the established grading criteria (e.g., interpreting particular intermediate output and a final "failed" result as "passed", according to an understanding of the dispute).

C.4 Summary

There is no limit on the number of test programs that can be challenged by an implementer. Although there is a risk that a petition will not be decided in the implementer’s favor, that risk can be managed so as to not affect validation by early submission of petitions. Any interested party may challenge an ACVC test program.

APPENDIX D. POINTS OF CONTACT Ada Joint Program Office (AJPO)

Ms. Joan McGarity
Ada Joint Program Office
Center For Computer Systems Engineering
Defense Information Systems Agency
5600 Columbia PikeFalls Church, VA 22041-2717
Tel: 703-681-2458
Email: ajpo@ncr.disa.mil

Ada Validation Facility Managers

Mr. Phil Brashear
Electronic Data Systems
1230 E. Dayton-Yellow Springs Road
Fairborn, OH 45324
Tel: 937-754-4848 or 937-754-4846
FAX: 937-754-4949
Email: edsavf@dysmailpo.deisoh.msd.eds.com

Mr. Jon Leigh
The National Computing Centre, Ltd.Oxford Road
Manchester, England MI7ED
Tel: +44-61-228-6333
Email: jon@ncc.co.uk

Mr. Michael Tonndorf
IABG, Software-Technologie IS24
Einsteinstr. 20
D-85521 0ttobrunn, Germany
Tel: +49-89-6088-2477
Email: tonndorf@iabg.de

Mr. Clyde Roby
Institute for Defense Analyses
1801 N. Beauregard St.
Alexandria, Va. 22311
Tel: 703-845-6666
FAX: 703-845-6788
Email: avo@sw-eng.falls-church.va.us

Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG)

Dr. Erhard Ploedereder
University of Stuttgart
Institute for Computer Science
Breitwiesenstr. 20-22D-70565
Stuttgart, Germany
Tel: +49 +711 7816-322
Fax: +49 +711 7816-380
Email: ploedere@informatik.uni-stuttgart.de

Ada Information Clearinghouse
P.O. Box 1866
Falls Church, VA 22041USA
Tel: 703-681-2466 OR 1-800-232-4211
Email: adainfo@sw-eng.falls-church.va.us
WWW: http://www.adaic.com/

Ada Compiler Validation Capability (ACVC)

The ACVC is available to the general public from an AVF; it is also available from the Ada Information Clearinghouse (AdaIC) on the Internet or via anonymous ftp.
URL: http://archive.adaic.com/compilers/acvc/

In either case, look for "compilers" and "acvc."
Questions concerning Ada validation or comments on ACVC test programs should be sent to:
acvc-comment@sw-eng.falls-church.va.us

[Ada95]ANSI/ISO/IEC 8652:1995 Ada 95 Reference Manual, January 1995 (supersedes [Ada83]).

[Ada83] American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652:1987 and FIPS 119, 1985.

[ANSI/IEEE 90]American National Standards Institute / Institute of Electrical and Electronic Engineers, Inc., Standard 610.12-1990; "ANSI/IEEE Standard Glossary of Software Engineering Terminology".

[FIPS] Federal Information Processing Standards publications relative to the Ada Programming Language are FIPS 119 and FIPS 119-1.

[ISO 74] International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms.

[ISO/IEC 91] International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities.

Reference Configuration Registration Form

Vendor:<name of vendor>

Compiler:<identification of Ada compiler>

Validation Certificate No.:<identifier for validation certificate>

ACVC Version:<ACVC version number>

Test Configuration(s):

Host Computer System: <host hardware and operating system>

Target Computer System: <target hardware and operating system>

I, the undersigned, representing the Vendor, declare that, for the items identified above, the Compiler was used in the Test Configuration to process the ACVC and that this Ada implementation produced the same results profile that is documented

Choose one of the following three clauses to end the above paragraph:

1. in the Validation Summary Report (VSR) that is identified on the validation certificate.
2. in the Validation Summary Report (VSR) that is identified on the validation certificate and the associated VSR Supplement of <yyyy-mm-dd>.
3. {either choice 1. or 2. above} except for the differences reported and explained on the VSR Supplement accompanying this registration request.

VSR Supplement

Vendor:<name of vendor>

In reference to:

Validation Certificate No.: <VC number>

Validation Summary Report: <VSR number>

Include the following line, if applicable:

VSR Supplement dated: <VSR Supplement date>

This document describes differences in the Results Profile of compilers derived by software maintenance from that of the cited validation reference above.

Choose only one of the following three paragraphs, whichever applies:

1.

The following compiler

<full, particular compiler designation>

used in the Base Configuration produced the same Results Profile that is documented by the reference above, except for the differences described below.

2.

The following compiler

<full, particular compiler designation>

>used in Reference Configuration <RC reference number> produced the same Results Profile that is documented by the reference above, except for the differences described below.

3.

The Ada implementation described in the accompanying Reference Configuration Registration Form produced the same Results Profile that is documented by the reference above, except for the differences described below.

DIFFERENCES IN ACVC CUSTOMIZATION

Explain all differences in the customization of the ACVC other than the setting of the NAME_SPECIFICATION(1,2,3) macros.

DIFFERENCES IN TEST RESULTS

Show all executable results that differ from those that have been previously documented by the VSR or VSR Supplement, and explain the reason for these differences. Explain any differences in compilation of executable tests (e.g., if some tests no longer can be successfully compiled, or if they newly can be).

I, the undersigned, representing the Vendor, declare that the above statements are true.