Ada Compiler Validation Procedures VERSION 5.0 F I N A L D R A F T 2 September 1997 PREPARED FOR Ada Joint Program Office Center For Computer Systems Engineering Defense Information Systems Agency 5600 Columbia Pike Falls Church, Virginia 22041-2717 USA EXECUTIVE SUMMARY 1 1. INTRODUCTION 1 2. GLOSSARY OF TERMS 2 3. ORGANIZATIONS AND RESPONSIBILITIES 5 3.1 Ada Joint Program Office (AJPO) 5 3.2 Ada Validation Organization (AVO) 5 3.3 Ada Validation Facilities (AVFs) 6 3.4 Customers 6 4. THE ADA COMPILER VALIDATION CAPABILITY 7 4.1 Versions 7 4.2 Applicability of ACVC Test Programs 7 4.3 Test Modifications 7 4.4 Test Withdrawal 8 4.5 Customization 8 4.6 Availability 8 5. VALIDATION TESTING 9 5.1 Validation Agreement 9 5.2 Prevalidation 10 5.2.1 Customer Testing 10 5.2.2 Submission of Results 10 5.2.3 Test Modifications 11 5.2.4 Test Issues 11 5.2.5 Test Issue Resolution 12 5.2.6 Incomplete Prevalidations 12 5.2.7 Successful Prevalidations 12 5.3 Validation Testing 12 5.4 Declaration of Conformance 13 5.5 Validation Summary Report (VSR) 13 5.5.1 VSR Production 13 5.5.2 VSR Availability 14 5.6 Successful Validation 14 5.7 Validation with Superseded Versions of Test Suite 14 5.8 Retention of Validation Records 14 5.9 Advertising Validated Status 15 5.10 Content of Ada Validation Certificate 15 6. SIGNIFICANCE OF VALIDATION 16 6.1 Host/Target Computer Systems Domain 16 6.2 Maintenance and Validation 16 6.3 Re-hosted Validated Compilers 17 6.4 Compiler Modes of Operation 17 6.5 Approved Results Profile 17 APPENDIX A. DECLARATION OF CONFORMANCE 19 APPENDIX B. IMPLEMENTER DISPUTE FORMAT 20 APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS 21 C.1 Introduction 21 C.2 Expedited Resolution Process 21 C.3 Types of Resolutions 22 C.4 Summary 22 APPENDIX D. POINTS OF CONTACT 23 APPENDIX E. REFERENCES 25 APPENDIX F. SAMPLE VALIDATION CERTIFICATE 26 EXECUTIVE SUMMARY The Ada Compiler Validation Procedures define the certification body for the Ada programming language and set forth its rules of procedure for testing implementations of the Ada language for conformity to the language standard. The process of testing implementations is known as validation. This version (5.0) of the Procedures is a revision to Version 4.2 to reflect two important changes in Ada Validation: the end of the period of transition in validation from the use of the original language standard [Ada83] to the revised standard [Ada95] (January 1995 to July 1997); and the transferal of validation authority for the Ada language from the National Institute for Standards and Technology (NIST) to the United States Department of Defense (DoD). This document supersedes all previous versions of the Ada Compiler Validation Procedures. During the transition period, which began with the publication of the revised Ada language standard, [Ada95], in January 1995, validation was conducted according to either the original or revised standard. Testing for the revised standard was introductory in both the limited number and extent of the test programs. Furthermore, there was an allowance for implementations to fail tests for language specifications of [Ada95] that are not present in [Ada83]. This validation lenience was intended to encourage implementers to bring useable partial implementations of Ada 95 to the market quickly; validation would still give a common conformity check on these implementations. With the completion of [Ada95] and near completion of the full validation test suite, the DoD transferred the certification authority to the NIST in October 1996; the NIST is the certification authority for other FIPS standards. However, the NIST changed its focus and announced that it would soon bring to a close all of its conformity-testing activities; the DoD then decided that it should continue to support Ada compiler validation. Moreover, the National Research Council issued a report on Ada recommending that the DoD sustain its investment in an Ada infrastructure, including support for validation. As a result, the Ada certification authority was assumed by the DoD in April 1997. The Ada certification body comprises the DoD’s Ada Joint Program Office (AJPO), a principal technical advisor called the Ada Validation Organization (AVO), and the AJPO-recognized test laboratories called Ada Validation Facilities (AVFs) which conduct validation testing. The AJPO is responsible for the maintenance of the validation test suite, the issuance of validation certificates, and the establishment of these Ada Compiler Validation Procedures. Ada Compiler Validation Capability (ACVC) The ACVC is a conformity test suite (and supporting documents); it is designed to ensure that validated Ada compilers achieve a high degree of conformity to the Ada language standard. The ACVC is customized by an AVF for each Ada implementation that is tested; customization consists of adjusting the ACVC appropriately for various implementation characteristics. The ACVC is maintained by the AJPO; the ACVC is upgraded according to changes in interpretations of the Ada standard by the language maintenance body (ISO Working Group 9, Ada Rapporteur Group) and the discovery of deficiencies in the test programs. New versions of the ACVC are released according to a schedule set by the AJPO. ACVC versions that tested conformity to [Ada83] were numbered “1.1” through “1.11”, the final version; versions that test conformity to [Ada95] are numbered “2.0”, “2.0.1”, and “2.1”. ACVC versions 2.0 & 2.0.1 were used during the transition period; ACVC 2.0.1 was an intermediate release with corrections to many errors of version 2.0, and also was used with slightly stronger validation requirements. ACVC 2.1 is the first version that is considered to be a full test of [Ada95]. (The successors to this version will be numbered “2.2”, “2.3”, and so on, as needed.) For the compiler of an Ada implementation to receive validated status under ACVC 2.1, the implementation must process each test program (for the “core” language) of a customized ACVC so that the result is graded passed, inapplicable, or unsupported by ACVC grading criteria. (Under ACVC versions 2.0 and 2.0.1, an implementation was allowed to fail any test program that tested language features introduced by the revised standard.) An AVF customizes the ACVC for a particular Ada implementation by appropriately setting test parameters, by removing “withdrawn” tests (tests ruled by the AVO to be in error) and certain inapplicable tests, by splitting as needed test files with multiple intended errors so to enable complete error detection, by making any other modifications to the tests as directed by the AVO, and by including the set of tests for each Specialized Needs Annex as requested by the validation customer. In addition to the specification of a “core” language, [Ada95] contains several Specialized Needs Annexes; these are optional language requirements designed to meet the particular needs of various general application domains, such as information-systems programming. An implementation of [Ada95] need not include implementation of any of these annexes, or it may implement only some of the features of these annexes. Whereas all ACVC test programs for the core language must be processed during validation, those for the Specialized Needs Annexes are processed only upon vendor request. A validation certificate is issued only if all tests for the core are correctly processed; it will additionally give credit for support of a Specialized Needs Annex to the extent that the relevant set of tests is correctly processed. Validation Validation involves interaction between the AVF customer and the Ada certification body. The testing process consists of well defined actions which, when completed successfully, result in the award of a validation certificate for the tested Ada compiler. The key actions in the validation of an Ada compiler are: 1. The customer and an AVF reach a formal agreement for validation, including the dates for the submittal of the results of customer-administered processing of the ACVC and for AVF witness testing. 2. The customer petitions for deviation from the requirements of each ACVC test program that is believed to be wrong for the candidate implementation(s). 3. The AVO rules on the customer petitions. 4. The customer processes the ACVC on the candidate implementation(s) and submits the results to the AVF. 5. The AVF analyzes the results of the customer’s independent processing of the ACVC. (If the results are not acceptable, the previous action must be repeated, and new results analyzed.) 6. The AVF conducts witness testing of the candidate Ada implementation(s), documenting this testing in a Validation Summary Report (VSR) which the AVF submits to the AVO for review. 7. The customer signs a Declaration of Conformance for each candidate Ada implementation. 8. The AVO reviews the VSR, with comment to the AVF. The AVO recommends to the AJPO that a validation certificate be issued for the tested compiler(s), if the testing is successful. 9. The AJPO issues a validation certificate for each tested compiler upon the successful completion of the preceding actions. Hence, successful testing of an Ada implementation concludes with the AJPO's awarding a validation certificate for an Ada compiler to the customer. This validation certificate grants validated status to that particular compiler, which will be listed on the AJPO's Validated Compilers List (VCL). The VCL may also carry a description of the domain of compatible computer systems in which the compiler should produce the same results documented by the VSR, as asserted by the vendor. The vendor may perform maintenance on the compiler and retain validated status for such derived versions of it, so long as the vendor ensures that they produce the same ACVC results as are documented in the VSR, when used on a registered “reference platform” (the computer systems used in the formal testing are the initial reference platform). This maintenance may even include adaptive maintenance that enables the compiler to run on entirely different host computers (i.e., re-hosting) or to target closely related target computers. The AJPO uses a registration process whereby a compiler vendor may register each computer system used to ensure that delivered versions of the compiler process the ACVC correctly. (These additional computer systems and that used for formal testing are known as “reference platforms”.) 1. INTRODUCTION This document provides operating policy and procedures of the Ada certification system. The Ada certification system is managed by the Ada Joint Program Office (AJPO) of the United States Department of Defense (DoD). The two other principals of the Ada certification body are the Ada Validation Organization (AVO) for technical support and the Ada Validation Facilities (AVFs) for performing witness testing of Ada implementations. These procedures address the validation of Ada compilers of implementations of [Ada95] and provide an operational definition of a validation Ada compiler. These procedures are effective September 1997 until they are superseded or revoked. Detailed procedures regarding the application of the Ada Compiler Validation Capability (ACVC) are given in the ACVC User’s Guide. The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. The AJPO established a certification system to realize the benefits of standardization which include the ability to transfer software and programming expertise between computer systems that use a conforming Ada compiler. It is important to note the scope and intent of validation. The purpose of validation is to ensure that Ada implementations achieve a high degree of conformity with the standard. Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada validation. Moreover, it is important to note that the ACVC conformity test is the processing of a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; validated implementations may fail to conform to the standard in ways peculiar to each, under particular circumstances. Validation testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada validation is to identify Ada processors which may be procured and used to develop application programs which meet the [Ada95] goals of portability and interoperability. The ACVC Test Suite is not designed to replace the vendor’s quality assurance testing or systematically to detect inconsistencies or “bugs.” The technical goal of Ada validation testing is primarily to verify that the Ada product correctly supports all required features. Rather than exhaustive testing of permutations of features, the ACVC Test Suite contains a carefully-chosen set of test cases which cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard. Neither is validation intended as a means of performance benchmarking. The VSR does not contain information about the speed, cost, or efficiency of executing the validation tests. 2. GLOSSARY OF TERMS Ada programming language: The language defined by reference [Ada95]. ACVC User’s Guide: A document that explains the technical details of processing the test programs and evaluating their results. Ada Compiler Validation Capability (ACVC): The means for testing conformity of Ada implementations, consisting of the test suite, the support programs, and the ACVC User’s Guide. Ada implementation: An Ada compiler with its host and target computers and operating systems (kernels) Ada Rapporteur Group (ARG): The Ada Rapporteur group (ARG) is a subgroup of ISO/IEC/JTC1/SC22/WG9, the International Organization for Standardization Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language. Ada Validation Facility (AVF): The part of the certification body that performs validation testing of Ada compilers. Certification body: [ISO/IEC 86] An impartial body, governmental or non-governmental, possessing the necessary competence and reliability to operate a certification system, and in which the interests of all parties concerned with the functioning of the system are represented. Certification system: [ISO/IEC 86] A system having its own rules of procedure and management for carrying out conformity certifications. Compiler: The software and any needed hardware that must be added to a given host and target machine to allow transformation of programs into executable form and execution thereof. Computer system: [IEEE 90] A system containing one or more computers and associated software. Configuration management: [IEEE 90] A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements. Conformity: [ISO/IEC 86] Fulfillment by a product, process or service of all requirements specified. [Note: Also see Section 1.1.3 of ANSI/ISO/IEC 8652:1995] Core language: The Sections 1-13 and Annexes A, B and J of [Ada95]. [See Section 1.1.2 of ANSI/ISO/IEC 8652:1995] Customer: An individual or corporate entity who has an agreement with an AVF that specifies the terms and conditions for AVF services (of any kind) to be performed. Customized test suite: The ACVC tests, adjusted as necessary, that must be used for validation testing of a given Ada implementation (see Section 4.5). Declaration of conformance: A formal statement from a customer assuring that conformity is realized on the Ada implementation for which validation status is requested. [See Appendix A for the format of a declaration of conformance.] Host machine: [IEEE 90] (l) A computer used to develop software intended for another computer (contrast with target machine). (2) A computer used to emulate another computer. (3) The computer on which a program or file is installed. (4) In a computer network, a computer that provides processing capabilities to users of the network. Inapplicable test: A test that contains one or more test objectives found to be irrelevant for the given Ada implementation. Instruction set: [IEEE 90] The complete set of instructions recognized by a given computer or provided by a given programming language. Operating system: [IEEE 90] A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. Perfective maintenance: [ANSI/IEEE 90] Maintenance performed to improve performance or maintainability. Prevalidation testing: Process of supplying to an AVF the results of processing an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests. Results profile: The aggregate of test results produced by processing the customized test suite according to given evaluation criteria (see Section 6). Software maintenance: [ANSI/IEEE 83] Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. Specialized Needs Annexes: Annexes C through H of [Ada95]. These Annexes define standards for additional functionality required by specific application areas. An Ada Implementation may support some or none of these annexes. Target machine: [IEEE 90] (1) The computer on which a program is intended to run. (2) A computer being emulated by another computer. Target run-time system: The set of subprograms that may be invoked by linking, loading, and executing object code generated by an Ada compiler. If these subprograms use or depend upon the services of an operating system, then the target runtime system includes those portions of that operating system. Test issue: Any problem arising during validation (see Section 5.2.5). Validation: The process of checking the conformity of an Ada implementation to the Ada programming language and of issuing a validation certificate for the implementation. Validated Ada compiler: The compiler of a validated Ada implementation. Validation Certificate (VC): A certificate issued by authority of the AJPO for a successfully tested Ada compiler. Validation Summary Report (VSR): A report produced by an AVF containing results that are observed from on-site, witness testing a specific Ada implementation or grouping of Ada implementations. Withdrawn test: A test that has been officially removed from the test suite because the test did not meet the test objective. In most cases, the test case will be rewritten and included in a future release of the test suite. Until then the test case is not used. 3. ORGANIZATIONS AND RESPONSIBILITIES This section specifies the roles of organizations that form the certification body and of customers who receive service from them. 3.1 Ada Joint Program Office (AJPO) The AJPO is the head of the certification body. It directs the certification system by: ? establishing the structure and roles of the certification body and designating its members; ? setting validation standards to be followed by the AVFs; ? controlling the use of the Ada certification mark and setting the conditions for the award of and issuing of validation certificates; ? contracting for the development of each ACVC version, overseeing ACVC quality control and configuration management, approving the content of the ACVC, and establishing the ACVC release schedule; ? resolving issues that may arise during validation when these issues cannot be resolved through the best efforts of the AVO and the AVF; ? publishing the official list of validated Ada compilers; and ? publishing these Ada Compiler Validation Procedures. The AJPO may issue an AVF charter to an organization that it has recognized as an accredited testing laboratory. The AJPO may issue a charter to an organization located in a country that has a Memorandum of Understanding (MoU) with the United States government covering the chartering of AVFs according to the rules specified in the MoU. An AVF charter shall remain in effect for the duration specified in the charter; however, a charter can be revoked by the AJPO, at any time, for due cause. The AJPO may direct an impartial body to conduct an audit at any time or prior to issuing an AVF charter. Audits are conducted in accordance with procedures established by the AJPO at the time of the audit and are tailored to reflect the purpose of the audit. 3.2 Ada Validation Organization (AVO) The AVO provides the technical and administrative support required to operate the certification system by: ? advising the AJPO and the AVFs with regard to the validation procedures; ? resolving issues that may arise during the validation process; ? reviewing all validation summary reports (VSRs) prepared by AVFs: ? reviewing each validation summary report (VSR), assessing the fulfillment of validation requirements by a customer, and recommending to the AJPO that a validation certificate be issued for the successful completion of validation; ? participating in the ACVC quality control and configuration management process; ? receiving and ruling on challenges to ACVC test programs, including making the withdrawal of, or requiring modifications to, such test programs; and ? convening meetings of the members of the certification body at appropriate intervals to discuss the process and to evaluate practices. 3.3 Ada Validation Facilities (AVFs) An AVF is a test laboratory chartered by the AJPO to conduct validation testing by: ? entering into a contract for validation services with a customer; ? consulting pre-validation analysis of a candidate Ada implementation’s results from processing the ACVC test suite, and conducting witness testing of Ada implementations; ? forwarding unresolved test issues to the AVO for review ? documenting validation testing in a VSR, forwarding the VSR to the AVO for review and comment, and distributing the VSR. 3.4 Customers Customers obtain services from the Ada certification body in matters concerning Ada validation. In requesting services of the Ada certification body, customers are to provide accurate and complete information to perform validation or to obtain other services. 4. THE Ada COMPILER VALIDATION CAPABILITY The ACVC is designed to demonstrate the conformity of an Ada implementation with the standard. The ACVC is distributed as a collection of test programs, support programs which facilitate processing the tests, and an ACVC User’s Guide that explains the criteria for evaluating the results. 4.1 Versions New ACVC versions are released periodically according to a schedule which is determined by the AJPO. Each new version incorporates changes to the ACVC as deemed necessary by the AJPO. These changes are made in order to reflect a revision of the standard, to incorporate ISO WG9 interpretations, or to address implementer or user comments. The test objectives and test programs for each new ACVC version are available for public review and comment before that version is issued for use in validation. Questions concerning Ada validation or comments on ACVC test programs should be submitted by e-mail to acvc-comment@sw-eng.falls-church.va.us or by FAX or regular mail to the AVO (see Appendix D, Points of Contact). 4.2 Applicability of ACVC Test Programs Each ACVC test program has one or more test objectives which are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada implementation (e.g., “check floating-point operations for digits 18”). These test programs generally contain an explicit indication of their applicability and the expected behavior of Ada implementations for which they do not apply. The determination of applicability is made according to the grading criteria in the ACVC or as a ruling by the AVO. All applicable test programs must be processed and passed according to the specified grading criteria. ACVC test programs for any Specialized Needs Annexes (C, D, E, F, G, and H of [Ada95]) will not be processed unless the AVF customer requests that they be processed. A summary of the results of these tests will be put on the validation certificate to indicate the extent of support that a compiler provides for the Annex. 4.3 Test Modifications The certification body strives to apply the ACVC as uniformly as practical to all Ada compilers. In order to apply common test objectives that depend on implementation- dependent characteristics (e.g., line lengths and numeric types), some test programs must be adjusted to a given implementation following the procedures given in the ACVC User’s Guide. These adjustments consist of the insertion of implementation dependent values in certain test programs at places prescribed by the ACVC User’s Guide. In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (e.g., the algorithm for recovering from syntax errors). The allowable changes for each Ada implementation are determined by the ACVC User’s Guide and the AVO, and may require AVF assistance — especially in the case of compiler error- recovery problems. 4.4 Test Withdrawal In any ACVC version, it is possible that a test program is based on assumptions that need not hold true for all Ada implementations or that a test program does not meet its objective. In these cases, the AVO may issue a correction to the evaluation criteria in the ACVC User’s Guide or the test program may be withdrawn from that version of the test suite. Any interested party may challenge a test program by sending a petition to the AVO. When an AVF customer prepares for validation, the customer may submit petitions against ACVC test programs through an AVF. The form for submitting a challenge is provided in Appendix B. 4.5 Customization A customized test suite is produced by the AVF for each Ada implementation that is a candidate for validation. This customization always consists of removing withdrawn tests and in making required modifications to test and support programs; and, it may include removal of some inapplicable tests as allowed by the ACVC User’s Guide. 4.6 Availability The ACVC is available to the general public from an AVF or from the AJPO on the Internet. See APPENDIX D for points of contact. 5. VALIDATION TESTING There are a number of steps that must be completed by a customer and the certification body so that the customer obtains a validation certificate and a VSR. The same ACVC version must be used to complete the steps described in this section. The AJPO announces the in-use and end-use period for each ACVC version. The AVF must begin on-site, witness, validation testing of the Ada implementation at the customer’s site before the current ACVC version expires or else validation with that ACVC version will not be allowed. Anyone intending to obtain a validation certificate should contact an AVF without delay for advice on the handling of the ACVC, on interpretation of the test grading criteria, and on the operational details of that AVF’s management practices. The steps are: 1. Validation Agreement 2. Prevalidation 3. Validation Testing 4. Declaration of Conformance 5. Validation Summary Report 6. Validation Certificate 5.1 Validation Agreement In order to obtain services from the certification body, an interested party must become a customer of an AVF by reaching a formal agreement. This agreement should address the following topics: ? identification of the Ada implementation(s) to be tested and the ACVC version to be used; ? a statement of work, including analysis of prevalidation testing, witness testing, validation, and preparation of the VSR; ? the format of data to be exchanged; ? a schedule of events and the site of validation; ? financial arrangements; ? retention of records; ? AVF liability; and ? confidentiality of validation information. The schedule for events, deliverables, and payments should take into account the fact that certain steps in the validation process require interaction with other members of the certification body (i.e., AVO or AJPO). The AVF will put forth its best effort to keep confidential a customer’s intent to obtain a validation certificate and the projected schedule for validation. This confidentiality will not be allowed to interfere with the normal review procedures of validation. If the customer requests confidentiality for reasons of national security or procurement sensitivity, the customer will provide to the AVF an official, written statement describing the request and the reason(s) for the request; the AVF will also obtain further guidance from the AVO. 5.2 Prevalidation The requirements of this step are discussed separately so that the customer understands the interaction that is required with an AVF. 5.2.1 Customer Testing After entering into a formal agreement, the customer either provides the necessary information for the AVF to prepare a customized test suite or, the customer may prepare a customized test suite according to instructions in the ACVC User’s Guide. The customer then processes all the tests in this customized test suite using the candidate Ada implementation or another Ada implementation that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of an option controlling the production of information output. Any other exception constitutes a test issue that must be resolved with the AVF (see Section 5.2.5). Test issues should be sent to the AVF for analysis as soon as they are known. The customer must provide (unless explicitly waived by the AVF) the prevalidation materials in a timely manner or by the agreement requirements between the customer and the AVF. Prevalidation materials include as a minimum the processing of an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests. 5.2.2 Submission of Results Upon completion of testing, the customer delivers the complete set of results in the agreed format to the AVF. These results are accompanied by the following information: ? a list of test programs that the customer claims are inapplicable, together with a rationale for inapplicability; ? a list of test programs that are challenged (see Section 4.4) together with rationale (see Appendix B for format); ? the necessary information for the AVF to prepare the customized test suite, if applicable; ? an annotated sample command script; ? the complete set of option settings used for processing the customized test suite, including the default settings; and ? complete and current documentation for implementation-dependent characteristics as required in Annex M of [Ada95]. 5.2.3 Test Modifications In order to meet the test objective, it may be required to modify the code, the processing method, or the grading of a test program. Only the AVO shall make the decision to use any of the following Test Modifications. Possible kinds of modification are: ? Code Modification: The source code of the test is changed. Examples of code modifications are the insertion of a pragma, the insertion of a representation clause, or the splitting of a B-test into several individual tests, if the compiler does not detect all intended errors in the original test. (This last example is the only exception to the rule that only the AVO shall make the decision to apply a Code Modification.) ? Processing Modification: The processing of the test by the Ada implementation for validation is changed. Examples of processing modification is the change of the compilation order for a test that consists of multiple compilations or the additional compilation of a specific support unit in the library. ? Grading Modification: The grading of a test result is changed. An example of a grading modification is the grading of a test other than what the output from REPORT.RESULT indicates. This may be required if the test makes assumptions about implementation features that are not supported by the implementation (e.g. the implementation of a file system on a bare target machine). 5.2.4 Test Issues A test issue may be any of the following: ? a missing or incomplete result to a test; ? a result presented in an inadequate form; ? a disagreement between the customer and the AVF as to the interpretation of a result; ? a change in the choice of options to be used during testing; ? a result that makes the Ada implementation fail a test program according to the current grading criteria; or ? an implementation-dependent characteristic that may affect the conformity of the Ada implementation. The material submitted by the customer is analyzed by the AVF and test issues are resolved. If the AVF and the customer cannot agree on a way to resolve a test issue, the issue will be referred to the AVO for a resolution (see Section 5.2.5). It may be justified to leave a test issue unresolved at prevalidation. For example, it may be impossible to check the processing of control characters by inspecting printed results. The AVF will note these unresolved issues and describe the results that will be expected during validation testing. It is also possible that the customer information provided for production of the customized test suite (see Section 5.2.1) was insufficient so that corrections to the customized test suite must be made and additional processing will be required. 5.2.5 Test Issue Resolution A customer may challenge the applicability or correctness of any particular ACVC test program. Such challenges should be presented to the AVF in the test-dispute format (see Appendix B). The AVF will forward challenges to the AVO for resolution; the AVO will strive to rule on the challenge within two weeks of receiving it. The AVO will report all challenges and rulings to the AJPO. (See Appendix C for a description of the Test Dispute and Resolution Process.) 5.2.6 Incomplete Prevalidations The AVF and the customer may agree that, at the customer’s risk, parts of the customized test suite need not be processed during prevalidation. The customer must certify that the results from a previous prevalidation submitted to the AVF or validation results obtained by the AVF are identical to those that would have been obtained by the customer. The normal practice is to submit complete prevalidation results. 5.2.7 Successful Prevalidations Prevalidation testing is successful if the analysis of results and the resolution of test issues show that the candidate Ada implementation passes the customized test suite. Prevalidation is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until validation testing by agreement between the AVF and the customer. 5.3 Validation Testing Upon successful completion of prevalidation, with or without caveats, the AVF witnesses testing of the Ada implementation at the site and time mutually agreed by the AVF and customer. The AVF prepares a customized test suite based upon customer information and any information collected during the resolution of test issues. The customized test suite is installed and processed in the physical presence of an AVF representative. If the AVF determines that the results agree with those obtained from prevalidation and are satisfactory with respect to the caveats, the testing has been successful; otherwise, re-testing will be required, unless the validation attempt is discontinued. 5.4 Declaration of Conformance At the start of the on-site, witness testing of the customer’s Ada implementation, the customer will provide to the AVF a completed declaration of conformance. The customer should provide the information contained in the declaration of conformance to the AVF prior to the on-site testing as part of the prevalidation effort. The declaration of conformance states that the organization that is responsible for the production, maintenance or distribution of the Ada compiler is offering a product that is in conformity with the Ada programming language. The declaration of conformance becomes part of the AVF records and is copied into the VSR. A validation certificate will not be issued unless a signed declaration of conformance has been provided to the AVO. (See Appendix A, for an example of the declaration of conformance.) 5.5 Validation Summary Report (VSR) A VSR is produced for each validation testing effort. A single VSR may cover validation testing of several Ada implementations, provided that they all have the same results profile. The VSR provides the following documentation pertaining to the validation effort: ? identification of the customer responsible for validation of the Ada implementation; ? identification of the organization responsible for the production, maintenance, or distribution of the Ada compiler or Ada implementation; ? identification of the Ada implementation tested; ? options provided by the Ada compiler and identification of the options used for testing; ? the inapplicable test programs and implementation-dependent characteristics exhibited by the test programs that established inapplicability; ? the implementation-dependent characteristics pertinent to the customized test suite; ? identification of withdrawn test programs; ? a description of all Test Modifications made for the validation testing; and ? a description of failed test programs, if applicable. 5.5.1 VSR Production The VSR is prepared by the AVF but includes material that is produced by the customer, such as the documented compiler and linker options used during the Ada validation process. Draft versions of the VSR are sent to the AVO for review. The final version of the VSR is signed by the AVF, the AVO, and the AJPO. 5.5.2 VSR Availability The final version of the VSR is available to the general public from the customer and from the AVF that produced it. The AVF may require payment of a fee for VSR reproduction and mailing costs. (See Appendix D for points of contact.) 5.6 Successful Validation For each successful validation, one certificate is issued by the AJPO. The information on the validation certificate describes the tested Ada implementation: the source of this information is the signed declaration of conformance that describes the tested Ada implementation that the AVF provides to the AVO after completion of testing. The customer will ensure that the information contained on the validation certificate does not infringe on the rights of third parties and may be required to provide a written statement of consent from any third party involved. Validation certificates for ACVC 2.1 will expire 31 March 2000. An entry in the AJPO’s Validated Compiler List will be made for each validation certificate; this entry will be removed when the certificate expires. A sample Validation Certificate is provided in Appendix F. 5.7 Validation with Superseded Versions of Test Suite The AJPO is willing to allow a validation for an Ada processor with superseded versions of the test suite. The result of such a validation is a VSR which may be shown to customers requiring this type of evaluation. The AJPO procedures require the use of the latest version of the ACVC for validation testing. Validation Certificates are issued only for testing done with a current version of the test suite. Testing with an expired test suite is done only upon special request for specific procurement requirements; however, no Validation Certificate will be issued. When testing is performed with an expired test suite, the AJPO will provide a letter in lieu of a Validation Certificate. Testing must be conducted using the procedures and tests that were in effect for the ACVC version requested. The letter will note whether or not the implementation described in the VSR passed all applicable tests of the ACVC version. Concurrently with, but not later than, the request for validation with a superseded ACVC version, the applicant may identify derived implementation(s) that satisfy the requirements for registration as specified by the Ada Compiler Validation Procedures in effect for that ACVC version. 5.8 Retention of Validation Records Each AVF shall determine the length of time that records and information for an Ada validation are retained (at a minimum, for the life of the Validation Certificate). For the AVO, the records and information used in preparing the VSR will typically be retained for six (6) months after expiration of the latest Validation Certificate. 5.9 Advertising Validated Status The customer will not advertise or make public claims that the Ada implementation is validated until after receiving a validation certificate or after receiving formal notification from the AVF that the AJPO has issued a validation certificate. A waiver of confidentiality must be signed by a customer who intends to advertise the completion of events that indicate progress toward completion of validation. If a waiver of confidentiality has been signed with the AVF, the AVF will respond to inquiries about the customer’s advertisements or public claims by acknowledging receipt of validation materials (i.e., a formal agreement, pre-validation results, or validation testing results) without judgment concerning the success of the validation. 5.10 Content of Ada Validation Certificate The Ada Validation Certificate identifies: ? the Ada compiler under test ? the AVF that performed the validation ? the host and target computer systems ? the VSR number ? the validation certificate number ? the dates of issue and expiration of the certificate ? the extent of support of the Specialized Needs Annexes. 6. SIGNIFICANCE OF VALIDATION An Ada compiler is typically designed to be used in some computer-system domain, i.e., on any of a set of host/target pairs; further, a compiler is usually provided with different modes of operation (also known as “options” or “switch” settings). In validation testing, the compiler is tested under one mode of operation on a particular member of this set. Thus, for an Ada implementation that is awarded a VC, the significance of validation testing is that it demonstrates that the compiler has a mode of operation in which it passes the Ada certification body's test for conformity to the Ada standard for some member of the intended domain; it thereby establishes an approved results profile of a conforming implementation for this domain. A compiler that is awarded a VC is called a base compiler; it is considered “validated”. Moreover, the validated status granted to this compiler extends to maintained versions of it (i.e., which operate in the same domain) and also to re-hosted versions (i.e., which operate in another domain that differs only with regard to the hosts), if these versions produce an approved results profile that is associated with the base compiler. A vendor that markets an Ada compiler as being “validated as per AJPO Procedures” for some computer systems domain warrants that the compiler has processed the ACVC and produced an approved results profile for some host-target pair of that domain, and that the vendor has a current (i.e., not expired) VC for a compiler with the same target domain (i.e., re-hosting the compiler is permitted — see 6.3). 6.1 Host/Target Computer Systems Domain The AJPO will carry the vendor's description of the intended computer systems domain on its Validated Compilers List. The vendor will submit this description to the AVO for approval. A computer systems domain shall comprise only those computer systems whose hardware implements compatible instruction sets, and whose operating systems are versions of the same operating system. The vendor may update the description of the domain as necessary to accommodate changes introduced by compiler maintenance. Common examples of compatible instruction sets and operating systems are two different computer models in a manufacturer's product line and an older and upgraded version of an operating system, or the computers produced by different manufacturers that use the same instruction set and operating systems, or any computer system and a simulation of it. 6.2 Maintenance and Validation It is expected that vendors will perform various kinds of software maintenance on their compilers in order to continually improve their products. Thus, the AJPO allows the validated status of a base compiler to be extended to maintained versions of this compiler. Maintenance may be corrective (to fix errors), perfective (to improve performance), or adaptive (to accommodate different operating environments). A maintained compiler may be marketed as being “validated as per AJPO Procedures” if the compiler produces an approved results profile for some host-target pair of the base compiler's domain. 6.3 Re-hosted Validated Compilers Validation testing in one host-target domain may be used as a basis for extending validated status to compilers for other host-target domains that had compatible targets (but incompatible hosts). This decision was based on the consideration that the target computer system is the principal determinant of the Ada implementation, and that re- hosting a compiler involves changes that are unlikely to introduce subtle non- conformities. If a vendor has a current VC for an Ada compiler, then the vendor may re-host that compiler for any computer-system domain and market it as being “validated as per AJPO Procedures” if the compiler produces an approved results profile for some host- target pair of that domain. The AJPO will carry the vendor's description of any re-hosted compiler's computer systems domain on its Validated Compilers List, subject to AVO approval as described in 6.1. 6.4 Compiler Modes of Operation An Ada compiler generally provides various modes of operation which are selected when the compiler is invoked. A validated compiler must provide at least one “standard” mode, as required by [Ada95], in which the Ada implementation conforms to [Ada95]. Similarly, a validated compiler has at least one mode in which it can process the ACVC to produce the approved results profile. A user should enquire of the vendor as to which modes of operation will produce the approved results profile. 6.5 Approved Results Profile A results profile is the complete set of results of an implementation's processing a customized ACVC test suite. The results profile comprises: all compiler or linker diagnostic output that corresponds to code that causes the compiler or linker to fail to (successfully) compile or link a test program (without regard to the exact placement or content of such output); all output from the executable tests' calls to the ACVC report procedures (of package Report: Test, Comment, Special_Action, Inapplicable, Failed, & Result); and the customization of the ACVC to enable it to be processed. The customization of an ACVC test suite comprises: the replacement of test-parameterization symbols with particular implementation-dependent values, the setting of ACVC-specified implementation-dependent constants, the removal of withdrawn tests, and any AVO- specified code or processing modifications. An approved results profile is one that the certification body has formally documented in a VSR and supplements. An approved results profile is applicable only to a particular vendor's compilers for the particular computer systems domains that have compatible targets. The results profile of the base compiler is documented in the VSR by the AVF upon the completion of validation testing. This results profile is associated with the base compiler and is generally applicable to maintenance versions of that compiler. However, in some cases the maintenance that is performed on a compiler will require that the customization of the ACVC be changed, or will result in changes to the output from processing the ACVC test suite, for the maintained compiler. Such differences in the results profile must be documented by the vendor and approved by the AVO in order for the different results profile to also be associated with the base compiler. APPENDIX A. DECLARATION OF CONFORMANCE Declaration of Conformance Customer: Certificate Awardee: Ada Validation Facility: ACVC Version: Ada Implementation: Ada Compiler Name and Version: Host Computer System: Target Computer System: Declaration: I, the undersigned, representing the Customer, declare that the Customer has made no deliberate deviations from the Ada language standard (ANSI/ISO/IEC 8652:1995). ________________________________________ __________________ Customer Signature Date Company Title ________________________________________ __________________ Certificate Awardee Signature Date Company Title APPENDIX B. IMPLEMENTER DISPUTE FORMAT [part A] Implementer: Configuration: ACVC Version: Pre-Validation Submittal Date: [Part A will be completed once by each implementer; part B will be completed for each dispute. It is not necessary for a pre-validation date to have been established. Part A information is treated as confidential.] [part B] Reference: Summary: Discussion: [In this Discussion, arguments should be specified using test line #s and references to pertinent sections of the Ada standard or Commentaries (Al-xxxx). The implementer must describe the behavior of the implementation for the test or tests that are disputed, stating the particular test messages that are produced. It is sufficient for the detailed description to be limited to the particular segment of test code that is disputed. Relevant source code with compiler messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.)] If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the AJPO to assess the reasonableness of these limitations. Failure to fully specify the points pertinent to a dispute might result in an adverse decision being made, with the disputer having to further argue the case with a second submittal to the AJPO. Yet it is possible that the Summary will suffice to adequately present a dispute. APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS C.1 Introduction A “deviation” is defined by the ACVC User’s Guide as any result from processing an ACVC test program that is not a passed, inapplicable, or unsupported result according to the established grading criteria. This intentionally broad definition of a “deviation” is intended to ensure that compiler implementers bring all deviant test results to the attention of the AVO or AVF, without assuming that such results are acceptable. In petitioning for acceptance of a deviation, the petitioner provides a rationale for each challenge made against a test program. Petitions are sent to the AVO, usually electronically, by the petitioner or by an AVF on behalf of their validation customer. For each deviation that is accepted (i.e., when the AVO rules in favor of the petition), generally some correction is indicated for the cited tests. The AVO may withdraw a test program or require that it be processed with some modification. The AVO will withdraw any test that is found to be incorrect to a degree that makes it unsuitable for validation. The withdrawal of a test consists of including it on a list of tests that are ignored for validations and applies only to a particular ACVC version. Additionally, withdrawn tests are usually not a part of the customized test suite used in validation. C.2 Expedited Resolution Process The AVO resolves challenges by any of three methods: 1. a resolution that was made previously is applied to the current petition (e.g., the same petition might be submitted at different times by different petitioners); 2. the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or 3. the resolution is determined based on the deliberations of the FRT. Although the Ada Compiler Validation Procedures do not set a limit on the length of time for reaching a resolution, the AVO attempts to rule on petitions within two weeks. The AVO also attempts to place priority on ruling on petitions from AVF customers who have a firmly scheduled date for validation testing. Implementers should submit challenges well in advance of a scheduled validation testing date (see Section 5). On receipt of a petition, the AVO checks whether the issue matches any that had been previously resolved. If the challenge is new, it is given an initial AVO analysis which involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous dispute deliberations. Often the AVO consults Ada experts in order to resolve any petition. The identity of the petitioner is not disclosed. Resolution of a petition made by the AVO. C.3 Types of Resolutions The resolution of a petition is either an acceptance or rejection of the petitioner’s arguments. Acceptance can result in either withdrawal of the test program from the ACVC, or in a “Code, Processing, or Grading” modification for validation. A dispute may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. Withdrawn tests have no effect on validation (they are generally not processed). If the challenge shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the AVO will direct that the test(s) be processed with a test modification. There are three types of test modification: Code, Processing, and Grading modifications. 1. A Code Modification is an actual change to the code of the test (e.g., adding a choice to an exception handler). 2. A Processing Modification is a change to the way in which the test is processed (e.g., re-ordering the compilation of component files of a multi- compilation test). 3. A Grading Modification is simply the grading of the observed results by other than the established grading criteria (e.g., interpreting particular intermediate output and a final “failed” result as “passed”, according to an understanding of the dispute). All test modifications are documented in the VSR. C.4 Summary There is no limit on the number of test programs that can be challenged by an implementer. Although there is a risk that a petition will not be decided in the implementer’s favor, that risk can be managed so as to not affect validation by early submission of petitions. Any interested party may challenge an ACVC test program. APPENDIX D. POINTS OF CONTACT Ada Joint Program Office (AJPO) Ms. Joan McGarity Ada Joint Program Office Center For Computer Systems Engineering Defense Information Systems Agency 5600 Columbia Pike Falls Church, VA 22041-2717 Tel: 703-681-2458 Email: ajpo@ncr.disa.mil Ada Validation Facility Managers Mr. Phil Brashear Electronic Data Systems 1230 E. Dayton-Yellow Springs Road Fairborn, OH 45324 Tel: 937-754-4848 or 937-754-4846 FAX: 937-754-4949 Email: edsavf@dysmailpo.deisoh.msd.eds.com Mr. Jon Leigh The National Computing Centre, Ltd. Oxford Road Manchester England, MI7ED Tel: +44-61-228-6333 Email: jon@ncc.co.uk Mr. Michael Tonndorf IABG, Software-Technologie IS24 Einsteinstr. 20 D-85521 0ttobrunn Germany Tel: +49-89-6088-2477 Email: tonndorf@iabg.de Ada Validation Organization Mr. Clyde Roby Institute for Defense Analyses 1801 N. Beauregard St. Alexandria, Va. 22311 Tel: 703-845-6666 FAX: 703-845-6788 Email: avo@sw-eng.falls-church.va.us Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG) Dr. Erhard Ploedereder University of Stuttgart Institute for Computer Science Breitwiesenstr. 20-22 D-70565 Stuttgart Germany Tel: +49 +711 7816-322 Fax: +49 +711 7816-380 Email: ploedere@informatik.uni-stuttgart.de Ada Information Clearinghouse and Validated Compilers List Ada Information Clearinghouse P.O. Box 1866 Falls Church, VA 22041 USA Tel: 703-681-2466 OR 1-800-232-4211 Email: adainfo@sw-eng.falls-church.va.us WWW: http://sw-eng.falls-church.va.us/AdaIC/ Ada Compiler Validation Capability (ACVC) The ACVC is available to the general public from an AVF; it is also available from the Ada Information Clearinghouse (AdaIC) on the Internet or via anonymous ftp. URL: http://sw-eng.falls-church.va.us/adaic/ ftp://sw-eng.falls-church.va.us/public/AdaIC In either case, look for “compilers” and “acvc.” Questions concerning Ada validation or comments on ACVC test programs should be sent to: acvc-comment@sw-eng.falls-church.va.us APPENDIX E. REFERENCES [Ada95] ANSI/ISO/IEC 8652:1995 Ada 95 Reference Manual, January 1995 (supersedes Ada83) [Ada83] American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652:1987 and FIPS 119, 1985. [ANSI/IEEE 90] American National Standards Institute / Institute of Electrical and Electronic Engineers, Inc., Standard 610.12-1990; “ANSI/IEEE Standard Glossary of Software Engineering Terminology”. [FIPS] Federal Information Processing Standards publications relative to the Ada Programming Language are FIPS 119 and FIPS 119-1 [ISO 74] International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms. [ISO/IEC 91] International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities. APPENDIX F. SAMPLE VALIDATION CERTIFICATE (not included in electronic version) Ada and Beyond: Software Policies for the Department of Defense, Computer Science and Telecommunications Board, National Research Council, LOC 96-71960, ISBN 0-309-05597-0, November 1, 1996. xxvii ii ES-3 26