Ada Compiler Validation Procedures VERSION 5.0 D R A F T April 15, 1997 PREPARED FOR Ada Joint Program Office Center For Computer Systems Engineering Defense Information Systems Agency 5600 Columbia Pike Falls Church, Virginia 22041-2717 USA EXECUTIVE SUMMARY 1 1. INTRODUCTION 1 2. GLOSSARY OF TERMS 2 3. ORGANIZATIONS AND RESPONSIBILITIES 5 3.1 Ada Joint Program Office (AJPO) 5 3.2 Ada Validation Organization (AVO) 5 3.3 Ada Validation Facilities (AVFs) 6 3.4 Customers 6 4. THE Ada COMPILER VALIDATION CAPABILITY 7 4.1 Versions 7 4.2 Applicability of ACVC Test Programs 7 4.3 Test Modifications 7 4.4 Test Withdrawal 8 4.5 Customization 8 4.6 Availability 8 5. VALIDATION TESTING 9 5.1 Validation Agreement 9 5.2 Prevalidation 10 5.2.1 Customer Testing 10 5.2.2 Submission of Results 10 5.2.3 Test Modifications 11 5.2.4 Test Issues 11 5.2.5 Test Issue Resolution 12 5.2.6 Incomplete Prevalidations 12 5.2.7 Successful Prevalidations 12 5.3 Validation Testing 12 5.4 Declaration of Conformance 12 5.5 Validation Summary Report (VSR) 13 5.5.1 VSR Production 13 5.5.2 VSR Availability 13 5.6 Successful Validation 14 5.7 Validation with Superseded Versions of Test Suite 14 5.8 Publication 14 5.9 Retention of Validation Records 15 5.10 Advertising Validated Status 15 5.11 Content of Ada Validation Certificate 15 6. VALIDATION APPLICABILITY 16 APPENDIX A. DECLARATION OF CONFORMANCE 17 APPENDIX B. IMPLEMENTER DISPUTE FORMAT 18 APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS 19 C.1 Introduction 19 C.2 Expedited Resolution Process 19 C.3 Types of Resolutions 20 C.4 Summary 20 APPENDIX D. POINTS OF CONTACT 21 APPENDIX E. REFERENCES 24 APPENDIX F. SAMPLE CERTIFICATE OF VALIDATION 25 APPENDIX G. SAMPLE SOLICITATION WORDING FOR Ada VALIDATIONS 26 EXECUTIVE SUMMARY The Ada Compiler Validation Procedures define the certification body for the Ada programming language and set forth its rules of procedure for testing implementations of the Ada language for conformity to the language standard. The process of testing implementations is known as validation. This version (5.0) of the Procedures is a revision to Version 4.2 to reflect two important changes in Ada Validation: the period of transition in validation from the use of the original language standard [Ada83] to the revised standard [Ada95] will end in July 1997; and the validation authority for the Ada language has been transferred from the National Institutes for Standards and Technology (NIST) to the United States Department of Defense. This document supersedes all previous versions of the Ada Compiler Validation Procedures and requires the use of ACVC Version 2.1 for all Ada compiler validations. During the transition period, which began with the publication of the revised Ada language standard, [Ada95], in January 1995, validation was conducted according to either the original or revised standard. Additionally, testing for the revised standard was made lenient in both the limited number and extent of the test programs as well as an allowance for implementations to fail tests for language specifications of [Ada95] that are not present in [Ada83]. This validation lenience was intended to encourage implementers to bring useable partial implementations of [Ada95] to the market quickly; validation would still give a common conformity check on these implementations. With the completion of [Ada95] and near completion of the full validation test suite, the Department of Defense transferred the certification authority to the NIST in October 1996, who are the certification authority for other languages. However, the NIST changed its focus and announced that it would soon bring to a close all of its conformity-testing activities; the DoD then decided that it should continue to support Ada validation. Moreover, the National Research Council issued a report on Ada recommending that the DoD sustain its investment in an Ada infrastructure, including support for validation. As a result, the Ada validation authority was (re-)assumed by the DoD in April 1997. The Ada certification body comprises the DoD’s Ada Joint Program Office (AJPO), a principal technical advisor called the Ada Validation Organization (AVO), and the AJPO-recognized test laboratories called Ada Validation Facilities (AVFs) which conduct validation testing . The AJPO is responsible for the maintenance of the validation test suite, the issuance of validation certificates, and the establishment of these Ada Compiler Validation Procedures. Ada Compiler Validation Capability (ACVC) The ACVC is a conformity test suite (and supporting documents); it is designed to ensure that validated Ada implementations achieve a high degree of conformity to the Ada language standard. The ACVC is customized by an AVF for each Ada implementation that is tested; customization consists of adjusting the ACVC appropriately for various implementation characteristics. The ACVC is maintained by the AJPO; the ACVC is upgraded according to changes in interpretations of the Ada standard by the language maintenance body (ISO Working Group 9, Ada Rapporteur Group) and the discovery of deficiencies in the test programs. New versions of the ACVC are released according to a schedule set by the AJPO. ACVC versions that tested conformity to [Ada83] were numbered “1.1” through “1.11”, the final version; versions that test conformity to [Ada95] are numbered “2.0”, “2.0.1”, and “2.1”. ACVC versions 2.0 & 2.0.1 were used during the transition period; ACVC 2.0.1 was an intermediate release with corrections to many errors of version 2.0, and also was used with slightly stronger validation requirements. ACVC 2.1 is the first version that is considered to be a full test of [Ada95]. (The successors to this version will be numbered “2.2”, “2.3”, and so on, as needed.) For an Ada implementation to receive validated status, it must process each test program of a customized ACVC so that the result is graded passed, inapplicable, or unsupported, by ACVC grading criteria. An AVF customizes the ACVC for a particular Ada implementation by appropriately setting test parameters, by removing “withdrawn” tests (tests ruled by the AVO to be in error) and certain inapplicable tests, by splitting as needed test files with multiple intended errors so to enable complete error detection, and by including the set of tests for a Specialized Needs Annex as requested by the validation customer. In addition to the specification of a “core” language, [Ada95] contains several Specialized Needs Annexes; these are optional language requirements designed to meet the particular needs of various general application domains, such as information systems programming. An implementation of [Ada95] need not include implementation of any of these annexes, or it may implement only some of the features of these annexes. Whereas all ACVC test programs for the core language must be processed during validation, those for the Specialized Needs Annexes are processed only upon vendor request. A validation certificate is issued only if all tests for the core are correctly processed; it will additionally give credit for support of a Specialized Needs Annex only if all tests for that are correctly processed. Validation Validation involves interaction between the AVF customer and the Ada certification body. The testing process consists of well defined actions which, when completed successfully, result in the award of a validation certificate for the test implementation. The key actions in the validation of an Ada implementation are: 1. The customer and an AVF reach a formal agreement for validation, including the dates for the submittal of the results of customer-administered processing of the ACVC and for AVF witness testing. 2. The customer signs a Declaration of Conformance for each candidate Ada implementation. 3. The customer petitions for deviation from the requirements of each ACVC test program that is believed to be wrong for the candidate implementation(s). 4. The AVO rules on the customer petitions. 5. The customer processes the ACVC on the candidate implementation(s) and submits the results to the AVF. 6. The AVF analyzes the results of the customer’s independent processing of the ACVC. 7. The AVF conducts witness testing of the candidate Ada implementation(s), documenting this testing in a Validation Summary Report which the AVF submits to the AVO for review. 8. The AVO reviews the VSR, with comment to the AVF; the AVO recommends to the AJPO that a validation certificate be issued for the tested implementations. 9. The AJPO issues a validation certificate for each tested implementation upon the successful completion of the preceding actions. Validation of an Ada implementation results in the AJPO’s awarding a validation certificate for that implementation to the customer. This certificate grants validated status to that particular implementation, which will be listed on the AJPO’s Validated Compilers List. This validated status is applicable to cognate implementations that meet certain conditions. In short, all of the customer’s implementations with a similar computer-systems environment and the same or maintained compiler are considered validated if they produce the ACVC results that are documented in the Validation Summary Report (and the customer declares this). 1. INTRODUCTION This document provides operating policy and procedures that are followed in administering the Ada certification system by the Ada Joint Program Office (AJPO) of the United States Department of Defense (DoD). This document is not intended to explain the detailed procedures that can be found in the documentation associated with the Ada processor validation system, commonly called the Ada Compiler Validation Capability (ACVC). The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. The AJPO established a certification system to realize the benefits of standardization which include the ability to transfer software and programming expertise between computer systems that use a conforming Ada compiler. The Ada certification system’s rules of procedure and management address the validation of Ada implementations by testing. These rules form an operational description of a validated Ada compiler. These procedures (version 5.0) address the validation process that is used for ACVC Version 2.1 (for [Ada95]). These procedures are effective 6 April 1997 until they are superseded or revoked. The principals of the Ada certification body are: the AJPO, for overall direction; the Ada Validation Organization (AVO), for technical support; and the Ada Validation Facilities (AVFs), for performing witness testing of Ada implementations. It is important to note the scope and intent of validation. The purpose of validation is to ensure that Ada implementations achieve a high degree of conformity with the standard. Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada validation. Moreover, it is important to note that the ACVC conformity test is the processing of a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; validated implementations will fail to conform to the standard in ways peculiar to each, under particular circumstances. Validation testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada validation is to identify Ada processors which may be procured and used to develop application programs which meet the [Ada95] goals of portability and interoperability. The ACVC Test Suite is not designed to replace the vendor’s quality assurance testing or systematically to detect inconsistencies or “bugs.” The technical goal of Ada validation testing is primarily to verify that the Ada product correctly supports all required features. Rather than exhaustive testing of permutations of features, the ACVC Test Suite contains a carefully-chosen set of test cases which cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard. Neither is validation intended as a means of performance benchmarking. The VSR does not contain information about the speed, cost, or efficiency of executing the validation tests. 2. GLOSSARY OF TERMS Ada programming language: The language defined by reference [Ada95]. ACVC User’s Guide: A document that explains the technical details of processing the test programs and evaluating their results. Ada compiler: The software and any needed hardware that must be added to a given host and target machine to allow transformation of Ada programs into executable form and execution thereof. Ada Compiler Evaluation System (ACES): A test suite designed to assist in the evaluation of Ada compilers and their implementation. The ACES does not test conformity to the Ada programming language but can be used to demonstrate the performance characteristics of an Ada implementation. Ada Compiler Validation Capability (ACVC): The means for testing compliance of Ada implementations, consisting of the test suite, the support programs, and the ACVC User’s Guide. Ada implementation: An Ada compiler with its host and target computers and operating systems (kernels) Ada Rapporteur Group (ARG): The Ada Rapporteur group (ARG) is a subgroup of ISO/IEC/JTC1/SC22/WG9, the International Organization for Standardization Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language. Ada Validation Facility (AVF): The part of the certification body that carries out the procedures required to establish the compliance of an Ada implementation. Adaptive maintenance: [ANSI/IEEE 90] Maintenance performed to make a software product usable in a changed environment. Applicable ACVC test: A test that is neither inapplicable nor withdrawn. Compare with inapplicable test, and withdrawn test. Base implementation: An Ada implementation that was validated by an AVF on-site witness testing (see Section 5). Baseline: [IEEE 90] A specification or product that has been formally reviewed and agreed upon, that thereafter serves as the basis for further software development or maintenance work, that can be changed only through formal change procedures. Certification body: [ISO/IEC 86] An impartial body, governmental or non-governmental, possessing the necessary competence and reliability to operate a certification system, and in which the interests of all parties concerned with the functioning of the system are represented. Certification system: [ISO/IEC 86] A system having its own rules of procedure and management for carrying out conformity certifications. Computer system: [IEEE 90] A system containing one or more computers and associated software. Configuration management: [IEEE 90] A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements (See Appendix E). Conformity: [ISO/IEC 86] Fulfillment by a product, process or service of all requirements specified. [Note: Also see Section 1.1.3 of ANSI/ISO/IEC 8652:1995] Core language: The Sections 1-13 and Annexes A, B and J of [Ada95]. [See Section 1.1.2 of ANSI/ISO/IEC 8652:1995] Corrective maintenance: [ANSI/IEEE 90] Maintenance performed specifically to overcome existing faults. Customer: An individual or corporate entity who enters into an agreement with an AVF that specifies the terms and conditions for AVF services (of any kind) to be performed. Customized test suite: The ACVC tests, adjusted as necessary, that must be used for validation testing of a given Ada implementation (see Section 4.5). Declaration of conformance: A formal statement from a customer assuring that conformity is realized on the Ada implementation for which validation status is requested. [See Appendix A for the format of a declaration of conformance.] Derived implementation: An Ada implementation that was obtained by performing maintenance on a base implementation that has a current validation certificate (see Section 6). Host machine: [IEEE 90] (l) A computer used to develop software intended for another computer (contrast with target machine). (2) A computer used to emulate another computer. (3) The computer on which a program or file is installed. (4) In a computer network, a computer that provides processing capabilities to users of the network. Inapplicable test: A test that contains one or more test objectives found to be irrelevant for the given Ada implementation. Instruction set: [IEEE 90] The complete set of instructions recognized by a given computer or provided by a given programming language. Kernel: [IEEE 90] (1) That portion of an operating system that is kept in main memory at all times. (2) A software module that encapsulates an elementary function or functions of a system. Operating system: [IEEE 90] A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. Perfective maintenance: [ANSI/IEEE 90] Maintenance performed to improve performance or maintainability. Prevalidation testing: Process of supplying to an AVF the results of processing an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests. Result profile: The aggregate of test results produced by processing the customized test suite according to given evaluation criteria (see Section 6). Software maintenance: [ANSI/IEEE 83] Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. Specialized Needs Annexes: Annexes C through H of [Ada95]. These Annexes define standards for additional functionality required by specific application areas. An Ada Implementation may support some or none of these annexes. Conformity to a special needs annex means that each capability required by the annex is provided as specified. Target machine: [IEEE 90] (1) The computer on which a program is intended to run. (2) A computer being emulated by another computer. Target run-time system: The set of subprograms that may be invoked by linking, loading, and executing object code generated by an Ada compiler. If these subprograms use or depend upon the services of an operating system, then the target runtime system includes those portions of that operating system. Test issue: Any problem arising during validation (see Section 5.2.5). Validation: The process of checking the conformity of an Ada implementation to the Ada programming language and of issuing a certificate for the implementation. Validated Ada implementation: An Ada implementation that has been validated successfully by AVF on- site witness testing (see Section 5). Validated Ada compiler: The compiler of a validated Ada implementation. Validation Certificate (VC): A certificate issued by authority of the AJPO for a tested Ada implementation that passes an ACVC version. Validation Summary Report (VSR): A report produced by an AVF containing results that are observed from on-site, witness testing a specific Ada implementation or grouping of Ada implementations. Withdrawn test: A test which has been officially removed from the test suite because the test did not meet the test objective. In most cases, the test case will be rewritten and included in a future release of the test suite. Until then the test case is not used. 3. ORGANIZATIONS AND RESPONSIBILITIES This section specifies the roles of organizations that form the certification body and of customers who receive service from them. 3.1 Ada Joint Program Office (AJPO) The AJPO provides policy guidance for the certification system by: 1. setting validation standards to be followed by all AVFs; 2. establishing the conditions for issuance, the life, and the scope of a validation certificate; 3. establishing the schedule for issuing versions of the ACVC; 4. producing an ACVC version; 5. approving the release of an ACVC version; 6. designating members of the certification body; 7. resolving issues that may arise during validation when these issues cannot be resolved through the best efforts of the AVO and the AVF; 8. distributing an ACVC version to the AVFs and the AVO; 9. maintaining the official list of validated Ada implementations; 10. issuing the validation certificates for Ada implementations validated by testing (see Section 5); 11. overseeing the ACVC quality control and configuration management process; 12. providing information to the public concerning the test objectives and number of test programs in each version of the test suite, and other information that promotes a public awareness of the test suite and evaluation criteria. 3.2 Ada Validation Organization (AVO) The AVO provides the technical and administrative support required to operate the certification system by: 1. advising the AJPO and the AVFs concerning requirements for modification to the validation procedures; 2. resolving issues that may arise during the validation process; 3. reviewing all validation summary reports (VSRs) prepared by AVFs: 4. recommending to the AJPO issuance of a validation certificate for Ada implementations validated by testing (see Section 5) 5. participating in the ACVC quality control and configuration management process; 6. deciding on the withdrawal of test programs from the ACVC version that is being used for validation; 7. convening meetings of the members of the certification body at appropriate intervals to discuss the process and to evaluate practices. 3.3 Ada Validation Facilities (AVFs) An AVF is chartered by the AJPO to conduct validation by: 1. adhering to validation procedures approved by the AJPO; 2. producing the VSR; 3. forwarding unresolved test issues to the AVO for review and analysis, with final resolution to be provided by the AJPO, if necessary; 4. providing advice on a customer’s registration request for a derived Ada implementation; and 5. striving to satisfy national accreditation criteria. The AJPO may issue an AVF charter to an organization that has been recognized as an accredited testing laboratory by the AJPO. The AJPO may issue a charter to an organization located in a country that has a Memorandum of Understanding (MoU) with the U.S. government covering the chartering of AVFs, according to the rules specified in the MoU. An AVF charter shall remain in effect for the duration specified in the charter; however, a charter can be revoked by the AJPO, at any time, for due cause. The AJPO may direct an impartial body to conduct an audit at any time or prior to issuing an AVF charter. Audits are conducted in accordance with procedures established by the AJPO at the time of the audit and are tailored to reflect the purpose of the audit. 3.4 Customers Customers are serviced by the Ada certification body in matters concerning Ada validation. In requesting services of the Ada certification body, customers are to provide accurate and complete information to perform validation or to obtain other services. 4. THE Ada COMPILER VALIDATION CAPABILITY The ACVC is designed to demonstrate the conformity of an Ada implementation with the standard. The ACVC is distributed as a collection of test programs, support programs which facilitate processing the tests, and an ACVC User’s Guide that explains the criteria for evaluating the results. The ACVC Version 2.1 is now available for Ada validation use. Compiler vendors must use ACVC Version 2.1 on or after July 6, 1997. 4.1 Versions New ACVC versions are released periodically according to a schedule which is determined by the AJPO. Each new version incorporates changes to the ACVC as deemed necessary by the AJPO. These changes are made in order to reflect a revision of the standard, to incorporate ISO WG9 interpretations, or to address implementer or user comments. The test objectives and test programs for each new ACVC version are available for public review and comment before that version is issued for use in validation. Ada validation related questions or comments on test objectives or test programs should be submitted by e- mail to acvc-comment@sw-eng.falls-church.va.us, or by FAX (703-681-2869) or by mail: Ada Information Clearinghouse (AdaIC), P.O. Box 1866, Falls Church, Virginia USA 22041; voice telephone: 800-232-4211 or 703-681-2466. 4.2 Applicability of ACVC Test Programs Each ACVC test program has one or more test objectives which are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada implementation (e.g., “check floating-point operations for digits 18”). These test programs generally contain an explicit indication of their applicability and the expected behavior of Ada implementations for which they do not apply. The determination of applicability is made according to the grading criteria in the ACVC or as a ruling by the AJPO. All applicable test programs must be processed and passed according to the specified grading criteria. For [Ada95] validations, ACVC test programs for special needs annexes (C, D, E, F, G, and H) will not be processed unless the AVF customer requests that they be processed. The result of these tests will not affect the issuance of a certificate for the core language but shall be made available as part of the public record of the Ada validation effort. Full support of a special needs annex is indicated on the certificate provided that the implementation passes all applicable test programs for that annex according to the specified grading criteria. 4.3 Test Modifications The certification body strives to apply the ACVC as uniformly as practical to all Ada implementations. In order to apply common test objectives that depend on implementation dependent characteristics (e.g., line lengths and numeric types), many test programs must be adjusted to a given implementation following the procedures given in the ACVC User’s Guide. These adjustments consist of the insertion of implementation dependent values in certain test programs at places prescribed by the ACVC User’s Guide. In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (e.g., the algorithm for recovering from syntax errors). The allowable changes for each Ada implementation are determined by an AVF after consultation with the AJPO, except in the case of error-recovery problems which an AVF may address without consultation. 4.4 Test Withdrawal In any ACVC version, it is possible that a test program is based on assumptions that need not hold true for all Ada implementations or that a test program does not meet its objective. In these cases, the AVO may issue a correction to the evaluation criteria in the ACVC User’s Guide or the test program may be withdrawn from that version of the test suite. Any interested party may challenge a test program by sending a rationale for the challenge to the Ada Rapporteur Group (ARG) (see appendix F for the address). When the ARG has accepted the challenged test for extended resolution, the AVO may withdraw the test. When an AVF customer prepares for validation, the customer must challenge a test program only through an AVF by asking for a review of its evaluation criteria or for its withdrawal. The form for submitting a challenge is provided in Appendix B. 4.5 Customization A customized test suite is produced by the AVF for each Ada implementation that is a candidate for validation. This customization always consists of removing withdrawn tests and in making required modifications to test and support programs; and, it may include removal of some inapplicable tests as allowed by the ACVC User’s Guide. 4.6 Availability The ACVC is available to the general public from an AVF or from the AJPO on the Internet. See APPENDIX D for points of contact. 5. VALIDATION TESTING There are a number of steps that must be completed by a customer and the certification body so that the customer obtains a validation certificate and a VSR. The same ACVC version must be used to complete the steps described in this section. The AJPO announces the in-use and end-use period for each ACVC version. The AVF must begin on-site, witness, validation testing of the Ada implementation at the customer’s site before the current ACVC version expires or else validation with that ACVC version will not be allowed. Anyone intending to obtain a validation certificate should contact an AVF without delay for advice on the handling of the ACVC, on interpretation of the test grading criteria, and on the operational details of that AVF’s management practices. The steps are: 1. Validation Agreement 2. Prevalidation 3. Validation Testing 4. Declaration of Conformance 5. Validation Summary Report 6. Validation Certificate 5.1 Validation Agreement In order to obtain services from the certification body, an interested party must become a customer of an AVF by reaching a formal agreement. This agreement should address the following topics: 1. identification of the Ada implementation(s) to be tested and the ACVC version to be used; 2. a statement of work, including analysis of prevalidation testing, witness testing, validation, and preparation of the VSR; 3. the format of data to be exchanged; 4. a schedule of events and the site of validation; 5. financial arrangements; 6. retention of records; 7. AVF liability; and 8. confidentiality of validation information. The schedule for events, deliverables, and payments should take into account the fact that certain steps in the validation process require interaction with other members of the certification body (e.g., AVO or AJPO). The AVF will put forth its best effort to keep confidential a customer’s intent to obtain a validation certificate and the projected schedule for validation. This confidentiality will not be allowed to interfere with the normal review procedures of validation. If the customer requests confidentiality for reasons of national security or procurement sensitivity, the customer will provide to the AVF an official, written statement describing the request and the reason(s) for the request; the AVF will also obtain further guidance from the AJPO. 5.2 Prevalidation The requirements of this step are discussed separately so that the customer understands the interaction that is required with an AVF. 5.2.1 Customer Testing After entering into a formal agreement, the customer either provides the necessary information for the AVF to prepare a customized test suite or, the customer may prepare a customized test suite according to instructions in the ACVC User’s Guide. The customer then processes all the tests in this customized test suite using the candidate Ada implementation or another Ada implementation that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of an option controlling the production of information output. Any other exception constitutes a test issue that must be resolved with the AVF (see Section 5.2.5). Test issues should be sent to the AVF for analysis as soon as they are known. The customer must provide (unless explicitly waived by the AVF) the prevalidation materials in a timely manner or by the agreement requirements between the customer and the AVF. Prevalidation materials include as a minimum the processing of an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests. 5.2.2 Submission of Results Upon completion of testing, the customer delivers the complete set of results in the agreed format to the AVF. These results are accompanied by the following information: 1. a list of test programs that the customer claims are inapplicable, together with a rationale for inapplicability; 2. a list of test programs that are disputed (see Section 4.4) together with rationale (see Appendix B for format); 3. the necessary information for the AVF to prepare the customized test suite, if applicable; 4. an annotated sample command script; 5. the complete set of option settings used for processing the customized test suite, including the default settings; and 6. complete and current documentation for implementation-dependent characteristics as required in Annex M of [Ada95]. 5.2.3 Test Modifications In order to comply with the test objective it may be required to modify the test source code, the test processing method, or the test evaluation method. Modifications are allowable because at the time of test writing not all possible execution environments of the test and the capabilities of the compiler could be foreseen. Only the AVO shall make the decision to use any of the following Test Modifications. Possible kinds of modification are: 1. Test Modification: The source code of the test is changed. Examples for test modifications are the insertion of a pragma, the insertion of a representation clause, or the splitting of a B-test into several individual tests, if the compiler does not detect all intended errors in the original test. (This last example is the only exception to the rule that only the AVO shall make the decision to apply a Test Modification.) 2. Processing Modification: The processing of the test by the Ada implementation for validation is changed. Examples for processing modification is the change of the compilation order for a test that consists of multiple compilations or the additional compilation of a specific support unit in the library. 3. Evaluation Modification: The evaluation of a test result is changed. An example for evaluation modification is the grading of a test other than the output from REPORT.RESULT indicates. This may be required if the test makes assumptions about implementation features that are not supported by the implementation (e.g. the implementation of a file system on a bare target machine). 5.2.4 Test Issues A test issue may be any of the following: 1. a missing or incomplete result to a test; 2. a result presented in an inadequate form; 3. a disagreement between the customer and the AVF as to the interpretation of a result; 4. a change in the choice of options to be used during testing; 5. a result that makes the Ada implementation fail the ACVC according to the current grading criteria; or 6. an implementation-dependent characteristic that may affect the conformity of the Ada implementation. The material submitted by the customer is analyzed by the AVF and test issues are resolved. If the AVF and the customer cannot agree on a way to resolve a test issue, the issue will be referred to the AVO for a resolution (see Section 5.2.5). It may be justified to leave a test issue unresolved at prevalidation. For example, it may be impossible to check the processing of control characters by inspecting printed results. The AVF will note these unresolved issues and describe the results that will be expected during validation testing. It is also possible that the customer information provided for production of the customized test suite (see Section 5.2.1) was insufficient so that corrections to the customized test suite must be made and additional processing will be required. 5.2.5 Test Issue Resolution A customer may challenge the applicability or correctness of any particular ACVC test program. Such challenges should be presented to the AVF in the test-dispute format (see Appendix B). The AVF will forward challenges to the AVO for resolution; the AVO will strive to rule on the challenge within two weeks of receiving it. The AVO will forward all challenges and rulings to the AJPO. (See Appendix C for a description of the Test Dispute and Resolution Process.) 5.2.6 Incomplete Prevalidations The AVF and the customer may agree that, at the customer’s risk, parts of the customized test suite need not be processed during prevalidation. The customer must certify that the results from a previous prevalidation submitted to the AVF or validation results obtained by the AVF are identical to those that would have been obtained by the customer. The normal practice is to submit complete prevalidation results. 5.2.7 Successful Prevalidations Prevalidation testing is successful if the analysis of results and the resolution of test issues show that the candidate Ada implementation passes the customized test suite. Prevalidation is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until validation testing by agreement between the AVF and the customer. 5.3 Validation Testing Upon successful completion of prevalidation, with or without caveats, the AVF witnesses testing of the Ada implementation at the site and time mutually agreed by the AVF and customer. The AVF prepares a customized test suite based upon customer information and any information collected during the resolution of test issues. The customized test suite is installed and processed in the physical presence of an AVF representative. If the AVF determines that the results agree with those obtained from prevalidation and are satisfactory with respect to the caveats, the testing has been successful; otherwise, re-testing will be required, unless the validation attempt is discontinued. 5.4 Declaration of Conformance At the start of the on-site, witness testing of the customer’s Ada implementation, the customer will provide to the AVF a completed, dated, and signed declaration of conformance. The customer should provide the information contained in the declaration of conformance to the AVF prior to the on-site testing as part of the prevalidation effort. The declaration of conformance states that the organization that is responsible for the production, maintenance or distribution of the Ada compiler is offering a product that is in conformity with the Ada programming language. The declaration of conformance becomes part of the AVF records and is copied into the VSR. A validation certificate will not be issued unless a signed declaration of conformance has been provided to the AVO (See Appendix A, for an example of the declaration of conformance.) 5.5 Validation Summary Report (VSR) A VSR is produced for each validation testing effort. A single VSR may cover validation testing of several Ada implementations, provided that they all have the same result profile. The VSR provides the following documentation pertaining to the validation effort: 1. identification of the customer responsible for validation of the Ada implementation; 2. identification of the organization responsible for the production, maintenance, or distribution of the Ada compiler or Ada implementation; 3. identification of the Ada implementation tested; 4. options provided by the Ada compiler and identification of the options used for testing; 5. the inapplicable test programs and implementation-dependent characteristics exhibited by the test programs that established inapplicability; 6. the implementation-dependent characteristics pertinent to the customized test suite; 7. description of implementation-dependent characteristics as detailed by Annex M [Ada95]; 8. withdrawn test programs; 9. modifications to test programs with an explanation for such modifications; and 10. a description of failed test programs, if applicable. 5.5.1 VSR Production The VSR is prepared by the AVF but includes material that is produced by the customer, such as the documented compiler and linker options used during the Ada validation process and Annex M required by [Ada95]. Draft versions of the VSR are sent to the AVO for review before and after validation testing. The final version of the VSR is signed by the AVF, the AVO, and the AJPO. 5.5.2 VSR Availability The final version of the VSR is available to the general public from the customer and from the AVF that produced it. The AVF may require payment of a fee for VSR reproduction and mailing costs. The VSR is available on-line whenever possible (see Appendix D for points of contact). There is no cost for VSRs obtained by electronic means. 5.6 Successful Validation For each successful validation, one certificate is issued by the AJPO. The information on the validation certificate describes the tested Ada implementation: the source of this information is the signed declaration of conformance that describes the tested Ada implementation that the AVF provides to the AVO after completion of testing. The customer will ensure that the information contained on the certificate does not infringe on the rights of third parties and may be required to provide a written statement of consent from any third party involved. Validation certificates for ACVC 2.1 will expire one year after a new test suite is released. The certificate indicates which specialized needs annexes, if any, are fully supported. An entry in the list of Ada implementations validated by testing will be made for each certificate issued. This entry will be removed when the certificate expires along with all entries for registered validations derived from it. A sample Certificate of Validation is provided in Appendix F. 5.7 Validation with Superseded Versions of Test Suite The AJPO is willing to conduct a validation for an Ada processor with superseded versions of the test suite. The result of such a validation is a VSR which will not be listed in the VCL, but which may be shown to customers requiring this type of evaluation. The AJPO procedures require the use of the latest version of the ACVC for validation testing. Certificates of Validation are issued only for testing done with the latest version of the test suite. Testing with an expired test suite is done only upon special request for specific procurement requirements. Certificates of Validation are not issued for tests performed with expired test suites. When testing is performed with an expired test suite, the AJPO will provide a letter in lieu of a Certificate of Validation. Testing must be conducted using the procedures and tests that were in effect for the ACVC version requested. The letter will note whether or not the implementation described in the VSR passed all applicable tests of the ACVC version as specified by the Ada Validation Procedures in effect for that ACVC version. The letter will be dated but shall not have an expiration date since the letter is not a certificate. Concurrently with but not later than the request for validation with a superseded ACVC version, the applicant may include derived implementation(s) which may be noted in an attachment to the letter should the derived request(s) satisfy the derived requirements as specified by the Ada Validation Procedures in effect for that ACVC version. 5.8 Publication VSRs registered with the AJPO shall be made available to the public. In no event, however, shall the name of the supplier or any of its trademarks and trade names be used in AJPO publications without the supplier’s prior written consent. With respect to publication in the VCL, the standard Validation Customer Agreement contains the supplier’s written consent. The AJPO and the supplier shall agree to confer and consult prior to the publication of data to assure that no Proprietary Data is released and that patent rights are not jeopardized. Prior to publishing a VSR, the supplier shall be offered an opportunity to review such proposed publication. 5.9 Retention of Validation Records Each AVF shall determine the length of time that records and information for an Ada validation are retained. For the AVO, the records and information used in preparing the VSR will typically be retained for six (6) months after expiration of the latest Certificate of Validation. 5.10 Advertising Validated Status The customer will not advertise or make public claims that the Ada implementation is validated until after receiving a validation certificate or after receiving formal notification from the AVF that the AJPO has issued a validation certificate. A waiver of confidentiality must be signed by a customer who intends to advertise the completion of events that indicate progress toward completion of validation. If a waiver of confidentiality has been signed with the AVF, the AVF will respond to inquiries about the customer’s advertisements or public claims by acknowledging receipt of validation materials (i.e., a formal agreement, pre-validation results, or validation testing results) without judgment concerning the success of the validation. 5.11 Content of Ada Validation Certificate The Ada Validation Certificate shall contain as a minimum the name of the Ada compiler under test, the name of the testing laboratory performing the validation, identification of the host computer and the host computer operating system, identification of the target computer and the target computer operating system (if present), the validation certificate number, identification of the VSR number, the issue date of the certificate, the expiration date of the certificate, identification of Specialized Needs Annexes fully supported (fully supported means that applicable tests for that annex are graded PASSED), and the test counts with category explanations (on an attached sheet if necessary). 6. VALIDATION APPLICABILITY Most Ada implementations are submitted for validation with the intent to be representative of some set of implementations that are similar in their defining components of Ada compiler, host machine, and target machine. A validation certificate identifies only the particular Ada implementation that was tested by the AVF (i.e., the base implementation); the validated status that it conveys is extensible to a range of cognate implementations for the life of the certificate. For the purpose of extending validated status to cognate implementations, the changes required to render the base Ada compiler fully functional on a different host machine are considered adaptive maintenance; thus, the principal defining characteristics of the set of cognate Ada implementations that may be considered validated based upon some validation (i.e., with reference to a particular validation certificate) are the compiler and target machine. For any base implementation with a current validation certificate, a cognate implementation is considered to be validated if it satisfies all of the following conditions: 1. The cognate compiler differs from the base compiler by changes that are within the scope of software maintenance. 2. The target machines of the base and cognate implementations have compatible instruction sets and operating systems or kernels. (There need be no such similarity between the host machines.) 3. The results profiles for the base and cognate implementations are the same. The changes that may be made to a base compiler for producing cognate compilers will be within the scope of software maintenance as applied to the domain of compiler construction. Changes must be either corrective, adaptive, or perfective. Examples of these changes are the correction of a compiler error, the adaptation to an operating system upgrade or the transfer of the compiler to another host machine, or the perfection of a garbage-collection algorithm. For a target machine, common examples of compatible instruction sets and operating systems are two different computer system models in a manufacturer’s product line or the computer systems produced by different manufacturers that use the same instruction set and operating systems, or any computer system and a simulated or emulated computer system that have the same instruction set. Two Ada implementations have the same results profile when: 1. they use the same customized test suite; and, 2. their output from processing the customized test suite is the same (with the possible exception of the precise content and position of diagnostic messages from the compiler). For cases where a cognate implementation’s results profile differs from that of the base implementation, the vendor may petition the certification body for acceptance of the cognate implementation as validated . The vendor will present the cognate implementation’s results profile in comparison to the base results profile, and explain each difference. If the certification body accepts the petition, a description of the differences in the results profile will be approved and filed with the implementation’s VSR for distribution to interested users upon request (the vendor will receive a copy for promotional use). APPENDIX A. DECLARATION OF CONFORMANCE Declaration of Conformance Customer: Certificate Awardee: Ada Validation Facility: ACVC Version: Ada Implementation: Ada Compiler Name and Version: Host Computer System: Target Computer System: Declaration: I, the undersigned, declare that my organization has no knowledge of deliberate deviations from the Ada Language Standard (ANSI/ISO/IEC 8652:1995, FIPS 119-1) other than the omission of features as documented in the Validation Summary Report. ______________ Customer Signature/Date Company Title _________________ Certificate Awardee Signature/Date Company Title APPENDIX B. IMPLEMENTER DISPUTE FORMAT [part A] Implementer: Configuration: ACVC Version: Pre-Validation Submittal Date: <:due date for in-house results> [Part A will be completed once by each implementer; part B will be completed for each dispute. It is not necessary for a pre-validation date to have been established. Part A information is treated as confidential.] [part B] Reference: Summary: Discussion: [In this Discussion, arguments should be specified using test line #s and references to pertinent sections of the Ada standard or Commentaries (Al-xxxx). The implementer must describe the behavior of the implementation for the test or tests that are disputed, stating the particular test messages that are produced. It is sufficient for the detailed description to be limited to the particular segment of test code that is disputed. Relevant source code with compiler messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.)] If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the AJPO to assess the reasonableness of these limitations. Failure to fully specify the points pertinent to a dispute might result in an adverse decision being made, with the disputer having to further argue the case with a second submittal to the AJPO. Yet it is possible that the Summary will suffice to adequately present a dispute. APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS C.1 Introduction A “dispute” is defined by the ACVC User’s Guide as any result from processing an ACVC test program that is not a passed, inapplicable, or unsupported result according to the established grading criteria. This intentionally broad definition of a “dispute” is to make certain that compiler implementers bring all deviant test results to the attention of the AVF, without assuming that such results would be accepted without special review. The compiler implementer also provides a rationale for each challenge being made to a particular test program. Disputes are forwarded to the AVO, usually electronically, by the AVF on behalf of their validation customer. For each dispute that is accepted (i.e., when the AVO rules in favor of the dispute), it is likely that some correction is indicated for the disputed tests. The AVO withdraws any test that is found to be incorrect to a degree that makes it unsuitable for validation. The withdrawal of a test consists of including it on a list of tests that are ignored for validations conducted with the current ACVC version. The AVO updates the list of withdrawn tests and distributes this list to the AVFs and to the AJPO. The AVO also maintains a database consisting of all test disputes and their resolution (AVO rulings) which is periodically provided to the Fast Reaction Team (FRT) and other interested parties. The AVO may also recommend that certain resolved disputes be considered further by the ARG. C.2 Expedited Resolution Process The AVO resolves disputes by any of three methods: 1. a resolution that was made previously is applied to the current dispute (e.g., the same dispute might be submitted at different times by different petitioners); 2. the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or 3. the resolution is determined based on the deliberations of the FRT. Although the Ada Compiler Validation Procedures do not set a limit on the length of time for reaching a resolution, the AVO attempts to resolve disputes within two weeks, an informal guideline that was established by the certification body. If a dispute is submitted to the AVO more than four weeks before the scheduled beginning of on-site testing, and the AVO is unable to issue a final resolution of the dispute within the two week period, then the dispute is presumed to have been resolved in the implementer’s favor. The disputed tests will be graded as inapplicable and the default resolution will be documented in the VSR. Such a default resolution will not be used as a basis for future disputes of the same test. The AVO also attempts to place priority on resolving disputes for AVF customers who have a firmly scheduled date for validation testing. Implementers should submit disputes well in advance of a scheduled validation testing date. (see Section 5) On receipt of a dispute, the AVO checks whether the issue matches any that had been previously resolved. If the dispute is new, it is given an initial AVO analysis which involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous dispute deliberations. A dispute is referred to the FRT when questions of interpretation arise and a resolution is not obvious. The AVO presents the dispute and any additional information resulting from an initial analysis to the FRT by e-mail without disclosing the identity of the petitioner. Deliberation of the dispute proceeds with the exchange of each expert’s opinion and analysis. The AVO participates in the deliberation by providing information as requested (e.g., ACVC tests or information from the petitioner), eliciting discussion from the experts, and making or challenging technical points raised in the discussion. Resolution of a dispute is made by the AVO and is generally based upon a consensus of the FRT members participating. There is no prescribed formality to the FRT deliberations, such as voting procedures or time limits on deliberation. The AVO might extend deliberation when a basis for resolving the dispute has not been made. However, the AVO will give its ruling on the dispute when a sufficient basis has been established, regardless of whether the FRT discussion continues. C.3 Types of Resolutions The resolution of a dispute is either an acceptance or rejection of the petitioner’s arguments. Acceptance can result in either withdrawal of the test program from the ACVC, or in a “Test, Processing, or Evaluation” modification for validation. A dispute may be rejected if it conflicts with the Ada standard or Ada Commentaries or if it is a test to which previously validated implementations conform and the petitioner has not provided compelling reasons for a deviation. A dispute may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. Withdrawn tests have no effect on validation (they are generally not processed). If the dispute shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the AVO will direct that the test(s) be processed with a test modification. There are three types of test modification: Test, Processing, and Evaluation modifications. 1. A Test Modification is an actual change to the code of the test (e.g., adding a choice to an exception handler). 2. A Processing Modification is a change to the way in which the test is processed (e.g., re-ordering the compilation of component files of a multi-compilation test). 3. An Evaluation Modification is simply the grading of the observed results by other than the established grading criteria (e.g., interpreting particular intermediate output and a final “failed” result as “passed”, according to an understanding of the dispute). All test modifications are documented in the VSR. C.4 Summary There is no limit on the number of disputed tests that can be submitted by an implementer. Although there is a risk that a dispute will not be decided in the implementer’s favor, that risk can be managed so as to not affect validation by early submission of disputes. Any interested party may dispute an ACVC test and its grading criteria. APPENDIX D. POINTS OF CONTACT Ada Joint Program Office (AJPO) Lt. Col. John Hamilton Ada Joint Program Office Center For Computer Systems Engineering Defense Information Systems Agency 5600 Columbia Pike Falls Church, VA 22041-2717 Tel: 703-681-2458 Email: hamilt2d @ncr.disa.mil Ada Validation Facility Managers Dr. William H. Dashiell The National Institute of Standards and Technology Building 820, Room 517 Gaithersburg, MD 20899 USA Tel: 301-975-2490 Email: dashiell@speckle.ncsl.nist.gov Mr. Jon Leigh The National Computing Centre, Ltd. Oxford Road Manchester England, MI7ED Tel: +44-61-228-6333 Email: jon@ncc.co.uk Mr. Michael Tonndorf IABG, Software-Technologie lTE Einsteinstr. 20 D-85521 0ttobrunn Germany Tel: +49-89-6088-2477 Email: tonndorf@iabg.de Ada Validation Organization Mr. Clyde Roby Institute for Defense Analyses 1801 N. Beauregard St. Alexandria, Va. 22311 Tel: 703-845-6639 Email: avo@sw-eng.falls-church.va.us Validated Compilers List Ada Information Clearinghouse P.O. Box 1866 Falls Church, VA 22041 USA Tel: 703-681-2466 OR 1-800-232-4211 Email: adainfo@sw-eng.falls-church.va.us WWW: http://sw-eng.falls-church.va.us/AdaIC/ Ada Compiler Evaluation System (ACES) Data Analysis Center for Software P.O. Box 120 Utica, NY 13503 Tel: 315-734-3696 Anonymous FTP to sw-eng.falls-church.va.us, directory is /public/AdaIC/testing/aces WWW: URL http://sw-eng.falls-church.va.us/AdaIC. Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG) Dr. Erhard Ploedereder University of Stuttgart Institute for Computer Science Breitwiesenstr. 20-22 D-70565 Stuttgart Germany Tel: +49 +711 7816-322 Fax: +49 +711 7816-380 Email: ploedere@informatik.uni-stuttgart.de Ada Information Clearinghouse Ada Information Clearinghouse P.O. Box 1866 Falls Church, VA 22041 USA Tel: 703-681-2466 OR 1-800-232-4211 Email: adainfo@sw-eng.falls-church.va.us WWW: http://sw-eng.falls-church.va.us/AdaIC/ Ada Validation Related Questions: Email: validation@sw-eng.falls-church.va.us Ada Compiler Validation Capability (ACVC) The ACVC is available to the general public from an AVF or from the Ada Information Clearinghouse (AdaIC) on the Internet. URL: http://sw-eng.falls-church.va.us/adaic/ or by anonymous FTP at: sw-eng.falls-church.va.us/public/AdaIC In either case, look for “testing” and “acvc.” Comments or questions about ACVC tests should be sent to: acvc-comment@sw-eng.falls-church.va.us APPENDIX E. REFERENCES [Ada95] ANSI/ISO/IEC 8652:1995 Reference Manual for the Ada Programming Language, February 1995 (supersedes Ada83) [Ada83] American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652:1987 and FIPS 119, 1985. [ANSI/IEEE 90] American National Standards Institute / Institute of Electrical and Electronic Engineers, Inc., Standard 610.12-1990; “ANSI/IEEE Standard Glossary of Software Engineering Terminology”. [FIPS] Federal Information Processing Standards publications relative to the Ada Programming Language are FIPS 119 and FIPS 119-1 [ISO 74] International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms. [ISO/IEC 91] International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities. APPENDIX F. SAMPLE CERTIFICATE OF VALIDATION (not included in electronic version) APPENDIX G. SAMPLE SOLICITATION WORDING FOR Ada VALIDATIONS This appendix provides suggested solicitation wording for validation for Ada language processors. Role of Procuring Agency It is not feasible for the AJPO to validate all Ada processors which conform to the specifications of ANSI/ISO/IEC 8652:1995, since formal validation can be expensive, time consuming, and resource intensive. The procuring agency has the authority to extrapolate from the test results published in the Validated Compiler List (VCL) to other hardware/software environments, based on additional research by the procuring agency, demonstrations or warranties by the Ada supplier, etc. The procuring agency, not the AJPO, has the responsibility of reviewing the Ada validations listed in the VCL and determining the applicability of these validations to the hardware/software environment involved in a specific procurement. The criteria for applicability of a Certificate should be appropriate to the size and timing of the procurement. The AJPO suggests that procuring agencies use one (or more) of the following terms for the three validation options: “Delayed Validation,” “Prior Validation Testing,” and “Prior Validation.” The option “Prior Validation Testing” allows Ada suppliers to offer tested products which may contain nonconformities; whereas “Prior Validation” requires tested products without nonconformities. “Delayed Validation” allows Ada suppliers to offer products which may not have been tested prior to contract award. The procuring agency shall select the appropriate validation option and shall specify whether a Validation Summary Report or Certificate of Validation is required. The agency shall specify appropriate time frames for validation and correction of nonconformities. The agency is advised to refer to the VCL for information about the validation status of Ada products. This information may be used to specify validation time frames that are not unduly restrictive of competition. The sample solicitation wording for the different validation options follows: “All implementations of ANSI/ISO/IEC 8652:1995 that are brought into the Federal inventory as a result of this document for which validation is specified, and those implementations used by vendors to develop programs or provide services shall be validated using the official Validation System specified by the Ada Joint Program Office (AJPO). Validation shall be in accordance with AJPO validation procedures for ANSI/ISO/IEC 8652:1995. The results of validation shall be used to confirm that the implementation meets the requirements of ANSI/ISO/IEC 8652:1995 as specified in this document. To be considered responsive the offeror shall: (1) Provide validated ANSI/ISO/IEC 8652:1995 implementations through ‘Delayed Validation’, ‘Prior Validation Testing’ or ‘Prior Validation’.” (Option 1) - “Delayed Validation” Delayed Validation - When an agency determines that it is acceptable for implementations of ANSI/ISO/IEC 8652:1995 to be offered that have not yet been tested for conformance to ANSI/ISO/IEC 8652:1995 the following ‘Delayed Validation’ solicitation wording option may be used: “The offeror shall certify in the offer that all implementations of ANSI/ISO/IEC 8652:1995, including applicable ANSI/ISO/IEC 8652:1995 options, offered in response to this document will be submitted for validation upon contract award. Implementations submitted for validation shall have the validation completed at the earliest possible date permitted by the AJPO validation procedures. Unless specified elsewhere, proof of submission for validation shall be in the form of a letter scheduling the validation and the subsequent delivery by the offeror of a AJPO registered report or AJPO Certificate of Validation. Proof of testing shall be provided in the form of a AJPO registered validation summary report. Proof of fully validated implementations shall be in the form of a AJPO Certificate of Validation.” (Option 2) - “Prior Validation Testing” Prior Validation Testing - When an agency determines that it is essential for implementations of ANSI/ISO/IEC 8652:1995 to be previously tested for conformance to ANSI/ISO/IEC 8652:1995 before being offered, and the nature of the requirement is such that an implementation of ANSI/ISO/IEC 8652:1995 may be initially offered that has not been fully validated (i.e., implementation has not fully demonstrated compliance to ANSI/ISO/IEC 8652:1995), the following ‘Prior Validation Testing’ solicitation wording option may be used: “The offeror shall certify in the offer that all implementations of ANSI/ISO/IEC 8652:1995, including applicable ANSI/ISO/IEC 8652:1995 options, offered in response to this document have been previously tested or validated and included on the current list of validated Ada compilers maintained by the Ada Joint Program Office (AJPO). Unless specified elsewhere, proof of testing shall be provided in the form of a AJPO registered validation summary report. Proof of fully validated implementations shall be in the form of a AJPO Certificate of Validation.” (Option 3) - “Prior Validation” Prior Validation - When an agency determines that it is essential for implementations of ANSI/ISO/IEC 8652:1995 be validated (i.e., implementation has been tested and has demonstrated compliance to the ANSI/ISO/IEC 8652:1995) for conformance to ANSI/ISO/IEC 8652:1995 prior to being offered the following ‘Prior Validation’ solicitation wording option may be used: “The offeror shall certify in the offer that all implementations of ANSI/ISO/IEC 8652:1995, including applicable ANSI/ISO/IEC 8652:1995 options, offered in response to this document have been previously validated and included on the current list of validated Ada compilers maintained by Ada Joint Program Office (AJPO). Unless specified elsewhere, proof of validation shall be in the form of a AJPO Certificate of Validation.” (2) Agree to correct all implementation nonconformance from the applicable ANSI/ISO/IEC 8652:1995 reflected in the validation summary report not previously covered by a waiver. All areas of nonconformance must be corrected within 12 months from the date of contract award unless otherwise specified elsewhere in this document. If an interpretation of the ANSI/ISO/IEC 8652:1995 is required, such a request for interpretation shall be made within 30 calendar days after contract award. Any corrections that are required as a result of decisions made under the procedures of ANSI/ISO/IEC 8652:1995 AJPO rulings shall be completed within 12 months of the date of the formal notification to the contractor of the approval of the interpretation. Proof of correction in either case shall be in the form of a AJPO Certificate of Validation showing that the implementations nonconformance has been corrected. Failure to make required corrections within the time limits set forth above shall be deemed a failure to deliver required software. The liquidated damages as specified for failure to deliver the operating system or other software shall apply.” iii ES-3 28