Ada Compiler Validation Procedures VERSION 5.0 D R A F T April 6, 1997 PREPARED BY National Institute of Standards and Technology Information Technology Laboratory Conformance Testing Group Building 820NN, Room 517 Gaithersburg, Maryland 20899 USA PREPARED FOR Ada Joint Program Office Center for Software Defense Information Systems Agency 5600 Columbia Pike Falls Church, Virginia USA EXECUTIVE SUMMARY The purpose of this executive summary is to provide an overview of the Ada compiler validation process. Anyone who obtains services from the Ada certification body must understand the definitions of terms and follow the specific rules provided in the body of this document. This Version (5.0) of Ada Compiler Validation Procedures has been updated to reflect the change from use of the Ada Compiler Validation Capability (ACVC) Version 1.11 for Ada83 and ACVC Version 2.0.1 for Ada95 to the exclusive use of ACVC Version 2.1 for Ada95. Also, this version of the procedures will reflect that beginning April 6, 1997, the Ada Joint Program Office (AJPO) will assume the responsibility from National Institute of Standards and Technology (NIST) as the point of contact for Ada certification issues. The NIST issued the Ada Compiler Validation Procedures Version 4.2 in September 1996 to reflect NIST's role as the Ada certification authority for the Ada language standard, ANSI/ISO/IEC 8652:1995. Version 4.2 of the Ada Compiler Validation Procedures also reflected the use of ACVC 2.0.1 during the transition period when compiler vendors incorporated the features provided by the revised Ada language standard, ANSI/ISO/IEC 8652:1995. The transition period was from February 1995, when Ada95 was published as an approved standard, until July 6, 1997, when the Ada95 Ada Compiler Validation Capability (ACVC) version 2.1 is required for use in the validation process. ACVC Version 2.1 shall be available for Ada validation use beginning April 6, 1997. During the transition period, two (2) ACVC versions were in use for validation testing - ACVC 1.11 for Ada83 and ACVC 2.0.1 for Ada95 and compiler vendors could have chosen to earn a certificate for either Ada83 or Ada95. This Ada Compiler Validation Procedures Version 5.0 supersedes all previous Ada Compiler Validation Procedures and requires the use of ACVC Version 2.1 for all Ada compiler validations. Certification System The Ada certification body consists of the Ada Joint Program Office (AJPO), the Ada Validation Organization (AVO) and AJPO recognized Ada Validation Facilities (AVFs). The AJPO provides validation guidance, issues the Ada validation certificates, maintains the Ada Compiler Validation Capability (ACVC), maintains the Ada Validated Compiler list, reviews all Validation Summary Reports (VSRs), reviews and adjudicates with assistance of the AVO all disputes regarding the Ada test instrument, and maintains current operating agreements with the AVFs (see Appendix E for points of contact). Ada Compiler Validation Capability (ACVC) The ACVC is designed to demonstrate the compliance of an Ada implementation with the Ada programming language. Some test programs may contain test objectives that are irrelevant for a particular Ada implementation and may be declared inapplicable, in whole or in part, for that implementation. A test program may be withdrawn from the ACVC by the AJPO when it is found that it is based on assumptions that need not hold true for all Ada implementations or that the test program does not meet its test objective. Any interested party may dispute a test program to the AJPO. An AVF customer must dispute a test program only through the AVF providing validation services. (See Sections 4.4, 5.2.4, 5.2.5 and Appendices B and C for discussion of the dispute process and formats for submitting a dispute.) The AVF customizes an ACVC for each Ada implementation that is validated by an AVF. An Ada implementation passes a given ACVC version if it processes each test of the customized ACVC version according to grading criteria for individual tests and the test result profile matches the passing requirement for a specific ACVC version. For Ada95 validations, ACVC test programs for Specialized Needs Annexes will not be processed unless the AVF customer requests that they be processed. The result of these tests will not affect the issuance of a certificate for the core language. Specialized Needs Annexes that are fully supported, indicated by passing all the applicable tests for an annex, are indicated on the certificate. Validation Testing The validation testing process involves interaction between the AVF customer and the Ada certification body. The testing process consists of well defined steps, concluding with the issuance of a validation certificate when the test results indicate zero non- conformities. In either case, a VSR is issued to document the validation effort. The steps in the testing process are: a. A formal validation agreement is established between the customer and an AVF for validation services (see Section 5.1). b. Prevalidation is completed (see Section 5.2). c. Validation testing is performed by an AVF at a mutually agreed upon site. That is, the AVF performs an on-site witness testing (see Section 5.3). d. A Declaration of Conformance is completed and signed by the customer not later than at the on-site AVF validation testing. A validation certificate will not be issued until a Declaration of Conformance has been completed (see Section 5.4 and Appendix A). e. A VSR is completed by the AVF to document the validation by testing (see Section 5.5). Registration of a Derived Ada Implementation An Ada implementation that has been derived from an implementation that has been validated by testing (a base implementation) may be given validated status by registration, if certain conditions are true (see Section 6). These conditions form the test for determining whether to register a derived implementation or to proceed with validation by testing. A registration request must be submitted to the AJPO in the form provided in Appendix D. If the AJPO accepts a registration request, the information will be included in the list of validated Ada implementations. Appendices Appendix A contains the Declaration of Conformance statement and is for Ada95 compilers being validated with an ACVC version 2.1 based upon Ada95. Appendix B gives the format for submitting a test dispute. Appendix C describes the dispute resolution process. Appendix D gives the format for a request to register a derived Ada implementation. Appendix E gives names and addresses for points of contact who participate in the Ada compiler certification body or who provide information about validated Ada compilers. Appendix F is a list of references used in this document. Appendix G is a sample Certificate of Validation. Appendix H provides sample solicitation wording for specifying validation of Ada compilers. 1. INTRODUCTION This document provides operating policy and procedures that are followed in administering the Ada Joint Program Office of the United States Department of Defense validation program for Ada processors. This document contains operating policy and is not intended to explain the detailed procedures that can be found in the documentation associated with the Ada processor validation system, commonly called the Ada Compiler Validation Capability (ACVC). The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. In view of the well known benefits of standardization, the AJPO established a certification system to prevent the proliferation of dialects of the Ada programming language and to encourage Ada implementations which conform to [Ada83] or to [Ada95]. Effective April 6, 1997, the AJPO operates the Ada certification system. The Ada certification system rules of procedure and management in this document address the validation of Ada implementations by testing and by registration. These rules form an operational description of a validated Ada compiler. These procedures (version 5.0) address the validation process that is used for ACVC Version 2.1 for ANSI/ISO/IEC 8652:1995. These procedures (version 5.0) are effective April 6, 1997 until being superseded or revoked. The principals of the certification body of the Ada certification system consist of the AJPO for overall direction, the Ada Validation Organization (AVO) for technical support, for guidance to AJPO from industry on Ada conformance testing issues, and the Ada Validation Facilities (AVFs) for performing validations. It is important to note the scope and intent of validation. Users of an Ada implementation are cautioned that the purpose of validation is to encourage conformity of Ada implementations with the standard and that characteristics other than those specified by the standard, such as performance or suitability for a particular application, are outside the scope of Ada validation. Moreover, users are cautioned that the yardstick of conformity testing is the collection of test programs contained in the ACVC. Thus, conformity is measured only within the limitation of these tests. A glossary of terms used in this document is provided in Section 2. Appendices to this document provide examples of documents used in validation, a description of the test dispute and resolution process, current points of contacts, and references. 1.1 Ada Joint Program Office (AJPO) Validated Compilers List (VCL) The Validated Compilers List is available in a softcopy (electronic) form. The most current information would be available either via the World Wide Web: URL: http://sw-eng.falls-church.va.us/AdaIC/compilers/ or for those users who do not have access to the World Wide Web, the VCL is available on the Internet via FTP as follows: type: ftp sw-eng.falls-church.va.us login as user: anonymous password: type: cd public/AdaIC/compilers 1.2 SCOPE OF Ada VALIDATION As a technical activity, an Ada validation has a very narrow scope. The results of validation apply only to the Ada processor tested, its operating environment, and the ACVC Test Suite version used. The results should be reproducible for the environment tested, but not necessarily for other environments. Therefore, it is important to document accurately the significant software and hardware components involved. 1.3 CAVEATS Validation testing does not warrant that the product tested is free of nonconformities, even if all tests are passed. The practical goal of Ada validation is to identify Ada processors which may be procured and used to develop application programs which meet the ANSI/ISO/IEC 8652:1995 goals of portability and interoperability. The ACVC Test Suite is not designed to replace the vendor's quality assurance testing or systematically to detect inconsistencies or "bugs." The technical goal of Ada validation testing is primarily to verify that the Ada product correctly supports all required features. Rather than exhaustive testing of permutations of features, the ACVC Test Suite contains a carefully-chosen set of test cases which cover the required syntax and demonstrate the correct implementation of each of the applicable general rules from the standard. Validation is not intended as a means of performance benchmarking. The VSR does not contain information about the speed, cost, or efficiency of executing the validation programs. 1.4 REGISTRAION OF OTHER ENVIRONMENTS The AJPO conformance testing program for Ada provides for two types of assurance regarding conformance to the standard. The first type is the assurance provided by a formal validation (third-party testing conducted by a recognized testing laboratory). This is referred to as validation testing (see Section 5). The second type is the assurance offered by the product supplier based on the supplier's self testing and a signed declaration of conformance. This is referred to as registration of other computer system environments (see Section 6). The second type is allowed only for a formally validated Ada processor which has been implemented on or adapted to additional computer system environments. Registration of other computer system environments is provided by the AJPO conformance testing program as a cost-efficient alternative to formal validation. Registration of other computer system environments is allowed only for environments which are very similar to the formally validated environment. 2. GLOSSARY OF TERMS Ada programming language: The language defined by reference [Ada95]. ACVC 2.1: The Ada Compiler Validation Capability Test Suite used to assess conformance to ANSI/ISO/IEC 8652:1995 at the initial publication of this document. ACVC User's Guide: A document that explains the technical details of processing the test programs and evaluating their results. Ada compiler: The software and any needed hardware that have to be added to a given host and target machine to allow transformation of Ada programs into executable form and execution thereof. Ada Compiler Evaluation System (ACES): A test suite designed to assist in the evaluation of Ada compilers and their implementation. The ACES does not test conformity to the Ada programming language but can be used to demonstrate the performance characteristics of an Ada implementation. Ada Compiler Validation Capability (ACVC): The means for testing compliance of Ada implementations, consisting of the test suite, the support programs, and the ACVC User's Guide. Ada implementation: An Ada compiler with its host and target computers and operating systems (kernels) Ada Issues: Working documents of the ARG, containing answers to questions of interpreting the Ada standard. These documents go though formal approval by the ARG and are intended as the basis for future Corrigenda to the Ada Standard. Ada Rapporteur Group (ARG): The Ada rapporteur group (ARG) is a subgroup of ISO-IEC/JTC1/SC22/WG9, the International Organization for Standardization Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language. Ada Validation Facility (AVF): The part of the certification body that carries out the procedures required to establish the compliance of an Ada implementation. Adaptive maintenance: [ANSI/IEEE 90] Maintenance performed to make a software product usable in a changed environment. Annexes: The parts of [Ada95] that follow Section 13. Applicable ACVC test: A test that is neither inapplicable nor withdrawn. Compare with inapplicable test, and withdrawn test. Base implementation: An Ada implementation that was validated by an AVF on-site witness testing (see Section 5). Baseline: [IEEE 90] A specification or product that has been formally reviewed and agreed upon, that thereafter serves as the basis for further software development or maintenance work, that can be changed only through formal change procedures. Certification body: [ISO/IEC 86] An impartial body, governmental or non-governmental, possessing the necessary competence and reliability to operate a certification system, and in which the interests of all parties concerned with the functioning of the system are represented. Certification system: [ISO/IEC 86] A system having its own rules of procedure and management for carrying out conformity certifications. Compliance of an Ada implementation: The ability of the implementation to pass an ACVC version. [Note: For the purposes of this document, compliance is a practical measure of conformity.] Computer system: [IEEE 90] A system containing one or more computers and associated software. Configuration management: [IEEE 90] A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements (See Appendix F). Conformity: [ISO/IEC 86] Fulfillment by a product, process or service of all requirements specified. [Note: Also see Section 1.1.3 of ANSI/ISO/IEC 8652:1995] Conformity testing: The process described in Section 5 of this document. Core language: The Sections 1-13 and Annexes A, B and J of [Ada95]. [See Section 1.1.2 of ANSI/ISO/IEC 8652:1995] Corrective maintenance: [ANSI/IEEE 90] Maintenance performed specifically to overcome existing faults. Customer: An individual or corporate entity who enters into an agreement with an AVF that specifies the terms and conditions for AVF services (of any kind) to be performed. Customized test suite: The ACVC tests, adjusted as necessary, that must be used for validation testing of a given Ada implementation (see Section 4.5). Declaration of conformance: A formal statement from a customer assuring that conformity is realized on the Ada implementation for which validation status is requested. [See Appendix A for the format of a declaration of conformance.] Derived implementation: An Ada implementation that was obtained by performing maintenance on a base implementation that has a current validation certificate (see Section 6). Fail an ACVC version: An Ada implementation fails an ACVC version if it fails one or more required tests of the ACVC version. Host machine: [IEEE 90] (l) A computer used to develop software intended for another computer (contrast with target machine). (2) A computer used to emulate another computer. (3) The computer on which a program or file is installed. (4) In a computer network, a computer that provides processing capabilities to users of the network. Inapplicable test: A test that contains one or more test objectives found to be irrelevant for the given Ada implementation. Instruction set: [IEEE 90] The complete set of instructions recognized by a given computer or provided by a given programming language. Kernel: [IEEE 90] (1) That portion of an operating system that is kept in main memory at all times. (2) A software module that encapsulates an elementary function or functions of a system. Operating system: [IEEE 90] A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. Pass an ACVC version: An Ada implementation passes a given ACVC version if it processes each test of the customized test suite in accordance with grading criteria for individual tests and the test result profile matches the passing requirement for a specific ACVC version. Perfective maintenance: [ANSI/IEEE 90] Maintenance performed to improve performance or maintainability. Prevalidation testing: Process of supplying to an AVF the results of processing an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests. Registration request: A formal request for extension of validated status to a derived implementation (See Appendix D for the required form). Result profile: The aggregate of test results produced by processing the customized test suite according to given evaluation criteria (see Section 6). Software maintenance: [ANSI/IEEE 83] Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. Specialized Needs Annexes: Annexes C through H of [Ada95]. These Annexes define standards for additional functionality required by specific application areas. An Ada Implementation may support some or none of these annexes. Conformity to a special needs annex means that each capability required by the annex is provided as specified. Target machine: [IEEE 90] (1) The computer on which a program is intended to run. (2) A computer being emulated by another computer. Target run-time system: The set of subprograms that may be invoked by linking, loading, and executing object code generated by an Ada compiler. If these subprograms use or depend upon the services of an operating system, then the target runtime system includes those portions of that operating system. Test issue: Any problem arising during validation (see Section 5.2.5). Validation: The process of checking the conformity of an Ada implementation to the Ada programming language and of issuing a certificate for the implementation. Validated Ada implementation: An Ada implementation that has been validated successfully either by AVF on-site witness testing (see Section 5) or by registration (see Section 6). Validated Ada compiler: The compiler of a validated Ada implementation. Validation Certificate (VC): A certificate issued by authority of the AJPO for a tested Ada implementation that passes an ACVC version. Validation Summary Report (VSR): A report produced by an AVF containing results that are observed from on-site, witness testing a specific Ada implementation or grouping of Ada implementations. Withdrawn test: A test which has been officially removed from the test suite because the test did not meet the test objective. In most cases, the test case will be rewritten and included in a future release of the test suite. Until then the test case is not used. 3. ORGANIZATIONS AND RESPONSIBILITIES This section specifies the roles of organizations that form the certification body and of customers who receive service from them. 3.1 Ada JOINT PROGRAM OFFICE (AJPO) The AJPO provides direction for the certification system by: a. setting validation standards to be followed by all AVFs; b. establishing the conditions for issuance, the life, and the scope of a validation certificate; c. establishing the schedule for issuing versions of the ACVC; d. approving the release of an ACVC version; e. designating members of certification body; f. resolving issues that may arise during validation when these issues cannot be resolved through the best efforts of the AVO and the AVF; g. maintaining the official list of validated Ada implementations; h. issuing the validation certificates for Ada implementations validated by testing (see Section 5) and the registration of derived implementations (see Section 6); i. overseeing the ACVC quality control and configuration management process; 3.2 Ada VALIDATION ORGANIZATION (AVO) The AVO provides the technical and administrative support required to operate the certification system by: a. advising the AJPO and the AVFs concerning requirements for modification to the validation procedures; b. resolving issues that may arise during the validation process; c. reviewing all validation summary reports (VSRs) prepared by AVFs: d. recommending to the AJPO issuance of a validation certificate for Ada implementations validated by testing (see Section 5) and the registration of derived implementations (see Section 6); e. participating in the ACVC quality control and configuration management process; f. deciding on the withdrawal of test programs from the ACVC version that is being used for validation; g. convening meetings of the members of the certification body at appropriate intervals to discuss the process and to evaluate practices. 3.3 Ada VALIDATION FACILITIES (AVFs) An AVF is chartered by the AJPO to conduct validation by: a. adhering to validation procedures approved by the AJPO; b. producing the VSR; c. forwarding unresolved test issues to the AVO for review and analysis, with final resolution to be provided by the AJPO, if necessary; d. providing advice on a customer's registration request for a derived Ada implementation; and e. striving to satisfy national accreditation criteria. The AJPO may issue an AVF charter to an organization that has been recognized as an accredited testing laboratory by the AJPO. The AJPO may issue a charter to an organization located in a country that has a Memorandum of Understanding (MoU) with the U.S. government covering the chartering of AVFs, according to the rules specified in the MoU. An AVF charter shall remain in effect for the duration specified in the charter; however, a charter can be revoked by the AJPO, at any time, for due cause. The AJPO may direct an impartial body to conduct an audit at any time or prior to issuing an AVF charter. Audits are conducted in accordance with procedures established by the AJPO at the time of the audit and are tailored to reflect the purpose of the audit. 3.4 CUSTOMERS Customers are serviced by the Ada certification body in matters concerning Ada validation. In requesting services of the Ada certification body, customers are to provide accurate and complete information to perform validation, to register derived implementations, or to obtain other services. 3.5 FAST-REACTION TEAM (FRT) The fast-reaction team is a small group of Ada language experts who advise the AVO on complex test issues by: a. assisting the AVO to issue a decision to the customer and AVF in a timely manner; and, b. contributing to ISO Working Group 9 language issue resolution. 4. THE Ada COMPILER VALIDATION CAPABILITY The ACVC is designed to demonstrate the conformity of an Ada implementation with the standard. The ACVC is distributed as a collection of test programs, support programs which facilitate processing the tests, and an ACVC User's Guide that explains the criteria for evaluating the results. The ACVC Version 2.1 will be available for Ada validation use beginning April 6, 1997. Compiler vendors must use ACVC Version 2.1 on or after July 6, 1997. 4.1 VERSIONS New ACVC versions are released periodically according to a schedule which is determined by the AJPO. Each new version incorporates changes to the ACVC as deemed necessary by the AJPO. These changes are made in order to reflect a revision of the standard, to incorporate ISO WG9 interpretations, or to address implementer or user comments. The test objectives and test programs for each new ACVC version are available for public review and comment before that version is issued for use in validation. Ada validation related questions or comments on test objectives, or test programs should be submitted by e-mail to acvc-comment@sw- eng.falls-church.va.us, or by FAX (703-681-2869) or by mail: Ada Information Clearinghouse (AdaIC), P.O. Box 1866, Falls Church, Virginia USA 22041; voice telephone: 800-232-4211 or 703-681- 2466. 4.2 APPLICABILITY OF ACVC TEST PROGRAMS Each ACVC test program has one or more test objectives which are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada implementation (e.g., "check floating-point operations for digits 18"). These test programs generally contain an explicit indication of their applicability and the expected behavior of Ada implementations for which they do not apply. The determination of applicability is made according to the grading criteria in the ACVC or as a ruling by the AJPO. All applicable test programs must be processed and passed according to the specified grading criteria. For [Ada95] validations, ACVC test programs for special needs annexes (C, D, E, F, G, and H) will not be processed unless the AVF customer requests that they be processed. The result of these tests will not affect the issuance of a certificate for the core language but shall be made available as part of the public record of the Ada validation effort. Full support of a special needs annex is indicated on the certificate provided that the implementation passes all applicable test programs for that annex according to the specified grading criteria. 4.3 TEST MODIFICATIONS The certification body strives to apply the ACVC as uniformly as practical to all Ada implementations. In order to apply common test objectives that depend on implementation dependent characteristics (e.g., line lengths and numeric types), many test programs must be adjusted to a given implementation following the procedures given in the ACVC User's Guide. These adjustments consist of the insertion of implementation dependent values in certain test programs at places prescribed by the ACVC User's Guide. In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (e.g., the algorithm for recovering from syntax errors). The allowable changes for each Ada implementation are determined by an AVF after consultation with the AJPO, except in the case of error-recovery problems which an AVF may address without consultation. 4.4 TEST WITHDRAWAL In any ACVC version, it is possible that a test program is based on assumptions that need not hold true for all Ada implementations or that a test program does not meet its objective. In these cases, the AVO may issue a correction to the evaluation criteria in the ACVC user's guide or the test program may be withdrawn from that version of the test suite. Any interested party may challenge a test program by sending a rationale for the challenge to the Ada rapporteur group (ARG) (see appendix G for the address). When the ARG has accepted the challenged test for extended resolution, the AVO may withdraw the test. When an AVF customer prepares for validation, the customer must challenge a test program only through an AVF by asking for a review of its evaluation criteria or for its withdrawal. The form for submitting a challenge is provided in Appendix B. 4.5 CUSTOMIZATION A customized test suite is produced by the AVF for each Ada implementation that is a candidate for validation. This customization always consists of removing withdrawn tests and in making required modifications to test and support programs; and, it may include removal of some inapplicable tests as allowed by the ACVC User's Guide. 4.6 AVAILABILITY The ACVC is available to the general public from an AVF or from the Ada Information Clearinghouse (AdaIC) on the Internet. See APPENDIX E. POINTS OF CONTACT. 5. VALIDATION TESTING There are a number of steps that must be completed by a customer and the certification body so that the customer obtains a validation certificate with a VSR or a Registered Validation Summary Report. The same ACVC version must be used to complete the steps described in this section. The AJPO announces the in- use and end-use period for each ACVC version. The AVF must begin on-site, witness, validation testing of the Ada implementation at the customer's site before the current ACVC version expires or else validation with that ACVC version will not be allowed. Anyone intending to obtain a validation certificate should contact an AVF without delay for advice on the handling of the ACVC, on interpretation of the test grading criteria, and on the operational details of that AVF's management practices. 5.1 VALIDATION AGREEMENT In order to obtain services from the certification body, an interested party must become a customer of an AVF by reaching a formal agreement. This agreement should address the following topics: a. identification of the Ada implementation(s) to be tested and the ACVC version to be used; b. a statement of work, including analysis of prevalidation testing, witness testing, and preparation of the VSR; c. the format of data to be exchanged; d. a schedule of events and the site of validation; e. financial arrangements; f. retention of records; g. AVF liability; and h. confidentiality of validation information. The schedule for events, deliverables, and payments should take into account the fact that certain steps in the validation process require interaction with other members of the certification body (e.g., AVO or AJPO). The AVF will put forth its best effort to keep confidential a customer's intent to obtain a validation certificate and the projected schedule for validation. This confidentiality will not be allowed to interfere with the normal review procedures of validation. If the customer requests confidentiality for reasons of national security or procurement sensitivity, the customer will provide to the AVF an official, written statement describing the request and the reason(s) for the request. The AVF will obtain further guidance from the AJPO. 5.2 PREVALIDATION The requirements of this step are discussed separately so that the customer understands the interaction that is required with an AVF. 5.2.1 CUSTOMER TESTING After entering into a formal agreement, the customer prepares a customized test suite. The customer then processes all the tests in this customized test suite using the candidate Ada implementation or another Ada implementation that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of an option controlling the production of information output. Any other exception constitutes a test issue that must be resolved with the AVF (see Section 5.2.5). Test issues should be sent to the AVF for analysis as soon as they are known. The customer must provide (unless explicitly waived by the AVF) the prevalidation materials in a timely manner or by the agreement requirements between the customer and the AVF. Prevalidation materials include as a minimum the processing of an appropriately customized test suite by the customer, a customer supplied Declaration of Conformance, and any disputed ACVC tests. 5.2.2 SUBMISSION OF RESULTS Upon completion of testing, the customer delivers the complete set of results in the agreed format to the AVF. These results are accompanied by the following information: a. a list of test programs that the customer claims are inapplicable, together with a rationale for inapplicability; b. a list of test programs that are disputed (see Section 4.4) together with rationale (see Appendix B for format); c. the necessary information for the AVF to prepare the customized test suite; d. an annotated sample command script; e. the complete set of option settings used for processing the customized test suite, including the default settings; and f. complete and current documentation for implementation- dependent characteristics as required in Annex M of [Ada95]. 5.2.3 TEST MODIFICATIONS In order to comply with the test objective it may be required to modify the test source code, the test processing method, or the test evaluation method. Modifications are allowable because at the time of test writing not all possible execution environments of the test and the capabilities of the compiler could be foreseen. Only the AVO shall make the decision to use any of the following Test Modifications. Possible kinds of modification are: - Test Modification: The source code of the test is changed. Examples for test modifications are the insertion of a pragma, the insertion of a representation clause, or the splitting of a B-test into several individual tests, if the compiler does not detect all intended errors in the original test. (This last example is the only exception to the rule that only the AVO shall make the decision to apply a Test Modification.) - Processing Modification: The processing of the test by the Ada implementation for validation is changed. Examples for processing modification is the change of the compilation order for a test that consists of multiple compilations or the additional compilation of a specific support unit in the library. - Evaluation Modification: The evaluation of a test result is changed. An example for evaluation modification is the grading of a test other than the output from REPORT.RESULT indicates. This may be required if the test makes assumptions about implementation features that are not supported by the implementation (e.g. the implementation of a file system on a bare target machine). 5.2.4 TEST ISSUES A test issue may be any of the following: a. a missing or incomplete result to a test; b. a result presented in an inadequate form; c. a disagreement between the customer and the AVF as to the interpretation of a result; d. a change in the choice of options to be used during testing; e. a result that makes the Ada implementation fail the ACVC according to the current grading criteria; or f. an implementation-dependent characteristic that may affect the conformity of the Ada implementation. The material submitted by the customer is analyzed by the AVF and test issues are resolved. If the AVF and the customer cannot agree on a way to resolve a test issue, the issue will be referred to the AVO for a resolution (see Section 5.2.5). It may be justified to leave a test issue unresolved at prevalidation. For example, it may be impossible to check the processing of control characters by inspecting printed results. The AVF will note these unresolved issues and describe the results that will be expected during validation testing. It is also possible that the customer information provided for production of the customized test suite (see Section 5.2.1) was insufficient so that corrections to the customized test suite must be made and additional processing will be required. 5.2.5 GRADING RULES FOR REPRESENTATION CLAUSE TESTS The AJPO grading rules for representation clause tests (Annexes C and D) for ACVC Version 2.1 are: a. For Annex C or D validations, ALL tests in the sub- directory of the category of recommended level of support required must be PASSED or judged inapplicable through the dispute process according to the existing validation procedures. b. For core validation, the representation clause tests that are rejected by the implementation prior to execution are graded as "Unsupported". The results are reported in the Validation Summary Report or the Registered Validation Summary Report or the Validation Certificate. c. For core validation, the representation clause tests that fail in execution are graded as "Failed" unless judged inapplicable by the test dispute process. Such a test failure is graded as a failure for the core validation. The results are reported in the Validation Summary Report . 5.2.6 TEST ISSUE RESOLUTION A customer may challenge the applicability or correctness of any particular ACVC test program. Such challenges should be presented to the AVF in the test-dispute format (see Appendix B). The AVF will forward challenges to the AVO for resolution; the AVO will strive to rule on the challenge within two weeks of receiving it. The AJPO will forward all challenges and rulings to the AJPO. (See Appendix C for a description of the Test Dispute and Resolution Process.) 5.2.7 INCOMPLETE PREVALIDATIONS The AVF and the customer may agree that, at the customer's risk, all or parts of the customized test suite need not be processed during prevalidation. The customer must certify that the results from a previous prevalidation submitted to the AVF or validation results obtained by the AVF are identical to those that would have been obtained by the customer. The normal practice is to submit complete prevalidation results. 5.2.8 SUCCESSFUL PREVALIDATIONS Prevalidation testing is successful if the analysis of results and the resolution of test issues show that the candidate Ada implementation passes the customized test suite. Prevalidation is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues is deferred until validation testing by agreement between the AVF and the customer. 5.3 VALIDATION TESTING Upon successful completion of prevalidation, with or without caveats, the AVF witnesses testing of the Ada implementation at the site and time mutually agreed by the AVF and customer. The AVF prepares a customized test suite based upon customer information and any information collected during the resolution of test issues. The customized test suite is installed and processed in the physical presence of an AVF representative. If the AVF determines that the results agree with those obtained from prevalidation and are satisfactory with respect to the caveats, the testing has been successful; otherwise, re-testing will be required, unless the validation attempt is discontinued. 5.4 DECLARATION OF CONFORMANCE At the start of the on-site, witness testing of the customer's Ada implementation, the customer will provide to the AVF a completed, dated, and signed declaration of conformance. The customer should provide the information contained in the declaration of conformance to the AVF prior to the on-site testing as part of the prevalidation effort. The declaration of conformance states that the organization that is responsible for the production, maintenance or distribution of the Ada compiler is offering a product that is in conformity with the Ada programming language. The declaration of conformance becomes part of the AVF records and is copied into the VSR. Neither a validation certificate nor a Registered Validation Summary Report will be issued unless a signed declaration of conformance has been provided to the AVO (See Appendix A, for an example of the declaration of conformance.) 5.5 VALIDATION SUMMARY REPORT (VSR) A VSR is produced for each validation testing effort. A single VSR may cover validation testing of several Ada implementations, provided that they all have the same result profile. The VSR provides the following documentation pertaining to the validation effort: a. identification of the customer responsible for validation of the Ada implementation; b. identification of the organization responsible for the production, maintenance, or distribution of the Ada compiler or Ada implementation; c. identification of the Ada implementation tested; d. identification of the options used for testing; e. the inapplicable test programs and implementation- dependent characteristics exhibited by the test programs that established inapplicability; f. the implementation-dependent characteristics pertinent to the customized test suite; g. withdrawn tests; h. modifications to test programs with an explanation for such modifications; and i. a description of failed test programs, if applicable. 5.5.1 VSR PRODUCTION The VSR is prepared by the AVF but includes material that is produced by the customer, such as the documented compiler and linker options used during the Ada validation process. Draft versions of the VSR are sent to the AVO for review before and after validation testing. The final version of the VSR is signed by the AVF, AVO, and the AJPO. 5.5.2 VSR AVAILABILITY The final version of the VSR is available to the general public from the customer and from the AVF that produced it. The AVF may require payment of a fee for VSR reproduction and mailing fee. The VSR is available on-line whenever possible (see APPENDIX E. POINTS OF CONTACT). There is no cost for VSRs obtained by electronic means. 5.6 SUCCESSFUL VALIDATION For each successful validation, one certificate is issued by the AJPO. The information on the validation certificate describes the tested Ada implementation: the source of this information is the signed declaration of conformance that describes the tested Ada implementation that the AVF provides to the AVO after completion of testing. The customer will ensure that the information contained on the certificate does not infringe on the rights of third parties and may be required to provide a written statement of consent from any third party involved. Validation certificates for ACVC 2.1 will expire two (2) years after issuance or one (1) year after a new test suite is released whichever is earlier. A validation certificate may be renewed by the original owner of the validation certificate if the requirements in 5.7 are fully satisfied. The certificate also indicates which specialized needs annexes, if any, are fully supported. An entry in the list of Ada implementations validated by testing will be made for each certificate issued. This entry will be removed when the certificate expires along with all entries for registered validations derived from it. A sample Certificate of Validation is provided in Appendix G. 5.7 EXTENSION FOR CERTIFICATE OF VALIDATION The requirement for revalidation may be waived and the Certificate of Validation extended at the option of AJPO for no more than two (2) extensions if the supplier submits a signed statement certifying that all of the following criteria are met: a. No substantive changes have been made to either the Ada processor or the system software that supports it. However, if continuing maintenance changes or adaptive changes (for interfacing to the operating system or integrated software) have been made, then AJPO may require revalidation. b. The Ada processor has been validated with the latest official Ada Test Suite release. c. No changes were made in the supporting software (operating system, programming language compiler(s), other software named in the VSR) since validation testing that alter the function or operation of the Ada processor. However, if continuing maintenance changes to the subject operating system or pertinent supporting software have been made, then the AJPO may require revalidation. d. A newly signed and dated Declaration of Conformance is submitted to the AJPO. Only the expiration date will be changed on the Certificate of Validation. Version or release numbers for all software will not be updated when a Certificate of Validation is extended. No more than two (2) one-year extensions will granted; then the Ada processor must go through complete validation testing. 5.8 VALIDATION WITH SUPERSEDED VERSIONS OF TEST SUITE The AJPO is willing to conduct a validation for an Ada processor with superseded versions of the test suite. The result of such a validation is a VSR which will not be listed in the VCL, but which may be shown to customers requiring this type of evaluation. The AJPO procedures require the use of the latest version of the Ada Validation Test Suite for validation testing. Certificates of Validation and Registered Validation Summary Reports are issued only for testing done with the latest version of the test suite. Testing with an expired test suite is done only upon special request for specific procurement requirements. Certificates of Validation or Registered Validation Summary Reports are not issued for tests performed with expired test suites. When testing is performed with an expired test suite, the AJPO will provide a letter in lieu of a Certificate of Validation or a Registered Validation Summary Report. Testing must be conducted using the procedures and tests that were in effect for the ACVC Test Suite Version requested. The letter will note whether or not the implementation described in the Validation Summary Report (VSR) passed all applicable tests of the ACVC Test Suite version as specified by the Ada Validation Procedures in effect for that ACVC test suite version. The letter will be dated but shall not have an expiration date since the letter is not a certificate. Concurrently with but not later than the request for validation with a superseded Ada test suite version, the applicant may include derived implementation(s) which may be noted in an attachment to the letter should the derived request(s) satisfy the derived requirements as specified by the Ada Validation Procedures in effect for that ACVC test suite version. 5.9 PUBLICATION VSRs registered with AJPO shall be made available to the public. In no event, however, shall the name of the supplier or any of its trademarks and trade names be used in AJPO publications without the supplier's prior written consent. With respect to publication in the VCL, the standard Validation Customer Agreement contains the supplier's written consent. AJPO and the supplier shall agree to confer and consult prior to the publication of data to assure that no Proprietary Data is released and that patent rights are not jeopardized. Prior to publishing a VSR, the supplier shall be offered an opportunity to review such proposed publication. 5.10 RETENTION OF VALIDATION RECORDS Each AVF shall determine the length of time that records and information for an Ada validation are retained. For the AVO, the records and information used in preparing the VSR will typically be retained for six (6) months after expiration of the latest Certificate of Validation. 5.11 ADVERTISING VALIDATED STATUS The customer will not advertise or make public claims that the Ada implementation is validated until after receiving a validation certificate or after receiving formal notification from the AVF that the AJPO has issued a validation certificate. A waiver of confidentiality must be signed by a customer who intends to advertise the completion of events that indicate progress toward completion of validation. If a waiver of confidentiality has been signed with the AVF, the AVF will respond to inquiries about the customer's advertisements or public claims by acknowledging receipt of validation materials (i.e., a formal agreement, pre- validation results, or validation testing results) without judgement concerning the success of the validation. 5.12 CONTENT OF Ada VALIDATION CERTIFICATE The Ada Validation Certificate shall contain as a minimum the name of the Ada compiler under test, the name of the testing laboratory performing the validation, identification of the host computer and the host computer operating system, identification of the target computer and the target computer operating system (if present), the validation certificate number, identification of validation summary report number, the issue date of the certificate, the expiration date of the certificate, identification of Specialized Needs Annexes fully supported (fully supported means that applicable tests for that annex are graded PASSED), and the test counts with category explanations (on an attached sheet if necessary). 6. REGISTRATION OF DERIVED IMPLEMENTATIONS It is expected that nearly every Ada implementation that is validated by testing is intended to be representative of a class of implementations that are similar in their defining components of Ada compiler, host machine, and target machine. Such implementations are known as derived implementations. Whereas a validation certificate identifies only the Ada implementation that was tested by the AVF (i.e., the base implementation), the validated status that it conveys is extensible to a range of derived implementations for the life of the certificate. The procedure for extending validated status to a derived implementation is called registration. This section prescribes the requirements for registration. For any base implementation with a current validation certificate, a derived implementation may obtain validated status by registration when it satisfies all of the following conditions: a. The Ada compiler was obtained from the Ada compiler of the base implementation by changes that are within the scope of accepted software maintenance practices. b. The target machine of the base and derived implementation have compatible instruction sets and operating system or kernel. c. The result profile for the derived implementation is either the same as the base implementation or, if there are differences, these differences are justified as being within the scope of accepted software maintenance practices. The changes that may be made to an Ada compiler for the purpose of derivation will be within the scope of software maintenance as applied to the domain of compiler construction. Changes must be classified as corrective, adaptive, or perfective. Examples of these changes are the correction of a compiler error, the adaptation to an operating system upgrade, the transfer of the compiler to another host machine, the addition of a floating point processor to a small target machine, or the perfection of a garbage collection algorithm. For the purposes of obtaining validated status by registration, the changes required to render the base Ada compiler fully functional on a different host machine or host operating system are considered adaptive maintenance. However, a change in a target machine may be outside the scope of acceptable maintenance. For a target machine, common examples of compatible instruction sets and operating systems are two different computer system models in a manufacturer's product line or the computer systems produced by different manufacturers that use the same instruction set and operating systems, or any computer system and a simulated or emulated computer system that have the same instruction set. 6.1 REGISTRATION REQUEST Any interested party may initiate the registration of a derived implementation by sending a registration request to the AVO or to an AVF. A registration request consists of two parts -a "letter" part and a "supporting information" part. Additionally the requester must provide any other information specifically requested by the AVF. The information requested may include test results from the supplier's self-testing with the ACVC. Appendix D specifies both the form and content of the information to be provided in these two parts. A registration request must be submitted to the AVF or the AVO no more than 30 days before the expiration of the base implementation's certificate. The registration request contain descriptions of the base and derived implementations, a declaration of conformity to the relevant version of the Ada programming language, compliance with the relevant version of the ACVC, a description of what maintenance and testing was performed, and any appropriate supporting documentation from hardware or software vendors. If any party in addition to the one submitting the request has a legal interest in the derived implementation, this interest will also be documented in the request. Two Ada implementations have the same result profile when: a. they use the same customized test suite; and, b. their output from processing the customized test suite is the same (with the possible exception of the precise content and position of error messages from the compiler). If the derived implementation has a different result profile than the base implementation, the registration request must describe and justify the differences. If the information requested cannot be provided or it does not support the supplier's claim, the supplier's registration request will be denied. 6.2 EVALUATION The AVF will not perform testing on derived implementations, but may require submission of the test results described in subparagraph c of Section 6. Any AVF may review the registration request for completeness and the plausibility of information and provide advice. The AJPO will evaluate all registration requests before adding the derived implementations to the list of validated Ada implementations maintained by the AJPO. 6.3 REGISTRATION Registration requests that are acceptable to the AVO will be registered as validated Ada implementations untested by an AVF. A validation certificate will not be issued for these derived implementations. The list of derived implementations and information provided in the registration request will be available to the public. 6.4 EXPIRATION OF VALIDATED STATUS A derived implementation loses its status of being validated if it is challenged successfully (see Section 6.7), upon expiration of the validation certificate of its base implementation, when registration is revoked by the customer, or when the derived implementation is no longer supported by the supplier. APPENDIX A. DECLARATION OF CONFORMANCE Declaration of Conformance Customer: Certificate Awardee: Ada Validation Facility: ACVC Version: Ada Implementation: Ada Compiler Name and Version: Host Computer System: Target Computer System: Declaration: I, the undersigned, declare that my organization has no knowledge of deliberate deviations from the Ada Language Standard (ANSI/ISO/IEC 8652:1995, FIPS 119-1) other than the omission of features as documented in the Validation Summary Report. ______________ Customer Signature/Date Company Title _________________ Certificate Awardee Signature/Date Company Title APPENDIX B. IMPLEMENTER DISPUTE FORMAT [part A] Implementer: Configuration: ACVC Version: Pre-Validation Submittal Date: <:due date for in-house results> [Part A will be completed once by each implementer; part B will be completed for each dispute. It is not necessary for a pre- validation date to have been established. Part A information is treated as confidential.] [part B] Reference: Summary: Discussion: [In this Discussion, arguments should be specified using test line #s and references to pertinent sections of the Ada standard or Commentaries (Al-xxxx). The implementer must describe the behavior of the implementation for the test or tests that are disputed, stating the particular test messages that are produced. It is sufficient for the detailed description to be limited to the particular segment of test code that is disputed. Relevant source code with compiler messages should be included. (For a group of tests that cause essentially the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.)] If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the AJPO to assess the reasonableness of these limitations. Failure to fully specify the points pertinent to a dispute might result in an adverse decision being made, with the disputer having to further argue the case with a second submittal to the AJPO. Yet it is possible that the Summary will suffice to adequately present a dispute. APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS C.1 Introduction A "dispute" is defined by the ACVC User's Guide as any result from processing an ACVC test program that is not a passed, inapplicable, or unsupported result according to the established grading criteria. This intentionally broad definition of a "dispute" is to make certain that compiler implementers bring all deviant test results to the attention of the AVF, without assuming that such results would be accepted without special review. The compiler implementer also provides a rationale for each challenge being made to a particular test program. Disputes are forwarded to the AVO, usually electronically, by the AVF on behalf of their validation customer. For each dispute that is accepted (i.e., when the AVO rules in favor of the dispute), it is likely that some correction is indicated for the disputed tests. The AVO withdraws any test that is found to be incorrect to a degree that makes it unsuitable for validation. The withdrawal of a test consists of including it on a list of tests that are ignored for validations conducted with the current ACVC version. The AVO updates the list of withdrawn tests, and distributes this list to the AVFs and to the AJPO. The AVO also maintains a database consisting of all test disputes and their resolution (AVO rulings) which is periodically provided to the FRT and other interested parties. The AVO may also recommend that certain resolved disputes be considered further by the ARG/URG. C.2 Expedited Resolution Process The AVO resolves disputes by any of three methods: a resolution that was made previously is applied to the current dispute (e.g., the same dispute might be submitted at different times by different petitioners); the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or, the resolution is determined based on the deliberations of the FRT. Although the Ada Compiler Validation Procedures do not set a limit on the length of time for reaching a resolution, the AVO attempts to resolve disputes within two weeks, an informal guideline that was established by the certification body. If a dispute is submitted to the AVO more than four weeks before the scheduled beginning of on-site testing, and the AVO is unable to issue a final resolution of the dispute within the two week period, then the dispute is presumed to have been resolved in the implementers favor. The disputed tests will be graded as inapplicable and the default resolution will be documented in the VSR. Such a default resolution will not be used as a basis for future disputes of the same test. The AVO also attempts to place priority on resolving disputes for AVF customers who have a firmly scheduled date for validation testing. Implementers should submit disputes well in advance of a scheduled validation testing date. (see Section 5) On receipt of a dispute, the AVO checks whether the issue matches any that had been previously resolved. If the dispute is new, it is given an initial AVO analysis which involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous dispute deliberations. A dispute is referred to the FRT when questions of interpretation arise and a resolution is not obvious. The AVO presents the dispute and any additional information resulting from an initial analysis to the FRT by e-mail without disclosing the identity of the petitioner. Deliberation of the dispute proceeds with the exchange of each expert's opinion and analysis. The AVO participates in the deliberation by providing information as requested (e.g., ACVC tests or information from the petitioner), eliciting discussion from the experts, and making or challenging technical points raised in the discussion. Resolution of a dispute is made by the AVO and is generally based upon a consensus of the FRT members participating. There is no prescribed formality to the FRT deliberations, such as voting procedures or time limits on deliberation. The AVO might extend deliberation when a basis for resolving the dispute has not been made. However, the AVO will give its ruling on the dispute when a sufficient basis has been established, regardless of whether the FRT discussion continues. C.3 Types of Resolutions The resolution of a dispute is either an acceptance or rejection of the petitioner's arguments. Acceptance can result in either withdrawal of the test program from the ACVC, or in a "Test, Processing, or Evaluation" modification for validation. A dispute may be rejected if it conflicts with the Ada standard or Ada Issues or if it is a test to which previously validated implementations conform and the petitioner has not provided compelling reasons for a deviation. A dispute may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. Withdrawn tests have no effect on validation (they are generally not processed). If the dispute shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the AVO will direct that the test(s) be processed with a test modification. There are three types of test modification: Test, Processing, and Evaluation modifications. A Test Modification is an actual change to the code of the test (e.g., adding a choice to an exception handler). A Processing Modification is a change to the way in which the test is processed (e.g., re-ordering the compilation of component files of a multi-compilation test). And an Evaluation Modification is simply the grading of the observed results by other than the established grading criteria (e.g., interpreting particular intermediate output and a final "failed" result as "passed", according to an understanding of the dispute). All test modifications are documented in the VSR. C.4 Summary There is no limit on the number of disputed tests that can be submitted by an implementer. Although there is a risk that a dispute will not be decided in the implementers favor, that risk can be managed so as to not affect validation by early submission of disputes. Any interested party may dispute an ACVC test and its grading criteria. APPENDIX D. REGISTRATION REQUEST A registration request must have two main parts: (1) a letter and (2) an enclosure consisting of specific information to support the request. PART I - LETTER The request letter should be addressed to the AVO (note: see Appendix E for address) even though the requester does seek advice from an AVF. The mandatory parts of the letter are: a. Certificate number and description of the Base Implementation, b. List of derived implementations that were fully tested with the customized ACVC used in the base validation. If this list is less than the list for which registration is requested item d must also be included in this letter. c. A signed and dated Declaration of Conformance. d. For the derived implementations that were not fully tested, the requester must declare that the same binary executable version of the compiler was fully tested with the same target architecture and operating system. e. If this statement applies, legal interests of (name) are affected by this registration request and their consent is documented in enclosure (number) to this letter. f. The Ada implementations for which registration is requested may, in fact, be derived according to the rules given in the Ada Validation Procedures. g. Signature h. Typed name of signer i. Date of signature j. Corporate title PART 2 - TEMPLATE FOR SUPPORTING INFORMATION Identification of Base Ada Implementation: 1. Certificate number: 2. Registration requested for: Note: This registration request pertains to all Ada implementations obtained by taking the Ada compiler and selecting any host and any target from the list below. Compiler: [name and version(s)/release number/product(s) range of versions/releases] Host: choose either (a) or (b) a. [make, specific members of series and/or model numbers] (e.g. IBM PS2 model 80-110 & 113) or, b. [all members of a model series] (e.g. IBM PS2 all models, or HP 9000 series 300) Operating System: [name and version/release number(s)] Target: choose (a), (b), or (c) a. [make, specific members of series and/or model numbers] or b. [all members of a model series] or c. [members of an instruction set architecture and board implementation(s)] (e.g. Motorola 68020-MVME133-1 & MVME130COF, Motorola 68030-VME147, Motorola 68030- MilSpec285) Operating System: choose either (a) or (b) or (c) a. [OS name and version/release number for all target machines listed] or b. [kernel identifier for all target machines listed] or c. [none for all target machines listed] Evidence that the Ada implementations listed above should be validated by registration: 1. State the type of software maintenance changes made to the compiler and describe what was done. corrective - maintenance performed specifically to overcome an existing fault. perfective - maintenance performed to improve performance or maintainability. adaptive - maintenance performed to make a software product usable in a changed environment. 2. Which host/target combinations were fully tested with the ACVC? Testing with "customer applications" or other types of tests is not equivalent to running the ACVC. 3. State whether the result profile is the same or different from the base. If different, list all ACVC test results that are different or exhibit different behavior and explain the difference. 4. What is the authoritative source used to determine the technical compatibility between the derived implementations and the base? If copies of technical manuals are supplied as the source of this information, give the enclosure number where this information can be found. Do not duplicate a previous submission of identical technical material, just reference its previous submission. If there has been a change in the previously submitted material, submit only the change. If there is no authoritative source of information given, this registration request may be refused. APPENDIX E. POINTS OF CONTACT Ada Joint Program Office (AJPO) Lt. Col. John Hamilton 5600 Columbia Pike Falls Church, VA 22041-2717 Tel: 703-681-2458 Email: hamilt2d @ncr.disa.mil Ada Validation Facility Managers Dr. William H. Dashiell The National Institute of Standards and Technology Building 820, Room 517 Gaithersburg, MD 20899 USA Tel: 301-975-2490 Email: dashiell@speckle.ncsl.nist.gov Mr. Jon Leigh The National Computing Centre, Ltd. Oxford Road Manchester England, MI7ED Tel: +44-61-228-6333 Email: jon@ncc.co.uk Mr. Michael Tonndorf IABG, Software-Technologie lTE Einsteinstr. 20 D-85521 0ttobrunn Germany Tel: +49-89-6088-2477 Email: tonndorf@iabg.de Ada Validation Organization Mr. Clyde Roby Institute for Defense Analyses 1801 N. Beauregard St. Alexandria, Va. 22311 Tel: 703-845-6639 Email: roby@ida.org Fast Reaction Team Mr. Dan Lehman (test challenges) Institute for Defense Analyses 1801 N. Beauregard St. Alexandria, Va. 22311 Tel: 703-845-6633 Email: avo@sw-eng.falls-church.va.us Validated Compilers List Ada Information Clearinghouse P.O. Box 1866 Falls Church, VA 22041 USA Tel: 703-681-2466 OR 1-800-232-4211 Email: adainfo@sw-eng.falls-church.va.us WWW: http://sw-eng.falls-church.va.us/AdaIC/ ACES Data Analysis Center for Software P.O. Box 120 Utica, NY 13503 Tel: 315-734-3696 Anonymous FTP to sw-eng.falls-church.va.us, directory is /public/AdaIC/testing/aces WWW: URL http://sw-eng.falls-church.va.us/AdaIC. Ada Rapporteur Group (ISO/IEC JTC1/SC22 WG9/ARG) Dr. Erhard Ploedereder University of Stuttgart Institute for Computer Science Breitwiesenstr. 20-22 D-70565 Stuttgart Germany Tel: +49 +711 7816-322 Fax: +49 +711 7816-380 Email: ploedere@informatik.uni-stuttgart.de Ada Information Clearinghouse Ada Information Clearinghouse P.O. Box 1866 Falls Church, VA 22041 USA Tel: 703-681-2466 OR 1-800-232-4211 Email: adainfo@sw-eng.falls-church.va.us WWW: http://sw-eng.falls-church.va.us/AdaIC/ Ada Validation Related Questions: Email: acvc-comment@sw-eng.falls-church.va.us OR adainfo@sw-eng.falls-church.va.us Validated Compiler Lists Ada Information Clearinghouse P.O. Box 1866 Falls Church, VA 22041 USA Tel: 703-681-2466 OR 1-800-232-4211 Email: adainfo@sw-eng.falls-church.va.us WWW: http://sw-eng.falls-church.va.us/AdaIC/ Ada Compiler Validation Capability (ACVC) The ACVC is available to the general public from an AVF or from the Ada Information Clearinghouse (AdaIC) on the Internet. URL: http://sw-eng.falls-church.va.us/adaic/ or by anonymous FTP at sw-eng.falls- church.va.us/public/AdaIC. In either case, look for "testing" and "acvc." APPENDIX F. REFERENCES [Ada95] ANSI/ISO/IEC 8652:1995 Reference Manual for the Ada Programming Language, February 1995 (supersedes Ada83] [Ada83] American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652:1987 and FIPS 119, 1985. [ANSI/IEEE 90] American National Standards Institute / Institute of Electrical and Electronic Engineers, Inc., Standard 610.12-1990; "ANSI/IEEE Standard Glossary of Software Engineering Terminology". [FIPS] Federal Information Processing Standards publications relative to the Ada Programming Language are FIPS 119 and FIPS 119-1 [ISO 74] International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms. [ISO/IEC 91] International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities. APPENDIX G. SAMPLE CERTIFICATE OF VALIDATION (not included in electronic version) APPENDIX H. SAMPLE SOLICITATION WORDING FOR Ada VALIDATIONS This appendix provides suggested solicitation wording for validation for Ada language processors Role of Procuring Agency It is not feasible for AJPO to validate all Ada processors which conform to the Specifications of ANSI/ISO/IEC 8652:1995, since formal validation can be expensive, time consuming, and resource intensive. The procuring agency has the authority to extrapolate from the test results published in the Validated Compiler List (VCL) to other hardware/software environments, based on additional research by the procuring agency, demonstrations or warranties by the Ada supplier, etc. The procuring agency, not AJPO, has the responsibility of reviewing the Ada validations listed in the VCL and determining the applicability of these validations to the hardware/software environment involved in a specific procurement. The criteria for applicability of a Certificate should be appropriate to the size and timing of the procurement. The AJPO suggests that procuring agencies use one (or more) of the following terms for three validation options: "Delayed Validation," "Prior Validation Testing," and "Prior Validation." The option "Prior Validation Testing" allows Ada suppliers to offer tested products which may contain nonconformities; whereas "Prior Validation" requires tested products without nonconformities. "Delayed Validation" allows Ada suppliers to offer products which may not have been tested prior to contract award. The procuring agency shall select the appropriate validation option and shall specify whether a Validation Summary Report or Certificate of Validation is required. The agency shall specify appropriate time frames for validation and correction of nonconformities. The agency is advised to refer to the VCL for information about the validation status of Ada products. This information may be used to specify validation time frames that are not unduly restrictive of competition. The sample solicitation wording for the different validation options follows: "All implementations of ANSI/ISO/IEC 8652:1995 that are brought into the Federal inventory as a result of this document for which validation is specified, and those implementations used by vendors to develop programs or provide services shall be validated using the official Validation System specified by the Ada Joint Program Office (AJPO). Validation shall be in accordance with AJPO validation procedures for ANSI/ISO/IEC 8652:1995. The results of validation shall be used to confirm that the implementation meets the requirements of ANSI/ISO/IEC 8652:1995 as specified in this document. To be considered responsive the offeror shall: (1) Provide validated ANSI/ISO/IEC 8652:1995 implementations through 'Delayed Validation', 'Prior Validation Testing' or 'Prior Validation'." (Option 1) - "Delayed Validation" Delayed Validation - When an agency determines that it is acceptable for implementations of ANSI/ISO/IEC 8652:1995 to be offered that have not yet been tested for conformance to ANSI/ISO/IEC 8652:1995 the following 'Delayed Validation' solicitation wording option may be used: "The offeror shall certify in the offer that all implementations of ANSI/ISO/IEC 8652:1995, including applicable ANSI/ISO/IEC 8652:1995 options, offered in response to this document will be submitted for validation upon contract award. Implementations submitted for validation shall have the validation completed at the earliest possible date permitted by the AJPO validation procedures. Unless specified elsewhere, proof of submission for validation shall be in the form of a letter scheduling the validation and the subsequent delivery by the offeror of a AJPO registered report or AJPO Certificate of Validation. Proof of testing shall be provided in the form of a AJPO registered validation summary report. Proof of fully validated implementations shall be in the form of a AJPO Certificate of Validation." (Option 2) - "Prior Validation Testing" Prior Validation Testing - When an agency determines that it is essential for implementations of ANSI/ISO/IEC 8652:1995 to be previously tested for conformance to ANSI/ISO/IEC 8652:1995 before being offered, and the nature of the requirement is such that an implementation of ANSI/ISO/IEC 8652:1995 may be initially offered that has not been fully validated (i.e., implementation has not fully demonstrated compliance to ANSI/ISO/IEC 8652:1995), the following 'Prior Validation Testing' solicitation wording option may be used: "The offeror shall certify in the offer that all implementations of ANSI/ISO/IEC 8652:1995, including applicable ANSI/ISO/IEC 8652:1995 options, offered in response to this document have been previously tested or validated and included on the current list of validated Ada compilers maintained by the Ada Joint Program Office (AJPO). Unless specified elsewhere, proof of testing shall be provided in the form of a AJPO registered validation summary report. Proof of fully validated implementations shall be in the form of a AJPO Certificate of Validation." (Option 3) - "Prior Validation" Prior Validation - When an agency determines that it is essential for implementations of ANSI/ISO/IEC 8652:1995 be validated (i.e., implementation has been tested and has demonstrated compliance to the ANSI/ISO/IEC 8652:1995) for conformance to ANSI/ISO/IEC 8652:1995 prior to being offered the following 'Prior Validation' solicitation wording option may be used: "The offeror shall certify in the offer that all implementations of ANSI/ISO/IEC 8652:1995, including applicable ANSI/ISO/IEC 8652:1995 options, offered in response to this document have been previously validated and included on the current list of validated Ada compilers maintained by Ada Joint Program Office (AJPO). Unless specified elsewhere, proof of validation shall be in the form of a AJPO Certificate of Validation." (2) Agree to correct all implementation nonconformance from the applicable ANSI/ISO/IEC 8652:1995 reflected in the validation summary report not previously covered by a waiver. All areas of nonconformance must be corrected within 12 months from the date of contract award unless otherwise specified elsewhere in this document. If an interpretation of the ANSI/ISO/IEC 8652:1995 is required, such a request for interpretation shall be made within 30 calendar days after contract award. Any corrections that are required as a result of decisions made under the procedures of ANSI/ISO/IEC 8652:1995 AJPO rulings shall be completed within 12 months of the date of the formal notification to the contractor of the approval of the interpretation. Proof of correction in either case shall be in the form of a AJPO Certificate of Validation showing that the implementations nonconformance has been corrected. Failure to make required corrections within the time limits set forth above shall be deemed a failure to deliver required software. The liquidated damages as specified for failure to deliver the operating system or other software shall apply."