Ada Compiler Validation Procedures, Version 4.1 December 1995 Prepared for the Defense Information Systems Agency Center for Software Prepared by the Institute for Defense Analysis EXECUTIVE SUMMARY 1. INTRODUCTION 2. GLOSSARY OF TERMS 3. ORGANIZATION AND RESPONSIBILITIES 3.1 Ada Joint Program Office (AJPO) 3.2 The Ada Validation Organization (AVO) 3.3 The ACVC Maintenance Organization (AMO) 3.4 Ada Validation Facilities (AVFs) 3.5 ACVC Reviewers 3.6 Fast-Reaction Team (FRT) 3.7 Customers 3.8 Project Manager 4. THE Ada COMPILER VALIDATION CAPABILITY 4.1 Versions 4.2 Applicability of ACVC Test Programs 4.3 Test Modifications 4.4 Test Withdrawal 4.5 Customization 4.6 Availability 5. VALIDATION BY TESTING 5.1 Step One: Validation Agreement 5.2 Step Two: Prevalidation 5.2.1 Customer Testing 5.2.2 Submission of Results 5.2.3 Test Issues 5.2.4 Test Issue Resolution 5.2.5 Incomplete Prevalidations 5.2.6 Successful Prevalidation 5.3 Step Three: Validation Testing 5.4 Step Four: Declaration of Conformance 5.5 Step Five: Validation Summary Report 5.5.1 VSR Production 5.5.2 VSR Availability 5.6 Step Six: Validation Certificate 5.7 Provisional Validation 5.8 Advertising Validated Status 5.9 Certification Mark 6. REGISTRATION OF DERIVED IMPLEMENTATIONS 6.1 Registration Request 6.2 Evaluation 6.3 Registration 6.4 Expiration of Validated Status 6.5 Challenges 6.6 Challenge Mark 6.7 Challenge Test APPENDIX A. DECLARATION OF CONFORMANCE APPENDIX B. IMPLEMENTER DISPUTE FORMAT APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS APPENDIX D. REGISTRATION REQUEST APPENDIX E. CERTIFICATION MARK FOR Ada 83 PRODUCTS APPENDIX F. PROJECT GUIDELINES APPENDIX G. POINTS OF CONTACT APPENDIX H. REFERENCES EXECUTIVE SUMMARY The purpose of this executive summary is to provide an overview of the Ada compiler validation process. Anyone who obtains services from the Ada certification body must understand the definition of terms and follow the specific rules provided in the body of this document. This version (4.1) of the Ada Compiler Validation Procedures has been updated to reflect the release and use of ACVC 2.0.1 during the transition period when compiler vendors will incorporate the features provided by the revised Ada language standard, ANSI/ISO/IEC 8652:1995 and FIPS 119-1. The transition period is from January 1995, when Ada 95 published as an approved standard, until 1 March 1997, when the Ada 95 Ada Compiler Validation Capability (ACVC) version 2.1 is required for use in the validation process. During the transition period, three ACVC versions will be "current" versions used for validation testing - ACVC 1.11 for Ada 83 and ACVC 2.0 replaced by 2.0.1 for Ada 95. During the transition period compiler vendors may choose to get a certificate for either Ada 83 or Ada 95. The mandatory criteria for obtaining an Ada 95 certificate will change when ACVC 2.0.1 is released. Validations with ACVC 2.0 required passing all applicable Ada 9X basic tests. Validations with ACVC 2.0.1 require passing all applicable Ada 9X basic tests and either all OOP or real-time tests. ACVC 2.0 will be withdrawn from use on 31 March 1996. ACVC 2.0.1 will be in use from 31 March 1996 until 1 March 1997. Certificates issued for compilers tested with ACVC 1.11, ACVC 2.0 or 2.0.1 will expire 31 March 1998. ACVC 2.1 will be available for public review three months before it becomes the official Ada 95 ACVC version on 1 March 1997. Certification System The Ada certification body consists of the Ada Joint Program Office (AJPO), the Ada Validation Organization (AVO), the ACVC Maintenance Organization (AMO), and the Ada Validation Facilities (AVFs). The AJPO, a component of the Department of Defense, provides validation guidance and issues Ada validation certificates in conjunction with the National Institute for Standards and Technology (NIST), Department of Commerce. During the transition period Ada 95 implementations validated by AVF testing will be issued certificates only by the AJPO. The AVO provides the technical guidance for operation of the Ada certification system. The AMO maintains the ACVC used by the five AVFs who conduct validations (see Appendix G for points of contact). The Ada certification body works with the Department of Commerce which has the responsibility for establishing and maintaining a certification system for the Federal Information Processing Standards (FIPS). Ada Compiler Validation Capability (ACVC) The ACVC is designed to demonstrate the compliance of an Ada implementation with the Ada programming language. Some test programs may contain test objectives that are irrelevant for a particular Ada implementation and may be declared inapplicable, in whole or in parts, for that implementation. Test programs may be withdrawn from the ACVC by the AVO when it is found that they are based on assumptions that need not hold true for all Ada implementations or that the test program does not meet its test objective. Any interested party may dispute a test program to the AVO and the Ada Rapporteur Group, ISO WG9. An AVF Customer must dispute a test program only through the AVF providing validation services. (See Sections 4.4, 5.2.3, 5.2.4 and Appendix B and C for discussion of the dispute process and formats for submitting a dispute.) The AVF customizes an ACVC for each Ada implementation that is validated by an AVF. An Ada implementation passes a given ACVC version if it processes each test of the customized ACVC version according to grading criteria for individual tests and the test result profile matches the passing requirement for a specific ACVC version. During the transition period, there are three exceptions to previous procedures for establishing the applicability of ACVC test programs. These exceptions are: a. For an [Ada 83] validation, a failure of an ACVC version 1.11 test program may be judged inapplicable when the reason for failure is caused by a correct but partial implementation of [Ada 95]. b. For an [Ada 95] validation with version 2.0, the mandatory criteria for obtaining a certificate is that all applicable Ada 9X basic tests must be passed. c. For an [Ada 95] validation with version 2.0.1, the mandatory criteria for obtaining a certificate is that all applicable Ada 9X basic tests must be passed and either all tests labeled OOP or real time must be passed. For [Ada 95] validations, ACVC test programs for Specialized Needs Annexes will not be processed unless the AVF customer requests that they be processed. The result of these tests will not affect the issuance of a certificate for the core language. Validation by Testing The validation testing process involves interaction between the AVF customer and the Ada certification body. The testing process consists of well defined steps, concluding with the issuance of a validation certificate and a Validation Summary Report (VSR). These steps are: a. Completion of a formal validation agreement between the customer and an AVF for validation services (see Section 5.1). b. Prevalidation is completed. (see Section 5.2). c. Validation testing is performed by an AVF at a mutually agreed upon site (see Section 5.3). d. A Declaration of Conformance is completed and signed by the customer not later than at validation testing. A validation certificate will not be issued until a Declaration of Conformance has been completed (see Section 5.4 and Appendix A). e. A VSR is completed by the AVF to document the validation by testing (see Section 5.5). f. A Validation Certificate is issued by authority of the AJPO for a successfully tested Ada implementation (see Section 5.6). An Ada implementation that fails 1 to 10 ACVC test programs may be provisionally validated for one year. During that 12-month period, the Ada implementation must be re-tested by the AVF and successfully pass the ACVC to obtain a validation certificate. (See Section 5.7.) Registration of a Derived Ada Implementation An Ada implementation that has been derived from an implementation that has been validated by testing (a base implementation) may be given validated status by registration, if certain conditions are true (see Section 6). These conditions are the test for determining whether to register a derived implementation or to proceed with validation by testing. A registration request must be submitted to the AVO in the form provided in Appendix D. If the AVO accepts a registration request, the information will be forwarded to the AJPO so that a derived implementation may be included in the list of validated Ada implementations. Appendices Appendix A contains three versions of the Declaration of Conformance statement. Version A1 is for Ada 83 compilers that are not being incrementally changed to incorporate Ada 95 features. A2 is for Ada 83 compilers that are being incrementally changed but are being validated with ACVC 1.11. A3 is for Ada 95 compilers being validated with an ACVC version based upon Ada 95. Appendix B gives the format for submitting a test dispute and Appendix C describes the dispute resolution process. Appendix D gives the format for a request to register a derived Ada implementation. Appendix E is a graphic display with standard text for the Ada 83 (E1) and for the Ada 95 (E2) product certification mark. Appendix F provides Project Guidelines for the use of baselined Ada compilers. These guidelines are normative for the DoD (United States) and optional for others. Appendix G gives names and addresses for points of contact who participate in the Ada compiler certification system or who provide information about validated products. Appendix H is a list of references used in this document. Editorial Style These written procedures are based on a framework for the validation process that has been refined over 10 years of operational experience. The style is based upon the Institute of Electrical and Electronics Engineering, Inc. (IEEE) editorial practice for technical documents that use terms defined to have a particular meaning. Defined terms are always in bold type whenever they appear in the body of the document. This practice has helped reduce the problems of understanding the meaning of words when translated into more than one written language (e.g. German, French) or a dialect of a written language (e.g., American English, Canadian English, British English). Rules and procedures are written to convey the essential amount of information needed to understand the legitimate rules for participating in the validation process. References listed in Appendix H are generally cited in the text enclosed in brackets, e.g. [Ada 95]. 1. INTRODUCTION The United States Department of Defense (DoD) sponsored the development of the Ada programming language and established the Ada Joint Program Office (AJPO) as part of an effort to support recognized principles of software engineering for a wide range of applications. In view of the well known benefits of standardization, the AJPO has established a certification system to prevent the proliferation of dialects of the Ada programming language and to encourage Ada implementations which conform to [Ada 83] or to [Ada 95]. The Ada certification system rules of procedure and management in this document address the validation of Ada implementations by testing and by registration. These rules form an operational description of a validated Ada compiler that is required by [DoD 91] and by the [FIRMR 87]. This version [4.1] addresses the validation process during the transition from [Ada 83] to [Ada 95]. Where the validation procedures for transition differ from past procedures, these difference are explained in appropriate sections. These procedures will be updated before the end of the transition period, 1 March 1997. The principals of the certification body of the Ada certification system consist of the AJPO for overall direction, the Ada validation organization (AVO) and ACVC maintenance organization (AMO) for technical support, and the Ada validation facilities (AVFs) for performing validations. The Ada certification body operates in conjunction with the U.S. Department of Commerce which has the responsibility for establishing and maintaining a certification system for the Federal Information Programming Standards (FIPS). It is important to note the scope and intent of validation. Users of an Ada implementation are cautioned that the purpose of validation is to encourage conformity of Ada implementations with the standard and that characteristics other than those specified by the standard, such as performance or suitability for a particular application, are outside the scope of Ada validation. Moreover, users are cautioned that the yardstick of conformity testing is the collection of test programs contained in the ACVC. Thus, conformity is measured only within the limitation of these tests. A glossary of terms used in this document is provided in Section 2. Terms defined in the glossary are signified in the body of the document by bold print (in the hardcopy or postscript print-out). Appendices to this document provide examples of documents used in validation, a description of the test dispute and resolution process, project guidelines for use of baselined Ada compilers, current points of contacts, and references. 2. GLOSSARY OF TERMS Ada programming language: The language defined by reference [Ada 83] or by [Ada 95]. Ada 9X basic: Tests derived from ACVC versions based upon [Ada 83] that are or have been made compatible with [Ada 95]. ACVC 2.0: The first [Ada 95] version of the ACVC consisting of Ada 9X basic and new tests for [Ada 95]. ACVC 2.0.1: A version of ACVC 2.0 containing corrected withdrawn test programs. ACVC maintenance organization (AMO): The part of the certification body that maintains the ACVC. ACVC reviewers: A part of the certification body that performs the functions of a Test Method Control Executive Committee along the lines of the NIST Test Method Control Procedures Model for programming languages. ACVC team: The group that produces the ACVC under contract to the AMO. ACVC user's guide: A document that explains the technical details of processing the test programs and evaluating their results. Ada compiler: The software and any needed hardware that have to be added to a given host and target machine to allow transformation of Ada programs into executable form and execution thereof. Ada compiler validation capability (ACVC): The means for testing compliance of Ada implementations, consisting of the test suite, the support programs, and the acvc user's guide. Ada compiler evaluation system (ACES): A test suite designed to assist in the evaluation of Ada compilers and their implementation. The ACES does not test conformity to the Ada programming language but can be used to demonstrate the performance characteristics of an Ada implementation. (See Appendix G for contact information.) Ada implementation: An Ada compiler with its host machine and its target machine. Ada Joint Program Office (AJPO): The part of the certification body that provides policy guidance for the Ada certification system. Ada rapporteur group (ARG): The Ada rapporteur group (ARG) is a subgroup of ISO-IEC/JTC1/SC22/WG9, the International Organization of Standards Working Group for Ada. Members of the ARG are appointed by the convener of the ISO working group for the purpose of resolving issues with respect to the interpretation of the Ada programming language. Ada validation facility (AVF): The part of the certification body that carries out the procedures required to establish the compliance of an Ada implementation. Ada validation organization (AVO): The part of the certification body that provides technical guidance for operation of the Ada certification system. adaptive maintenance: [ANSI/IEEE 90] Maintenance performed to make a software product usable in a changed environment. annexes: The parts of [Ada 95] that follow Section 13. applicable acvc test: A test that is neither inapplicable nor withdrawn. Compare with inapplicable test program and withdrawn test program. base implementation: An Ada implementation that was validated by testing (see Section 5). baseline: [IEEE 90] A specification or product that has been formally reviewed and agreed upon, that thereafter serves as the basis for further software development or maintenance work, that can be changed only through formal change procedures. certification body: [ISO/IEC 86] An impartial body, governmental or non-governmental, possessing the necessary competence and reliability to operate a certification system, and in which the interests of all parties concerned with the functioning of the system are represented. certification mark: A mark which may be used only on products directly associated with the Ada compiler for which the certification mark was awarded. certification system: [ISO/IEC 86] A system having its own rules of procedure and management for carrying out conformity certifications. compliance of an Ada implementation: The ability of the implementation to pass an ACVC version. [Note: For the purposes of this document, compliance is a practical measure of conformity.] computer system: [IEEE 90] A system containing one or more computers and associated software. configuration management: [IEEE 90] A discipline applying technical and administrative direction and surveillance to: identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specific requirements. (See Appendix F). conformity: [ISO/IEC 86] Fulfillment by a product, process or service of all requirements specified. [Note: Also see Section 1.1.2 in the ANSI/MIL-STD-1815A-1983 and Section 1.1.3 of ANSI/ISO/IEC-8652:1995] conformity testing: The process described in Section 5 of this document. core language: The Sections 1-13 and Annexes A, B and J of [Ada 95]. [See Section 1.1.2 of ANSI/ISO/IEC-8652:1995] corrective maintenance: [ANSI/IEEE 90] Maintenance performed specifically to overcome existing faults. customer: An individual or corporate entity who enters into an agreement with an AVF that specifies the terms and conditions for AVF services (of any kind) to be performed. customized test suite: The ACVC tests, adjusted as necessary, that must be used for a given Ada implementation (see Section 4.5). declaration of conformance: A formal statement from a customer assuring that conformity is realized or attainable on the Ada implementation for which validation status is requested. [See Appendix A for the formats of a declaration of conformance for the transition period.] derived implementation: An Ada Implementation that was obtained from a base implementation that has a current validation certificate (see Section 6). fail an acvc version: The Ada implementation fails one or more test of the ACVC version. fast reaction team (FRT): The part of the certification body that provides expertise for the expeditious resolution of test issues. (See Appendix C). host machine: [IEEE 90] (1) A computer used to develop software intended for another computer (contrast with target machine) (2) A computer used to emulate another computer. (3) The computer on which a program or file is installed. (4) In a computer network, a computer that provides processing capabilities to users of the network. inapplicable test: A test that contains one or more test objectives found to be irrelevant for the given Ada implementation. instruction set: [IEEE 90] The complete set of instructions recognized by a given computer or provided by a given programming language. kernel: [IEEE 90] (1) That portion of an operating system that is kept in main memory at all times. (2) A software module that encapsulates an elementary function or functions of a system. operating system: [IEEE 90] A collection of software, firmware, and hardware elements that controls the execution of computer programs and provides such services as computer resource allocation, job control, input/output control, and file management in a computer system. pass an ACVC version: An Ada implementation passes a given ACVC version if it processes each test of the customized test suite in accordance with grading criteria for individual tests and the test result profile matches the passing requirement for a specific ACVC version. perfective maintenance: [ANSI/IEEE 90] Maintenance performed to improve performance or maintainability. prevalidation testing: Processing of an appropriately customized test suite by the customer. project compiler: A validated Ada compiler that is baselined for a project in accordance with applicable configuration management practices. A project compiler may be used for the life of the project (see Appendix F). provisional validation status (PVS): Conferred by the AJPO for tested Ada implementations that fail an ACVC version. No validation certificate is issued.(See Section 5.7.) registration request: A formal request for extension of validated status to a derived implementation. (See Appendix D for the required form). result profile: The result of processing the customized test suite according to given evaluation criteria (see Section 6). software maintenance: [ANSI/IEEE 83] Modification of a software product after delivery to correct faults, to improve performance, or to adapt the product to a changed environment. specialized needs annexes: Annexes C through H of [Ada 95]. These Annexes define standards for additional functionality required by specific application areas. An Ada Implementation may support some or none of these annexes. Conformity to a special needs annex means that each capability required by the annex is provided as specified. target machine: [IEEE 90] (1) The computer on which a program is intended to run. (2) A computer being emulated by another computer. target run-time system: The set of sub-programs that may be invoked by linking, loading, and executing object code generated by an Ada compiler. If these sub-programs use or depend upon the services of an operating system, then the target run-time system includes those portions of that operating system. test issue: Any problem arising during validation (see Section 5.2.3). validation: The process of checking the conformity of an Ada implementation to the Ada programming language and of issuing a certificate for the implementation. validated Ada implementation: An Ada implementation that has been validated successfully either by AVF testing (see Section 5) or by registration (see Section 6). validated Ada compiler: The compiler of a validated Ada implementation. validation certificate (VC): Issued by authority of the AJPO for tested Ada implementations that pass an ACVC version. validation summary report (VSR): A report produced by an AVF containing results that are observed from testing a specific Ada implementation or grouping of Ada implementations. withdrawn test: A test found to be incorrect and not used in conformity testing. A test may be incorrect because it has an invalid test objective, fails to meet its test objective, or contains erroneous or illegal use of the Ada programming language. 3. ORGANIZATION AND RESPONSIBILITIES This section specifies the role of organizations that form the certification body, of customers who receive service from them, and of project managers who use Ada implementations to develop or maintain software. 3.1 Ada Joint Program Office (AJPO) The AJPO provides guidance for the certification system by: a. setting validation standards to be followed by all AVFs; b. establishing the conditions for issuance, the life, and the scope of a validation certificate; c. establishing the schedule for issuing versions of the ACVC; d. approving the release of an ACVC version; e. designating members of the certification body; f. resolving issues that may arise during validation when these issues cannot be resolved through the best efforts of the AVO and AVF; g. maintaining the official list of validated Ada implementations; and h. issuing documents pertaining to validation. 3.2 The Ada Validation Organization (AVO) The AVO provides the technical and administrative support required to operate the certification system by: a. advising the AJPO and AVFs concerning requirements for modification to the validation procedures; b. resolving issues that may arise during the validation process; c. reviewing all validation summary reports (VSRs) prepared by AVFs; d. recommending to the AJPO issuance of a validation certificate for Ada implementations validated by testing (see Section 5) and the registration of derived implementations (see Section 6); e. participating in the ACVC quality control and configurationmanagement process; f. deciding on the withdrawal of test programs from the ACVC version that is being used for validation; and g. convening meetings of the members of the certification body at appropriate intervals to discuss the process and to evaluate practices. 3.3 The ACVC Maintenance Organization (AMO) The AMO and its ACVC team provide the administrative and technical support required to supply the ACVC for use in the operation of the certification system by: a. producing an ACVC version according to a schedule established by the AJPO; b. performing ACVC quality control and configuration management; c. distributing an ACVC version to AVFs and the AVO; d. distributing an ACVC version to the U.S. National Technical Information Service (NTIS), a service of the U.S. Department of Commerce, for further distribution to the public; and e. providing information to the public concerning the test objectives and number of test programs in each version of the test suite, and other information that promotes a public awareness of the test suite and evaluation criteria. 3.4 Ada Validation Facilities (AVFs) An AVF is chartered by the AJPO to conduct validation by: a. adhering to validation procedures approved by the AJPO; b. producing the VSR; c. forwarding unresolved test issues to the AVO for review and analysis, with final resolution to be provided by the AJPO, if necessary; d. providing advice on a customer's registration request for a derived Ada implementation; and e. striving to satisfy national accreditation criteria. The AJPO may issue an AVF charter to an organization that has been recognized as an accredited testing laboratory by the U.S. Department of Commerce, National Institute of Standards and Technology (NIST). The AJPO may issue a charter to an organization located in a country that has a Memorandum of Understanding (MoU) with the U.S. government covering the chartering of AVFs, according to the rules specified in the MoU. An AVF charter may remain in effect indefinitely; however, a charter can be revoked by the AJPO, at any time, for due cause. The AJPO may direct an impartial body to conduct an audit at any time or prior to issuing an AVF charter. Audits are conducted in accordance with procedures established by the AJPO at the time of the audit and are tailored to reflect the purpose of the audit. 3.5 ACVC Reviewers The ACVC reviewers are the group chartered by the AJPO to address quality and configuration management issues of the ACVC test programs by: a. assisting in refining overall testing philosophy and priorities for test coverage; b. providing expert technical review of the test objectives and test programs during their development; c. ensuring that advances in the interpretation of the Ada programming language are reflected appropriately in the test objectives and test programs; d. providing liaison between the certification system and the ISO Working Group for the Ada Standard (i.e. ISO- IEC/JTC1/SC22/WG9); e. reviewing and acting upon comments received from compiler vendors and other interested parties; and, f. establishing the configuration baseline for a proposed change to an ACVC version. Accordingly, the ACVC reviewers cooperate closely with the AVO and the AMO. In particular, the ACVC reviewers monitor the resolution of test program disputes (see Section 5.2.3). 3.6 Fast-Reaction Team (FRT) The fast-reaction team is a small group of Ada language experts who advise the AVO on complex test issues by: a. assisting the AVO to issue a decision to the customer and AVF in a timely manner; and, b. contributing to ISO Working Group 9 language issue resolution. 3.7 Customers Customers are serviced by the Ada certification body in matters concerning Ada validation. In requesting services of the Ada certification body, customers are to provide accurate and complete information to perform validation, to register derived implementations, or to obtain other services. 3.8 Project Manager Project managers are responsible for the acquisition of an Ada compiler that is validated according to the procedures set forth in this document. The Ada compiler selected for software development or for maintenance of existing software may be baselined as a project compiler that will be controlled by the project configuration management process. A project manager may also obtain a copy of the ACVC used to validate the project compiler so as to periodically test it if customizing changes have been made. The project manager may perform the ACVC testing or obtain these services from an AVF. A project manager plays an important role in preventing the proliferation of dialects of the Ada programming language. 4. THE Ada COMPILER VALIDATION CAPABILITY The ACVC is designed to demonstrate the conformity of an Ada implementation with the standard. The ACVC is distributed as a collection of test programs, support programs which facilitate processing the tests, and an ACVC user's guide that explains the criteria for evaluating the results. During the transition to [Ada 95], the last test suite for [Ada 83], ACVC 1.11, and the second version of the test suite for [Ada 95], ACVC 2.0.1, will be in use until 1 March 1997. Compiler vendors may choose to obtain an [Ada 83] validation certificate or an [Ada 95] validation certificate until 1 March 1997 when the third version of the [Ada 95] ACVC (version 2.1) becomes the only test suite for use in Ada validation. 4.1 Versions A new ACVC version is released periodically according to a schedule which is determined by the AJPO. Each new version incorporates changes to the ACVC as determined necessary by the AJPO. These changes are made in order to reflect a revision of the standard, to incorporate ISO WG9 interpretations, or to address implementer or user comments. The test objectives and test programs for each new ACVC version are available for public review and comment before that version is issued for use in validation. Comments on test objectives or test programs should be submitted by e-mail to acvc-comment @sw-eng.falls-church.va.us 4.2 Applicability of ACVC Test Programs Each ACVC test program has one or more test objectives which are described in a comment in the test program. Some test objectives might address language features that are not required to be supported by every Ada implementation (e.g., "check floating-point operations for digits 18"). These test programs generally contain an explicit indication of their applicability and the expected behavior of Ada implementations for which they do not apply. The determination of applicability is made according to the grading criteria in the ACVC or as a ruling by the AVO. All applicable test programs must be processed and passed according to the specified grading criteria. ACVC test programs for a specialized needs annex of [Ada 95] need not be processed unless the AVF customer requests that a specific annex be tested. During the transition period, there are three exceptions to previous criteria for passing an ACVC version. These exceptions are: a. For an [Ada 83] validation, a failure of an ACVC version 1.11 test program may be judged inapplicable when the reason for failure is caused by a correct but partial implementation of [Ada 95]. b. For an [Ada 95] validation with ACVC version 2.0 all applicable Ada 9X basic tests must be passed. c. For an [Ada 95] validation with ACVC version 2.0.1 all applicable Ada 9X basic tests and either all tests labeled as OOP or real-time must be passed. For [Ada 95] validations, ACVC test programs for special needs annexes will not be processed unless the AVF customer requests that they be processed. The result of these tests will not effect the issuance of a certificate for the core language. 4.3 Test Modifications The certification body strives to apply the ACVC as uniformly as practical to all Ada implementations. In order to apply common test objectives that depend on implementation dependent characteristics (e.g., line lengths and numeric types), many test programs must be adjusted to a given implementation following the procedures given in the ACVC user's guide. These adjustments consist of the insertion of implementation dependent values in certain test programs at places prescribed by the ACVC user's guide. In addition to the anticipated test modifications, other changes may be required in order to remove conflicts between a test program and implementation-dependent characteristics (e.g., the algorithm for recovering from syntax errors). The allowable changes for each Ada implementation are determined by an AVF after consultation with the AVO, except in the case of error-recovery problems which an AVF may address without consultation. 4.4 Test Withdrawal In any ACVC version, it is possible that a test program is based on assumptions that need not hold true for all Ada implementations or that a test program does not meet its objective. In these cases, the AVO may issue a correction to the evaluation criteria in the ACVC user's guide or the test program may be withdrawn from that version of the test suite. Any interested party may challenge a test program by sending a rationale for the challenge to the Ada rapporteur group (ARG) (see appendix G for the address). When the ARG has accepted the challenged test for extended resolution, the AVO may withdraw the test. When an AVF customer is preparing for validation, the customer must challenge a test program only through an AVF by asking for a review of its evaluation criteria or for its withdrawal. The form for submitting a challenge is provided in Appendix B. 4.5 Customization A customized test suite is produced by the AVF for each Ada implementation that is a candidate for validation. This customization always consists of removing withdrawn tests and in making required modifications to test and support programs; and, it may include removal of some inapplicable tests as allowed by the ACVC user's guide. 4.6 Availability The ACVC is available to the general public from the NTIS according to U.S. Department of Commerce policies and rules for payment of fees. The ACVC is also available to from an AVF and the Internet. 5. VALIDATION BY TESTING There are six steps that must be completed by a customer and the certification body so that the customer obtains a validation certificate and a VSR. The same ACVC version must be used to complete the steps described in this section. The ACVC version used for validation testing must be current at the time customer testing is started (see 5.2.1). The AJPO announces the in-use and end-use period for each ACVC version. The AVF must begin validation testing the Ada implementation at the customer's site before the current ACVC version expires or else validation with that ACVC version will not be allowed. Anyone intending to obtain a validation certificate should contact an AVF without delay for advice on the handling of the ACVC, on interpretation of the test grading criteria, and on the operational details of that AVF's management practices. 5.1 Step One: Validation Agreement In order to obtain services from the certification body, an interested party must become a customer of an AVF by reaching a formal agreement. This agreement should address the following topics: a. identification of the Ada implementation to be tested and the ACVC version to be used; b. a statement of work, including analysis of prevalidation testing, validation, and preparation of the VSR; c. the format of data to be exchanged; d. a schedule of events and the site of validation; e. financial arrangements; f. retention of records; g. AVF liability; and h. confidentiality of validation information. The schedule for events, deliverables, and payments should take into account the fact that certain steps in the validation process require interaction with other members of the certification body (e.g., AVO or AJPO). The AVF will put forth its best effort to keep confidential a customer's intent to obtain a validation certificate and the projected schedule for validation. This confidentiality will not be allowed to interfere with the normal review procedures of validation. If the customer requests confidentiality for reasons of national security, the customer will provide to the AVF an official statement of the security level that applies to the validation, and the AVF will obtain further guidance from the AJPO. 5.2 Step Two: Prevalidation The requirements of this step are discussed separately so that the customer understands the interaction that is required with an AVF. 5.2.1 Customer Testing After entering into a formal agreement, the customer provides the necessary information for the AVF to prepare a customized test suite. The customer then processes all the tests in this customized test suite using the candidate Ada implementation or another Ada implementation that produces the same result. If the implementation provides for options in the way programs are processed, then the same set of options must be chosen for all test programs, with the possible exception of an option controlling the production of information output. Any other exception constitutes a test issue that must be resolved with the AVF (see Section 5.2.3). Test issues should be sent to the AVF for analysis as soon as they are known. 5.2.2 Submission of Results Upon completion of testing, the customer delivers the complete set of results in the agreed format to the AVF. These results are accompanied by the following information: a. a list of test programs that the customer claims are inapplicable, together with a rationale for inapplicability; b. a list of test programs that are disputed (see Section 4.4) together with rationale (see Appendix B for format); c. an annotated sample command script; d. the complete set of option settings used for processing the customized test suite, including the default settings; and e. complete and current documentation for implementation-dependent characteristics as required in Appendix F of [Ada 83] or Annex M of [Ada 95]. 5.2.3 Test Issues A test issue may be any of the following: a. a missing or incomplete result to a test; b. a result presented in an inadequate form; c. a disagreement between the customer and the AVF as to the interpretation of a result; d. a change in the choice of options to be used during testing; e. a result that makes the Ada implementation fail the ACVC according to the current grading criteria; or f. an implementation-dependent characteristic that may affect the conformity of the Ada implementation. The material submitted by the customer is analyzed by the AVF and test issues resolved. If the AVF and the customer cannot agree on a way to resolve a test issue, the issue will be referred to the AVO for a resolution (see Section 5.2.4). It may be justified to leave a test issue unresolved at prevalidation. For example, it may be impossible to check the processing of control characters by inspecting printed results. The AVF will note these unresolved issues and describe the results that will be expected during validation testing. It is also possible that the customer information provided for production of the customized test suite (see Section 5.2.1) was insufficient so that corrections to the customized test suite must be made and additional processing will be required. 5.2.4 Test Issue Resolution A customer may challenge the applicability or correctness of any particular ACVC test program. Such challenges should be presented to the AVF in the test-dispute format (see Appendix B). The AVF will forward challenges to the AVO for resolution; the AVO will strive to rule on the challenge within two weeks of receiving it. The AVO will forward all challenges and rulings to the ACVC reviewers, the AMO and the ACVC team. (See Appendix C for a description of the Test Dispute and Resolution Process.) 5.2.5 Incomplete Prevalidations The AVF and the customer may agree that, at the customer's risk, parts of the customized test suite need not be processed. The customer must certify that the results from a previous prevalidation submitted to the AVF or validation results obtained by the AVF are identical to those that would have been obtained by the customer. The normal practice is to submit complete prevalidation results. 5.2.6 Successful Prevalidation Prevalidation testing is successful if the analysis of results and the resolution of test issues show that the candidate Ada implementation passes the customized test suite. Prevalidation is successful with caveats if the results are satisfactory except that they were incomplete or if resolution of some test issues are deferred until validation testing by agreement between the AVF and the customer. 5.3 Step Three: Validation Testing Upon successful completion of prevalidation, with or without caveats, the AVF witnesses testing of the Ada implementation at the site and time mutually agreed by the AVF and customer. The AVF prepares a customized test suite based upon customer information and any information collected during the resolution of test issues. The customized test suite is installed and processed under AVF supervision. If the AVF determines that the results agree with those obtained from prevalidation and are satisfactory with respect to the caveats, the testing has been successful; otherwise, re-testing will be required, unless the validation attempt is discontinued. (See Section 5.7 for Provisional Validation.) 5.4 Step Four: Declaration of Conformance At some time during the validation but not later than at validation testing, the customer will complete and sign a declaration of conformance. The declaration states that the organization that is responsible for the production, maintenance or distribution of the Ada compiler is offering a product that is in conformity with the Ada programming language. The declaration becomes part of the AVF records and is copied into the VSR. A validation certificate will not be issued unless a signed declaration of conformance has been provided to the AVO. (See Appendix A, for examples of the declaration of conformance that may be used through the transition period.) 5.5 Step Five: Validation Summary Report A VSR is produced for each validation testing effort. A single VSR may cover validation testing of several Ada implementations, provided that they all have the same result profile. The VSR provides the following documentation pertaining to the validation effort: a. identification of the customer responsible for validation of the Ada implementation; b. identification of the organization responsible for the production, maintenance, or distribution of the Ada compiler or Ada implementation; c. identification of the Ada implementation tested; d. options provided by the Ada compiler and identity of the options used for testing; e. the inapplicable test programs and implementation-dependent characteristics exhibited by the test programs that established inapplicability; f. the implementation-dependent characteristics pertinent to the customized test suite; g. description of implementation-dependent characteristics as detailed by Appendix F of [Ada 83]. Implementation-dependent characteristics detailed by Annex M of [Ada 95] should be provided in vendor documentation for users but need not be provided in a VSR in the same format and detail called for in Annex M ; h. withdrawn test programs; i. modifications to test programs with an explanation for such modifications; and j. a description of failed test programs, if applicable. (See Provisional Validation, Section 5.7.) 5.5.1 VSR Production The VSR is prepared by the AVF but includes material that is produced by the customer, such as Appendix F required by [Ada 83] and applicable implementation-dependent characteristics selected from Annex M in [Ada 95]. A draft of the VSR is sent to the AVO for approval before or after validation testing. The final version of the VSR is signed by the AVF, the AVO, and the AJPO. 5.5.2 VSR Availability The final version of the VSR is available to the general public from NTIS, from the customer and from the AVF that produced it. The AVF may require payment of a fee for VSR reproduction and mailing costs. 5.6 Step Six: Validation Certificate For each successful validation, one certificate is issued by the AJPO. During the transition period, a page will be attached to each [Ada 95] certificate that provides a numerical summary of results for each category of tests. The information on the validation certificate describes the tested Ada implementation: the source of this information is the signed declaration of conformance that describes the tested Ada implementation that the AVF provides to the AVO after completion of testing. The customer will ensure that the information contained on the certificate does not infringe on the rights of third parties and may be required to provide a written statement of consent from any third party involved. Validation certificates for ACVC 1.11, for ACVC 2.0 and ACVC 2.0.1 will expire 31 March 1998. An entry in the list of Ada implementations validated by testing will be made for each certificate issued. This entry will be removed when the certificate expires. 5.7 Provisional Validation When prevalidation testing has been unsuccessful, a customer may petition the AVO to continue with validation by identifying up to ten ACVC test program failures to dispute on non-technical grounds. The AVO will present the dispute to the FRT as a petition for validation with test failures. The test program(s) will be processed during validation testing and the compiler behavior will be documented in a VSR that is registered with the AJPO as documentation of a non-conforming Ada implementation. Provisional validation status (PVS) will be award for that Ada implementation for a period not to exceed 12 months after the date validation testing was completed. The customer will arrange with the AVF to witness the result of error correction in the base implementation. If the compiler then passes the previously failed tests, a VC will be issued and the VSR will be revised. The list of PVS Ada implementations will be maintained by the AJPO as public information. VSRs for Ada implementations awarded PVS will be plainly marked as being registered with the AJPO. These VSRs will be reviewed by the AVO and will be publicly available from the AVF, AVO and AJPO. The AVO may deny repeated petitions from the same customer for provisional validation of an Ada implementation. 5.8 Advertising Validated Status The customer will not advertise or make public claims that the Ada implementation is validated until after receiving a validation certificate or after receiving formal notification from the AVF that the AJPO has issued a validation certificate. A waiver of confidentiality must be signed by a customer who intends to advertise the completion of events that indicate progress toward completion of validation. If a waiver of confidentiality has been signed with the AVF, the AVF will respond to inquiries about the customer's advertisements or public claims by acknowledging receipt of validation materials (i.e., a formal agreement, pre-validation results, or validation testing results) without judgement concerning the success of the validation. 5.9 Certification Mark The certification mark (see Appendix E for reproduction) may only be used on products directly associated with validated Ada compilers, such as disks, tapes, packaging, advertising, reference manuals and any other associated documentation where a significant portion relates to a validated Ada compiler. This unique mark distinguishes compilers validated in accordance with the rules in this document. The certification mark can be reproduced in any size, color, or combination of colors. 6. REGISTRATION OF DERIVED IMPLEMENTATIONS It is expected that nearly every Ada implementation that is validated by testing is intended to be representative of a class of implementations that are similar in their defining components of Ada compiler, host machine, and target machine. Such implementations are known as derived implementations. Whereas a validation certificate identifies only the Ada implementation that was tested by the AVF (i.e., the base implementation), the validated status that it conveys is extensible to a range of derived implementations for the life of the certificate. The procedure for extending validated status to a derived implementation is called registration. This section prescribes the requirements for registration. For any base implementation with a current validation certificate, a derived implementation may obtain validated status by registration when it satisfies all of the following conditions: a. The Ada compiler was obtained from the Ada compiler of the base implementation by changes that are within the scope of accepted software maintenance practices. b. The target machine of the base and derived implementation have compatible instruction sets and operating system or kernel. c. The result profile for the derived implementation is either the same as the base implementation or, if there are differences, these differences are justified as being within the scope of accepted software maintenance practices. The changes that may be made to an Ada compiler for the purpose of derivation will be within the scope of software maintenance as applied to the domain of compiler construction. Changes must be classified as corrective, adaptive, or perfective. Examples of these changes are the correction of a compiler error, the adaption to an operating system upgrade, the transfer of the compiler to another host machine, the addition of a floating point processor to a small target machine, or the perfection of a garbage collection algorithm. For the purposes of obtaining validated status by registration, the changes required to render the base Ada compiler fully functional on a different host machine or host operating system is considered adaptive maintenance. These procedures impose no restrictions on the host machine of a derived implementation. Experience shows that re-hosting Ada compilers does not present significant difficulty in maintaining implementation conformity. However, a change in a target machine may be ouside the scope of acceptable maintenace. For a target machine, common examples of compatible instruction sets and operating systems are two different computer system models in a manufacturer's product line or the computer systems produced by different manufacturers that use the same instruction set and operating systems, or any computer system and a simulated or emulated computer system that are the same instruction set. 6.1 Registration Request Any interested party may initiate the registration of a derived implementation by sending a registration request to the AVO or to an AVF. A registration request consists of two parts - a "letter" part and a "supporting information" part. Appendix D specifies both the form and content of the information to be provided in these two parts. A registration request must contain descriptions of the base and derived implementations, a declaration of conformity to the relevant version of the Ada programming language, compliance with the relevant version of the ACVC, a description of what maintenance and testing was performed, and any appropriate supporting documentation from hardware or software vendors. If any party in addition to the one submitting the request has a legal interest in the derived implementation, this interest will also be documented in the request. Two Ada implementations have the same result profile when: a. they use the same customized test suite; and, b. their output from processing the customized test suite is the same (with the possible exception of the precise content and position of error messages from the compiler). If the derived implementation has a different result profile than the base implementation, the registration request must describe and justify the differences. 6.2 Evaluation The AVF will not perform testing on derived implementations. Any AVF may review the registration request for completeness and the plausibility of information and provide advice. The AVO will evaluate all registration requests before recommending that the derived implementations be added to the list of validated Ada implementations maintained by the AJPO. 6.3 Registration Registration requests that are acceptable to the AVO will be registered as validated Ada implementations untested by an AVF. A validation certificate will not be issued for these derived implementations but the customer may use the certification mark awarded to the base implementation. The list of derived implementations and information provided in the registration request will be available to the public. 6.4 Expiration of Validated Status A derived implementation loses its status of being validated if it is challenged successfully (see Section 6.7), upon expiration of the validation certificate of its base implementation, when registration is revoked by the customer, or the derived implementation is no longer supported by the supplier. 6.5 Challenges Any derived implementation may be challenged by any interested party through an AVF. The challenger will pay a challenge fee to the AVF and will submit a challenge request that: a. identifies the derived implementation being challenged; b. names one ACVC test from the customized test suite together with its implementation dependent parameters, if any; and c. describes in what way the implementation will fail this test. The AVF will send this challenge to the originator of the registration asking for comment. The challenge will be considered settled if the registration is revoked by the registration originator; otherwise, the challenge will be settled as outlined in Section 6.7. 6.6 Challenge Mark The AVF will inform the AVO that a challenge for a given derived implementation has been received. The derived implementation will then be marked as "challenged" on the list of derived implementations maintained by the AJPO. Information pertaining to the challenge may be requested by any interested party and received from the AVF. It should be noted that a challenge mark applies only to the derived implementation that was named and does not indicate any judgement about the conformity of the challenged implementation. 6.7 Challenge Test The AVF will conclude a formal agreement with the challenger that covers the AVF's cost for performing a challenge test. For challenge testing, the challenged derived implementation will be tested against the named ACVC test. The challenger will provide access to the challenged derived implementation and appropriate expertise to facilitate the AVF test. The AVO will be informed of the test result. Depending on its result, the AVO will settle the challenge by either removing the challenge mark or the derived implementation from the list of validated Ada compilers. APPENDIX A. DECLARATION OF CONFORMANCE A1 -- Ada 83 Declaration of Conformance Customer: Certificate Awardee: Ada Validation Facility: ACVC Version: Ada Implementation Ada Compiler Name and Version: Host Computer System: Target Computer System: Declaration: I the undersigned, declare that I have no knowledge of deliberate deviations from the Ada Language Standard (ANSI/MIL-STD- 1815A, ISO 8652-1987, FIPS 119) as tested in this validation and documented in the Validation Summary Report. Customer Signature / Date Company Title Certificate Awardee Signature / Date Company Title Note: If the Customer and the Certificate Awardee are the same, only the customer signature is needed. A2 - Ada 83 Transitioning Declaration of Conformance Customer: Certificate Awardee: Ada Validation Facility: ACVC Version: Ada Implementation Ada Compiler Name and Version: Host Computer System: Target Computer System: Declaration: I the undersigned, declare that I have no knowledge of deliberate deviations from the Ada Language Standard (ANSI/MIL-STD- 1815A, ISO 8652-1987, FIPS 119) other than modifications necessary to incorporate some features of (ANSI/ISO/IEC 8652:1995) untested by this validation. Any non- conforming characteristics are due to implementation errors. Customer Signature / Date Company Title Certificate Awardee Signature / Date Company Title Note: If the Customer and the Certificate Awardee are the same, only the customer signature is needed. A3 -- Ada 95 Declaration of Conformance Customer: Certificate Awardee: Ada Validation Facility: ACVC Version: Ada Implementation Ada Compiler Name and Version: Host Computer System: Target Computer System: Declaration: I the undersigned, declare that I have no knowledge of deliberate deviations from the Ada Language Standard (ANSI/ISO/IEC 8652:1995, FIPS 119-1) other than the omission of features as documented in the Validation Summary Report. Customer Signature / Date Company Title Certificate Awardee Signature / Date Company Title Note: If the Customer and the Certificate Awardee are the same, only the customer signature is needed. APPENDIX B. IMPLEMENTER DISPUTE FORMAT [part A] Implementer: Configuration: ACVC Version: Pre-Validation Submittal Date: [Part A will be completed once by each implementer; part B will be completed for each dispute. It is not necessary for a pre-validation date to have been established. Part A information is treated as confidential.] [part B] Reference: Summary: Discussion: [In this Discussion, arguments should be specified using test line #s and references to pertinent sections of the Ada standard or Commentaries (AI-xxxx).The implementer must describe the behavior of the implementation for the test or tests that are disputed, stating the particular test messages that are produced. It is sufficient for the detailed description to be limited to the particular segment of test code that is disputed. Relevant source code with compiler messages should be included. (For a group of tests that cause much the same behavior, it is sufficient for a detailed description to be given for one of them, with the relevant line numbers given for the like problems in the related tests.) If the argument depends upon implementation constraints of hardware or software (e.g., characteristics of the operating system), then these should be specified; the particular computer and operating system should be identified. It is especially important that implementations that fail to pass some test due to capacity limitations be described in enough detail for the AVO to assess the reasonableness of these limitations. Failure to fully specify the points pertinent to a dispute might result in an adverse decision being made, with the disputer having to further argue the case with a second submittal to the AVO. Yet it is possible that the Summary will suffice to adequately present a dispute. APPENDIX C. TEST DISPUTE AND RESOLUTION PROCESS C.1 Introduction A "dispute" is defined by the ACVC User's Guide as any result from processing an ACVC test program that is not a passed or inapplicable result according to the established grading criteria. This intentionally broad definition of a "dispute" is to make certain that compiler implementers bring all deviant test results to the attention of the AVF, without assuming that such results would be accepted without special review. The compiler implementer also provides a rationale for each challenge being made to a particular test program. Disputes are forwarded to the AVO, usually electronically, by the AVF on behalf of their validation customer. For each dispute that is accepted (i.e., when the AVO rules in favor of the dispute), it is likely that some correction is indicated for the disputed tests. The AVO withdraws any test that is found to be incorrect to a degree that makes it unsuitable for validation. The withdrawal of a test consists of including it on a list of tests that are ignored for validations conducted with the current ACVC version. The AVO updates the list of withdrawn tests, and distributes this list to the AVFs, the AMO, ACVC Reviewers, and ACVC Team. The AVO also maintains a database consisting of all test disputes and their resolution (AVO rulings) which is periodically provided to the ACVC Reviewers, AMO, ACVC Team, and the FRT. The AVO may also recommend that certain resolved disputes be considered further by the ARG/URG, even though the Chairs of both the ARG and URG currently participate in deliberations of the FRT. C.2 Expedited Resolution Process The AVO resolves disputes by any of three methods: a resolution that was made previously is applied to the current dispute (e.g., the same dispute might be submitted at different times by different petitioners); the resolution can be determined unequivocally based on the Ada standard or Ada Commentaries; or, the resolution is determined based on the deliberations of the FRT. Although the Ada Compiler Validation Procedures do not set a limit on the length of time for reaching a resolution, the AVO attempts to resolve disputes within two weeks, an informal guideline that was established by the certification body. The AVO also attempts to place priority on resolving disputes for AVF customers who have a firmly scheduled date for validation testing. Implementers should submit disputes well in advance of a scheduled validation testing date. (see Section 5) On receipt of a dispute, the AVO checks whether the issue matches any that had been previously resolved. If the dispute is new, it is given an initial AVO analysis which involves research using the Ada Commentaries in conjunction with the Ada standard and references to previous dispute deliberations. A dispute is referred to the FRT when questions of interpretation arise and a resolution is not obvious. The AVO presents the dispute and any additional information resulting from an initial analysis to the FRT by e-mail without disclosing the identity of the petitioner. Deliberation of the dispute proceeds with the exchange of each expert's opinion and analysis. The AVO participates in the deliberation by providing information as requested (e.g., ACVC tests or information from the petitioner), eliciting discussion from the experts, and making or challenging technical points raised in the discussion. In general, where an issue receives support from some members of the FRT, the dispute is accepted. There is no prescribed formality to the FRT deliberations, such as voting procedures or time limits on deliberation. The AVO might extend deliberation when a basis for resolving the dispute has not been made. However, the AVO will give its ruling on the dispute when a sufficient basis has been established, regardless of whether the FRT discussion continues. C.3 Types of Resolutions The resolution of a dispute is either an acceptance or rejection of the petitioner's arguments. Acceptance can result in either withdrawal of the test program from the ACVC or in a "Test, Processing, or Evaluation" modification for validation. A dispute may be rejected if it conflicts with the Ada standard or Ada Commentaries or if it is a test to which previously validated implementations conform and the petitioner has not provided compelling reasons for a deviation. A dispute may lead to the withdrawal of a test program if the test is shown to be incorrect to a degree that wrongly influences implementation. Withdrawn tests have no effect on validation (they are generally not processed). If the dispute shows the affected test program(s) to be incorrect in only a minor, limited degree, generally the AVO will direct that the test(s) be processed with a test modification. There are three types of test modification: Test, Processing, and Evaluation modifications. A Test Modification is an actual change to the code of the test (e.g., adding a choice to an exception handler). A Processing Modification is a change to the way in which the test is processed (e.g., re-ordering the compilation of component files of a multi-compilation test). And an Evaluation Modification is simply the grading of the observed results by other than the established grading criteria (e.g., interpreting particular intermediate output and a final "failed" result as "passed", according to an understanding of the dispute). All test modifications are documented in the VSR. C.4 Summary There is no limit on the number of disputed tests that can be submitted by an implementer. Although there is a risk that a dispute will not be decided in the implementer's favor, that risk can be managed so as to not affect validation by early submission of disputes. The validity of grading criteria for ACVC tests was very important for Ada 83, and will take on even greater importance for Ada 95 usage based tests. Any interested party may dispute an ACVC test and its grading criteria. APPENDIX D. REGISTRATION REQUEST A registration request must have two main parts: (1) a letter and (2) an enclosure consisting of specific information to support the request. PART 1 - LETTER The request letter should be addressed to the Ada Validation Organization (AVO) (note: see Appendix G for address) even though the requester does seek advice from an AVF. The mandatory parts of the letter are: a. Certificate number and description of the Base Implementation, b. List of derived implementations that were fully tested with the customized ACVC used in the base validation. If this list is less than the list for which registration is requested, item 4 must also be included in this letter. c. Select the appropriate statement to declare conformity: a. Ada 83 compiler: I the undersigned, declare that I have no knowledge of deliberate deviations from the Ada Language Standard [ANSI/MIL_STD-1815A/ISO 8652:1987/FIPS119] in the implementation. b. Ada 83 migrating compiler: I the undersigned, declare that I have no knowledge of deliberate deviations from the Ada Language Standard [ANSI/MIL-STD- 1815A/ISO 8652:1987/ FIPS 119] other than modifications necessary to incorporate some features of ANSI/ISO/IEC 8652:1995 untested by a validation. Any non- conforming characteristics are due to implementation errors. c. ISO 95 compiler: I the undersigned, declare that I have no knowledge of deliberate deviations from the Ada Language Standard [ANSI/ISO/IEC 8652:1995, FIPS 119-1] other than the omission of features documented in the Validation Summary Report for the base implementation. d. For the derived implementations that were not fully tested, we declare that, to the best of our knowledge, these implementations will pass the customized ACVC used in the base validation. Any possibly observed deviations are the consequence of implementation errors. e. If this statement applies, legal interests of (name) are affected by this registration request and their consent is documented in enclosure (number) to this letter. f. The Ada implementations for which registration is requested may, in fact, be derived according to the rules given in the Ada Validation Procedures. g. Signature h. Corporate Title PART 2 - TEMPLATE FOR SUPPORTING INFORMATION a. Identification of Base Ada Implementation: a. Certificate number:_________________ b. Registration requested for: Note: This registration request pertains to all Ada implementations obtained by taking the Ada compiler and selecting any host and any target from the list below. Compiler: [name and version(s)/release number/product(s) range of versions/releases] Host: choose either (a) or (b) a. [make, specific members of series and/or model numbers] (e.g. IBM PS2 model 80-110 & 113) or, b. [all members of a model series] (e.g. IBM PS2 all models, or HP 9000 series 300) Operating System: [name and version/release number(s)] Target: choose (a), (b), or (c) a. [make, specific members of series and/or model numbers] or b. [all members of a model series] or c. [members of an instruction set architecture and board implementation(s)] (e.g. Motorola 68020-MVME133-1 & MVME130COF, Motorola 68030-MVME147, Motorola 68030-MilSpec285) Operating System: choose (a), (b) or (c) a. [OS name and version/release number for all target machines listed] or b. [kernel identifier for all target machines listed] or c. [none for all target machines listed] PART 3 - EVIDENCE THAT THE ADA IMPLEMENTATIONS LISTED ABOVE SHOULD BE VALIDATED BY REGISTRATION: a. State the type of software maintenance changes made to the compiler and describe what was done. - corrective - maintenance performed specifically to overcome an existing fault. - perfective - maintenance performed to improve performance or maintainability. - adaptive - maintenance performed to make a software product usable in a changed environment. b. Which host/target combinations were fully tested with the ACVC? Testing with "customer applications" or other types of tests is not equivalent to running the ACVC. If a sub-set of ACVC tests were run, so state and indicate the sub-set. c. State whether the result profile is the same or different from the base. If different, list all ACVC test results that are different or exhibit different behavior and explain the difference. d. What is the authoritative source used to determine the technical compatibility between the derived implementations and the base? If copies of technical manuals are supplied as the source of this information, give the enclosure number where this information can be found. Do not duplicate a previous submission of identical technical material, just reference its previous submission. If there has been a change in the previously submitted material, submit only the change. If there is no authoritative source of information given, this registration request may be refused. APPENDIX E. (CERTIFICATION MARK FOR Ada 83 PRODUCTS AND ISO 95 PRODUCTS) APPENDIX F. PROJECT GUIDELINES F.1 Background The general validation procedures given in the preceding sections of this document describe the process of establishing the conformity of an Ada implementation before it is offered for sale in the market place. In the U.S., Federal Information Resources Management Regulations [FIRMRs] require that a compiler must have validated status when it becomes part of the Federal Government's software inventory. The FIRMRs also require that application software supplied to the Federal Government for government maintenance will be the product of a validated compiler. In other countries, there may be established government or commercial requirements that are similar to the FIRMRs in intent. Since validated status is limited to the life of a particular validation certificate, suppliers of compilers or software must periodically complete the steps in the validation process so as to have a current validation certificate for Ada implementations. However, there are practical project-level difficulties associated with maintaining an always current validated status for all compilers being used to develop or maintain software. This section provides guidelines for project managers that are consistent with the goal of preventing the proliferation of dialects of the Ada programming language. These guidelines should be tailored by organizations that manage software projects according to their quality assurance policies and configuration management procedures. These guidelines are normative for the U.S. DoD and optional for others. F.2 Acquisition Requirements A project manager will identify the requirement for a validated Ada compiler as an action within the context of project milestones. This requirement will be met when a validated Ada implementation has been delivered for the project. If the Ada implementation selected for the project is not validated, the project manager is responsible for requesting that validated status will be obtained in accordance with the general rules of procedure for validation by testing or registration as early as possible in the software development process. F.3 Baselined Project Compiler The project manager should determine whether the Ada compiler(s) used on the project will be upgraded or replaced by the supplier on a validation schedule which is independent of project milestones; or, whether the validation certificate will be allowed to expire before these milestones are reached. The project manager will baseline the validated compiler and the ACVC at a given version when it is more cost-effective for the project to forego replacements of the compiler with later validated versions. If the compiler being used on the project does not have validated status (e.g., the version was derived but not registered by the vendor), it must have validated status, by testing or registration, to qualify as a project compiler. When the validated compiler has been baselined, it then becomes a project compiler for the lifetime of that project, or until the project manager establishes a requirement for validation prior to reaching a particular project milestone. When a validated Ada compiler has been baselined for a project, configuration management procedures must be established to ensure complete documentation of any derivations from it or any deviations from ACVC test results obtained previously from the baselined compiler. See Section 6 for a discussion of rules for determining if changes to the Ada implementation are within the scope of generally accepted software maintenance practices. Changes can be made to a baselined compiler but these changes may constitute a new baseline for configuration management purposes. The project manager (government or contractor) is responsible for determining when a new baseline as been constructed. F.4 Project Registration The project manager should notify the AJPO when a validated Ada implementation has been selected for a particular project as a baselined version that will be used for an extended period or for the life of the project. Notification may take the form of the registration request generally used by compiler vendors with the addition of information that identifies the project. (See Appendix D for the registration request form and Section 6 for general rules.) Experience has shown that formalizing the declaration of a project compiler is useful for a project manager, as well as for the AJPO, when disputes arise concerning the validated status of compilers that have been used on a project. F.5 Embedded Applications Some software development projects include applications that execute on a target machine that is embedded in a larger system. A compiler used to generate object code for the embedded target machine is considered to be a validated project compiler only if all of the following conditions are satisfied: a. The project compiler was a validated Ada compiler or was derived from a validated Ada compiler. b. All mandatory features of the Ada programming language that can be supported, or are emulated on the embedded target machine, are supported by the compiler for the target. That is, compilers for the restricted target shall not be arbitrarily constrained to subset implementations of the Ada programming language. During software development of embedded applications, project managers must ensure that all run-time systems used to generate code are managed as configuration items consisting of libraries and documentation for all versions that will be delivered for operational use. All limitations of the restricted target should be documented in Appendix F for the Ada implementation. F.6 Re-testing The project manager should ensure that the project compiler retains a known degree of compliance with the standard throughout the remaining life of the project by periodically re-testing the project compiler using the ACVC version used to originally establish the conformity of the base implementation or a derived implementation. The project manager should determine whether this testing will be done by project personnel or by an AVF, so as to obtain a VSR that will document the result of re-testing. The services of an AVF are desirable when project personnel are unfamiliar with the testing procedure and interpretation of test results. (A VC will not be issued for a project compiler unless it passes a current ACVC version and validation is conducted as described in Section 5.) When a software release has been produced on several project compilers, ACVC re-testing requirements apply to each of these compilers. Records of compiler maintenance changes and the ACVC result profile will be maintained for each project compiler. These records will be current and available for inspection by the government. Examination of such records should be a routine action for audit teams. F.7 Ada 83 to 95 Transition The last [Ada 83] ACVC version that can be baselined with a project compiler is ACVC version 1.11. Project managers who decide to baseline a validated Ada compiler for [Ada 83] must do so before expiration of the validation certificates issued for ACVC 1.11 validations. APPENDIX G. POINTS OF CONTACT Ada Validation Facility Managers -Dr. William Dashiell National Institute of Standards and Technology National Computer Systems Laboratory Building 225, Room A266 Gaithersburg, MD 20899 phone: 301-975-2490 net: nist-avf@sw-eng.falls-church.va.us -Mr.Brian Andrews 88 Communication Group Wright-Patterson Air Force Base Ohio 45433-6503 phone: 513-255-4472 net: wp-avf@msrc.wpafb.af.mil -Mr. Jon Leigh The National Computing Centre, Ltd. Oxford Road Manchester England, M17ED phone: +44-61-228-6333 net: uk-avf@sw-eng.falls-church.va.us -Mr. Alphonse Philippe AFNOR Tour Europe, cedex 7 F-92080 Paris la Defence France phone: +33-1-42-91-55-55 or 57-96 net: afnor@sw-eng.falls-church.va.us -Mr. Michael Tonndorf IABG, Software-Technologie ITE Einsteinstr. 20 D-85521 Ottobrunn Germany phone: +49-89-6088-2477 net: tonndorf@sw-eng.falls-church.va.us (EUnet:tonndorf@ite.iabg.de) ACVC Maintenance Organization (Ada 95 development) -Mr.Brian Andrews 88 Communication Group Wright-Patterson Air Force Base Ohio 45433-6503 phone: 513-255-4472 net: wp-avf@msrc.wpafb.af.mil ACVC Team (Ada 95) -Mr. Mike Middlemas SAIC 10770 WaterRidge Circle MS 213 San Diego, CA 92121 phone: 619-552-5326 net: middlema@sw-eng.falls-church.va.us ACVC Reviewers (Ada 9X) -Dr. Nelson Weiderman (Chair) 127 Schooner Dr. Wakefield, RI 02789 phone: 401-783-6863 net: nhw@sei.cmu.edu Ada Rapporteur Group (ISOWG9) -Dr. Erhard Ploedereder (Chair) net: ploede@grimm.informatik.uni-stuttgart.de net: comp.lang.ada@sw-eng.falls-church.va.us Ada Joint Program Office (AJPO) -Dr. Charles Engle DISA/Center for Software 5600 Columbia Pike Falls Church, VA 22041 phone: 703-681-2107 net:engle1c @ncr.disa.mil Ada Validation Organization -Ms. Audrey A. Hook Institute for Defense Analyses 1801 N. Beauregard St. Alexandria, Va. 22311 phone: 703-845-6639 net: hook@ida.org Fast Reaction Team -Mr. Dan Lehman (test challenges) Institute for Defense Analyses 1801 N. Beauregard St. Alexandria, Va. 22311 phone: 703-845-6633 net: avo@sw-eng.falls-church.va.us Ada Information Clearinghouse (for the AJPO) -Ms. Susan Carlson (industry and government inquiries) Ada Information Clearinghouse IIT Research Institute 4600 Forbes Blvd. Lanham, Md. 20706-4312 phone: 703-681-2464 net: adainfo@sw-eng.falls-church.va.us URL http://sw-eng.falls-church.va.us/AdaIC/ ACVC and VSR Distribution -National Technical Information Service U.S. Department of Commerce 5285 Port Royal Road Springfield, Va. 22161 Telephone: 703-487-4650 Validated Compiler Lists -AJPO - Official Ada lists, updated monthly. Ada Information Clearinghouse (for the AJPO) IIT Research Institute 4600 Forbes Boulevard Lanham, MD 20706-4312 phone: 703-681-2466 net: adainfo@sw-eng.falls-church.va.us ACES is available from: -Data Analysis Center for Software P.O. Box 120 Utica, NY 13503 phone: 315-734-3696 via anonymous FTP to sw-eng.falls-church.va.us, directory is /public/AdaIC/testing/aces via WWW: URL http://sw-eng.falls-church.va.us/AdaIC. APPENDIX H. REFERENCES [Ada 95] ANSI/ISO/IEC 8652:1995 Reference Manual for the Ada Programming Language, February 1995 supercedes Ada 83] [Ada 83] American National Standards Institute and United States Department of Defense: ANSI/MIL-STD-1815A Reference Manual for The Ada Programming Language, 1983 Note: This standard is identical with ISO/8652-1987 and FIPS 119, 1985. [ANSI/IEEE 90] American National Standards Institute and Institute of Electrical and Electronics Engineers, Inc., Standard 610.12-1990; "ANSI/IEEE Standard Glossary of Software Engineering Terminology". [DoD 91] Department of Defense Instruction: Number 5000.2, January 1, 1991 Subject: Defense Acquisition Management Policies and Procedures [FIPS] Federal Information Processing Standard publications relative to the Ada Programming Language are FIPS 119 and FIPS 119-1 [FIRMR 90] General Services Administration, Federal Information Resources Management Regulations (FIRMR) 201.20.303 and 201.39.1002, 28 December 1990, Subject: Federal Standards; and, Associated Federal ADP and Telecommunications Standards Index (updated semi-annually), Subject: Federal Information Processing Standards; Implementation, Validation and Conformance. [ISO 74] International Standards Organization: ISO 2382/I-1974 Data Processing - Vocabulary - Section 01: Fundamental Terms. [ISO/IEC 91] International Standards Organization: ISO/IEC, Guide 2, 6th edition 1991 - General Terms and Their Definitions Concerning Standardization and Related Activities.