1 INTRODUCTION

1.1 DOCUMENT IDENTIFICATION

This is the Usability Enhancements Document for the Ada Compiler Evaluation System (ACES), Version 2.1. It was produced by CTA INCORPORATED as CDRL A0014, Task 15 (ACES Version 2.1 Development), contract number F33600-92-D090125. This contract is directed by the Business Software Branch (88 CG/SCTL), United States Air Force, Wright-Patterson AFB, Ohio.

1.2 PURPOSE

This document states the requirements for the revision of the Support Software Computer Software Configuration Item (CSCI) of the ACES. The principal changes anticipated in this CSCI are the provision of default processing and reporting options, the capability to select predefined sets of performance tests, and the capability to add user-defined benchmarks to the testing and analysis process.

1.3 OVERVIEW

This document is divided into 9 sections, describing the system, the CSCI, relevant documents, and the requirements. The following list presents a brief description of the contents of each section.

* Section 1 Introduction This is the current section, which identifies and describes the purpose and structure of the document.

* Section 2 Applicable Documents This section lists documents that are referenced in or otherwise relevant to the Enhancements document.

* Section 3 System Overview This section gives a high-level description of the purpose and structure of the Ada Compiler Evaluation System.

* Section 4 CSCI Overview This section gives a summary description of the purpose and structure of the Support Software CSCI of the ACES.

* Section 5 Requirements for Processing New Performance Tests This section states the requirements for incorporating new performance tests into the test control and analysis software.

* Section 6 Requirements for Default Choices This section states the requirements for providing default choices for selecting, processing, and analyzing the results of performance tests.

* Section 7 User-Defined Benchmark Inclusion This section states the requirements for enabling the user to process and analyze the results of performance tests that are not part of the ACES.

* Section 8 Other Changes This section states the requirements for changes to the Support Software CSCI that are not covered in the previous sections.

* Section 9 Abbreviations and Acronyms This section defines abbreviations and acronyms that are used in this and other ACES documents.

In the Requirements sections (Sections 5 through 8), each requirement has an associated 3-part identifier of the form "SPT-x-dddd". In these identifiers, "SPT" indicates that the requirement is for the Support Software CSCI. (Requirements for the Operational Support CSCI use "OPS" similarly.) The "x" component is replaced by "P" if the requirement is for the Pretest CSC, "H" for the Harness CSC, "A" for the Analysis CSC, "Q" for the Quick-Look CSC, or "D" for documentation. The "dddd" component is a numerical encoding of the section, subsection, and paragraph number where the requirement appears in this document. Thus, for example, "SPT-H-5103" identifies a Support Software CSCI requirement, allocated to the Harness CSC, appearing in this document as paragraph 5.1.3.

2 APPLICABLE DOCUMENTS

The following Government and non-Government documents are referenced or are relevant to this Objectives Document.

2.1 GOVERNMENT DOCUMENTS

ANSI/MIL-STD-1815A Reference Manual for the Ada Programming Language (LRM)

Readers Guide for Ada Compiler Evaluation System (ACES), Version 2.0 High Order Language Control Facility Business Software Support Branch 88 CG/SCTL Wright-Patterson AFB OH

Version Description Document (VDD) for Ada Compiler Evaluation System (ACES), Version 2.0 High Order Language Control Facility Business Software Support Branch 88 CG/SCTL Wright-Patterson AFB OH

Primer for Ada Compiler Evaluation System (ACES), Version 2.0 High Order Language Control Facility Business Software Support Branch 88 CG/SCTL Wright-Patterson AFB OH

User's Guide for Ada Compiler Evaluation System (ACES), Version 2.0 High Order Language Control Facility Business Software Support Branch 88 CG/SCTL Wright-Patterson AFB OH

Operational Software Objectives Document for Ada Compiler Evaluation System (ACES), Version 2.0 High Order Language Control Facility Business Software Support Branch 88 CG/SCTL Wright-Patterson AFB OH

2.2 NON-GOVERNMENT DOCUMENTS

ISO/IEC 8652 (1995) Programming Language Ada, Language and Standard Libraries (RM 95)

3 SYSTEM OVERVIEW

The Ada Compiler Evaluation System (ACES) is a set of software tools, source code, scripted scenarios, and documentation. Its purpose is to help assess quality factors of Ada implementations. (An Ada implementation consists of a host computer, a target computer, operating systems for the two computers, and an Ada compilation system.) The ACES can provide compile-time performance data, run-time performance data, and system usability data. The ACES analysis programs can be used to compare the performance of two or more Ada implementations, or they can be used to indicate ways to get the best performance out of a particular implementation.

The ACES is sponsored by the Ada Joint Program Office (AJPO). Version 1.0 was developed by Boeing Defense and Space Group; subsequent versions have been (are being) developed by CTA INCORPORATED.

The ACES software is organized into two Computer Software Configuration Items (CSCIs), referred to as the Operational Software and the Support Software. The Operational Software CSCI consists of performance tests, usability assessor software, and sample command scripts for processing these software modules. The Support Software CSCI consists of library support units, test management software, analysis software, and a Quick-Look (restricted performance testing) facility.

4 CSCI OVERVIEW

4.1 SUPPORT SOFTWARE ORGANIZATION (VERSION 2.0)

The Support Software CSCI, Version 2.0, consists of five Computer Software Components (CSCs), as described in the following list.

* The Pretest CSC consists of programs and data files used to prepare the ACES for executing performance tests on a particular implementation.

* The Utilities CSC consists of program library units that are shared by the performance tests and Support Software units, tools for performing baseline measurements, tools for preparing individual tests for compilation, and test programs for these tools and shared library units.

* The Harness CSC consists of a program and data files that allow the user to control and track the performance testing process.

* The Analysis CSC consists of programs and data files used for data reduction, analysis, and reporting.

* The Quick-Look CSC consists of simplified adaptation and control software, a restricted set of performance tests, and a simple reporting facility. It provides the capability to quickly prepare and execute the limited set of tests, producing only execution time data.

4.2 DIRECTION FOR VERSION 2.1

As directed by the sponsor, Version 2.1 of the Support Software is to provide the following additional capabilities:

* Process and analyze results of new performance tests added to the Operational Software CSCI;

* Provide default choices for performance testing options, test selection, and reporting options; and

* Process and analyze results of performance tests that are provided by the user.

5 REQUIREMENTS FOR PROCESSING NEW PERFORMANCE TESTS

A number of performance tests, most specific to Ada 95, will be added to the ACES in Version 2.1. The Support Software must distinguish Ada 95 tests from those that require only Ada 83 features, must provide the user with means of selecting the new tests, must generate scripts that process them, and must analyze their results. The requirements in the following subsections address these needs.

Addition of new tests does not result in any new requirements for the Pretest, Utilities, or Quick-Look CSCs.

5.1 HARNESS REQUIREMENTS

The following requirements apply to the Harness software.

5.1.1 SPT-H-5101

The Harness shall recognize new performance test groups and subgroups, as determined by new and renamed performance tests.

5.1.2 SPT-H-5102

The Harness shall recognize names of new default main programs that incorporate new and renamed tests.

5.1.3 SPT-H-5103

Main programs that incorporate new or renamed tests shall not include any previously existing tests.

5.1.4 SPT-H-5104

The Harness shall display all new test names that are appropriate for the implementation being tested.

5.1.5 SPT-H-5105

The Harness shall display tests specific to Ada 95 only if the user has identified the implementation as an Ada 95 implementation.

5.1.6 SPT-H-5106

The Harness shall display group and subgroup names only if the groups and subgroups contain tests applicable to the language version of the implementation under test.

5.2 ANALYSIS REQUIREMENTS

The following paragraphs state the requirements that must be met by the Analysis CSC in order to incorporate new performance tests into the ACES.

5.2.1 SPT-A-5201

The Analysis software shall recognize the names of new performance test groups, subgroups, tests, and default main programs.

5.2.2 SPT-A-5202

The Menu program shall allow selection of groups consisting entirely of Ada 95 tests if and only if at least one selected implementation has been identified as an Ada 95 implementation.

5.2.3 SPT-A-5203

The Comparative Analysis program shall incorporate results from new performance test groups, subgroups, default main programs, and tests into its analyses and reports.

5.2.4 SPT-A-5204

The Single-System Analysis program shall provide comparisons of sets of new tests and of sets containing both new and previously existing tests as appropriate.

6 REQUIREMENTS FOR DEFAULT CHOICES

6.1 DEFAULT PERFORMANCE TESTING OPTIONS

This section states the requirements for default selection of performance testing options. These options are set in the Pretest and Quick-Look activities. Therefore, requirements for default testing options apply to the Pretest and Quick-Look CSCs.

6.1.1 SPT-P-6101

The Pretest software shall provide the user with the opportunity to exercise the same degree of detailed control of the pretest activity process that is provided by the Version 2.0 software.

6.1.2 SPT-P-6102

The number of questions presented to the user by the Pretest software shall be minimized by the use of default values and values that are specific to the operating system and compiler family being tested.

6.1.3 SPT-P-6103

The Pretest software shall base the default values for the following upon the operating system and compiler family being used. If no operating system or compiler family is identified, the user shall be required to supply the values.

* Command to execute a compiled program on the host system or on a self- hosted system;
* Command to delete a file;
* Suffix required for the delete command;
* Command to invoke a command script file;
* Command to copy a file;
* Command to rename a file;
* Symbol for the current directory;
* Symbol used as a separator in directory paths;
* Symbol used as the terminator of a directory path;
* Special first line of a script file (allow for multiple lines);
* Special symbol for the beginning of a script file line;
* Symbol or word to indicate the beginning of a comment line;
* Command to echo a string or value to the output file;
* Delimiter for a string being echoed to the output file;
* Suffix used to direct echoed string to a particular file;
* Symbol for the first parameter to a script file;
* Special form of the first parameter symbol for use in an echoed string;
* Symbol for the second parameter to a script file;
* Command to identify the Ada library for compiling to run on the host system (or a self-hosted system);
* Suffix expected for an Ada source code file;
* Command to compile a program (with default options) to be executed on the host system (or a self-hosted system);
* Command to compile (with full optimization) a program to be run on the target system (or a self-hosted system);
* Command to compile (with no optimization) a program to be run on the target system (or a self-hosted system);
* Command to compile (with full checking enabled) a program to be run on the target system (or a self-hosted system);
* Command to link a program to run on the target system (or a self-hosted system);
* Command to delete a compilation unit from the current Ada library;
* Command to delete the body (only) of a compilation unit from the current Ada library; and
* Command to delete the specification (only) of a compilation unit from the current Ada library.

6.1.4 SPT-P-6104

For commands and values that depend on the operating system and compiler family, the Pretest software shall make use of default responses applicable to the following combinations:

Operating System Compiler Family

DOS ThomSoft/Alsys DOS GNAT DOS Unspecified UNIX ThomSoft/Alsys UNIX Rational/VADS UNIX GNAT UNIX Unspecified VMS VAX-Ada VMS Unspecified Unspecified ThomSoft/Alsys Unspecified GNAT Unspecified Rational/VADS

6.1.5 SPT-P-6105

The Pretest software shall provide instructions or shall refer to documentation containing instructions for providing user-defined defaults for the commands and values that are specified for the above combinations of operating system and compiler family.

6.1.6 SPT-P-6106

The Pretest software shall ask the user to provide the following information in all cases:

* The operating system being used (for the host system);
* The compiler family being used;
* Whether the system is for cross-compilation or is self-hosted;
* Whether the system implements Ada 83 or Ada 95; and
* The directory path for the "aces" root directory.

6.1.7 SPT-P-6107

The Pretest software shall assume the indicated values for the following:

* Execution time measurement (EL: Elapsed time)
* Compilation time measurement (EL: Elapsed time)
* Code size measurement technique (L: Label'Address)
* Host or self-hosted executable suffix (.exe)
* Bit size of largest integer type (32)
* Default file name extension for script files will be:
- DOS - .bat
- UNIX - .unx
- VMS - .com

6.1.8 SPT-P-6108

The Pretest software shall assume that the Portable math packages are to be used if the system is an Ada 83 implementation; otherwise, it shall assume the existence of the math packages required by the Ada 95 standard.

6.1.9 SPT-P-6109

For cross-compilation systems only, the Pretest software shall require the user to provide the following commands and values:

* Command to define the Ada library for target system compilations;
* Command to compile a program to be run on the target system;
* Command to link a program to be run on the target system;
* Suffix for an executable file linked to run on the target system;
* Command to execute a compiled program on the target system;
* Command to delete the specification (only) of a unit from the target library;
* Command to delete the body (only) of a unit from the target library; and
* Command to delete both the specification and body of a unit from the target
library.

6.1.10 SPT-P-6110

The Pretest software shall neither ask for nor use the following information that is requested by the Version 2.0 software:

* Command to execute a program under the debugger;
* Command to capture the status of a completed process;
* Command to export the status of a completing process;
* Command to compile with debugging enabled;
* Command to link a program for execution under the debugger;
* Method of managing the input and output files for the Include tool;
* Command for defining a logical symbol; and
* Operator used in defining a logical symbol.

6.1.11 SPT-Q-6111

The Quick-Look software shall provide the user with the opportunity to exercise the same degree of detailed control of the Quick-Look activity process that is provided by the Version 2.0 software. 6.1.12 SPT-Q-6112

The number of questions presented to the user by the Quick-Look software shall be minimized by the use of default values and values that are specific to the operating system and compiler family being tested.

6.1.13 SPT-Q-6113

The Quick-Look software shall base the default values for the following upon the operating system and compiler family being used. If no operating system or compiler family is identified, the user shall be required to supply the values.

* Command to execute a compiled program on the host system or on a self- hosted system;
* Command to delete a file;
* Suffix required for the delete command;
* Command to invoke a command script file;
* Command to copy a file;
* Command to rename a file;
* Symbol for the current directory;
* Symbol used as a separator in directory paths;
* Symbol used as the terminator of a directory path;
* Special first line of a script file (allow for multiple lines);
* Special symbol for the beginning of a script file line;
* Symbol or word to indicate the beginning of a comment line;
* Command to echo a string or value to the output file;
* Delimiter for a string being echoed to the output file;
* Suffix used to direct echoed string to a particular file;
* Symbol for the first parameter to a script file;
* Special form of the first parameter symbol for use in an echoed string;
* Symbol for the second parameter to a script file;
* Command to identify the Ada library for compiling to run on the host system (or a self-hosted system);
* Suffix expected for an Ada source code file;
* Command to compile a program (with default options) to be executed on the host system (or a self-hosted system);
* Command to compile (with full optimization) a program to be run on the target system (or a self-hosted system);
* Command to link a program to run on the target system (or a self-hosted system); and
* Command to delete a compilation unit from the current Ada library.

6.1.14 SPT-Q-6114

For commands and values that depend on the operating system and compiler family, the Quick-Look software shall make use of default responses applicable to the following combinations:

Operating System Compiler Family

DOS ThomSoft/Alsys
DOS GNAT
DOS Unspecified
UNIX ThomSoft/Alsys
UNIX Rational/VADS
UNIX GNAT
UNIX Unspecified
VMS VAX-Ada
VMS Unspecified
Unspecified ThomSoft/Alsys
Unspecified GNAT
Unspecified Rational/VADS

6.1.15 SPT-Q-6115

The Quick-Look software shall provide instructions or shall refer to documentation containing instructions for providing user-defined defaults for the commands and values that are specified for the above combinations of operating system and compiler family.

6.1.16 SPT-Q-6116

The Quick-Look software shall ask the user to provide the following information in all cases:

* The operating system being used (for the host system);
* The compiler family being used;
* Whether the system is for cross-compilation or is self-hosted;
* Whether the system implements Ada 83 or Ada 95; and
* The directory path for the "aces" root directory.

6.1.17 SPT-Q-6117

The Quick-Look software shall assume the indicated values for the following:

* Host or self-hosted executable suffix (.exe)
* Bit size of largest integer type (32)
* Default file name extension for script files will be:
- DOS - .bat
- UNIX - .unx
- VMS - .com

6.1.18 SPT-Q-6118

For cross-compilation systems only, the Quick-Look software shall require the user to provide the following commands and values:

* Command to define the Ada library for target system compilations;
* Command to compile a program to be run on the target system;
* Command to link a program to be run on the target system;
* Suffix for an executable file linked to run on the target system;
* Command to execute a compiled program on the target system;
* Command to delete the specification (only) of a unit from the target library;
* Command to delete the body (only) of a unit from the target library; and
* Command to delete both the specification and body of a unit from the target
library.

6.1.19 SPT-Q-6119

The Quick-Look software shall neither ask for nor use the following information that is requested by the Version 2.0 software:

* Method of managing the input and output files for the Include tool;
* Command for defining a logical symbol; and
* Operator used in defining a logical symbol.

6.1.20 SPT-D-6120

The Primer shall define default options to help the user who starts at download and ends with reports. The Primer shall make accommodations for those users who enter at an intermediate stage, referencing those files that are necessary at each point.

6.1.21 SPT-D-6121

The Primer shall note that for Pretest, the user has the option of unzipping only the files needed to compile and run Setup, Harness and Analysis. Test files can be added later at the user's discretion if there is a problem with space.

6.1.22 SPT-P-6122

The support.zip file shall be revised to unload the initial files needed for Pretest into a "working" subdirectory of the "aces" directory.

6.1.23 SPT-P-6123

Pretest shall give the user the opportunity to modify the default or previously constructed environment file by rerunning Setup.

6.1.24 SPT-P-6124

Setup shall be modified so that the Ada source files required for Pretest will be copied from the distribution subdirectories to the "source" subdirectory where they can be accessed for compilation. All other required files shall be copied from the distribution directories (note SPT-D-8109) to "working". This will be done at the beginning of the Pretest activity and all files will be renamed at that point if needed.

6.1.25 SPT-D-6125

The Pretest section of the Primer shall be redesigned so that each step has four parts:

1. PREPARATION: what needs to be done before executing the step

2. EXECUTION: (if all of the preparation is complete, user can now run the script) include comments such as expected time on a system

3. RESULTS: list of files --- vertical rather than horizontal. In the case of long lists, multiple columns can be used.

4. DEFAULT: a special section for those running in the default mode.

6.1.26 SPT-D-6126

The Primer shall indicate that the user need only unzip the support.zip file, go to the "working" directory, make a library, compile and link the Setup files and run Setup. After that the automated steps can be executed.

6.1.27 SPT-Q-6127

Setup shall be modified so that the Ada source files required for Quick-Look will be copied from the distribution subdirectories to the "source" subdirectory where they can be accessed for compilation. All other required files shall be copied from the distribution directories (note SPT-D-8109) to "ql_work". This will be done at the beginning of the Quick-Look activity and all files will be renamed at that point if needed.

6.1.28 SPT-D-6128

The Quick-Look section of the Primer shall be redesigned so that each step has four parts:

1. PREPARATION: what needs to be done before executing the step

2. EXECUTION: (if all of the preparation is complete, user can now run the script) include comments such as expected time on a system

3. RESULTS: list of files --- vertical rather than horizontal. In the case of long lists, multiple columns can be used.

4. DEFAULT: a special section for those running in the default mode.

6.1.29 SPT-D-6129

The Primer shall indicate that the user need only unzip the support.zip file, go to the "working" directory, make a library, compile and link the Quick-Look Setup file and run the Quick-Look version of Setup. After that the automated steps can be executed.

6.1.30 SPT-Q-6130

The support.zip file shall be revised to unload the initial files needed for Quick-Look into a "ql_work" subdirectory of the "aces" directory.

6.2 PREDEFINED TEST SET SELECTION

This section states the requirements for selection of predefined sets of performance tests. Test selection is performed solely by the Harness. Therefore, all the requirements for selection of predefined test sets apply to the Harness CSC.

6.2.1 SPT-H-6201

The Harness software shall provide the user an option of selecting tests based on the performance issues of interest to the user.

6.2.2 SPT-H-6202

The Harness shall provide the user with the option of bypassing the performance issues menu and proceeding directly to the Harness menu.

6.2.3 SPT-H-6203

The Harness shall provide a default set of performance issues, which shall be preselected when the performance issues menu first appears.

6.2.4 SPT-H-6204

The Harness shall allow the user to select or de-select any of the performance issues.

6.2.5 SPT-H-6205

The Harness shall provide at least the following options on the performance issues menu:

- select all performance issue selections;

- deselect all performance issue selections;

- request help; and

- quit the performance issues menu (exit to the Groups menu).

6.2.6 SPT-H-6206

If the performance issues menu is not used, the ACES shall work in the same manner as in ACES, Version 2.0. If a test.sel file exists, Harness shall use that as the initial test selection. Otherwise, no tests will be selected when the Harness menu is initially displayed.

6.2.7 SPT-D-6207

The Primer, Section 4, shall reflect the modifications described elsewhere in this document. In particular, the algorithm shall be modified to add a description of the additional activities described in Sections 6.2.1 through 6.2.5.

6.2.8 SPT-D-6208

Section 4.1.1 of the Primer shall describe any additional prerequisites as a result of the requirements of Sections 6.2.1 through 6.2.5.

6.2.9 SPT-D-6209

The Primer shall contain one or more sections (probably numbered 4.1.2, with later sections renumbered) describing the predefined test selection process described in Sections 6.2.1 through 6.2.5.

6.2.10 SPT-D-6210

Section 6.1 of the User's Guide shall reflect the modifications to Harness described in Sections 6.2.1 through 6.2.5.

6.2.11 SPT-D-6211

Section 6.3 of the User's Guide shall be modified to reflect the modifications to Harness described in Sections 6.2.1 through 6.2.5.

6.2.12 SPT-D-6212

The User's Guide shall contain a new section (probably numbered 6.4, with later sections renumbered) describing the predefined test selection added to Harness.

6.3 DEFAULT REPORTING OPTIONS

Reporting options are selected during the Analysis activity. Therefore, all the requirements for default reporting options apply to the Analysis CSC.

6.3.1 SPT-A-6301

The Analysis software shall provide default reports which shall be designed to provide the user with the least amount of information which would also be usable. The first CA report shall present the System Factors for all test groups. The second shall present the number of tests successfully implemented. 6.3.2 SPT-A-6302

The default metric for Comparative Analysis reports shall be execution time.

6.3.3 SPT-A-6303

The Analysis software shall provide default reports which shall be designed to provide the user with the least amount of information which would also be usable. The SSA report shall present a ratio of the smallest to the largest reported Language Features for each test comparison.

7 USER-DEFINED BENCHMARK INCLUSION

This section states the requirements for enabling the user to incorporate performance tests that are not part of the ACES. The ability to select, process, and analyze results of such tests affects the operation of the Harness, Analysis, and Quick-Look. Therefore, all the requirements for user-defined benchmark inclusion apply to the Harness, Analysis, and Quick-Look CSCs.

7.1 QUICK-LOOK REQUIREMENTS

7.1.1 SPT-Q-7101

The Quick-Look facility shall be flexible enough to allow additional User-Defined benchmark tests to be added.

7.1.2 SPT-Q-7102

The Quick-Look facility shall be able to report on the results of additional User-Defined benchmark tests.

7.2 HARNESS REQUIREMENTS

7.2.1 SPT-H-7201

The Harness shall provide a mechanism for selecting User-Defined benchmarks in a fashion similar to the way that tests are currently handled in the ACES.

7.2.2 SPT-H-7202

The Harness shall provide a mechanism for generating command scripts that will execute User-Defined benchmarks in a fashion similar to the way that command scripts are currently generated by the Harness.

7.3 ANALYSIS REQUIREMENTS

7.3.1 SPT-A-7301

The ACES shall provide a method to perform Comparative Analysis on User-Defined benchmarks.

7.3.2 SPT-A-7302

The ACES shall provide a method to perform Single-System analysis on User-Defined benchmarks.

7.4 DOCUMENTATION REQUIREMENTS

7.4.1 SPT-D-7401

The ACES documentation shall provide a detailed description of how benchmark tests are currently constructed.

7.4.2 SPT-D-7402

The ACES documentation shall provide a detailed description of what a user must do in order to create a user defined benchmark.

7.4.3 SPT-D-7403

The ACES documentation shall provide a detailed description of what the user must do in order to incorporate a user defined benchmark into the current ACES structure.

7.4.4 SPT-D-7404

The ACES documentation shall provide a detailed description of what the user must do in order to capture the results generated by a user defined benchmark.

7.4.5 SPT-D-7405

The ACES documentation shall provide a detailed description of what the user must do in order to perform Comparative Analysis using results from a user defined benchmark.

7.4.6 SPT-D-7406

The ACES documentation shall provide a detailed description of what the user must do in order to perform Single System Analysis using results from a user defined benchmark.

7.4.7 SPT-D-7407

The ACES documentation shall provide examples of the steps that the user must perform when incorporating user defined benchmarks.

8 OTHER CHANGES

This section states the requirements for changes to the Support Software CSCI that are not covered in the previous sections.

8.1 PRETEST AND UTILITIES REQUIREMENTS

8.1.1 SPT-D-8101

The Primer shall indicate to the user at each step what files must be present before that step can be executed. This shall be indicated either directly or in the form of an exact reference to the required information. The user shall not be expected to be conversant with all of the documentation before reaching the point in question. This shall be consistent with each possible combination of choices and shall include but not be limited to Section 3.2.4.a which does not differentiate between what files need to be in the directory if CPU rather than EL is selected.

8.1.2 SPT-D-8102

The Primer shall present a cross reference index for ACES Documentation that is verified and updated.

8.1.3 SPT-D-8103

The Primer documentation shall list at the end of Section 3 all of the referenced files, their actual file names, and which steps use them.

8.1.4 SPT-D-8104

The Primer shall properly and clearly document the corresponding software. This will eliminate existing errors and clarify existing documentation. These shall include but will not be limited to the following:

- Section 2.1.2: Clarify references to Harness tests or testing Harness.

- Section 2.2: Incorrect counts of software and file names when it refers to "Versions" of ACES and to seven compressed files.

8.1.5 SPT-D-8105

The Primer shall define all Primer-specific notation. This shall include but is not limited to the last paragraph of Section 2 which gives no sample for # sign but uses it in Section 2.1.3 and other places.

8.1.6 SPT-D-8106

The User' Guide shall properly and clearly document the corresponding software to eliminate existing errors and clarify existing documentation. These shall include but will not be limited to the following:

- Section 5.1.4 (first bullet) which refers to Ada 95 MATCH packages in places rather than Ada 95 MATH packages.

8.1.7 SPT-P-8107

Setup shall inform the user which (83/95) compiler version is selected in any existing environment file. It now only informs the user of the selection of cross/self compiler.

8.1.8 SPT-D-8108

The Primer shall indicate the room needed to house the uncompressed files as in Section 2.2, paragraph 4 ("The distributed...working directory.") and elsewhere as needed.

8.1.9 SPT-D-8109

The Primer shall indicate the correct orientation of files. If they are unzipped according to instructions, an "aces" subdirectory will be created off the user's directory and the working, source, support, tests, assessrs, qlook, and docs subdirectories will be created off the newly created aces directory. The individual test subdirectories will be created off the tests directory and the individual assessor subdirectories will be created off the assessrs directory. This is incorrect in the present Figure 2-1 and in the text following that figure.

8.1.10 SPT-D-8110

The Primer shall indicate at what point the Setup executable can actually be run. Sections 3.1, 3.1.1, and 3.1.2 say what to do before running Setup and what to do once Setup is run but the user can read on for several paragraphs before realizing that setup should have been executed.

8.1.11 SPT-D-8111

The Primer shall clarify the directions for UNIX users. These shall include but not be limited to:

- Section 3.1.3.2 paragraph 1: The copying and renaming are now done automatically;

- Section 3.1.3.2 paragraph 3: Bourne shell or C shell; and

- Section 3.1.3.2 paragraph 4: Need to mention that the example given will run in the back ground.

8.1.12 SPT-D-8112

The Primer shall indicate instances in which there may be a difference in what the user can expect when running Ada 83 as opposed to running Ada 95. These shall include but not be limited to:

- Section 3.2.11.c: If one is running an Ada 95 compiler, the files specific to Ada 95 will be present as well.

8.1.13 SPT-D-8113

The Primer shall indicate which files will be automatically deleted in each step, and which can or should be manually deleted. This shall include but not be limited to Section 3.2.e.

8.1.14 SPT-D-8114

The User' Guide shall present a clear and succinct description of each item covered. This shall include but not be limited to Section 5.1.1 paragraphs one and two which need to be rewritten and combined. The second paragraph looks as if it should be in the middle of the first.

8.1.15 SPT-D-8115

The Primer shall make better and more consistent use of CAPITALIZATION, underlines, italics, and boldface. This shall include but not be limited to Section 3.2.5.a in which "source" should be in boldface.

8.1.16 SPT-P-8116

Setup shall make a backup copy of any existing environment file before it offer the user a chance to make changes. It shall also offer the user the chance to revert to the old version by replacing quit with two choices: (1) generate new file with the new selections; or (2) exit and save none of the changes; go back to the old version.

8.1.17 SPT-P-8117

The Pretest software unit "zg_incld" (four versions) shall specify the type CHARACTER for at least one of the literals in the code "for i in 'a' .. 'z' loop".

8.1.18 SPT-D-8118

The Quick-Look section of the Primer shall match the Pretest section where appropriate. Note Primer 8.2.2.2.d.

8.2 HARNESS REQUIREMENTS

8.2.1 SPT-H-8201

The Harness software shall indicate, in its error messages in UPDATING, the name of the file causing the error rather than the name of the file following the file that caused the error.

8.2.2 SPT-D-8202

The User' Guide shall be careful of maintaining adequate references so that they are complete and accurate. This shall include but not be limited to:

- The last paragraph of Section 5.4.3 which refers to Sections 9.1.6 and 6.13 of the User' Guide as if they are in the Reader's Guide and gives no indication as to where in the Reader's Guide one can find a discussion of the interpretation of such modified tests.

8.2.3 SPT-D-8203

The Harness documentation shall list at the end of each section all default file names, their original locations, when they are needed, how they are referenced, when they are revised by a specific part of the program, and if the user changes the file name, where else changes need to be made to be consistent.

8.2.4 SPT-D-8204

The User' Guide shall warn the user that if either the "test.sel" or the "test.sum" file is corrupted, Harness will exit gracefully but the condition can be cured by deleting the offending file and running Harness without it. Harness will happily build another. References are made in Section 6.2.5 and Section 6.2.6.

8.2.5 SPT-D-8205

The User' Guide shall explain "status" in reference to Harness as referenced in Section 6.4.4, Section 6.4.5, Section 6.5.1 and elsewhere. If "status" is explained in detail elsewhere, then that section needs to be referenced at this point.

8.2.6 SPT-D-8206

The User' Guide shall clarify whether the "current subgroup" can be in a different group than the "current group". This shall include but not be limited to:

- Section 6.5.12

8.2.7 SPT-D-8207

The User' Guide shall demonstrate consistency between its text and its graphics. This shall include but not be limited to:

- Section 6.5.2 which refers to "in the groups menu below, the classical group ... number (3)" however, the screen in Figure 6-4 shows groups 11-17!

8.2.8 SPT-D-8208

The User' Guide shall fully explain each and every file it references. This shall include but not be limited to:

- Section 6.9.3 which displays in Figure 6-11 the result of the Get_Test_Name_List (GT) command and prompts the user for a "Test file name" but does not indicate what information in what format should be in that file and whether there is a default or sample file that the user can reference.

8.2.9 SPT-D-8209

The User' Guide shall provide the user with adequate information on how to modify files. This shall include but not be limited to text based data files (e.g., zh_??.txt, za_cosys.txt, zh_??.lib) that the user is to modify but that expect a specific number of white space delimiters between what appear on the screen to be columns of figures.

8.2.10 SPT-D-8210

The User' Guide shall present information about the ACES in a clear, concise, logical and factual manner. This clarification shall include but not be limited to:

- Section 6.13.

- The problem referred to in the last paragraph of Section 6.1 which mentions an "inconsistency in the Harness".

8.3 ANALYSIS REQUIREMENTS

8.3.1 SPT-A-8301

The Menu software shall allow the user to advance to the next screen without having to type "n" in addition to the command where multiple selections before going to the next screen are not possible.

8.3.2 SPT-A-8302

The Analysis software, on the MAIN MENU screen, shall tell the user to select a command or a tool and a command (separating them with a comma) instead of telling the user to "select one tool separated by a comma."

8.3.3 SPT-A-8303

The Analysis software shall be designed to exit gracefully if multiple errors come up after all selections are made instead of having the user resort to a keyboard abortion command.

8.3.4 SPT-A-8304

The Analysis software shall be designed to give adequate error messages and exit gracefully if it is missing a file so that the user will know what files are required for it to function properly. The software shall indicate if the file is missing or if it cannot locate the name it was given to find and where to make that change.

8.3.5 SPT-A-8305

The Analysis software shall be redesigned so that the GROUPS menu either fits on one screen or appears on two screens in a way that is easy to use. Presently the title and "1. Applications" scroll off screen when Ada_95 is an option.

8.3.6 SPT-A-8306

The Analysis software shall be redesigned so that the RUN OR SAVE REQUEST menu will either indicate "no reports chosen" (and therefore no output is to be expected) or state what has been chosen and what will be produced.

8.3.7 SPT-A-8307

The Analysis software shall be revised so that when a screen indicates that one of a group of items must be selected, then the user will really have to make a selection before going to the next screen.

This occurs on at least the following screens:

- SUMMARY OF ALL GROUPS LEVEL REPORT OPTIONS:

- SSA MAIN REPORT OPTIONS

- SSA HLS REPORT OPTIONS

8.3.8 SPT-A-8308

The Analysis software shall present a new menu that is a combination of the present MAIN and INITIAL menus.

8.3.9 SPT-D-8309

The Analysis software shall define the help menu for BUILD/MODIFY SYSTEM NAME FILE (GENERAL INFORMATION) so that the file names explained there are the same as those that appear in selections.

8.3.10 SPT-D-8310

The Analysis documentation shall list at the end of each section all default file names, their original locations, when they are needed, how they are referenced, when they are revised by a specific part of the program, and if the user changes the file name, where else changes need to be made to be consistent.

8.3.11 SPT-A-8311

The Analysis software menus shall use language consistent with the actual reports. This shall include but not be limited to:

- Selection 7 of the SUMMARY OF ALL GROUPS LEVEL REPORT OPTIONS, which indicates on the screen "errors" and on the report "successes".

8.3.12 SPT-D-8312

The Primer shall give clear, concise, and logical explanations for ACES. This shall include but not be limited to the following section:

- Section 3.2.12.c: "introduces this step" --- for which paragraph and for what purpose?

- Section 3.2.13.e.3: system1 is an Ada 83. What if they are testing an Ada 95?

8.3.13 SPT-D-8313

The User' Guide shall provide thorough explanations of all of the possibilities to which it refers or provide adequate reference to resources in which the information can be easily found. This shall include but not be limited to:

- Section 9.3 par 4 refers to za_anlys.com which cannot be found.

- Figure 9-27 indicates that the View/Modify systems selection is number 5 when it is actually two more than the number of systems listed.

8.3.14 SPT-D-8314

The User' Guide shall fully explain the importance of data files, how to manually construct them, what programs need which ones and will modify them, and where sample/dummy files can be found. This shall include but not be limited to:

- Section 9.3.2 tells the user to construct the file za_cosys.txt and refers the reader to Section 9.1.4 but both sections fail to mention the fact that a dummy comes with the support.zip file and can be easily modified when running the Analysis MENU program.

8.3.15 SPT-D-8315

The User' Guide shall fully explain how Condense, CA and SSA work both with the Menu and separately and what files are needed for either menu or independent use. This shall include but not be limited to clarification of several places in Section 9, including Section 9.1.1, which refers to Figure 9-1 and the fact that Condense, CA and SSA can be run independently of the Menu but give no more details as to where the files indicated in Figure 9-1 can be found. It also needs to cross reference the fact that Step 12 in the Primer will compile this automatically. It also refers to "za_anlys.com" which is not part of the test suite. Much of the information in Section 9 tends to ignore the fact that the Primer and the menu created in Step 12 of Pretest exist, as well as the fact that running the files created by Harness will produce log files that can be used by Analysis.

8.3.16 SPT-D-8316

The Reader's Guide shall clarify its use of the phrase "classical benchmark tests" by making appropriate reference to the Classical (Cl_*) performance tests. It shall further refer to "benchmark test suite" as the "ACES performance tests". This shall include, but not be limited to:

- the Abstract

- Section 2.4.2.4: The paragraph "Represent classic benchmark ... available for other languages"

- Section 3.2.8.1

- Section 5, paragraph 1

- Section 5.4.2, paragraph 2

8.3.17 SPT-D-8317

The Reader's Guide shall list at the end of each section all default file names, their original locations, when they are needed, how they are referenced, when they are revised by a specific part of the program, and if the user changes the file name, where else changes need to be made to be consistent. This shall include but not be limited to:

- Section 5.2.2 in which no default names are given for the No Data, Exceptional Data, or Multiple Results Reports.

8.3.18 SPT-D-8318

The Reader's Guide shall give a listing of all of the possible Analysis reports and the implications of the results along with cautions as to what could cause differences that would make the report misleading. This shall include clarification of but not be limited to:

- Section 5.3.2.1.2 (and others) which refer to Figure X-x, name the report that is displayed and sometimes give a short explanation of what each shows. These should be expanded in their definitions and separated so that they follow the reports that they describe.

8.3.19 SPT-D-8319

The Reader's Guide shall accurately reflect changes caused by new versions. This shall include but not be limited to:

- The third paragraph (Gps:) of Section 5.3.2.2.3 which refers to the number of defined groups as 17.

8.3.20 SPT-D-8320

The Reader's Guide shall explain in appropriate detail each of the possibilities to which it refers. This shall include but not be limited to:

- The fifth paragraph of Section 5.3.2.2.3 (NoData:) which refers to the user entering the appropriate code but does not tell what codes are appropriate or where they can be entered or what will happen if they are or are not entered.

8.3.21 SPT-D-8321

The User' Guide shall adequately explain all of the adaptations and limitations. This shall include but not be limited to Section 9.6 and especially the last limitation ("The Analysis... batch mode"). Since only 17 groups can now be displayed on a crt screen, what is meant by menu, (26) items, (25) systems, and groups. In addition, references to sections explaining requirements shall be instituted.

8.3.22 SPT-A-8322

The Analysis menus shall be revised as necessary to be more self explanatory. This shall include but not be limited to the use of white space to separate selection areas and the re- wording of selections to reflect the actual content of the output.

8.4 QUICK-LOOK REQUIREMENTS

8.4.1 SPT-D-8401

The Primer shall correctly, concisely and adequately explain the Quick-Look section of the ACES. This shall include but not be limited to:

- the last sentence of the paragraph following Section 8.2.2.j which refers to "compilable Ada --".

8.4.2 SPT-Q-8402

The Quick-Look software unit "zg_incld" shall specify the type CHARACTER for at least one of the literals in the code "for i in 'a' .. 'z' loop".

8.5 OTHER DOCUMENTATION (READER'S GUIDE)

8.5.1 SPT-D-8501

The Reader's Guide shall present information in a clear, concise, logical, and factual manner. This shall include, but not be limited to:

- Section 3.2.2: This section refers to pragmas in the first list, but does not indicate which list is the first one.

- Section 3.2.2: All lists shall be vertical. If they are too long, multiple columns are acceptable.

- Section 3.2.4.1.7: The statistics in paragraph 3 are incorrect. They should be the opposite, i.e., exchange the words inner and outer.

- Section 3.2.7.3: "...until the I/O completes" transitive verb needs an object.

9 ABBREVIATIONS AND ACRONYMS

This section defines abbreviations and acronyms that are used in this and other ACES documents.

AARM...Annotated Ada Reference Manual
ACEC...Ada Compiler Evaluation Capability
ACES...Ada Compiler Evaluation System
ACVC...Ada Compiler Validation Capability
AES...Ada Evaluation System (UK Ministry of Defence)
AJPO...Ada Joint Program Office
CA...Comparative Analysis
CISC...Complex Instruction Set Computer
CPU...Central Processing Unit
CRC...Cyclic Redundancy Check
CSC...Computer Software Component
CSCI...Computer Software Configuration Item
DEC...Digital Equipment Corporation
HOLCF...High Order Language Control Facility
I/O...Input/Output
ISA...Instruction Set Architecture
LRM...(Ada) Language Reference Manual (ANSI/MIL-STD-1815A)
MCCR...Mission Critical Computer Resource
MIP...Million Instructions Per second (a measure of hardware speed)
MIS...Management Information System
MoD...Ministry of Defence
NUMWG...Numerics Working Group (ACM SIGAda organization)
RAM...Random Access Memory
RISC...Reduced Instruction Set Computer
RM...(Ada 95) Reference Manual (ISO/IEC 8652 (1995))
ROM...Read Only Memory
RTS...Run-Time System
SSA...Single System Analysis (ACES analysis tool)
URG...Uniformity Rapporteur Group
VAX...Virtual Address eXtension (DEC family of processors)
VDD...Version Description Document
VLIW...Very Long Instruction Word
VMS...Virtual Memory System (DEC operating system for VAX processors)