Ada Compiler Evaluation System
ACES 2.1
Primer
FINAL
Contract Number F33600-92-D-0125
CDRL A0014
Prepared for:
High Order Language Control Facility
Business Software Support Branch
88 CG/SCTL
Wright-Patterson AFB OH 45433-5707
Prepared by:
CTA INCORPORATED
5100 Springfield Pike, Suite 100
Dayton OH 45431
Availability of the Ada Compiler Evaluation System
The ACES software and documentation are available by anonymous FTP from the host "sw-eng.falls-church.va.us" in the directory "public/AdaIC/testing/aces/v2.0" and from other Ada-related hosts. Document files are included in PostScript format and as ASCII text. The ACES files are also accessible via the World-Wide Web. The home page URL is "http://sw-eng.falls-church.va.us/AdaIC/testing/aces/".
For further information about the ACES, contact the High Order Language Control Facility. As of 1 March 1995, the appropriate contact is:
Mr. Brian Andrews
88 CG/SCTL
3810 Communications, Suite 1
Wright-Patterson AFB, OH 45433-5707
(513) 255-4472
Go to: Section Summary or Table of Contents
This document serves as an introduction to the nature and use of the Ada Compiler Evaluation System (ACES), Version 2.1. The intent of the Primer is to provide the minimum information needed to assess an implementation of the Ada language using the ACES and/or the ACES Quick-Look Facility. It addresses only those issues that are expected to arise in a straight-forward application of the ACES. Other documents in the ACES documentation set should be referenced if issues arise that are not covered in the Primer.
The Primer is meant to be usable by any person who has knowledge of the operating systems and compilation system being tested. Previous ACES experience is not necessary.
The document consists of eight sections, each of which is briefly described in the following list:
* Section 1, "Overview of ACES"
Summarizes the purpose and structure of the ACES and gives a high-level view of its use.
* Section 2, "File Names and Installation"
Describes the ACES files, the default directory structure, and the installation procedures.
Describes the use of the automated Pretest process that must be completed before performance testing can begin.
* Section 4, "Performance Testing"
Describes the use of the Harness tool in executing and tracking the performance tests.
* Section 5, "Performance Analysis"
Describes the use of the Condense, Comparative Analysis, and Single System Analysis tools in analyzing the data produced by the performance tests.
Describes the use of the Assessor software and scenarios for determining system capacities and the capabilities of the symbolic debugger, the library management system, and the diagnostics system.
* Section 7, "Interpretation of Results"
Describes the most useful reports that are produced by the Analysis tools and the assessors.
Describes the use of the ACES Quick-Look facility which provides a subset of key ACES performance tests.
The user is expected to refer to other documents when issues arise that are not anticipated in straight-forward uses of the ACES. Information on the execution of the tests and tools is maintained in the User's Guide. Information on the use of the results, including discussions of analysis techniques, is maintained in the Reader's Guide.
If the user wishes to do only performance testing, then he/she should read Primer Sections 1 through 4. (Note that it is necessary to perform the steps described in Section 3 before doing performance testing.) If he/she then wishes to use the Analysis tools, Section 5 is important. Section 7 will be helpful in this case, also. If the user does not wish to use the ACES analysis techniques, then he/she should follow the instructions in the Condense subsection of Section 7, choosing to produce comma-delimited output. These files can then be imported into the analysis tools of the user's choice.
If the user wishes to use only the Assessors, then he/she should use Primer Sections 1 through 3 and Section 6. Note that some of the steps described in Section 3 must be performed before using the Assessors. The Section 7 paragraphs on the Assessors are also useful in this case.
If the user wishes to use the Quick-Look facility, he/she should use Primer Section 8.
LIST OF TABLES
The Ada Compiler Evaluation System (ACES) is a set of software tools, source code, scripted scenarios, and documentation. Its purpose is to help assess quality factors of Ada implementations. (An Ada implementation consists of a host computer, a target computer, operating systems for the two computers, and an Ada compilation system.) The ACES can provide compile-time performance data, run-time performance data, and system usability data. The ACES analysis programs can be used to compare the performance of two or more Ada implementations, or they can be used to indicate ways to get the best performance out of a particular implementation.
The ACES represents the merger of two evaluation systems: The Ada Compiler Evaluation Capability (ACEC) and the Ada Evaluation System (AES). Each of the systems boasted some unique strengths, yet there was some duplication between the two. The goal of the ACES merger was to take advantage of each systems' strengths and reduce the amount of duplication.
The Ada Joint Program Office (AJPO) of the United States Department of Defense (DoD) funded the development of the ACEC. The ACEC went through several releases, culminating with Version 3.1 (1992). The Air Force Wright Laboratory provided management, and the Boeing Defense and Space Group (Wichita, Kansas) performed the technical work. Boeing designed the ACEC to be portable to all validated Ada implementations with minimal adaptation. The Air Force distributed dozens of copies of the ACEC to U.S. and foreign Government organizations and their contractors.
The United Kingdom's Ministry of Defence (MoD) sponsored the AES. The British Standards Institute and Software Sciences Ltd. accomplished the development of AES versions 1.0 and 2.0. The AES test management tools were its strength. Evaluation services using the AES were provided for fee.
In June 1991, DoD and MoD agreed to merge the two evaluation suites and produce a single, internationally available suite. The AJPO provided funding for the merger effort, which was managed by the High Order Language Control Facility (HOLCF) at Wright-Patterson Air Force Base. In 1993, the HOLCF contractor, CTA INCORPORATED, delivered the ACES as the merger of the ACEC and the AES. Boeing Defense and Space Group again performed the actual development work under contract to CTA. In 1994, the HOLCF and CTA enhanced the ACES with this Primer, comma-delimited report files, and an automated process for the Pretest activity. The ACES was further enhanced in 1995 with a Quick-Look facility which provides a fast and easy (yet less complete) process for evaluating an Ada implementation.
The current version (2.1) provides more Ada 95 testing; allows the user more default choices for processing options, test selection, and analysis report selection; and provides a mechanism for incorporating user-defined performance tests into the ACES test selection and analysis processes.
From the user's point of view, the ACES consists of five components, as follows:
a. Support software and data
This component is composed of approximately 350 files, classified as Setup files, Harness files, and Analysis files.
These files are used to build and test a core set of library units, command scripts, and data files. These are used throughout the evaluation.
Approximately 90 of the support files contain source code and data for the Harness program. This tool aids the user in tracking the run-time status of the performance tests and in creating command scripts for compiling and executing the tests.
The remaining support files contain source code and data files for the Analysis tools. These programs are used to compare performance test results for two or more implementations and to evaluate the relative performance of various features of one implementation.
b. Performance test software
This component comprises approximately 2120 files. These files include source code for the tests and their support units along with sample command scripts for executing the tests. There are 21 groups of performance tests, each group addressing a particular set of performance issues.
c. Assessor software and data
This component is comprised of approximately 335 files, representing software modules, data files, and scripted user scenarios. Three of the assessors evaluate the usability of the diagnostic system, the library management system, and the symbolic debugger. The fourth determines compile-time and run-time capacity limits.
d. Quick-Look software and data
This component comprises approximately 155 files comprised of software modules and data files. These files contain the software necessary to setup the software tests, the software tests (selected subset of the full ACES), and reporting software.
e. Documentation
The ACES documentation set consists of this Primer, a User's Guide, a Reader's Guide, and a Version Description Document. Documents are distributed in both ASCII text form and PostScript form. With appendices, there are approximately 40 files. The User's Guide provides detailed instructions for the user. The Reader's Guide provides information on the analysis reports, the measurement and analysis techniques, and the interpretation of the results. The Version Description Document provides descriptions of the tests, the association between (long) test names and (short) file names, and descriptions of tests that are known to cause difficulties on some implementations.
Portability across validated Ada implementations was a primary design goal of the ACES. In particular, all user interface input and output are performed using Text_IO. No assumptions are made about terminal capabilities except those that apply to the least sophisticated terminals. Thus, users who are accustomed to windowing facilities, "hot-key" definitions, and point-and-click interfaces may perceive the ACES user interface as primitive.
The ACES was designed with three general evaluation goals:
a. Evaluation for implementation selection;
b. Evaluation of a selected implementation; and
c. Comparing releases of an implementation.
The ACES user must start with a set of clearly defined goals for each evaluation. These goals lead to evaluation criteria, which should be carefully documented. Whatever the goal(s), it is not appropriate to perform the testing, generate the analysis reports, read all of the data, and then decide what information is or is not important.
Figure 1-1 illustrates the paths through the ACES evaluation process. The first step is performing the Pretest activity. (The Pretest automation tool, ZP_SETUP, is useful here.) The results of the Pretest are recorded on the Pretest Report.
If the user wishes to run the performance tests, then the next step is the use of the Harness program (compiled during the Pretest activity). This program generates command scripts for compiling and executing the performance tests. The user then executes these scripts and captures the results in log files. The Harness program reads these log files and gives execution status information so that the user can choose to re-run selected tests or go on to other tests. This process is iterative.
When the user has run as many performance tests as desired, he/she runs the Analysis Menu program, first selecting the Condense tool. Condense reads the log files and produces the Analysis Database files. The user then selects either Comparative Analysis (in which case Analysis Database files for other programs must be available) or Single-System Analysis. These Analysis programs produce extensive reports, as directed by the user.
The user may elect to run the Assessors before, after, or independently of the performance tests, provided that at least part of the Pretest has been completed. The Assessors may be run in any order. The results from each are recorded on the appropriate Assessor Report form.
The user may elect to run the Quick-Look facility before, after, or independently of the performance tests. The Quick-Look facility provides a setup process very similar, yet less complex than that done as part of the Performance Testing and Assessor pretest activity.
Figure 1-1 Overview of the ACES Evaluation Process
The testing process varies according to the goals of the evaluation. For each of the goals described in Section 1.3, the corresponding paragraph below identifies some assumptions, suggests some questions that are useful in setting up evaluation criteria, identifies the activities appropriate to the goal, points to the Primer section that discusses each activity, and identifies decisions that must be made.
Assumptions
This goal assumes that a set of candidate implementations has been identified and made available. All activities except analysis must be performed on each implementation. The analysis step may be carried out on any one implementation.
Criteria
* Which implementation best meets the minimum performance levels required by the project (e.g., for task rendezvous time, task scheduling time, or input/output time)?
* Which implementation makes the tradeoffs (regarding compilation speed, linking speed, execution speed, and code size) that best fit the needs of the project?
* Which implementation provides the best combination (for the project) of functionality of the debugger, library management system, and diagnostic system?
* Which implementation supports the most important capacity limit (both compile-time and run-time) that are required by the project?
* Which implementation gives the best (in terms of project goals) performance in those language areas that are of particular importance to the project (e.g., generics or tasking)?
Activities
* Pretest/Setup (Section 3)
* Performance Testing (Section 4)
* Condense (Section 5.2)
* Comparative Analysis (Section 5.3)
* Assessors as determined by the evaluation criteria (Section 6)
* Interpretation (Section 7)
Decisions
* Whether, and how, to measure code size (Section 3.2.2, Section 5.2)
* Whether to measure elapsed or CPU time (Section 3.2.3)
* Which performance test groups to use (Section 4.1)
* Which comparison reports to request (Section 5.3)
Assumptions
This goal assumes that the organization or project has already selected an implementation and has it available for evaluation. The intent of the evaluation is to support the establishment of procedures that would make the most efficient use of the implementation.
Criteria
* Should the software be organized as many small units or fewer large units?
* Should complete system "builds" be done frequently or infrequently?
* Should the use of tasking or generics be avoided or encouraged?
* Should automatically generated code be manually optimized for certain cases or should the compilation system be expected to handle optimization?
* Should coding and optimization emphasize execution speed or code size?
* Should unchecked deallocation be used?
* Should particular data structures and data references be avoided or encouraged?
* Should exception handling be used generously or sparingly?
* Should if-then-elsif control be used in preference to case statements?
* Should subprogram calls be expanded in line?
* Should deployed code be compiled without checks?
Activities
* Pretest/Setup (Section 3)
* Performance Testing (Section 4)
* Condense (Section 5.2)
* Single System Analysis (Section 5.4)
* Assessors as determined by the evaluation criteria (Section 6)
* Interpretation (Section 7)
Decisions
* Whether, and how, to measure code size (Section 3.2.2, Section 5.2)
* Whether to measure elapsed or CPU time (Section 3.2.3)
* Which performance test groups to use (Section 4.1)
* Which analysis reports to request (Section 5.3)
Assumptions
This goal assumes that two releases of the same implementation are being compared to determine the performance gains or losses. The differences in the implementations may be due to hardware changes, operating system (OS) upgrades, or compiler upgrades.
Criteria
* Is the new hardware (or OS) faster or slower?
* Has compilation or linking speed changed?
* Has execution time changed?
* Has code size changed?
* Are new optimizations detected?
* Has the performance of particular language features changed?
Activities
* Pretest/Setup (Section 3)
* Performance Testing (Section 4)
* Condense (Section 5.2)
* Comparative Analysis (Section 5.4)
* Interpretation (Section 7)
Decisions
* Whether, and how, to measure code size (Section 3.2.2, Section 5.2)
* Whether to measure elapsed or CPU time (Section 3.2.3)
* Which comparison reports to request (Section 5.4)
As distributed, the ACES files are arranged in the directory structure outlined below. The numbers that follow behind the directory name indicate the current number of files (excluding sub-directory files) that are found in that directory.
aces (1)
assessrs
capacity (63)
debugger (79)
diagnost (101)
library (91)
docs (38) (documentation files)
qlook (156)
support (354) (Pretest, harness, and analysis files)
tests
ap ( 75)
ar (142)
cl ( 90)
do (106)
dr (290)
dt ( 51)
gn ( 28)
in ( 12)
io (131)
ms ( 18)
oo ( 10)
op (329)
po ( 98)
pt ( 10)
sr ( 65)
st ( 92)
su ( 82)
sy (296)
tk (131)
xh ( 61)
ud ( 1)
In this Primer, references to this directory structure will be given in UNIX style, rooted at the "aces" directory. For example, the files for the Classical (cl) group of performance tests are in the directory referred to as "aces/tests/cl".
The Primer uses the asterisk ("*") in file names as a "wild-card" specifier, matching any sequence of characters. The question mark ("?") is sometimes used as a limited wild-card character, matching any single character. Finally, the sharp symbol ("#") is sometimes used as a limited wild-card character, matching only those characters that are specified in the section using this character. Some examples follow:
Pattern Sample Matches
zc_*.com zc_lnk.com, zc_setli.com zg_glob3.el? zg_glob3.elg, zg_glob3.ell *.ada cl_sodum.ada, zp_setup.ada z?_*.el? zg_glob3.elg, zc_incld.ell aces/tests/# aces/tests/cl, aces/tests/ap
The ACES is distributed as a set of approximately 3000 files, most of whose names have the form "pp_ident.typ". The file names have been kept short and simple in order to achieve the goal of portability. The first two characters of the file name identify the category to which the file belongs, as discussed in subsections 2.1.1 through 2.1.7. In nearly all cases, the third character is an underscore, though there are a few exceptions. Characters 4 through 8 (or 3 through 8 if there is no underscore) identify the purpose of the file. This identification depends on the file category. The three-character suffix often identifies the type of contents of the file; for example, "ada" denotes syntactically complete Ada source code, "inc" denotes incomplete Ada code (the timing loop code must be included), and "txt" denotes a data file. In this Primer, we assume that all letters in files names are lowercase; however, some systems will translate them to uppercase.
These files are all in the "aces/support" directory. Each setup file name begins with the letter 'z'. File names in some other categories also begin with 'z', as discussed in later paragraphs. The first two characters of every setup file are one of the following combinations:
a. "zc" : Nearly all of these files (names ending in ".tpl") are templates for script files. The "setup" program (see Sections 3 and 8) uses them to generate implementation-specific scripts for fairly primitive operations (e.g., compile, link, copy source file, or delete library unit). The remaining files (names ending in ".cp", ".el", or ".ada") are source code files used in setting up the testing environment.
b. "zg" : Most of these files are source code files used in setting up the testing environment. These include a sample assembly language routine for determining an address (for VAX) and a sample Ada subprogram for determining CPU time (for VAX-Ada). A few of the "zg" files (names containing "start", "stop", and "verfy") are fragments of timing code that are automatically inserted into performance test programs before compilation.
c. "zm" : These files contain Ada source code providing access to certain mathematical functions, including a random number generator.
d. "zp" : These files contain Ada source code, sample data, and script templates for compiling, linking, and testing the common library units, the performance testing harness, and the analysis tools.
These files are all in the "aces/support" directory. Each harness file name begins with the letters "zh", and contains Ada units or data for creating, testing, and using the Harness program. Creating the Harness program is discussed in Section 3.2.11, and its use is discussed in Section 4.
There are 21 groups of performance test files, each with its own prefix. The Harness program is used to create command scripts for completing, compiling, linking, and executing all the tests in the group. Incomplete test files (names ending with ".inc") lack the timing code, which is inserted by running a preprocessor program ("zg_incld"). Performance test file prefixes are "ap", "ar", "cl", "do", "dr", "dt", "gn", "io", "in", "ms", "oo, "op", "po", "pt", "sr", "st", "su", "sy", "tk", "xh", and "ud". The files for the group with prefix "##" are in the directory "aces/tests/##". The performance test files are typically used in conjunction with the Harness program, whose use is discussed in Section 4.
There are four assessors, one each for the symbolic debugger ("yb*" files), compile-time and run-time capacities ("yc*" files), the diagnostics system ("yd*" files), and the library management system ("yl*" files). Each assessor includes Ada software files, sample command scripts, documentation files, and a reporting template. The files are in the appropriately named subdirectories of "aces/assessrs". In addition, the capacity assessor has a data file that can be edited to select testing limits. The assessors are discussed in Section 6.
There are four programs in the analysis set: a user interface (Menu), a data extraction program (Condense), Comparative Analysis (CA), and Single System Analysis (SSA). CA will be used when evaluating for selection or comparing releases of an implementation. SSA will be used when evaluating a selected implementation. All the analysis program file names, including Ada source files, command scripts, and data files, begin with "za". These files are in the directory "aces/support". Their use is discussed in Section 5.
There are 156 files in the Quick-Look set: Setup software and data files, testing software, reporting software, and documentation. Note that much of this code and the documentation duplicates that found in other directories. This was done so that the Quick-Look facility could be accessed early by the users.
All of the ACES documentation is distributed in two formats, ASCII and PostScript. The documentation file names all begin with "zd"; they are found in the directory "aces/docs". The ASCII versions have the extension ".txt" and the PostScript versions have the extension ".ps".
The versions of ACES that are available on the Internet are stored in 11 compressed files, using the for-public-use "zip" software. The "zip" software is subject to the following notice:
"Copyright (C) 1990-1993 Mark Adler, Richard B. Wales, Jean-loup Gailly, Kai Uwe Rommel, Igor Mandrichenko and John Bush. Permission is granted to any individual or institution to use, copy, or redistribute this software so long as all of the original files are included, that it is not sold for profit, and that this copyright notice is retained."
This compression program and its companion "unzip" program are compatible with recent versions of PKZIP for DOS systems. They are available for many platforms, including UNIX, VMS, DOS, Windows, OS/2, and Macintosh. The software and its documentation may be obtained from the following sites, among others:
ftp.uu.net:/pub/archiving/zip/...
wuarchive.wustl.edu:/mirrors/misc/unix/...
wuarchive.wustl.edu:/packages/compression/...
wuarchive.wustl.edu:/mirrors/misc/vaxvms/...
wuarchive.wustl.edu:/mirrors/msdos/zip/...
wuarchive.wustl.edu:/mirrors/msdos/windows3/...
ftp.informatik.tu-muenchen.de:/pub/comp/os/os2/archiver/...
sumex-aim.stanford.edu:/info-mac/util/mac-unzip-50pl.hqx
The directories containing the compressed ACES files also contain a plain-text file called "README". This file describes the compressed files and their contents. If the user does not have sufficient disk space to store all the ACES files, he/she may wish to download only those files needed for various steps. For example, one obtains all of the files necessary for the entire Pretest process by downloading and unzipping only the "support.zip" file. It contains all of the files needed for creation and initial testing of the ACES including the performance test files from the Sort subgroup of the Classical test group (cl_so*).
The compressed files are set up to create a subdirectory tree whose top level is named "aces", and to place files in the appropriate subdirectories. This subdirectory tree is depicted in Figure 2-1. Files needed for more than one purpose are duplicated. When "qlook.zip" is unzipped, it creates the "aces/qlook" directory. One of the files placed in this directory is named "ql_work.zip". This is a compressed file containing all the files necessary for compiling and executing the Quick-Look version of the Setup program (discussed in Section 8). Likewise, when "support.zip" is unzipped, the "aces/working" directory will contain a compressed file, "zp_work.zip", containing all the files necessary for compiling and executing the full testing version of the Setup program (discussed in Section 3).
The "aces" distribution directory should be protected against modification. The testing procedures described in this document (and the software that supports these procedures) assume that the user has established both "source" and "working" directories separate from the "distribution" directory. The software copies needed files from the "aces" distribution directory into the source directory, where they may be modified if required.
Figure 2-1 Distribution Directory Structure
Before beginning performance testing or assessor execution, the user must perform some preliminary activities known as the Pretest phase. The Pretest is designed to achieve the following goals:
(a) Create command scripts for performing basic operations such as invoking the compiler with various option combinations, invoking the linker, and removing units from the library;
(b) Compile support units that will be used throughout the evaluation process; and
(c) Test these support units.
The first task in the setup process is to build the SETUP executable program. The SETUP program queries the user for basic information concerning operating system commands, compiler and linker commands, and various conventions for command script files. Using this information, SETUP creates the command scripts described in (a) above. SETUP also creates command scripts for the Pretest steps. The user executes these command scripts, captures the results, and verifies the expected results. The user must occasionally make certain decisions based on the results.
The ACES user must be familiar with the commands that are used by the implementation's operating system(s) and the Ada compilation system. The SETUP program has predefined commands for a variety of compilers and operating systems. In addition, it provides some examples.
Should the user need a manual method for running the ACES, the symbols and commands are further explained in Appendix A of the ACES User's Guide. This process many need to be used if major complications arise that would prevent the use of the automated Setup process.
This setup section makes references to three different categories of directories, "distribution", "source", and "working". The "distribution" category refers to the various directories in which the original ACES files reside. These files must not be modified, and are kept separate for this reason. The "source" category refers to one specific directory into which copies of the "distribution" files are placed. These files are occasionally modified by the user. Therefore, it is necessary to keep this directory separate from the "distribution" directories. The "working" category refers to one specific directory from which programs are executed and into which the system generated files are placed. The system often makes copies of the "source" files and uses these "working" copies for the actual processing. The files in the working directory are often modified and deleted by the system. Therefore, it is necessary to keep this directory separate from the "distribution" and "source" directories. The user must specify a path name for both the "distribution" directory and the "source" directory while running the SETUP program (below).
The following sections discuss the use of SETUP and provide the details needed for each Pretest step.
At the beginning of each of the Pretest step sections is a list of the files that must be in the specified source and working directories in order for the step to run properly. The command scripts will automatically copy most files from the source directory into the working directory (exceptions are noted when they occur). The command scripts are set up to delete files from the working directory, and library units from the Ada library when they are no longer necessary for the Pretest.
The source code for the SETUP program is contained in the two distribution files "zp_setup.ada", and "zp_defs.ada". The program requires several input files and produces several output files.
Perform the following steps to prepare SETUP for execution:
(a) Create an Ada library and identify it as the current Ada library;
(b) Unzip the file "zp_work.zip" into the working directory. This file can be found in either the "support.zip" file or in the DISTRIBUTION directory "aces/support". The following files should be in the working directory:
zc_adack.tpl zc_adaho.tpl zc_adano.tpl zc_adaop.tpl zc_adasp.tpl zc_cmpck.tpl zc_cmpno.tpl zc_cmpop.tpl zc_cmpsp.tpl zc_cpada.tpl zc_cpyck.tpl zc_cpyno.tpl zc_cpyop.tpl zc_cpysp.tpl zc_delbo.tpl zc_delsb.tpl zc_delso.tpl zc_dlada.tpl zc_dlexc.tpl zc_dlhex.tpl zc_holi.tpl zc_horun.tpl zc_linho.tpl zc_link.tpl zc_link1.tpl zc_link2.tpl zc_lnk.tpl zc_time1.tpl zc_time2.tpl zc_run.tpl zc_setli.tpl zg_incld.tpl zh_usr.tpl zp_defs.ada zp_envrn.def zp_envrn.dgn zp_envrn.dns zp_envrn.dth zp_envrn.ngn zp_envrn.nth zp_envrn.nvr zp_envrn.udf zp_envrn.ugn zp_envrn.uns zp_envrn.uth zp_envrn.uvr zp_envrn.vns zp_envrn.vva zp_quest.txt zp_scrnm.txt zp_setup.ada zp_stp00.tpl zp_stp01.tpl zp_stp02.tpl zp_stp03.tpl zp_stp04.tpl zp_stp05.tpl zp_stp06.tpl zp_stp07.tpl zp_stp08.tpl zp_stp09.tpl zp_stp10.tpl zp_stp11.tpl zp_stp12.tpl zp_stp13.tpl zp_stp14.tpl zp_stp15.tpl
To make the Setup process faster and less cumbersome, various default files have been provided. The Setup program will access these files at the user's request during the initial question phase of the program.
Predefined information for your operating system/compiler implementation can be found in the following files. If yours is not listed you can edit one of the existing files for your use. The file "zp_envrn.udf" contains the symbols, but none of the values for Setup. It is provided for your convenience.
zp_envrn.dgn DOS/GNAT
zp_envrn.dns DOS/non-specified compiler
zp_envrn.dth DOS/Thomsoft (Alsys)
zp_envrn.ngn non-specified operating system/GNAT
zp_envrn.nth non-specified operating system/Thomsoft (Alsys)
zp_envrn.nvr non-specified operating system/Rational (Verdix VADS)
zp_envrn.udf user defined operating system and compiler
zp_envrn.ugn UNIX/GNAT
zp_envrn.uns UNIX/non-specified compiler
zp_envrn.uth UNIX/Thomsoft (Alsys)
zp_envrn.uvr UNIX/Rational (Verdix VADS)
zp_envrn.vns VMS/non-specified compiler
zp_envrn.vva VMS/Vax-Ada
The "zp_envrn.???" files contain defaults for various compiler/operating system combinations. The file "zp_envrn.udf" is a file that can be customized by the user for an implementation for which there is no default. The file "zp_envrn.def" contains general defaults that can be used by all systems when running a simple version of Setup. It is suggested for first time users.
(c) If necessary, rename "zp_setup.ada" and "zp_defs.ada" to meet any naming requirements of the implementation;
(d) Compile "zp_setup.ada";
(e) Compile "zp_defs.ada"; and
(f) Link the main program "SETUP" to form an executable.
Before executing SETUP, the user is advised to consider the choice of measurement techniques and mathematical function options.
On choosing measurement options:
There are four classes of choices that must be made during the execution of the Setup program, relating to the execution time measurement technique, the compilation and linking time measurement techniques, the code size measurement techniques, and the implementation of the mathematical functions. The following paragraphs discuss these choices.
(a) Execution time measurement: Elapsed or CPU
Execution time may be measured using elapsed (wall-clock) time or CPU time. Elapsed time measurements make use of the "Calendar.Clock" function. If the target (or self-targetted) system has more than one user or has many daemons contending for the processor, using elapsed time may result in many unstable measurements. In this case, CPU time measurements may be better. To obtain CPU time measurements, the user must modify the file (in the source directory) "zg_cpu.ada", so that the function "zg_cpu" returns accumulated CPU time (on the target system) as a value of type Duration. (This function must apply to the target system.)
(b) Compilation and linking time measurement: Elapsed or CPU
Again, these measurements may use "Calendar.Clock", representing elapsed time. If the user wishes to use CPU time, she/he must provide a host-based CPU time function. For a self-hosted system, this function must be in a file named "zg_cpu.ada". For a cross-compilation system, the required file name is "zg_hocpu.ada". In Pretest Step 4, a test is made to determine whether the user-defined "zg_cpu" function is appropriate for measuring compilation and link time.
NOTE: If elapsed time is chosen for execution-time measurements, then elapsed time MUST also be used for compilation and linking time.
(c) Code size measurement: Label'Address or Get Address
The user should initially choose the Label'Address method of measuring code size. If the result of Pretest Step 2 shows that this technique is not appropriate for the system being testing, then the answer to the corresponding Setup question should be changed (by re-running Setup and choosing Option One). In this case, the user must provide a Get Address function ("zg_getad") that returns the address from which the function was called. If code size measurements are not desired, this function may return a constant value, and all sizes will be reported as zero.
(d) Mathematical functions
There are several options for the mathematical functions required by the performance tests and the support programs. When running Setup, you will be asked questions that determine which option is used.
1. Ada 95: If the implementation is for Ada 95, then the mathematical functions required by the standard will be used, so no choices are necessary.
2. NUMWG: If an Ada 83 implementation supports the Generic_Elementary_Functions package recommended by the Numerics Working Group (NUMWG), then this packages should be used. Choose the NUMWG option. In the case of a cross-compilation system, this option may be chosen for either the target or the host, or both systems.
3. VENDOR: It is possible that an Ada 83 implementation supports all the functionality of the NUMWG recommendation, but does not use the same package specification, choose the VENDOR option. (In the cross-compiler case, the VENDOR option must be chosen for both the target and host system, or for neither.) In this case, "call-through" package files, "zm_math.ada" and "zm_dblma.ada" must be created. The files "zm_math.dec" and "zm_dblma.dec" are provided as examples (based on the Digital Equipment Corporation (DEC) VAX-Ada compiler).
4. PORTABLE: If an Ada 83 implementation supports neither the NUMWG nor the VENDOR option, then choose the PORTABLE option. In this case, the ACES mathematical packages (that should be portable to most implementations) are used. In the case of a cross-compilation system, it is possible to choose the PORTABLE option for either the target or the host, or both systems.
Run the SETUP executable. It first looks for the existence of the file "zp_envrn.txt". This file will not exist the first time that SETUP is executed from any given working directory. One of the results of running SETUP is the building of the "zp_envrn.txt" file. This file contains the questions and answers for a series of symbols that define a particular implementation's environment. If the file does not exist, the user is led through a series of questions that identify the implementation's environment. If the file does exist, the user is given the following options:
1. Modify some of the existing answers;
2. Use the existing values and regenerate the scripts;
3. Answer all the questions all over again.
Two special cases need to be discussed. Once the user determines that a compiler is either a cross compiler or a self-hosted compiler, the environment file is locked into that decision. The same is true for the Ada 83 vs. Ada 95 implementation question. The answers to these two questions determine another level of questions that are appropriate for the selections made. In order to change the values of either cross vs. self-hosted or Ada 83 vs. Ada 95, the user needs to rerun the SETUP executable and select Option 3. (Answer all questions.)
As described above, responses are stored in the file "zp_envrn.txt". This file may be edited to correct any mistakes, by rerunning Setup and Selecting Option 1, "modify some of the existing answers". After running SETUP, the following files can be deleted from the working directory:
zc_*.tpl all zp_envrn.* files OTHER THAN zp_envrn.txt
zp_quest.txt
zp_scrnm.txt
zp_setup.ada
zp_stp*.tpl
zh_usr.tpl
If SETUP ever needs to be run again, ensure that the files discussed in Section 3.1.1(b) are available in the working directory.
Run command scripts in batch mode (via the "submit" command) except when instructed to run a command script interactively. Use the "/log = [ ] log file name" notation on the "submit" command line to ensure that the log file is written in the working directory.
When SETUP prompts for a "special first line" respond with "SET NOON" so that the command scripts continue to run even if errors occur.
Be reminded that when submitting a batch job, the system first runs the "login.com" file, so it is necessary to set the default directory to the working directory. Issue the following commands in the "login.com" file:
$ if F$MODE ( ) .eq. "BATCH" then
$ SET DEFAULT working
Many Ada compilers for UNIX require that Ada source code files have the suffix ".a". If manually copying ".ada" files into the working directory, be sure to rename them appropriately. The step files will do this automatically.
Since there are both script and executable files "zg_incld", it is necessary to specify a file name suffix for script files. Since the ACES - distributed scripts use ".unx", this is a good choice.
The command scripts generated by SETUP and the Harness program will not normally be executable, until the UNIX "chmod" command is run to make these scripts executable. Note that the scripts supplied with the ACES have been tested under the C shell.
Scripts may be run in foreground, background, or batch mode. In any case, all of the output should be directed to a log file. For example, "zp_stp01.unx >& zp_stp01.log &", is a UNIX command to execute "zp_stp01.unx" in the background and write output to a log file named "zp_stp01.log".
In order to capture test output, use the ">>PRETEST.LOG" as suggested in the appropriate question asked by SETUP. To keep separate log files for separate steps, rename PRETEST.LOG appropriately after running each step.
For the Alsys compiler, when entering compile and link options as part of a batch command, do not depend on default values. Explicitly add each option and value. For example: when asked for the command to compile without optimization, include it as part of the command even though no optimization is the default.
The default environment files for GNAT contain invocations of three non-standard scripts, "gnat-comp", "gnat-compop", and "remove-unit". The GNAT user will have to replace these commands or supply appropriately named scripts. The purposes of these scripts are as follows:
gnat-comp Apply GNATCHOP and execute the resulting script to compile the file named by the parameter, using the default gcc options.
gnat-compop Apply GNATCHOP and execute the resulting script to compile the file named by the parameter, using a gcc option to optimize.
remove-unit Remove the specification and body of the unit named by the parameter from the "library".
Figure 3-1, 3-2, and 3-3 contain UNIX versions of these scripts, as used by the ACES development team. Note that the gnat-comp script (Figure 3-1) assumes that the file named in the parameter has no suffix.
#!/bin/csh
#
# Compile without optimization
echo Compiling $1
gnatchop -s -w $1
$1.sh
unalias rm
rm -f $1.sh
Figure 3-1 Sample "gnat-comp" script
#!/bin/csh
#
# Compile with optimization
echo Compiling $1
gnatchop -s -w $1
$1.sh -02
unalias rm
rm -f $1.sh
Figure 3-2 Sample "gnat-compop" script
#!/bin/csh
unalias rm
rm -f $1*.ads
rm -f $1*.adb
rm -f $1*.ali
rm -f $1*.o
Figure 3-3 Sample "remove-unit" script
The pretest subsections below follow the general scenario described here. Note that not all subsections are needed in all steps.
STEP NUMBER: [SECTION NUMBER] [STEP NUMBER] [STEP TITLE/NAME]
DESCRIPTION: What this step is designed to accomplish.
PREREQUISITE: Which steps or what other actions must have been performed before this step can be successfully executed.
REQUIRED/OPTIONAL: The Pretest steps are designed to be run sequentially, but occasionally some steps are not required. This section explains the status of this step.
FILES NEEDED IN SOURCE DIRECTORY: This should have been accomplished when executing Step 0.
FILES NEEDED IN WORKING DIRECTORY: Should have been accomplished by a previous, required step. When rerunning Setup, verify that needed files were not deleted by subsequent steps.
PREPARATION: Tasks that must be done and issues that need to be decided before running the step file.
EXECUTION: Actually issuing the command to run the step file and sending output to a log file. Note that each step may not produce a log file with each implementation. See Sections 3.1.3.1, 3.1.3.2, and 3.1.3.3 for suggestions on generating log files.
RESULTS EXPECTED: What you should find in the log file, on the screen or in the working directory (source directory for Step 0). If the results do not match the description of the expected results, see the discussion of the corresponding Pretest step in the User's Guide.
FOLLOW-UP NEEDED: Decisions that need to be made or actions that need to be taken as a result of that step's output.
FILES NO LONGER NEEDED IN WORKING DIRECTORY: Files that may be deleted from the working directory to make it less crowded. Remember that these files may be needed if Setup is rerun.
DESCRIPTION: This step will automatically copy files from the "distribution" to the "source" directory.
PREREQUISITE: Compiling, linking, and executing Setup.
REQUIRED/OPTIONAL: If this cannot be used for a specific implementation, either edit the command file to meet user needs or manually copy all of the files named in "zp_stp00.tpl" into the "source" directory.
FILES NEEDED IN SOURCE DIRECTORY: None.
FILES NEEDED IN WORKING DIRECTORY: zp_stp00.(command script)
PREPARATION: None.
EXECUTION: Run the file.
RESULTS EXPECTED: Copies of all of the files that are in either "support.zip" or "/aces/support/" should now be in "source".
FOLLOW-UP NEEDED: Verify that the files were copied.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp00 (command file)
zp_stp00 (log file)
DESCRIPTION: This step compiles two global packages into the library. It also compiles, links, and executes a simple program that tests the ability of Text_IO to output strings, integers, and floating-point values.
PREREQUISITE: Low-level command scripts must have been generated by SETUP and Step 0 must have been executed.
REQUIRED/OPTIONAL: This step MUST be completed before performance testing can be attempted.
FILES NEEDED IN SOURCE DIRECTORY:
zg_glob1.ada zp_talk.ada
zg_glob2.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp01 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp01" command script and send the results to a log file.
RESULTS EXPECTED: A log file with the following information:
Display floating point numbers 1.50000E+01 1.500E+1 15.0
Display floating point numbers -1.23450E+04 -1.235E+4 -12345.000
Display integers 100 22222
FOLLOW-UP NEEDED: Verify results.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp01 (command script)
zp_stp01 (log file)
DESCRIPTION: This step determines whether applying the 'ADDRESS attribute to a label can be used to measure code size for the implementation being assessed.
PREREQUISITE: Step 1 must have been completed.
REQUIRED/OPTIONAL: If the user already knows whether label'Address is supported, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_label.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp02 (command script)
PREPARATION: A subprogram in file "zp_label.ada" may require modification if the target system's type System.ADDRESS is not suitable for unchecked-conversion to a 32-bit integer type. If this situation arises, modify the function "pkg01_zp_label.Subtract_Address", which calculates the difference between two addresses and expresses the result as a floating point value.
EXECUTION: Execute the "zp_stp02" command script and send the results to a log file.
RESULTS EXPECTED: The program "zp_label" should have been compiled, linked, and executed, and one of the following results generated:
1. "Label'ADDRESS attribute works correctly
SIZE IN 8 BIT BYTES = (some number in the range 4 to 16)";
2. "The difference between two label'ADDRESSes is outside the expected range of 4 to 16 bytes. Please compare this result with a system map or machine code listing to verify accuracy. SIZE in 8 BIT BYTES = (number)"; or
3. "zp_label.ada" does not compile.
FOLLOW-UP NEEDED:
1. Verify the expected results.
In the first case, the ACES can use the default (and preferred) method of measuring code size (i.e., subtracting the results of applying the 'ADDRESS attribute to statement labels).
In the second and third cases, the ACES cannot use the default method of measuring code size. In these cases, the user must decide whether to provide an alternate method of measuring code size, as described below, or choose not to measure code size.
To provide an alternate method of measuring code size, the user must define the function "zg_getad". This function must return the address from which it was called. The Ada declaration of this function appears in the files "zg_glob3.elg" and "zg_glob3.cpg", one of which will be used in Step 5. In these files, it is assumed that the function's body is written in assembly code, and that a "pragma interface" and a "pragma import_function" are used to reference the body. Note that the use (or even the existence) of such a pragma is implementation dependent, and that modification may be necessary.
A sample function (for VAX systems) is given in the file "zg_getad.mar". In Step 5, it may be necessary to modify the pragmas in the "zg_glob3" file, and it may be necessary to modify the command script for Step 5 to account for translating the function body and making it available to the Ada library.
In order to understand the implementation's interface to assembly language routines, we recommend that a simple Ada function be compiled with the option to generate embedded assembly code. This code can serve as a starting point for writing the "zg_getad" function.
If the user chooses not to measure code size, there is another choice to be made. If zp_label reports that the size is 0, then the user may continue to use the default method, and all sizes will be reported as 0. If "zp_label.ada" fails to compile, then the user must define the function "zg_getad" such that it always returns the same address. This will result in all sizes being reported as 0.
2. Record the method of measuring code size, as follows:
During SETUP the user is asked to identify the code measurement method. The results of this step either confirm or invalidate the method declared. If this test step confirms the user's selection, the user can continue. If the declared method is invalid, the user must rerun SETUP and modify the value. To do this run the SETUP executable. Select Option one (i.e., modify the answers to some of the existing answers). Identify the question for Code Measurement Technique. Modify this answer appropriately.
The answer to this question is used as part of one or more file names. If your operating system is case-sensitive, you must use the same case as is used for the names of your source code files.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp02 (command file)
zp_stp02 (log file)
DESCRIPTION: This step compiles units that provide checks on the behavior of the CPU time function or the Calendar.Clock function, depending on whether CPU time or elapsed time is to be measured. The actual checking is performed in Step 4.
PREREQUISITE: Step 1 must have been completed.
REQUIRED/OPTIONAL: If CPU time measurements are NOT being used and the user knows that Calendar.Clock is accurate, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
Required for all cases:
zp_tcal1.ada
zp_tcal2.ada
Required for CPU time measurement cases:
zg_cpu.ada
zg_hocpu.ada (if you are using a cross-compiler)
zp_tcal3.ada
zp_tcal4.ada (includes source for zp_tcal5 also) only if CPU is used for compilation time)
FILES NEEDED IN WORKING DIRECTORY:
zp_stp03 (command script)
PREPARATION: The first task in this step is to decide whether to measure elapsed time or CPU time (see User's Guide, Section 5.1.2 for some discussion of the issues). If CPU time is to be used, a function that reports CPU time as a value of type DURATION must be supplied. For cross-compilers, it may be necessary to supply two functions, one for the host system and one for the target system. There is a VAX-Ada example in the file "zg_cpu.dec". (Note: This file "zg_cpu.dec" can be copied into your subdirectory and renamed, as needed for use on your system.) The file name for a self-hosted system must be "zg_cpu.ada". For a cross-compilation system, the file name for the host system must be "zg_hocpu.ada" and the file name for the target system must be "zg_cpu.ada".
If CPU time measurements are being used, the command script generated for this step is not complete. The user needs to insert commands to compile or assemble non-Ada units required by zg_cpu and to link programs zp_tcal3, zp_tcal4, and zp_tcal5. See User's Guide, Section 5.1.3 and the zp_stp03 command script file for more information.
When SETUP was executed, the user was asked to identify both the method used to measure compilation time and the method to measure execution time. The user was asked whether CPU (cp) time or elapsed wall-clock time (el) was to be used for each. These letters (i.e., the answers "cp", "el") are used as part of one or more file names. If your operating system is case-sensitive, you must use the same case as is used for the names of your source code files.
NOTE THAT IT IS NOT PERMISSIBLE TO MEASURE COMPILATION AS CPU TIME WHEN MEASURING EXECUTION WITH ELAPSED TIME.
If either compilation or execution time is to be measured in CPU time, modify the "zp_stp03" command script to account for compiling or assembling non-Ada code. It may also be necessary to edit the "link" command scripts (zc_lnk, zc_link, zc_link1, zc_link2, zc_linho, and zc_linkd) to account for linking any non-Ada units used in this process.
EXECUTION: Execute the "zp_stp03" command script.
RESULTS EXPECTED:
Executables in working directory:
zp_tcal1
zp_tcal2
Additional executable files expected in the working directory if using CPU time:
zp_tcal3
zp_tcal4
zp_tcal5
FOLLOW-UP NEEDED:
Verify the following:
Confirm that zp_tcal1 and zp_tcal2 were successfully compiled and linked for running on both the host and the target (or the common host/target).
If time is to be measured in CPU time, verify that zp_tcal3, zp_tcal4, and zp_tcal5 were successfully compiled and linked for running on the host (or the common host/target).
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp03 (command script)
zp_stp03 (log file)
DESCRIPTION: This step checks the accuracy of the CPU time measurement functions or the elapsed time measurement functions, depending on the method chosen for measuring compilation and execution time.
PREREQUISITE: Steps 1 and 3 must have been completed.
REQUIRED/OPTIONAL: If the user already knows that the selected timing functions are accurate, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY: Those copied in Step 0.
FILES NEEDED IN WORKING DIRECTORY:
zp_tcal1 executable
zp_tcal2 executable
zp_tcal3 executable (only if using CPU time)
zp_tcal4 executable (only if using CPU time)
zp_tcal5 executable (only if using CPU time)
zp_stp04 (command script)
PREPARATION:
Identify the paragraph below that describes the system being assessed and run the tests as described.
* Self-hosted systems:
+ Measuring elapsed time:
- Run zp_tcal1 and zp_tcal2 interactively, using a stop-watch to check the timings.
+ Measuring CPU time:
- Run zp_tcal1 and zp_tcal2 interactively, using a stop-watch to check the timings.
- Run zp_tcal3 interactively, preferably as the only process running on the system.
- If the system supports concurrent jobs, run two copies of zp_tcal3 at the same time. For UNIX systems, start one copy in the background using the "&" and immediately start another copy in the foreground. For VMS systems, start one copy in the background using the SPAWN/NOWAIT command and immediately start another copy in the foreground.
* Cross-compilation systems:
+ Measuring elapsed time:
- Run zp_tcal1 and zp_tcal2 interactively on the host system, using a stop-watch to check the timings.
- Run zp_tcal1 and zp_tcal2 interactively on the target system, using a stop-watch to check the timings.
+ Measuring CPU time:
- Run zp_tcal1 and zp_tcal2 interactively on the host system, using a stop-watch to check the timings.
- Run zp_tcal3 interactively on the host system, preferably as the only process running on the system.
- If the host system supports concurrent jobs, run two copies of zp_tcal3 on the host system at the same time. For UNIX systems, start one copy in the background using the "&" and immediately start another copy in the foreground. For VMS systems, start one copy in the background using the SPAWN/NOWAIT command and immediately start another copy in the foreground.
- Run zp_tcal1 and zp_tcal2 interactively on the target system, using a stop-watch to check the timings.
- Run zp_tcal3 interactively on the target system, preferably as the only process running on the system.
- If the target system supports concurrent jobs, run two copies of zp_tcal3 on the target at the same time (as suggested above).
EXECUTION: Execute the "zp_stp04" command script.
Note that this script does nothing if elapsed time measurements have been chosen for execution timings.
RESULTS EXPECTED:
For zp_tcal1 and zp_tcal2, a message should be displayed every 60 seconds for 15 minutes.
For zp_tcal3, the execution as the only process should report CPU time that is less than, but near, the elapsed time. During the second execution, each copy should report CPU time that is roughly half of that reported during the first execution.
For zp_tcal4 and zp_tcal5, see the following section.
FOLLOW-UP NEEDED:
For zp_tcal1 and zp_tcal2, a message should be displayed every 60 seconds for 15 minutes. If the intervals between messages vary noticeably, then Calendar.Clock is not reliable. See the User's Guide, Section 5.1.3.
For zp_tcal3, the execution as the only process should report CPU time that is less than, but near, the elapsed time. During the second execution, each copy should report CPU time that is roughly half of that reported during the first execution.
When the zp_stp04 command script is executed, it runs two programs: zp_tcal4 and zp_tcal5. The results from these programs are used to determine whether the CPU time function returns a value relative to the beginning of the process or relative to the beginning of the program. For more details see the User's Guide Section 5.1.3.
If the results of zp_tcal5 show that the CPU time function returns PROGRAM time (which is NOT appropriate for measuring compilation and linking time), then one option is to measure compilation and linking with elapsed time, rather than CPU time. If this option is chosen, rerun SETUP and modify the value of the answer. The other option is to measure compilation and linking with CPU time. If this option is chosen, it is necessary for the user to provide a different zg_cpu function that somehow retrieves process time.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp04 (command script) zp_tcal1 (executable)
zp_stp04 (log file) zp_tcal2 (executable)
zp_tcal3 (executable)
zp_tcal4 (executable)
DESCRIPTION: This step compiles the remaining global packages, a random number generator package, and a mathematics package. It also creates executables for producing baseline timing data, for including timing loop code into the distributed performance test sources, and for creating compilation and linking time stamps.
PREREQUISITE: Step 1 must have been completed.
REQUIRED/OPTIONAL: This step MUST be executed before performance testing can begin.
FILES NEEDED IN SOURCE DIRECTORY: The answers given by the user while running Setup put file names in the zp_stp00 command script to be copied to the "source" directory. Examine each of the following descriptions to ensure that the correct files, as described below were copied into "source".
1. For ALL cases:
zg_glob4.ada zg_glob5.ada zg_glob6.ada zm_math.ada zm_dblma.ada zc_data.ada
(Note: That is the VENDOR option was chosen, then the "zm_math.ada" and "zm_dblma.ada" files are the "call-through" packages discussed in Section 3.1.1.)
2. Random number generator (select one of the following):
If the implementation supports the declaration of 32-bit integer types:
zm_ran32.ada
If the implementation does NOT support the declaration of 32-bit integer types:
zm_ran16.ada
3. Math packages (select files if needed):
a. Ada 95 standard, the following files are required:
zm_math.a95 zm_dblma.a95
b. Ada 83 implementation:
1) If the PORTABLE option was chosen for a self-hosted system or for either HOST or TARGET of a cross-compilation system, then the following files are required:
zm_math.por zm_depen.por zm_genma.ada
2) If the PORTABLE option was chosen for a self-hosted system or for the TARGET or a cross-compilation system, then the following file is required:
zm_dblma.por
4. Cross/self-hosted compiler (select one of the following):
a. If the implementation is CROSS-compilation system, then the following file is required:
zg_iniso.cro
b. If the implementation is a SELF-hosted system, then the following file is required:
zg_iniso.sel
5. Measuring execution time and code size (select one of the following groups):
a. execution time: CPU
code size: Label'Address
The following files are required:
b. execution time: CPU
code size: Get Address
The following files are required:
zg_glob3.cpg zg_init.cpg zg_incld.cpg zc_time1.cp
zc_time2.cp zc_link1.cp zc_link2.cp
c. execution time: ELAPSED
code size: Label'Address
The following files are required:
zg_glob3.ell zg_init.ell zg_incld.ell zc_time1.el
zc_time2.el zc_link1.el zc_link2.el
d. execution time: ELAPSED
code size: Get Address
The following files are required:
zg_glob3.elg zg_init.elg zg_incld.elg zc_time1.el
zc_time2.el zc_link1.el zc_link2.el
6. If ELAPSED time measurements were chosen for COMPILATION and link time, then the following files are required:
zc_time1.el zc_time2.el zc_link1.el zc_link2.el
FILES NEEDED IN WORKING DIRECTORY:
zp_stp05 (command script)
PREPARATION: If some method other than label'Address is being used to measure code size (i.e., option "g"), then modify the "zp_stp05" command script to account for translating and linking the "zg_getad" units. See the comments in the zp_stp05 command script file for hints on how to do this.
EXECUTION: Execute the "zp_stp05" command script and send the results to a log file.
RESULTS EXPECTED:
For all CROSS-COMPILERs, the HOST library should contain the following units:
For a CROSS-COMPILER using the PORTABLE option for the HOST, the HOST library should contain the following unit:
zm_math.math_dep
* Note that this is a subunit, and may not be listed separately.
For all CROSS-COMPILERs, the TARGET library should contain the following units:
proc_spoil_glob4_variables zg_glob1 zm_math
zg_glob2 zm_ran
zg_glob3
zg_glob4
zg_glob5
zg_glob6
For a CROSS-COMPILER using the PORTABLE option for the TARGET, the TARGET library should contain the following units:
mathematical_exceptions zm_dblma.math_dep
zm_depen
zm_genma
zm_math.math_dep
* Note that the two ".math_dep" subunits may not be listed separately.
For all SELF-HOSTED COMPILERS the library should contain the following units:
proc_spoil_glob4_variables zg_glob1 zm_math
zg_glob2 zm_ran
zg_glob3
zg_glob4
zg_glob5
zg_glob6
For SELF-HOSTED COMPILERS using the PORTABLE option, the library should contain the following units:
mathematical_exceptions zm_dblma.math_dep
zm_depen
zm_genma
zm_math.math_dep
* Note that the two subunits ".math_dep" may not be listed separately.
The following executable to be run on the host should have been created in the working directory:
zc_link1 zc_time1 zg_incld
zc_link2 zc_time2
FOLLOW-UP NEEDED: If the implementation is a CROSS-COMPILER, read through the log file looking for instructions on how to extract the following files from the log files. These files are used when executing performance tests to check the consistency of timing measurements.
zg_cpy.op zg_cpy.no zg_cpy.ck zg_cpy.sp
If the implementation is a self-hosted compiler, verify that the following files were created:
zg_cpy.op zg_cpy.no zg_cpy.ck zg_cpy.sp
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp05 (command script)
zp_stp05 (log file)
DESCRIPTION: This step helps to determine the minimum number of times each test must be executed to eliminate the effects of clock "jitter" and other "noise" on the system. As a consequence of this step, it may be necessary to modify and re-compile the body of the global package "zg_glob3".
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user wishes to use the default minimum numbers, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY: The appropriate copy of the following file should be in the source directory:
zp_basic.??? The suffix of this file is either ".ell", ".elg", ".cpl", or ".cpg".
FILES NEEDED IN WORKING DIRECTORY: zp_stp06 (command script)
PREPARATION: Determine that the appropriate version at zp_basic is in the source directory. Verify it against the options selected in Setup and described in Section 3.1.1 and 3.2.5 (step 5).
EXECUTION: Execute the "zp_stp06" command script and send the results to a log file.
RESULTS EXPECTED: A log file with the results of the execution of the script.
FOLLOW-UP NEEDED:
Study the output of the command script. It may recommend changing the value of BASIC_ITERATION_COUNT. Normally, the default value of 2**17-1 is acceptable. If change appears necessary, follow the procedure below:
1. Get a copy of the version of zg_glob3 that was used in Step 5. Extract the body of package zg_glob3 into another file. Locate the beginning of the package body's statement-part. These are the first two lines:
BEGIN
excessive_time := 1800.0; -- 30 minutes
Immediately after the BEGIN, add statements assigning the appropriate values (see below) to BASIC_ITERATION_COUNT_LOWER and BASIC_ITERATION_COUNT_UPPER.
Suppose the desired value of BASIC_ITERATION_COUNT is 2**N - 1.
If N <= 14 then use
BASIC_ITERATION_COUNT_LOWER := 2**N - 1;
BASIC_ITERATION_COUNT_UPPER := 0;
otherwise, use
BASIC_ITERATION_COUNT_LOWER := 2**14 - 1;
BASIC_ITERATION_COUNT_UPPER := 2**(N - 14) - 1;
(Do not use "N"; instead, replace "N" with the value of the exponent as recommended in the output.)
2. Compile the body (only) of zg_glob3, as modified.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp06 (command script)
zp_stp06 (log file)
The source file (if any) containing the revised body of zg_glob3.
DESCRIPTION: This step tests the functionality of the PORTABLE version of the mathematics package.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: This step is optional, depending on the versions of "zm_math" used in Step 5. If the PORTABLE option was chosen for either the host or the target, this step should be performed to verify that the package "zm_depen" works properly on that system. If the user is already satisfied that the package works properly, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_dptst.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp07 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp07" command script and read the results to a log file.
RESULTS EXPECTED: Log file.
FOLLOW-UP NEEDED: Examine the output. Discrepancies are flagged with the string "<<<ERROR>>>". See User's Guide Section 5.1.6.2 for a discussion of errors output by the "zp_dptst" file. If it appears that too many errors occur, then re-do Step 5 with a different math option.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp07 (command script)
DESCRIPTION: This step tests the accuracy of the mathematical functions provided by the single precision math package.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied with the accuracy of the single precision math function, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_mtasn.ada zp_mtexp.ada zp_mtsin.ada
zp_mtatn.ada zp_mtln.ada zp_mtsqr.ada
zp_mtchn.ada zp_mtpwr.ada zp_mttan.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp08 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp08" command script and send the results to a log file.
RESULTS EXPECTED: Log file.
FOLLOW-UP NEEDED: Verify that the accuracy is sufficient. Errors are indicated in the log file with the string ">>> error". If there are several errors, it may be necessary to return to Step 5 and choose one of the other mathematics package options.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp08 (command script)
zp_stp08 (log file)
DESCRIPTION: This step tests the accuracy of the mathematical functions provided by the double precision math package.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied with the double precision math function, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_dmchn.ada zp_dmexp.ada zp_dmsin.ada
zp_dmasn.ada zp_dmln.ada zp_dmsqr.ada
zp_dmatn.ada zp_dmpwr.ada zp_dmtan.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp09 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp09" command script and send the results to a log file.
RESULTS EXPECTED: Log file.
FOLLOW-UP NEEDED: Verify that the accuracy is sufficient. Errors are indicated in the log file with the string ">>> error". If there are several errors, it may be necessary to return to Step 5 and choose one of the other mathematics package options.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp09 (command script)
zp_stp09 (log file)
DESCRIPTION: This step tests the executable that is responsible for including timing loop code into the distributed performance test sources.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user has previously verified that the Include preprocessor is working correctly, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_intst.inc
One version of each of the following:
zg_stop2.### zp_intst.###
The suffix for these files is either ".ell", ".elg", ".cpl", or ".cpg".
FILES NEEDED IN WORKING DIRECTORY: zp_stp10 (command script)
PREPARATION: Determine that the correct files, appropriate for the options selected in Setup and described in Sections 3.1.1 and 3.2.5 (step 5), were copied into source.
EXECUTION: Execute the "zp_stp10" command script.
RESULTS EXPECTED:
- The working directory should contain the following files:
zp_intst (Ada source) zp_intst (".ell", ".elg", ".cpl", or ".cpg")
FOLLOW-UP NEEDED:
- Compare these two files, with an automated differencing tool or manually. There should be no differences.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_intst (Ada source) zp_stp10 (command script)
zp_intst (".ell", ".elg", ".cpl", or ".cpg") zp_stp10 (log file)
DESCRIPTION: This step compiles and tests the Harness program. The Harness program is used to create command scripts for groups of performance tests and to track their execution status. The command script file generated in this step copies the necessary files and builds the executable Harness. The user then runs Harness to create a command script for part of one performance test subgroup. Finally, the command script created by Harness runs the tests in the subgroup.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: This step MUST be completed before performance test is begun.
FILES NEEDED IN SOURCE DIRECTORY:
zh_*.lib zh_*.txt (Ada 83 Implementation)
zh_*.tpl zh_*.a95 (Ada 95 Implementation)
zh_hrn*.ada zh_prfmi.txt (Ada 83 Implementation)
cl_so*.* zh_prfmi.a95 (Ada 95 Implementation)
zh_cosys.txt
One version of each of the following files:
zg_start.### zg_stop0.### zg_stop2.### zg_verfy.###
The extension of these files is either ".ell", ".elg", ".cpl", or ".cpg".
FILES NEEDED IN WORKING DIRECTORY:
zp_stp11 (command script)
PREPARATION: Determine that the files, appropriate for the options selected in Setup and described in Sections 3.1.1 and 3.2.5 (step 5), were copied into source.
EXECUTION: Execute the "zp_stp11" command script.
RESULTS EXPECTED:
The following files should have been generated in the working directory in all cases:
One version of each of the following files:
zg_start.### zg_stop0.### zg_stop2.### zg_verfy.###
The extension of these files is either ".ell", ".elg", ".cpl", ".cpg" as appropriate for the options selected in Setup and described in Sections 3.1.1 and 3.2.5 (step 5).
zh_ap.lib zh_in.lib zh_st.txt
zh_ap.txt zh_in.txt zh_su.lib
zh_ar.lib zh_io.lib zh_su.txt
zh_ar.txt zh_io.txt zh_tk.lib
zh_cl.lib zh_ms.lib zh_tk.txt
zh_cl.txt zh_ms.txt zh_ud.lib
zh_do.lib zh_op.lib zh_ud.txt
zh_do.txt zh_op.txt zh_xh.lib
zh_dr.lib zh_po.lib zh_xh.txt
zh_dr.txt zh_po.txt zh_sy.tpl
zh_dt.lib zh_prfmi.txt zh_sy.txt
zh_dt.txt zh_sr.lib zh_sy000.tpl
zh_gn.lib zh_sr.txt zh_sy_cu.tpl
zh_gn.txt zh_st.lib harness (executable)
If you are running an Ada 95 compiler, the following files should have been generated in the working directory:
zh_oo.lib zh_oo.txt zh_pt.lib zh_pt.txt
FOLLOW-UP NEEDED:
(a) Run the "harness" executable.
1. Acknowledge any "no existing database" messages by pressing RETURN.
2. At the first screen, select by group, subgroup and tests by entering "1".
3. At the first full screen, select the Classical group (enter "3"). When the screen is refreshed, note the "+" beside group 3.
4. Change to the Subgroups screen by entering "Show Subgroups" (or "ss").
5. Bring up the next page of the Subgroups screen by entering "Next". Continue, if necessary, until the Sort subgroup (12) is listed.
6. Select all tests in the Sort subgroup by entering "12/all" or "12/*". Note that the screen now shows the number of tests chosen is equal to 14.
7. Enter "Build" to build a command script for the selected tests.
8. Enter "6" to toggle the "save/delete library units" ("DELETED" is the preferred mode during this step).
9. Note the name (item 8) of the command script file to be generated.
10. Enter "Do" to actually start the build process.
11. Enter "Quit" (or "q") to exit the harness program. Answer "Yes" to the question regarding the saving of the selection information.
More details on running the harness may be found in Section 4 of this document. Still more details are available in the User's Guide, Section 6.
(b) Run the newly created command script file (see step 9 above), capturing the output in a log file named "zp_cl_so.log".
(c) Verify the following:
The log file should show the results of compiling the cl_so* tests, linking them into 5 main programs, and executing the main programs.
There should be time stamps marked with "\aces begin\" and "\aces end\", followed by either "el" or "cp" (elapsed time or CPU time).
There should also be an execution time report for each of the 14 test problems (provided that all of them were successfully compiled and executed).
(d) Run the "harness" executable interactively.
Acknowledge any "no existing database" messages by pressing RETURN.
At the first screen, select by group, subgroup, and test by entering "1".
Enter "Update" to read the log file just generated.
Identify the log file, by entering:
1 = zp_cl_so.log
(If the log file is not in the current directory, include a directory path in the file name.)
Enter "Do" to cause the log file to be read.
(e) Verify the reported status of the tests. Enter the "Show Chosen" ("SC") command and note the status values. There should be a "*1" in each column where a status could be determined for a test. "No data available" entries indicate tests that failed somehow. If all tests report "No data available", then something has probably gone wrong. See the User's Guide, Section 6.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp11 (command script)
zp_stp11 (log file)
Keep the "zp_cl_so.log" log file for use in Step 13.
DESCRIPTION: This step compiles and links the analysis tools (Condense, Comparative Analysis, Single System Analysis, and Menu).
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: This step may be deferred until the user is ready to analyze performance test results.
FILES NEEDED IN SOURCE DIRECTORY:
za_ca01.ada za_cn06.ada za_co16.ada za_mn04.ada
za_ca02.ada za_cn07.ada za_co17.ada za_mn05.ada
za_ca03.ada za_co01.ada za_co18.ada za_mn06.ada
za_ca04.ada za_co02.ada za_co19.ada za_sa01.ada
za_ca05.ada za_co03.ada za_co20.ada za_sa02.ada
za_ca06.ada za_co04.ada za_co21.ada za_sa03.ada
za_ca07.ada za_co05.ada za_co22.ada za_sa04.ada
za_ca08.ada za_co06.ada za_co23.ada za_sa05.ada
za_ca09.ada za_co07.ada za_co24.ada za_sa06.ada
za_ca10.ada za_co08.ada za_co25.ada za_sa07.ada
za_ca11.ada za_co09.ada za_co26.ada za_sa08.ada
za_ca12.ada za_co10.ada za_co27.ada za_sa09.ada
za_ca13.ada za_co11.ada za_co28.ada za_sa10.ada
za_cn01.ada za_co12.ada za_mn01.ada za_sa11.ada
za_cn02.ada za_co13.ada za_mn02.ada za_sa12.ada
za_cn03.ada za_co14.ada za_mn03.ada za_sa13.ada
za_cn04.ada za_co15.ada
za_cn05.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp12 (command script)
PREPARATION:
There is an assumption that the host compilation system has sufficient capacity to compile and link all the units, an assumption that is known to fail for some systems. If it is not possible to do this, there are a couple of options. First, the analysis tools do not have to run on either the host or the target. They can be used on any Ada implementation that will support them, provided a mechanism exists for transferring the performance test log files to that implementation. Second, the analysis tools may be compiled and linked separately. See Section 9.1.1 of the User's Guide for details.
EXECUTION:
Execute the "zp_stp12" command script.
RESULTS EXPECTED: Menu executable.
FOLLOW-UP NEEDED:
Verify that the "menu" executable was created. If not, then the implementation may not have the capacity to compile and link all the analysis tools into one executable. In such case, see Section 9.1.1 of the User's Guide.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp12 (command script)
zp_stp12 (log file)
DESCRIPTION: This step tests the Condense tool that was built in Step 12.
PREREQUISITE: Steps 11 and 12 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied that Condense works properly (or if analysis is not being performed on this system), then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
za_cosys.txt za_cowgt.txt zp_sys_1.e00
FILES NEEDED IN WORKING DIRECTORY:
zp_cl_so.log (created in Step 11)
zp_stp13 (command script)
PREPARATION: Verify that the proper Log file exists.
EXECUTION:
Execute the "zp_stp13" command script.
RESULTS EXPECTED:
The following files should have been copied into the working directory:
za_cosys.txt za_cowgt.txt zp_sys_1.e00
FOLLOW-UP NEEDED:
(a) Run the "menu" executable interactively.
1. On the MAIN MENU screen, enter "1,4,Next" to select the Condense Option, interactively select reports, and go to the next screen.
2. On the SYSTEMS MENU screen, enter "1,Next" to select the first system (System 1), and go to the next screen.
3. On the CONDENSE: REPORT OPTIONS screen, enter "2,Next" to select the EXCEPTIONAL_DATA_REPORT, and go to the next screen.
4. On the RUN or SAVE REQUEST screen, check that the Current Selections are :
CONDENSE
System 1
EXCEPTIONAL_DATA_REPORT
Produce textual data
Append log to existing cmp database
Append log to existing exe database
5. Enter "1,Do" to start Condense.
6. Enter "Quit" to exit the "menu" executable.
(b) Verify that two "database" files ("zp_cl_so.c00" and "zp_cl_so.e00") were created. Compare the format of "zp_cl_so.e00" to that of "zp_sys_1.e00" to ensure that Condense formatted the data correctly. Note that the data will be different.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp13 (command script)
zp_stp13 (log file)
DESCRIPTION: This step tests the Comparative Analysis tool that was built in Step 12. It uses sample database files that are included in the ACES distribution.
PREREQUISITE: Steps 12 and 13 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied that Comparative Analysis works properly (or if analysis is not being performed on this system), then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
za_caari.tim za_cacla.tim zp_sys_2.e00 zp_sys_3.e00
FILES NEEDED IN WORKING DIRECTORY: zp_stp14 (command script)
The following files should have been created in Step 13.
za_cosys.txt za_cowgt.txt zp_sys_1.e00
PREPARATION: Verify that the proper files exist.
EXECUTION: Execute the "zp_stp14" command script.
RESULTS EXPECTED:
The following files should be in the working directory:
za_caari.tim zp_sys_1.e00
za_cacla.tim zp_sys_2.e00
za_cosys.txt zp_sys_3.e00
za_cowgt.txt
FOLLOW-UP NEEDED:
(a) Run the "menu" executable interactively.
1. At the MAIN MENU screen, enter "2,4,Next" to go to select the Comparative Analysis Option, interactively select reports, and go to the next screen.
2. At the SYSTEMS MENU screen, enter "2,3,4,Next" (or "2..4,Next") to select systems "sys_1," "sys_2," and "sys_3", and go to the next screen.
3. At the METRICS MENU screen, enter "1,Next" to select the Execution Time results, and go to the next screen.
4. At the GROUPS MENU - PAGE ONE OF TWO screen, enter "2,3,Next" to select the Arithmetic and Classical groups, and go to the next screen.
5. At the GROUPS MENU - PAGE TWO OF TWO screen, enter "Next" to go to the next screen.
6. At the COMPARATIVE ANALYSIS (CA): REPORTS screen, enter "1,Next" to select Group Level Reports, and go to the next screen.
7. At the CA: GROUP-LEVEL OPTIONS - FOR REPORTS OF SELECTED METRICS screen, enter "Next" to acknowledge the default selection, and go to the next screen.
8. At the eighth screen, verify that the selections are correct:
PROGRAM : COMPARATIVE ANALYSIS
SYSTEMS : sys_1, sys_2, sys_3
METRICS : EXECUTION_TIME
GROUPS : ARITHMETIC, CLASSICAL
OPTIONS : Text Reports
Group-Level Report : full
Output line length : 80
To correct any mistakes, use the "Previous" option to move to the appropriate screen where corrections can be made.
9. Enter "1,Do" to cause the analysis to be performed immediately.
10. Enter "Quit" to exit the program.
(b) Verify that the following files have been created in the working directory:
arithm00.tim classi00.tim za_cadb.txt
(c) Verify that the data in "arithm00.tim" is essentially the same as that in the file "za_caari.tim" and that the data in "classi00.tim" is essentially the same as that in "za_cacla.tim". Expect differences in dates, and there may be rounding differences in other values, but verify that the numeric values are essentially the same.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
arithm00.tim za_caari.tim zp_stp14 (command script) zp_sys_1.e00
classi00.tim za_cacla.tim zp_stp14 (log file) zp_sys_2.e00
za_cadb.txt zp_sys_3.e00
DESCRIPTION: This step tests the Single System Analysis tool that was built in Step 12. This test uses sample data provided as part of the ACES distribution.
PREREQUISITE: Steps 12 and 13 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied that Single System Analysis works properly (or if analysis is not being performed on this system), then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
za_salft.ssa za_saopt.ssa za_sasty.ssa za_sarts.ssa
za_smple.e00 za_smple.c00 zp_salft.ssa zp_smple.e00
zp_smple.toc zp_smple.rep
FILES NEEDED IN WORKING DIRECTORY:
za_cosys.txt za_cowgt.txt
PREPARATION: Verify that the proper files exist.
EXECUTION: Execute the "zp_stp15" command script.
RESULTS EXPECTED:
The following files have been created in the working directory:
za_salft.ssa za_smple.e00 zp_smple.toc zp_smple.rep
FOLLOW-UP NEEDED:
(a) Run the "menu" executable.
1. On the MAIN MENU screen, enter "3,4,Next" to select the Single System Analysis option, interactively select reports, and go to the next screen.
2. On the SYSTEM MENU screen, enter "5,Next" to select the Sample system, and go to the next screen.
3. On the SINGLE SYSTEM ANALYSIS (SSA): HIGH LEVEL SUMMARY REPORT OPTIONS screen, enter "Next" to ignore High Level Summary, and go to the next screen.
4. On the SINGLE SYSTEM ANALYSIS (SSA): MAIN REPORT OPTIONS screen, enter "1,3,10,Next" to select Main Report, Language Feature Overhead, and Write Statistical Tables, and go to the next screen.
5. On the RUN OR SAVE REQUEST screen, verify that the selections are correct:
PROGRAM: SINGLE_SYSTEM_ANALYSIS
SYSTEMS: Sample
OPTIONS: Main Report:
Sections: LANGUAGEFEATURES
Options: STATISTICALTABLES
To correct any mistakes, use the "Previous" option to move to the appropriate screen where corrections can be made.
6. Enter "1,Do" to cause the analysis to be performed immediately.
7. Enter "Quit" to exit the program.
(b) Verify that the following sets of files are identical. Expect differences in dates.
sample.toc and zp_smple.toc
sample.rep and zp_smple.rep
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
sample.rep za_salft.ssa zp_smple.rep zp_stp15 (command file)
sample.toc za_smple.e00 zp_smple.toc zp_stp15 (log file)
This section assumes that the Pretest activity (Section 3) has been successfully executed.
Performance testing is supported by the Harness program that was created during the Pretest activity. If the implementation does not support the Harness, then command scripts (distributed with the ACES) will need to be adapted for this implementation. These command scripts have names of the form "##.unx" and "##.com", where "##" is the two-letter group abbreviation. Thus "ap.unx" and "ap.com" are sample command scripts for compiling and executing the Applications (ap) group. The ".unx" versions were built for use on UNIX systems (Sun Ada (VADS)). The ".com" versions were built for use on VMS systems (A VAX-Ada implementation was used to test these scripts).
For further details on running the performance tests without the Harness, see the User's Guide, Section 7.
The general pattern for running performance tests is described in the algorithm below. The steps marked (ZP_SETUP) have already been done by the ZP_SETUP program; the steps marked (Harness) are most easily executed using the HARNESS program.
Create low-level command scripts (ZP_SETUP)
Adapt Harness control files (ZP_SETUP)
REPEAT
Select a set of tests to be run
Copy the necessary test files into the working directory (Manual)
(These files are the files "##*.*", where "##" is
the two-letter group abbreviation)
Generate a command script for these tests (HARNESS)
Execute the command script and capture the results in
a log file
Examine the log file (HARNESS)
Check the status of the tests (HARNESS)
(Decide which ones to re-run)
UNTIL Finished OR Available time expires
The following paragraphs explain how to prepare for using the Harness, how to select tests, and how to generate command scripts for compiling and executing the selected tests. More details on using the Harness appear in Section 6 of the User's Guide.
The files for a selected test group need to be in your source subdirectory. Harness will automatically copy them into your working directory while changing their (".ada") extension to the system default extension you selected in Setup.
The Harness program should have already been compiled, linked, and tested (Pretest Step 11). As part of this process, a number of data files should have been generated or copied from the source directory, as discussed below:
1. Files with names of the form "zh_??.lib", where the "??" is replaced by the two letter abbreviation of each test group except for the Systematic Compile Speed (sy) group. The file "zh_??.lib" is used when a command script being generated for group "??" includes the deletion of library units. There are 17 for version 83 and 20 for version 95.
2. Files with names of the form "zh_??.txt", where the "??" is replaced by the two letter abbreviation of each test group except for the Systematic Compile Speed (sy) group. There are 17 for version 83 and 20 for version 95.
3. Two files, "zh_sy000.tpl" and "zh_sy_cu.tpl", are used to define processing of the Systematic Compile Speed group.
4. Two files are generated by ZP_SETUP during Pretest Step 11, named "zh_cosys.txt" and "zh_usr.txt". These files may be edited, if necessary, to change the default file names and sizes (in "zh_cosys.txt") or primitive symbols and low-level commands (in "zh_usr.txt").
If the implementation requires more than one command line for a process that is only allocated a single command in "zh_usr.txt", then create a command script containing the needed lines. In place of the single command line allowed in "zh_usr.txt", use the command to invoke this command script. For example, if using a cross-compiler under VMS, then running an executable might involve several steps: download the executable, start the target, and retrieve the results. Place these three commands in a command script (say "runtarg.com") and use "@runtarg.com" as the value for "Run_Command" in "zh_usr.txt". (Note the inclusion of '@', the VMS symbol for invoking a command script.)
Sets of tests are selected for two primary purposes: to generate command scripts for compiling, linking, and executing the tests or to view the status of tests that have already been attempted.
Among the commands that are accepted by the Harness program are the "Show Group" (SG), "Show Subgroup" (SS), "Show Tests" (ST) and "Show Chosen" (SC) commands. The screen that is displayed at the beginning of execution lists the groups (use the command "Next" (N) to view the next page of the list). This is the same screen that appears in response to the SG command. A group must first be selected in order to use the SS and ST commands:
Select a Group
While on the Groups screen, enter a group number to select a group. Precede the group number with a ('-') to deselect a group. The most recently selected group is the current group.
When on the Groups, Subgroups, or Tests screen, tests may be selected for viewing status or for building command scripts. The command for selecting tests depends on the screen. In each case, the actual tests may be indicated by a wild-card ("all" or "*"), by a range (e.g., "42..65"), or by a list (e.g., "42, 43, 44"). The following example assumes that Group 4 is the current group.
Select Tests
While on the Groups screen, enter "4/*/*" to select all tests in Group 4.
While on the Subgroups screen, enter "*/*" to select all tests in Group 4.
While on the Tests screen, enter "*" to select all tests in Group 4.
Tests may be deselected by using the minus ('-') as a prefix to the test number.
Any combination of tests may be selected, including sets of tests that cross group boundaries. However, if any test in a subgroup is selected and not all the tests in that subgroup are selected, then a command script that is built for that selection will not include any commands for collecting compilation/linking time data. This is especially important when re-running tests that did not give usable data on previous attempts.
In the rest of Section 4, it is assumed that the complete subgroups have been selected.
If a command script has already been executed and the status of the tests is being viewed (as discussed below), then the user may wish to select some of the tests, generate a new command script for just those, and re-run them. Again, Section 4 assumes that the user always selects a complete subgroup.
After a set of tests has been selected, the command "Build" brings up the "Building Command Files" screen. There are up to 9 fields that may be changed, as follows:
1. Entering "1" will toggle a "switch" that determines whether the generated script will contain commands that collect compilation and linking time.
2. Entering "2=<number>" will change the maximum number of test problems to be bound into a single executable. The default maximum is 9; if any other value is specified, the default grouping of test problems into executables is overridden, and compilation/linking time measurements will not be collected.
3. Entering "3" will toggle a "switch" that determines whether the script will contain commands to delete all source code files from the working directory after processing. This includes the files produced by the "Include" preprocessor, the subgroup "dummy" files generated by the Harness, and the support package files. (See Section 4.2 (2).)
4. Entering "4" will toggle a "switch" that determines whether the script will contain commands to delete all the "Include" files from the working directory after the "Include" preprocessor has been run on them. (See Section 4.2 (1).) This includes the unprocessed forms of the source files and the main program unit files generated by the harness.
5. Entering "5" will toggle a "switch" that determines whether the script will contain commands to delete the executables from the working directory after they are run. These executables are created when the command script is executed.
6. Entering "6" will toggle a "switch" that determines whether the script will contain commands to delete the library units from the Ada program library after the linking is completed.
7. This field appears only if tests from more than one group have been selected. Entering "7" will toggle a "switch" that determines whether one script will be generated to process all the tests, or a separate script will be generated to process the tests in each group.
8. If the user requests that more than one script file be generated, then entering "8=<string>" sets the file suffix for each script file to the value of <string>. In this case, the file name prefix for each script file will be the two-letter abbreviation of the group.
If a single script file is being generated, then entering "8=<string>" sets the entire file name (including suffix) to the value of <string>.
9. Entering "9=<directory path>" causes the command script(s) to be generated in the directory indicated by the value of <directory path>.
Enter "Do" to invoke the command script generator. When the generation is complete, one or more messages will be displayed concerning the database files for all selected groups, and the "Building command files" screen will reappear. The user may enter "Cancel" to return to the previous screen or "Quit" to exit the HARNESS program.
Once the appropriate command scripts have been generated, the user must execute the command scripts, capturing output in a log file for later analysis. Recall that the source files for the group should have been copied into the source directory BEFORE executing the command scripts. Certain tests require special treatment, as discussed below.
Some of the performance tests must not be run in background mode or batch mode, but must be run interactively from the keyboard. If a windowing terminal with cut-and-paste capabilities is available, the time and code measurement data can be cut from the output window and pasted into a "log" file window. If such a facility is not available, it will be necessary to copy the time and code measurement data as it is displayed on the screen. Note that it is not wise to suspend processing while copying the data for one test, since this may interfere with the timing of a later test in the same main program. For fast systems, it may be necessary to specify one test per main program when building the testing scripts. (Of course, compilation and linking time cannot be captured in this case.)
Table 4-1 lists the interactive tests, along with the files in which the test source code is contained and the main program (using the default assignment of tests to main programs) containing the executable code.
Table 4-1 Tests That Must Be Run Interactively
Test Name Source File Name Default Main Program Name
io_tx_async_01 io_tx01_.inc io_txm01
io_tx_async_02 io_tx02_.inc "
io_tx_cio_01 io_tx03_.inc io_txm02
io_tx_cio_02 io_tx04_.inc "
io_tx_cio_03 io_tx05_.inc "
io_tx_cio_04 io_tx06_.inc "
io_tx_cio_05 io_tx07_.inc "
io_tx_cio_06 io_tx08_.inc "
io_tx_cio_07 io_tx09_.inc "
io_tx_cio_08 io_tx10_.inc io_txm03
io_tx_cio_09 io_tx11_.inc "
io_tx_cio_10 io_tx12_.inc "
io_tx_cio_11 io_tx13_.inc "
io_tx_cio_12 io_tx14_.inc "
io_tx_cio_13 io_tx15_.inc "
io_tx_cio_14 io_tx16_.inc "
io_tx_io_24 io_tx33_.inc io_txm07
io_tx_io_25 io_tx34_.inc "
io_tx_io_26 io_tx35_.inc "
io_tx_io_27 io_tx36_.inc "
io_tx_io_28 io_tx37_.inc "
io_tx_io_29 io_tx38_.inc "
io_tx_io_30 io_tx39_.inc "
tk_la_async_04 tk_la02_.inc tk_lam02
The Input-Output ("io") tests measure the time required to write to the console (terminal). If this timing is important to the purpose of the evaluation, then these tests really should be executed from a terminal that is connected in the exact fashion that would be used for the project.
The Tasking ("tk") test examines the behavior of a program while one task waits for (asynchronous) input from the keyboard. If the test does not terminate within 10 minutes, the user is expected to enter the character 'X' to satisfy the waiting task and allow program termination. (This is a way to account for those systems in which the entire program is suspended while a task waits for keyboard input.)
The test "ms_il_interface_lang_assembly" requires that an assembly language module be provided. The test contains a PRAGMA INTERFACE that is expected to enable calling the assembly language procedure. The source code may have to be modified if the implementation requires additional information for a subprogram imported by PRAGMA INTERFACE, or if the name "assembler" is not the appropriate name for interfacing to assembly language. The user must examine the source code file ("ms_il01_.inc") to determine whether the pragma meets the requirements of the implementation. In addition, the command script controlling the compilation of this test may have to be modified to assemble the required module and make it known to the Ada library. Locate the string "User adaptation required" and modify the next two commands. The first command means to assembly the assembly language module in the file "ms_ilnul" (with whatever file-naming conventions are required.) The second creates an object library named "ms_illib", containing the newly assembled object file. For some implementations, the appropriate command would add the object file to the Ada library as a "foreign" unit. Note that the string "User adaptation required" appears again, followed by a link command that references the object library created above. The implementation under test may require quite a different syntax, or may not require a special link command.
There are seven tests that contain deliberately illegal symbols. In each case, the line containing the illegal symbol must be modified by the user. These tests, their source files, and their default main program names are given in Table 4-2.
Table 4-2 Tests Containing Illegal Symbols
Test Name Source File Name Default Main Program Name
dt_dp_delay_zero_01 dt_dp16_.inc dt_dpm04
dt_dp_delay_zero_02 dt_dp17_.inc "
dt_dp_delay_zero_03 dt_dp18_.inc "
dt_dp_delay_zero_04 dt_dp19_.inc dt_dpm05
dt_dp_delay_zero_05 dt_dp20_.inc "
dt_dp_delay_zero_06 dt_dp21_.inc "
tk_lf_task_48 tk_lf20_.inc tk_lfm20
In the Delay and Timing ("dt") tests, the symbol "<>" appears in an illegal context, requiring that the user modify the source file. The test's purpose is to examine the effect of time-sliced task scheduling. If the implementation supports a pragma for this purpose (such as the supplied example using PRAGMA TIME_SLICE), then the pragma should be used in place of the line containing the illegal symbol. Some implementations may achieve the same effect with compiler or linker options; in this case, the user should delete (or comment) the line containing the illegal symbol and also must modify the command script controlling the compilation, linking and execution of the test. The source code contains comments describing the size of the time-slice quantum that should be used. Note that in the last three of these tests, the specified time-slice quantum should be 0.0.
In the Tasking ("tk") test, the symbol "<>" is used illegally in two lines. In each case, the line must be replaced by an implementation-dependent method of associating a task entry with an interrupt. An example is included in the illegal line, using an address clause to associate the entry with the interrupt that is associated with the hexadecimal address 40. The actual address is highly implementation-dependent. Some implementations may require one or more pragmas in this context. Some implementations may simply not permit such an association.
The Systematic Compile Speed ("sy") tests pose several difficulties. First, it is not possible to generate a command script for just selected tests. The Harness program always generates three scripts ("sy<scripts>", "sy_cu<script>", and "sy_1000<script>", where <script> is the script file suffix specified by the user). The first of these compiles and executes all the "sy" tests except for the Compilation Unit Size subgroup. Responsibility for the tests in the omitted subgroup is divided between the command scripts "sy_cu<script>" and "sy_1000<script>". This division of control of the "sy" tests into three scripts is done because these tests may require more time and resources than the evaluator can afford.
Note that these tests have been observed to monopolize the CPU on multiuser systems, to consume all free disk space, and to crash some operating systems.
The second problem is that the command scripts, even those generated by the Harness program, require extensive user modification. In each of the scripts, modifications are indicated by comments containing the string "~~~~" (four occurrences of the "tilde" character - ASCII code 126 decimal). The adaptations are described in comments appearing before the modification flag. Adaptations that must be made include, but may not be limited to, the following:
* Compiler option to include the source code in the library
* Compiler option to generate warnings
* Method of submitting a list of files to the compiler at once
* Compiler option to generate a listing file
* Compiler option to check syntax only
* Compiler option to check syntax and semantics only
* Compiler option to suppress warnings
* Recompile command
* Recompile option to force closure
* Method of deleting deeply nested subunits
(where the expanded name is longer than a script line)
* Method of determining the total size of the program library
* Method of determining the size of selected library units
(with or without wild-card symbols)
* Method of obtaining a list of library units
* Method of determining whether a specific unit is in library
* Method of deleting library units using wild-card symbols
* Method of creating an Ada library
* Method of deleting an Ada library
* Method of deleting all but most recent version of a file
* Method of assigning a character value to a script variable
* Method of opening a file so a script can read from it
* Method of reading a value from a file into a script variable
* Method of writing the value of a script variable to a file
* Method of closing a file that was opened by a script
* Method of assigning the value of one script variable to another
* Method of comparing a script variable's value to a literal
(using an IF .. THEN .. ENDIF kind of structure)
* Method of checking the status of the previous command
(using an IF .. THEN .. ELSE .. ENDIF kind of structure)
In this section, it is assumed that the command scripts have been executed and that the test output has been captured in a "log" file.
To extract status data from a log file, enter "Update" (or "U") while on any Harness screen.
(Note that this command is not displayed in the list of available commands on a screen.)
On the next screen, specify the name of the log file by typing "1 = <name of log file>". Then enter "Do" to cause the status data to be extracted from the file.
After updating the status from the log file, the Groups screen (the opening screen, reached from other levels by the Show Groups command) will show the number of tests in each group having each status value. Use the Show Subgroups and Show Tests commands to determine which specific tests have particular status values. The following list gives the possible status values and their meanings.
Va Valid non-zero execution time (the desired result)
Nu (Null) Execution times of 0.0 (leave alone)
Dy Measures time for delay statement (leave alone)
Un (Unreliable) Timing measurement was variable (try again)
Vr Verification error (try again)
Ng Timing ERROR: large negative time (probably should give up)
Xc (Excessive time) Storage reclamation takes too long (leave alone)
Pk Packaging error - might be ok (run separately from default main)
Cm Compile time error (log file might show a solvable problem)
Ln Link time error (log file might show a solvable problem)
Rn Run time error (test problem did not complete execution)
Dp System dependent test (failure due to known system dependency)
In Execution/compile inconsistency (recompile whole main program)
No No data available (not run yet or unknown failure)
NA Not Applicable (Set by user)
Wd Withdrawn test
After checking the status, the user may want to look at the log file to determine what happened to some tests. Depending on what is learned, the user may want to use Set Status to change the status.
Often, tests that are classified as Unreliable or Verification error will produce Valid results when run again, especially if rerun while the system is not busy. If the executables have not been saved (or if they need to be re-compiled), then the user should select the appropriate tests and generate new scripts. If compile-time results are desired, then be sure not to disturb the default structure for binding test problems into executable programs. The simplest way to do this is to select all the tests in the subgroup containing the test problem having an unsatisfactory status, and generate scripts containing whole subgroups. If this is not feasible, then determine which tests to select in order to guarantee that the default binding structure is not disturbed. This is a two-step process, in which the file names must first be identified and then the test problem names.
First, determine which of the "include" files belong to the same main program as the test in question. These files have names of the form "gg_ss??_.inc" (where "gg" is the group abbreviation and "ss" is the subgroup abbreviation), and the names all appear in the script file between the line "begin main gg_ssm??" and the line "main program".
Second, determine which test problem names correspond to these file names. Appendix B of the Version Description Document provides this correspondence.
Whether the user selects all the tests in the subgroup or uses the above two-step process to identify the tests for selection, the user should now build one or more scripts, execute them again, and update the status from the resulting log files.
The ACES Analysis software performs statistical analysis on the performance test results. There are 4 components of the Performance Analysis, as described below: (Note that no statistical analysis is performed on Assessor results.)
* Condense The Condense module extracts data from log files and creates "database" files of the compile-time and run-time data.
* CA Comparative Analysis compares the results of performance tests run on two or more systems and reports the summary statistics showing the relative efficiencies of the tested systems.
* SSA Single System Analysis compares the results from groups of related tests (all run on the same system) and reports the relative performance costs (for that system) of design and coding choices.
* Menu The Menu program provides a user interface for selecting Analysis tools and reports.
If the system capacities allow, all four components are linked together, with the Menu component as the main driver. In this case, the user makes appropriate selections of systems, groups, and reports, and the desired analysis component is invoked from Menu. In the discussions below, we assume this scenario. If your system does not allow linking all the components into one executable, see User's Guide Section 9.2 for help.
Note that the Condense tool must be executed before running either Comparative or Single System Analysis, and whenever new tests results need to be added to the database.
The choice between Comparative Analysis and Single System Analysis depends on the purpose of the evaluation. If the intent is to determine the relative performance of two or more systems, then Comparative Analysis is appropriate. If the purpose is to determine how best to use an existing system, then Single System Analysis is needed.
The Menu program screens allow three kinds of input: selection/deselection
of options, value entry, and command selection. To select an
option, enter its number. To select several options, separate
the numbers with commas or use an Ada-like range notation (e.g.,
"2..5"). To de-select one or more options, precede
the number or range with a minus sign (e.g., "-2, -3",
or "-2..5"). If an option contains ":=",
you may specify a new value by typing the option number, a blank,
":=", and the desired value
(e.g., "2 := NewFile.TXT"). To make a command, type
the command or an abbreviation (e.g., "Next" or "N").
Each screen includes a "Help" command that provides
information specific to that screen.
The Main program uses two basic (required) files. The first, "za_cowgt.txt", contains the names of all the groups, subgroups, and tests, along with weighting factors. See User's Guide Section 9.1.5 for a discussion of weights.
The second file, "za_cosys.txt", contains system names and file names that are specific to your evaluation. You may supply the required data by editing the sample file before running Menu, by using the Menu program to create the file interactively, or by using the Menu program to modify the data in the sample. We recommend the third option.
To modify the system names file interactively, choose the option "View/Modify system name file" (available on the SYSTEMS MENU screen). When this option is selected, the system presents default values for several file names. These may be changed, but are generally satisfactory. Choosing "Next" brings up a sequence of identical screens, one for each system identified in the system names file. You may replace any of the values given. Note that for each system, you must specify at least one log file or at least one condensed data file. To delete a system from the list, choose "Clear". To add a system, choose "Next" until a system with all blank names appears. Enter "Do" at any screen to store ALL system data (including any systems that you have not already viewed).
The Condense component reads the log files containing the output of performance tests, extracts the needed compile-time and run-time data, and creates two "database" files containing the data, formatted according to the requirements of the Comparative Analysis and Single System Analysis tools.
Condense recognizes each log file line that begins with "\aces" (or "\ACES") as a line that should be processed. The processing for such a line depends on the words following this initial string. For compile-time results, Condense performs the necessary subtractions and sums the compilation times for the units making up a main program, as well as determining the linking time for the program. These compile-time results are written to a text file, referred to as the "compile database". The name of this file is given in the system names file; it initially defaults to the log file name (without its suffix plus the suffix ".c00". Each time Condense is run, the file suffix is updated to represent a version number (as ".c01", ".c02", etc.).
For run-time results, Condense writes the formatted data to a text file, referred to as the "execution database". The default name of this file is the log file name (without its suffix) plus the suffix ".e00". This name is updated in the same way as the compile database.
To extract data from a log file, select the CONDENSE option (Option 1) on the MAIN MENU screen and the desired Report Selection Method (e.g., "1,4,n"). The Report Selection Methods provide the user with the option of either (4) select reports interactively, (5) use the default report selection, or (6) select reports from an existing request file.
On the SYSTEMS MENU screen, select the system(s) whose log files are to be read. (The log file names are specified in the system names file.)
The CONDENSE: REPORT OPTIONS screen, illustrated in Figure 5-1, asks the user to make several choices. Options 1, 2, and 3 identify the reports that Condense can generate while creating the database files. It is not necessary to select any of these reports. The reports provide the following information:
* No Data Lists subgroups and tests for which no data was found in the log file.
* Exceptional Data Lists data found to be not usable for analysis.
* Multiple Results Lists tests having more than one set of results.
Selecting Option 4 is the same as selecting Options 1, 2, and 3.
Selecting Option 5 causes Condense to generate comma-delimited forms of the database files. These files are suitable for importing into spreadsheets or other analysis tools. The names of these files will always be the system name plus one of the suffixes ".ccd" (for "Compile Comma-Delimited") or ".ecd" (for "Execution Comma-Delimited").
WARNING: The normal database files must have been created before attempting to create comma-delimited files. That is, Condense must have already been run at least once to produce the database files.
------------------CONDENSE: REPORT OPTIONS----------------
1. -NO_DATA_REPORT := .nda
2. -EXCEPTIONAL_DATA_REPORT := .exc
3. -MULTIPLE_RESULTS_REPORT := .mul
4. -All Above
5. -Comma delimited data
Compilation/Link database alternatives (SELECT 1):
6. -Create new database from log data
7. +Append log data to existing database (DEFAULT)
Execution/Code Size database alternatives (SELECT 1):
8. -Create new database from log data
9. +Append log data to existing database (DEFAULT)
10. -Merge Harness database
Figure 5-1 Report Options Screen
Options 6 through 9 allow the user to decide whether to create a new database file (for compile-time or run-time data) or to add the new data to the existing database. If the database files have not yet been created, then the choice to append new data to existing files actually creates new files; thus, the only time a choice needs to be made is when the user decides to ignore existing database files in favor of creating new ones.
Option 10 enables importing data from the Harness databases instead of from log files. (Note that the Harness databases do not contain compile-time data.) If this selection is made, then the system names file ("za_cosys.txt") must contain a path name for the harness files.
The RUN OR SAVE REQUEST screen displays the user choices and gives the opportunity to run Condense or to save the selections in a "request" file. To run Condense from Menu, enter "1,do". To return to a previous screen to change selections, enter "p" as many times as necessary (or "m" to return to the main menu). The request file options are used when Condense is linked alone as an executable; in this mode, Condense expects to find the selection data in the request file (whose name is given in the system names file).
The Comparative Analysis (CA) tool performs statistical comparisons of the compile-time and run-time results of two or more tested implementations. The user may select reports that provide differing levels of detail and may choose to analyze results representing selected test groups and selected metrics.
Before running Comparative Analysis, edit "za_cosys.txt" to give the correct system names and database file names. Run Condense to produce database files for the kinds of data to be analyzed: run-time data or compile-time data. The default file names for these database files have the suffixes ".e00" (execution data) and ".c00" (compilation data). The "00" may change; whenever Condense is run for either run-time or compile-time data from a system, the number represented by "00" is incremented and "za_cosys.txt" is updated to show the new suffix.
To run Comparative Analysis, select the COMPARATIVE_ANALYSIS option (Option 2) on the MAIN MENU screen and the desired report selection method (e.g., "2,4,n"). The Report Selection Methods provide the user with the option to either (4) select reports interactively, (5) use the default report selection, or (6) select reports from an existing request file.
On the SYSTEMS MENU screen, select the system(s) to be compared.
The METRICS MENU screen allows the user to choose one or more of the five metrics reported by CA. (For information on the characters following the ":=", enter "Help".) Since most evaluators are interested in execution time, Option 1 would usually be selected. If the user is interested in compilation time, consider whether the appropriate metric is really the total compilation and link time (Option 5). Since the assignment of operations to the compiler and to the linker varies from one system to another, it may well be that the total is the metric that the user really wants to consider. Note that the user may choose to generate reports for all five metrics by selecting Option 6.
The GROUPS MENU screens (PAGE ONE OF TWO and PAGE TWO OF TWO) present the performance test groups. (Again, for information on the names to the right of the ":=", enter "Help".) Selecting only the groups that are relevant to the user's purpose eliminates the other group comparisons from the analysis, reducing both the calculation time and the size of the reports.
The COMPARATIVE ANALYSIS (CA): REPORTS screen presents the choices of reports to be generated. Each Group Level Report provides detail on individual tests, along with summary statistics for the group. The statistical analysis and summaries apply to all the raw measurements for the group.
The Summary Of All Groups Report applies the same kind of analysis, beginning with the summary statistics that were produced for the individual groups. Thus, the user cannot produce a Summary Of All Groups Report unless the user generates Group Level Reports for those groups, either at the same time or in a previous run.
The Special Report is used when performing an Application Profile evaluation. See the User's Guide, Section 9.4, and the Reader's Guide, Section 5.3.2.3 for more information on this mode of evaluation.
Note that the user may choose to produce text reports, which can be printed, and/or comma-delimited reports, which can be imported into most analysis and presentation tools (such as spreadsheet programs). In either case, the user may choose to have all the text reports written to one file or to have all the comma-delimited reports written to a single file. The default is to write each group report (whether text or comma-delimited) to a separate file.
The user may also find it useful to lengthen the Text Report output line. The default of 80 characters works for 1-3 system, 87 works for 4 systems, and an additional 11 characters for each succeeding system is sufficient.
If the user chooses Group Level Reports from the CA Reports Menu, the CA: GROUP_LEVEL OPTIONS FOR REPORTS ON SELECTED METRICS screen will be presented. Here, the user may choose the Summary Report only (OPTION 1), or a more detailed report. The full report (Option 3) includes the summary statistics (with confidence intervals and sample size information), raw data, residual data, and other detailed tables. Choosing Option 1 (Summary Report Only) suppresses output of the detailed tables, leaving only the summary statistics, confidence intervals, and sample size information. Options 4 and 5 allow the user to gather two more detailed reports. Choosing Option 2 has the same effect as choosing Options 3, 4, and 5.
If the user chooses the Group Level Reports and the Summary Of All Groups, the CA: SUMMARY_OF_ALL_GROUPS_LEVEL REPORT OPTIONS screen will be presented. Here the user may choose the level of detail for the Summary Of All Groups Report. Choosing Option 1 is equivalent to choosing every option on the menu. Choosing Option 2 is equivalent to choosing both Options 3 and 4. The Vertical and Horizontal Bar Charts present the same information; user preference should dictate which of these to choose (if either). The information summarizes, in a graphical form, the comparative summary statistics for the various systems. Choosing Option 5 is equivalent to choosing all of Options 6, 7, 8, and 9. These are the intermediate summaries, including summary statistics and error data, sorted and grouped by group and by system. Note that the user must choose at least one of Options 6 and 7, as well as at least one of Options 8 and 9. Also note that Option 10 is automatically selected if any other option is chosen.
The RUN OR SAVE REQUEST screen displays the user choices and gives the opportunity to run CA or to save the selections in a "request" file. To run CA from Menu, enter "1,do". To return to a previous screen to change selections, enter "p" as many times as necessary (or "m" to return to the main menu). The request file options are used when CA is linked above as an executable; in this mode, CA expects to find selection data in the request file (whose name is given in the system name file).
The Single System Analysis tool (SSA) generates reports that are useful for the evaluation of a single Ada implementation. This is accomplished by presenting "average" metrics, statistical comparisons of sets of tests using different techniques for achieving the same goals, and summaries of special "ancillary" information output by selected tests.
Before running Single System Analysis, edit "za_cosys.txt" to give the correct system names and database file names. Run Condense to produce database files for the kinds of data to be analyzed: execution data (".e??") or compilation data (".c??"). In addition, SSA requires certain other data files in order to produce reports in selected areas, as follows:
* Language Feature Overhead za_salft.ssa
* Optimization za_saopt.ssa
* Run-Time System Behavior za_sarts.ssa
* Coding Style Variations za_sasty.ssa
These files must be copied from "aces/support" to the working directory.
To run Single System Analysis, select the SINGLE_SYSTEM_ANALYSIS option (Option 3) on the MAIN MENU screen and the desired report selection method (e.g., 3,4,n). The Report Selection Methods provide the user with the option to either (4) select reports interactively, (5) use the default report selections, or (6) select reports from an existing request file.
On the SYSTEMS MENU screen, select the system(s) to be analyzed.
The SINGLE SYSTEM ANALYSIS (SSA): HIGH LEVEL SUMMARY REPORT OPTIONS screen allows the user to choose the high-level summary reports: execution time, code size, compilation (plus linking) time, or errors. The user must choose at least one of these options; choosing Option 2 is equivalent to choosing all of Options 3 through 6. Option 1 allows the user to change the suffix of the report file; the default is to create a summary file whose prefix is the system name and whose suffix is ".hls".
The SINGLE SYSTEM ANALYSIS (SSA): MAIN REPORT OPTIONS screen allows the user to select the desired sections and level of details to be included in the reports.
The system will analyze results from similar tests to provide data on the overhead of language features, effectiveness of optimizations, cost of using selected services typically provided by the Ada run-time system, and performance effects of certain alternative coding styles. The ancillary data provided by selected tests is also part of the Main Report. This menu allows the user to selectively obtain reports in these areas. At least one of Options 3 through 7 must be selected; choosing Option 2 is equivalent to choosing all of Options 3 through 7. The Main Report is written to a file whose prefix is the system name and whose default suffix is ".rep". The suffix may be changed through Option 1. A Table of Contents is also generated in a file whose prefix is the system name and whose suffix is ".toc".
In addition to choosing one or more of the areas named above, the user must also select the detail to be included in the reports (via options). At least one report option (Options 9 through 12) must be selected. The part of the report devoted to a particular set of similar tests is made up of one or more of these report options. The heart of SSA is in the Statistical Tables (Option 10), but the other options can be extremely useful as well. See the Reader's Guide, Section 5.4.4, for more information.
The RUN or SAVE REQUEST screen displays the user choices and gives the opportunity to run SSA or to save the selections in a "request" file. To run SSA from Menu, enter "1,do". To return to a previous screen to change selections, enter "p" as many times as necessary (or "m" to return to the main menu). The request file options are used when SSA is linked alone as an executable; in this mode SSA expects to find selections data in the request file (whose name is given in the system names file).
The assessors are used to evaluate the capabilities and quality of subsystems that are not directly related to performance of the compiler or the generated code. The current version of ACES includes four assessors, each of which is discussed in its own section, as indicated below:
* The Symbolic Debugger Assessor (Section 6.2) determines the capabilities of the symbolic debugger, if one is supplied.
* The Diagnostic Assessor (Section 6.3) evaluates the quality of the diagnostic messages produced at compile time, link time, and execution time.
* The Program Library Assessor (Section 6.4) determines the capabilities of the library management system and produces some timing data for comparing different library usage scenarios.
* The Capacity Assessor (Section 6.5) attempts to determine compile-time and run-time capacities of the system.
Each assessor consists of a set of software modules, a scripted scenario for using these modules, and a report form for recording the results. Using an assessor follows the following general pattern:
* Get access to the required software modules.
* Adapt the required command scripts.
* Perform the steps identified by the assessor scenario.
* Answer the questions on the report form.
Many of the questions are of a somewhat subjective nature; however, it is expected that two users of roughly equivalent experience, using the assessors on the same system, would choose the same answers in most cases.
The Symbolic Assessor provides 29 scenarios, each of which consists of some source code and a set of actions to be requested of the debugger. Detailed instructions for these scenarios are given in the file "yb_readm.txt", which is reproduced as Appendix C of the User's Guide. The file "yb_tmplt.txt" (which is also reproduced in the same appendix) contains the report form for this assessor.
To prepare for executing the Symbolic Debugger Assessor, make sure that all the "yb_*.ada" files are available. If the compilation system requires that source code files have some suffix other than ".ada", then these files must be renamed or copied. In this case, adaptation of the command script files includes changing the ".ada" suffixes.
Before executing the assessor, you must also have compiled the "zg_glob*" files and the "zm_*" math packages must have been compiled into the library. (The math packages are needed only for module "yb_25_a" of the assessor.) See Step 5 of the Pretest (Section 3 of the Primer). If the system requires that a special compilation command be used for all modules in a program to be submitted to the debugger, then you will have to recompile the "zg_glob*" and "zm_*" modules with this command.
The following script files must also be available. The "zc_*" scripts should have been generated (or adapted) during the Pretest activity. If not, then the ACES-supplied versions (either "*.com" or "*.unx") will have to be used (or adapted).
* zc_adab - Compile a file with the "debug" option
* zc_cpydb - Copy the "debug" baseline timing file
* zc_linkd - Link a file with the "debug" option
* zc_setli - Identify the Ada library for compilation and linking
* yb_cmp1 - Compile and link one debugger assessor test
Note that the command script file "yb_cmp1" was neither generated nor adapted during Pretest.
Get printed copies of the sample command files whose names are of the form "yb_##*.com", where each "#" is replaced by a digit and "*" is replaced by any string. These VMS command files are meant to be used to guide the user in entering debugger commands interactively. (The file "yb_29_b.com" is an exception. It is used as a test of the debugger's ability to accept commands from an input file. Therefore, it must be adapted to the requirements of the debugger being tested.)
One of the steps (Scenario 12) requires an assembly language module. A VAX example is provided in the file "ms_ilnul.mar", in the directory containing the miscellaneous (ms) group. Use this example to produce an equivalent assembly language program for the system being tested.
In many of the debugger scenarios, the instructions provide line numbers at which to set breakpoints. These line numbers are relative to the beginning of the source file. Several debuggers use different schemes for numbering lines; for example, a system might use numbers relative to the start of each compilation unit, or it might use Ada statement numbers. To simplify adapting to systems using other numbering techniques, all the ACES debugger source programs where line number breakpoints are set include comments on the right margin of each source line containing the file relative line number. (In addition, many breakpoints are given both by line number and by Ada label.) This will make it straightforward for a user to determine what number to use by searching the compilation listing file for the line number comment and observing how the system identifies the line.
See the file "yb_readm.txt" (User's Guide, Appendix C) for detailed instructions on performing each of the 29 scenarios making up the Symbolic Debugger Assessor.
The Diagnostic Assessor contains source code that contains deliberate errors, providing the user with the opportunity to evaluate the quality of the diagnostic messages. Some of the errors should be reported at compilation time, some at link time, and some at run time. For each error, the user is asked a sequence of 6 yes/no questions, as follows:
1. Is a diagnostic message printed?
2. Is the message in the general area of difficulty?
3. Is the message at the correct specific location?
4. Does the text of the message clearly define the difficulty?
5. Is relevant non-local information listed where appropriate?
6. Is error recovery appropriate?
The user should answer these questions in the context of error correction. That is, the questions are asking whether the location and content of the messages are sufficient to help the Ada programmer correct the error. Question 6 refers to the compiler's ability to continue source code analysis after reporting an error. For run-time errors that raise exceptions, raising the exception is the only appropriate recovery.
Detailed instructions for using the assessor are provided in the file "yd_readm.txt", which is reproduced in Appendix D of the User's Guide. The file "yd_tmplt.txt" (which is also reproduced in the same appendix) contains the report form for this assessor.
To prepare for executing the Diagnostic Assessor, make sure that all the "yd_*" files are available. The file "yd_tmplt.txt" contains the report form for the assessor. If the compilation system requires that source code files have some suffix other than ".ada", then these files must be renamed or copied. In this case, adaptation of the command script files includes changing the ".ada" suffixes.
Before executing the assessor, the "zg_glob1" and "zg_glob4" files must have been compiled into the library. See Step 5 of the Pretest (either in the User's Guide Appendix B or in Section 3 of the Primer).
The following script files must also be available. If using a VMS or UNIX system, the ACES-supplied ".com" or ".unx" version is needed. Adapt any commands that are specific to the compilation system. If using some other operating system, produce equivalent scripts for the system. Use whichever example is closest to the system's requirements (or whichever is more familiar).
* yd_drivr - Submits the tests for compilation and linking, as required
* yd_linkr - Called from "yd_drivr" to link the link-time and run-time tests
* yd_fillr - Used to exhaust the disk space (needed by test "yd_lt04a")
(Best used on a disk with only a small amount of free space)
* yd_compl - Used to submit a file to the compiler
* zc_delsb - Removes a unit from the library
The following paragraphs give a top-level view of Diagnostic Assessor execution. For detailed instructions, see Appendix D of the User's Guide.
COMPILE/LINK-TIME TESTS
Once the files are available and the command scripts have been adapted, run the "yd_drivr" script, capturing all output (especially error output) for later analysis. Read this output in conjunction with the comments in the source code that describe the expected errors. Answer the questions in the "yd_tmplt" file.
RUN-TIME TESTS
Run the following tests interactively. They are expected to produce run-time error messages. If possible, capture the output in a log file so the error output can be studied. Be prepared to terminate any test that seems to be taking inordinately long (relative to the time for tests that terminate with error messages).
* yd_rt01a - Deadlock detection. (May require termination)
* yd_rt02a - Access errors: dereference through deallocated type
* yd_rt02b - Access errors: different type object access
* yd_rt02c - Access errors: access through bad address
* yd_rt02d - Access errors: access through deallocated object
* yd_rt03a - Exhausting storage (may crash system or require termination)
* yd_rt04a - Unhandled exception from main; no handler
* yd_rt04b - Unhandled exception from main, unnamed RAISE
* yd_rt05a - Uninitialized variables (optimized)
* yd_rt05b - Uninitialized variables (unoptimized)
The Program Library Assessor is meant to determine the capabilities of the implementation's subsystem for managing the library unit information required by the LRM. In addition, the report form for this assessor provides space for recording certain data on timing and disk usage; the required data is produced when processing the Systematic Compile Speed group of performance tests.
The assessor is largely automated, with the activity being driven by command scripts that are provided by ACES (scripts may require adaptation to the system being tested).
To prepare for executing the Program Library Assessor, make sure that all the "yl_*" files are available. The file "yl_tmplt.txt" contains the report form for the assessor. If the compilation system requires that source code files have some suffix other than ".ada", then these files must be renamed or copied. In this case, adaptation of the command scripts includes changing all the ".ada" suffixes.
Before executing this assessor, the "zg_glob*" files must have been compiled into the library. See Step 5 of the Pretest (either in the User's Guide Appendix B or in Section 3 of the Primer).
Before executing the assessor, parts of the Systematic Compile Speed group of performance tests should have been run. If they have not been run, they should be executed before beginning this assessor. See Section 4.2.4 for details. If the user is not interested in the timing and disk usage data provided by the Systematic Compile Speed tests, this assessor may be used without running these performance tests. However, of the 90 questions on the report form, nearly 40 require data from running the performance tests.
The command script files listed below must be ready to execute. For the first seven listed, use either the VMS versions ("*.com") or the UNIX versions ("*.unx"). If using an operating system other than one of these, adapt the one that is closer to the system's requirements. (Note that these first seven scripts were generated or adapted at the beginning of the Pretest activity.) Only VMS versions of the other command script files are provided, so UNIX users must also make adaptations of these. In any case, individual commands will probably have to be adapted to the requirements of the compilation system.
* zc_adano - Compiles a unit (should enter source into library, if possible)
* zc_adadb - (VMS only) alternate to "zc_adano"
* zc_delsb - Deletes the specification and body of a unit from the library
* zc_delbo - Deletes the body (only) of a unit from the library
* zc_setli - Identifies the Ada library to be used
* zc_link - Links to form the executable, with time stamps before and after link; used in scenario 6 to help check for concurrent library access
* zc_lnk - Links to form the executable
* yl_compl - Uses "zc_adadb" and "zc_lnk" to compile and link a test program; then executes the program
* yl_drivr - Invokes the individual scenario scripts
* yl_ib##? - Individual scenario scripts ("##" represents a sequence of two digits, and "?" represents an optional sequencing character)
After adapting the scripts listed above, execute "yl_drivr". If possible, capture the output in a log file. Read the results and answer the questions in the report form. The report file will direct the user to run some of the scenario scripts interactively. For those questions that have a test name and either "t" or "s" (or both) in the answer position, enter the time ("t") or space ("s") measurement from the Systematic Compile Speed group log file.
For detailed instructions by scenario, see Appendix E of the User's Guide, also provided as file "yl_readm.txt".
The ACES Capacity Assessor contains 41 tests (32 for compile/link time limits and nine for runtime limits), each of which tests a language feature. The tests rely on a shared set of programs and command files (for compilation/link time test and for runtime tests) which the user will adapt to the compilation system being evaluated. The test programs for the individual language features should require only implementation-dependent adaptation if a user chooses to work around errors or limitations in a system being evaluated. The common file "yc_parms.dat" (read by "yc_front.ada" and "yc_rtglg.ada") contains the testing limits for all the individual tests; this file is initially distributed with a set of default limits, which can be modified by the users through a text editor. The different tests are independent and can be run (or rerun) in any order. If any of the parameters are omitted or in the wrong format an error message will be printed when the procedure "yc_front" or "yc_rtglg" executes.
Detailed instructions for this assessor are given in the file "yc_readm.txt". Before running the capacity assessor tests, the user must have executed Pretest Step 5. This will have entered into the program library the "global" units required by the Capacity Assessor sample test programs.
To prepare for executing the Capacity Assessor, make sure that all the "yc_*.ada" files are available. If the compilation system requires that source code files have some suffix other than ".ada", then these files must be renamed or copied. In this case, adaptation of the command script files include changing the ".ada" suffixes.
Before executing the assessor, the user must have executed Pretest Step 5. This will have entered into the program library the "global" units required by the Capacity Assessor sample test programs.
The following script files must also be available. The "zc_*" scripts should have been generated (or adapted) during the Pretest activity. If not, then the ACES-supplied versions (either "*.com" or "*.unx") will have to be used or adapted.
* zc_setli - Used to point to the Ada library
* zc_adaop - Used to submit a file to the compiler in optimize mode
* zc_adack - Used to submit a file to the compiler
* zc_lnk - Used to link the tests
* zc_delsbv - Used to delete the spec & body of an Ada library unit
* zc_delso - Used to delete the spec only of an Ada library unit
The basic procedure for executing the capacity tests is:
* Adapt the shared command files and procedures for tests.
* Review the parameter file ("yc_parms.dat") and make any desired changes.
* Run the individual feature tests using "yc_drivr". Rerun as required. It is not expected that users will typically be able to derive a set of limits and run all the tests sequentially to completion. Users may review their choice of limits based on experience in initial testing: tests which timed out may be rerun with larger time limits, or test which quickly passed their maximum specified limits may be retested with larger limits.
The command file to execute all the tests may crash or halt when executing some test problem--in this case, the user will have to adapt the command file to execute the remaining test problems.
* Record the results from running each test in the report form (yc_tmp). The individual test output has been designed so that, if the test problem terminates normally (without the program or the operating system crashing) the last output from the test program will be information which the user will want to enter into the report form in the desired format (this permits users with a suitable host editing environment to "cut" the last output block from the program and "paste" it into the report form).
The Ada files associated with compile-time capacity test "xx" are named "yc_ctxxg.ada". For example, "yc_ct01g.ada" is the test generator Ada source for testing the "Number of Names in a Compilation Unit". On systems with limited disk space, a user could load only those files needed for a particular test onto the host system and modify "yc_drivr.com" to run only the selected test.
The Ada files associated with run-time capacity test "xx" are named "yc_rtxxg.ada". For example, "yc_rt01g.ada" is the test generator Ada source for testing the "Number of Elements in an Array". On systems with limited disk space, a user could load only those files needed for a particular test onto the host or target system.
Fill out the descriptive information on the report form. The report form, "yc_tmplt.txt", asks for information on the identity of the system being tested, the hardware and software configuration used, the name of the tester, the date, and any other information a user may wish to note.
This section identifies the analysis reports that are most likely to be used in preparing summary presentations of ACES evaluations. It also gives some indication of where the user should look to gain a more detailed understanding of the data produced by the analysis tools.
The user is warned against summarizing ACES analysis data at too high a level. The temptation to characterize performance by giving a single number is strong, but it is simply not realistic. For example, it is easy to state (and managers sometimes would like to hear) that one implementation executes similar code in 65% of the time that another implementation requires. However, such a statement does not take into account the often large variations observed when comparing execution time by individual performance test groups. The ACES team maintains that, in the presence of such variations, comparisons on a group-by-group basis provide an execution performance profile that provides much more useful information.
The principal statistic used in Comparative Analysis is referred to as the "system factor". This is a statistic that can be calculated for any data set that is organized in rows and columns. (In ACES reports, a row represents either a single test or a single performance test group; and a column represents one system that has been tested.) For each row, a "problem factor" (the row mean) is calculated, and the row is "normalized" by dividing each entry by the problem factor. (Note that, if each row of the original array were a scalar multiple of each other row, then the result would be an array in which all the rows were identical.) Next, a system factor is estimated for each column, and each entry of the column is divided by the system factor. The system factor represents a "typical" value for the entries of the column in much the same way that the row mean represents a typical value for the entries of the row. The estimation technique is chosen to provide a good fit to a "product" model in which each column of the array is a scalar multiple of each of the other columns.
After these calculations, we have the following relationship among the problem factor, the system factor, the original (raw) values, and the value remaining in the array (the residual):
raw value = system factor * problem factor * residual
If the raw data fits the product model perfectly, then each residual would be 1.0, and the raw value would be the system factor times the problem factor. In this case, we could summarize the relative performance of two systems by stating that one system's time was a (scalar) multiple of the other's time. The multiple would be given by the ratio of the two system factors.
Comparative Analysis produces two principal reports, the Group Level Report and the Summary of All Groups Report. The Group Level Report contains five sets of system factors for each group (one set for each metric collected by ACES). These system factors are based on the raw test data. The Summary of All Groups Report produces five sets of system factors for the entire performance test suite (one set for each metric). Each of these sets of summary system factors is based on an array with one row for each group and one column for each system. For each metric, the raw data in the summary analysis is the set of group-wise system factors for that metric.
For a quick look at the relative performance of two or more systems, the summary report is most useful. As indicated above, there are actually five such reports, one for each metric. This discussion uses the execution time metric as an example, not because it is more important than the others, but because the language is slightly easier to use.
The summary report contains bar-graph comparisons of the system factors for each of the tested systems, comparisons of selected (widely recognized) benchmarks, and other high-level tables. However, the most useful sections of the summary report are illustrated by Figure 7-1 and 7-2, taken from an analysis of four system evaluations. These results are for execution timing.
Figure 7-1 contains two tables that are printed side by side. The Significant Differences table (left section) contains an asterisk in each position where there is a statistically significant difference between the overall results from the system named in the row and those from the system named in the column. (Significance is at the 5% level.) The Data Summary table (right section) gives the number of groups processed for each system and the number of tests (for each system) whose results fall into particular categories. We note that System A had results for only 16 of the 21 groups. For System A, 1477 tests gave valid timing results; 24 reported failures at compilation time; 23 reported failures at run time; and 3 were excluded because no other system reported results for the same tests. No data was reported for 104 tests, and no explanation was available to the analysis software. The 114 tests in the "Other" category include the tests in the omitted group, along with tests giving results that were not usable (for various reasons).
========================================================================
---- Significant Diff = * | ---- Data Summary: Total n = 1783
========================================================================
A B C D | Gps Valid NoData Comp RunTim Exclu Other
------------------------------------------------------------------------
Sys A * * * | 16 1477 104 24 23 3 114
Sys B * * * | 20 1543 120 39 18 0 53
Sys C * * * | 20 1629 32 16 0 17 86
Sys D * * * | 20 1645 36 5 0 7 87
========================================================================
Figure 7-1 Significant Differences and Data Summary
Figure 7-2 presents the system factors for each system's performance on each group. Note that, in every case (except the missing groups), System A has the smallest system factor (that is, the smallest execution times). We might interpret the first row as telling us that, for tests in the Application group, System B typically took 11 times as long (1.22 / 0.11) to execute as did System A; System C typically took 18 times as long (2.01 / 0.11) as System A; and System D typically took 7 times as long as System A. Note that an individual system factor has no meaning on its own; the system factors are measures of RELATIVE typical performance.
========================================================================
Raw Data: | Sys A Sys B Sys C Sys D | Wgts
------------------------------------------------------------------------
application | 0.11 1.22 2.01 0.78 | 1.0
arithmetic | 0.21 1.49 2.78 0.91 | 1.0
classical | 0.14 1.95 2.38 0.85 | 1.0
data_storage | 0.15 2.54 2.40 0.83 | 1.0
data_structures | 0.15 1.54 2.77 0.93 | 1.0
delays_and_timing | 0.38 0.92 3.36 1.01 | 1.0
exception_handling | 0.13 0.99 3.30 1.27 | 1.0
generics | 0.09 1.22 2.39 0.91 | 1.0
input_output | 0.05 1.82 2.95 0.85 | 1.0
interfaces | 0.11 1.72 2.78 0.83 | 1.0
miscellaneous | 0.18 1.63 2.62 0.82 | 1.0
object_oriented | 0.17 1.68 2.56 0.83 | 1.0
optimizations | 0.16 1.74 2.50 0.84 | 1.0
program_organization | 0.12 1.41 1.95 0.87 | 1.0
protected_types | 0.13 1.64 2.12 0.85 | 1.0
statements | 0.15 1.88 2.29 0.82 | 1.0
storage_reclamation | missing 0.67 1.75 0.88 | 1.0
subprograms | 0.12 1.45 2.82 1.01 | 1.0
systematic_compile_speed| 0.16 0.84 2.71 1.58 | 1.0
tasking | 0.48 0.55 2.47 1.38 | 1.0
user_defined | missing missing missing missing | 1.0
========================================================================
Figure 7-2 Group System Factors
Figure 7-3 contains another view of the same data. This figure is an example of a graph that could be produced directly from the comma-delimited form of the Raw Data table above, using a popular spreadsheet program. (Due to space limitations, only the first 10 groups are presented.) The graph shows the four relative performance profiles for the four systems.
Figure 7-3 Performance Profiles
There are 18 group reports, each of which presents the data for one performance test group. In each case, there are five sub-reports (one for each of the five metrics). Each of the resulting 90 sub-reports contains data similar to that presented in Figure 7-1 and 7-2, with the raw data tables containing the results from individual tests.
Figure 7-4 presents the "residual table". The residual table displays the array produced from the raw data by dividing off the problem factors and system factors, as discussed in Section 7.1. It is important to note the deviations from 1.00. (Recall that if the data gave a perfect fit to the product model, then each residual would be 1.00.) The Residual tables include symbolic flags for values that are greater ("+") or less ("-") than would be expected for a particular system's performance on a specific test. (Double flags indicate even greater deviations than do single flags.) Such residuals are referred to as "outliers"; a system having an outlier on a particular test has reacted to that test in an unexpected way.
The residual table, through its flagging of outliers, draws the user's attention to test results that need special attention. For example, System A shows a residual of 10.22 (very high) for test "gn_in_enum_io_06". If fast execution of programs using generics are important to the project requesting this assessment, then this test and its treatment by System A should be closely examined. Perhaps the test contains some usage of generics that is not found in the other tests of the group (except, perhaps, for the very next test). Examining the assembly code for this test might also be enlightening, showing a real inefficiency in the compiler. Thus, the Residual tables are the place to start an in-depth study of the performance test results.
========================================================================
---- Residual * System Factor * Row Mean = Actual
========================================================================
Residuals: | Sys A Sys B Sys C Sys D | Means
------------------------------------------------------------------------
instantiation (in)
enum_io_01 | 0.59 2.44++ 0.92 0.94 | 33.37
enum_io_02 | 0.50-- 2.16++ 0.94 1.01 | 75.38
enum_io_03 | 0.51- 2.21++ 0.94 0.98 | 73.75
enum_io_04 | 1.00 0.18-- 1.03 0.97 | 23.74
enum_io_05 | 0.88 0.19-- 1.04 1.09 | 49.67
enum_io_06 | 10.22++ 0.15-- 0.87 0.90 | 22.01
enum_io_07 | 7.44++ 0.16-- 0.89 1.00 | 54.94
enum_io_08 | missing 1.94++ 0.28-- 1.36 | 1950.11
enum_io_09 | missing 1.06 3.11++ 0.92 | 3277.03
subprogram (su)
subprogram_01 | 0.52- 0.95 1.03 1.01 | 4211.32
subprogram_02 | 0.62 0.98 1.25 1.14 | 11.27
subprogram_03 | 1.01 1.38 0.84 0.91 | 6.70
subprogram_04_a | 0.98 0.75 1.12 1.11 | 7.06
subprogram_05 | 0.71 0.99 1.02 1.00 | 12.05
subprogram_06 | 1.11 1.35 0.64 0.99 | 5.55
subprogram_07 | 0.85 0.73 1.22 1.32 | 10.25
subprogram_08 | 1.24 1.46 0.70 0.79 | 5.03
subprogram_09 | 1.02 1.18 0.55- 0.98 | 6.41
subprogram_10 | 1.58 1.35 0.65 0.72 | 5.46
subprogram_11 | 0.70 0.99 1.41 1.01 | 12.18
subprogram_12 | 0.68 1.02 1.05 1.25 | 22.51
subprogram_13 | 1.21 1.62 0.71 0.73 | 7.86
subprogram_14 | 0.66 0.95 1.06 1.02 | 23.03
subprogram_15 | 1.18 1.63 0.72 0.74 | 7.81
========================================================================
System Factor | 0.09 1.22 2.39 0.91 |
========================================================================
Figure 7-4 Residual Table
The Single System Analysis program (SSA) treats data that does not lend itself to easy summarization. Although a high-level summary report can be produced, its primary purpose is to summarize and give examples of absolute execution times, compilation times, link times, cost expansion rates, and failures. The bulk of the main report, on the other hand, presents comparisons of performance data from similar tests.
SSA produces the reports listed below. The High Level Report and the Main Report are discussed in the following paragraphs.
* Missing Data Report (Lists tests that are needed for SSA but are not available)
* High Level Report (Gives summaries and examples of performance data)
* Table of Contents (Lists contents of the Main Report)
* Main Report (Comparisons of performance data for similar tests; ancillary data output by selected tests)
This reports gives the performance data (execution time, compilation time, and code size) for selected benchmarks. It also provides summaries (average, extremes, and frequency distributions) for execution time, compilation time, and code expansion rates. Compilation time is summarized as both lines per minute and semicolons per minute. Code expansion rates are given as bytes per line and as bytes per semicolon. The High Level Report also summarizes failures (both compile-time and run-time failures), reporting by performance test group and failure category.
There are five sections in the Main Report: Language Feature Overhead, Optimizations, Run Time System Behavior, Coding Style Variations, and Ancillary Data. The first four of these are devoted to comparison group reports, which compare the results (execution time, code size, and total compilation and linking time) from groups of similar tests. The Ancillary Data section contains information produced as the tests run.
There are two forms of comparison group reports: the first is used to indicate simple differences among the performance results of the tests in the group, and the second is used to indicate whether selected optimizations are performed.
--------------------------------------------------------------------------------------
Single Shared Scalar Variable, Pair, Shared Access
--------------------------------------------------------------------------------------
Description
--------------------------------------------------------------------------------------
These problems reference variables mentioned in a PRAGMA SHARED statement.
Problem AR_IO_INTEGER_OPER_14 references one shared scalar variable. Problem
AR_IO_INTEGER_OPER_15 references two shared scalar variables. Problem
DR_CO_ACCESS_OPER_06 references two shared variables of an access type.
--------------------------------------------------------------------------------------
Test Execution Bar Similar
Name Time (Microsec) Chart Groups
--------------------------------------------------------------------------------------
ar_io_integer_oper_15 0.8 **************** |
ar_io_integer_oper_14 0.9 ****************** |
dr_co_access_oper_06 1.2 ************************ |
--------------------------------------------------------------------------------------
Test Code Bar
Name Size (Bits) Chart
--------------------------------------------------------------------------------------
ar_io_integer_oper_15 64.0 *******************
dr_co_access_oper_06 72.0 **********************
ar_io_integer_oper_14 80.0 ************************
--------------------------------------------------------------------------------------
Figure 7-5 Simple Differences Comparison Group Report
Figure 7-5 is extracted from one of the comparison group reports of the first kind. The alignment of the vertical bar characters (|) in the Execution Time portion of the report indicates that the two tests "ar_io_integer_oper_14" and "ar_io_integer_oper_15" took essentially the same time to execute (the difference between the times is not statistically significant), while the test "dr_co_access_oper_06" was significantly slower. Note that the report can only point our statistical significance; the user must decide whether the difference is important to the project. The Code Size section shows the differences in code size for the three tests. Again, the user must decide whether the size difference is important.
The second kind of comparison group report is illustrated in Figure 7-6. This kind of report compares two tests of the same functionality, one being hand-optimized and the other being amenable to optimization by the compilation system. The decision (as to whether the potential optimization was performed) is based on the statistical significance of the difference between execution times. If the probability that the times are the same is greater than 90%, then the conclusion is that the optimization was performed ("Yes"). If the probability that the times are the same is less than 10%, the conclusion is that the optimization was not performed ("No"). If the probability is between 10% and 90%, the software does not make a decision ("Maybe"). The probability is based on the number of executions and the standard deviation of the sample of results. Again, the software can only report statistical significance; only the user can decide how much difference is important for the project.
--------------------------------------------------------------------------------------
t'SUCC and t'PRED Functions (use of machine idiom INC/DEC)
--------------------------------------------------------------------------------------
Description
--------------------------------------------------------------------------------------
This pair of problem tests for the use of INCREMENT and DECREMENT instructions to code
Ada's SUCC and PRED functions.
--------------------------------------------------------------------------------------
Time : dr_te_type_enum_01 ( 1.5 ) vs
st_ic_if_condition_02 ( 1.4 ) Maybe
--------------------------------------------------------------------------------------
Size : dr_te_type_enum_01 ( 168.0 ) vs
st_ic_if_condition_02 ( 136.0 )
--------------------------------------------------------------------------------------
dr_te_type_enum_01 => IF hue < black, THEN
hue := color'succ(hue);
END IF;
IF hue > white THEN
hue := color'pred(hue);
END IF;
-- PRED and SUCC functions on enumeration types.
--------------------------------------------------------------------------------------
st_ic_if_condition_02 => IF ei < 6 THEN ei := ei + 1; END IF;
IF ei > 0 THEN ei := ei - 1; END IF;
-- Same computations as in dr_te_type_enum_01 on integers.
--------------------------------------------------------------------------------------
Figure 7-6 Optimization Comparison Group Report
The data presented in the Ancillary Data section of the Main Report is specific to individual tests and is produced by them at execution time. The user should scan this material to see what is of potential importance to the project. Among the topics addressed are the following:
* Timing data specific to some of the classical benchmarks
* Tasking data (control block size, switching time, scheduling method)
* Whether generic code appears to be shared
* Availability of deallocated heap space
* Methods of passing parameters
Much of the data produced by the assessors is in the form of answers provided by the user. The questions are limited in an attempt to make it likely that any two engineers with Ada experience (but no ACES experience) would give the same answers. Nevertheless, the reader of the data must realize that the results are somewhat subjective.
The kind of data produced varies from one assessor to another. The following paragraphs briefly discuss each assessor's output.
The results from using the Symbolic Debugger Assessor are in the form of answers to 117 questions. Each question asks whether a certain capability is supported. Acceptable answers are "Yes", "No", "Partial", and "NA" (Not Applicable). The "Partial" response is used when the debugger can supply some, but not all, of the requested data. The "NA" response is used only in those cases where the question assumes the existence of a previously tested capability that was found to be unsupported.
The general topics of the questions and the number of questions per topic are listed in Table 7-1. For a detailed breakdown, see the User's Guide, Appendix C (also provided as file "yb_tmplt.txt". Please note that the topics in Table 7-1 are intended as an indication of the kinds of information that may be extracted from the report; the actual topics in the report template are more detailed.
Table 7-1 Debugger Assessor Topics
Topic Questions
Breakpoints 21
Examination of Variables 20
Watchpoints 13
Exceptions 10
Single Stepping 8
System State Information 9
Modification of Variables 6
History 5
Applicability to Tasks 5
Integration With Other Tools 5
User Interface 5
Execution Time Degradation (Time Measurement) 4
Control Flow 3
Access to Machine Properties 2
Virtual Calendar 1
When using the results of the Symbolic Debugger Assessor, the user should consider the value of each capability (especially those not supported) to the project. It is quite possible that some capabilities are not important for a particular development effort, and the appropriateness of a particular Ada implementation for an effort should not be judged on the basis of unneeded capabilities.
In the Diagnostic Assessor, 91 source code files are submitted to the compiler. Some of the program units contain deliberately incorrect code that should be reported as a compile-time error, and some contain questionable code that should be reported as a compile-time warning. The remaining source code files should be accepted by the compiler without any diagnostic message. Attempts are made to link various combinations of the resulting library units. Some of these attempts should cause error reports at link time; others should produce executable programs that result in run-time errors. In each case, six questions are asked about each expected diagnostic message. Acceptable responses are "Yes", "No", "Partial", and "NA" (Not Applicable). The "NA" response is reserved for questions that assume a "Yes" response to a prior question that was actually answered "No". The "Partial" response is used when part, but not all, of the expected information was presented. The questions are as follows:
1. Is a diagnostic message printed?
2. Is the message in the general area of difficulty?
3. Is the message at the correct specific location?
4. Does the text of the message clearly define the difficulty?
5. Is relevant non-local information listed where appropriate?
6. Is error recovery appropriate?
The Diagnostic Assessor report template includes four tables (one for each diagnostic category) that may be filled in with the percent of "Yes" and "Partial" answers.
The Program Library Assessor provides three kinds of data: capabilities, time measurements, and space measurements. The capability data is in the form of "Yes", "No", and "Unknown" answers to questions, based on the results of attempting specified actions. The time and space measurements are manually extracted from the log files showing the results of running the Systematic Compile Speed (sy) group of performance tests. There are 53 capability questions and 33 questions that require performance data. Table 7-2 gives the general topics for which data is gathered. For more detail, see Appendix E of the User's Guide, which is also provided as the file "yl_tmplt.txt". Please note that the topics in Table 7-2 are intended as an indication of the kinds of information that may be extracted from the report; the actual topics in the report are more detailed.
Table 7-2 Library Assessor Topics
Topic Questions
Recompilation and different versions 22
Library contents and status inquiries 19
(3 time measurements)
Support for large programs 16
(all time and space measurements)
Unit deletion 9
(8 time measurements)
Generic unit time and space 6
Space reclamation 4
(1 space measurement)
Library integrity 2
Concurrent access 4
Other capabilities and measurements 6
When using the results of the Program Library Assessor, the user should consider the value of each capability (especially those not supported) to the project. It is quite possible that some capabilities are not important for a particular development effort, and the appropriateness of a particular Ada implementation for an effort should not be judged on the basis of unneeded capabilities.
The Capacity Assessor produces data on both compilation-time capacities and run-time capacities. The method of determining these capacities (by automatically generating source code with varying values for a parameter representing the target feature) does not necessarily determine precise limits or even upper bounds. (In many cases, there are no fixed limits; the maximum parameter value may depend on the context.) For each capacity test, there are several possible outcomes, as follows:
* An actual limit is found. (The test succeeds with a given parameter value and fails with the next larger value.)
* An upper bound, but not an actual limit, is found. (The test fails for some parameter value, but time expires before the search method identifies a specific limit.)
* A legitimate parameter value, but no upper bound, is found. (Either the specified maximum time expires without any failure or the test succeeds for the specified maximum parameter value.)
--------------------------------------------------------------------------------------
<test id> <descriptive test name>
INPUT PARAMETERS TEST RANGE MIN/MAX : <value> / <value>
MAX PROCESSING TIME(SEC) : <value>
RESULTS TIME LIMIT EXPIRED(Y/N) : <Y or N>
LIMIT FOUND(Y/N) : <Y or N>
MIN FAILED/MAX PASSED : <value> / <value>
ELAPSED TIME(SEC) : <value>
USER COMMENTS :
--------------------------------------------------------------------------------------
Figure 7-7 Capacity Test Output Format
Each capacity test produces output as illustrated in Figure 7-7. (It is expected that this output will be transferred into the report file.) Note that the user has specified the input parameters, including the minimum and maximum parameter values to be tested ("Test Range Min/Max") and the maximum processing time. The test then reports whether the time limit expired, whether an actual limit was found, the minimum parameter value leading to failure, the maximum parameter value leading to success, and the actual elapsed time. From these results, it is easy to determine which of the cases described above actually occurred.
In most applications of the Capacity Assessor, there is no need to know an actual limit. It is generally sufficient to know that the implementation has the capacity to process the largest parameter value expected for the project. If the time required by the testing process is affordable, then specifying maximum values that satisfy the needs of the project (with a reasonable margin for safety).
The Capacity Assessor attempts to find capacity limits for 32 compilation-time features and 9 run-time features. See the User's Guide, Appendix F for details.
This section details the use of the Ada Compiler Evaluation System (ACES) Quick-Look Facility. It gives a brief discussion of the relationship between the Quick-Look and the entire ACES toolset, with indications as to which should be used for different purposes. Following this discussion, the document presents step-by-step instructions for using Quick-Look. For further information about the ACES, the user is referred to the ACES Primer, the ACES User's Guide, and the ACES Reader's Guide. These documents may be found in "docs.zip" (the compressed ACES Internet distribution file containing the entire ACES document set.)
The Quick-Look Facility allows the user to quickly provide execution-speed data for an Ada implementation. The tests chosen for the Quick-Look Facility provide coverage of the important Ada features, including some Ada 95 features, and a variety of classical and application tests. The Quick-Look Facility is stand-alone software that is part of the ACES. It is much simpler to use than the full ACES, but the results are quite limited. The Quick-Look contains fewer than 75 tests, and only reports execution time. No analysis is performed on the results. Users familiar with the Performance Issues Working Group (PIWG) benchmark set will recognize that the Quick-Look facility provides the same degree of information as the PIWG set, with about the same effort. However, the PIWG set includes tests for Ada 83 only, while the Quick-Look contains some tests for Ada 95 features.
Users trying to determine which of several Ada implementations best meets the needs of a project will want Quick-Look results from all these implementations. To choose the most efficient system from those that have satisfactory Quick-Look results, the full ACES is needed. The full test set contains over 1900 performance tests, grouped into 21 categories according to the kind of language feature being tested. The full test set software reports compilation time, linking time, total compilation and linking time, execution time, and code size. Analysis software is also provided that can perform statistical comparisons of the performance results of several implementations. For each metric, the analysis reports a set of summary statistics ("system factors") for each test group. These system factors are calculated so that ratio comparisons are meaningful.
The Quick-Look user will perform the following steps:
a. Obtain a copy of the Quick-Look software.
See "Availability of the Ada Compiler Evaluation System" section at the beginning of this document.
b. Unzip the compressed file "ql_work.zip" into your working directory. It contains all of the files needed to compile Setup.
Quick-Look expects a "source" directory separate from your "working" directory. This is used to hold files so that they can be copied only as needed into your "working" directory to properly execute Quick-Look.
Note the sample directory structure (Figure 2.1) Quick-Look expects for a distribution directory. If you do not wish to create a similar structure so that Step 0 can automatically copy files into your "source" directory, you can either modify Step 0 or hand copy the files as in Version 2.0.
c. Perform the steps necessary to establish an Ada library for your "working" directory.
d. Compile, link, and execute the Setup program.
This program solicits implementation-specific information from the user and generates command scripts that compile, link, and execute the support programs and performance tests making up the Quick-Look.
e. Execute the top-level scripts (ql_stp*) generated in Step d.
These scripts invoke several simpler scripts that perform copying, compilation, linking, and execution of the support programs and performance tests.
f. Execute the Report program to extract the performance and ancillary data from the log file generated in Step c.
The following paragraphs describe these steps in more detail.
The Quick-Look files are available via anonymous FTP from the AJPO host. Connect to "sw-eng.falls-church.va.us", give "anonymous" as your username, and give your complete e-mail address (as prompted) for your password. Change directory to "public/AdaIC/testing/aces/v2.0". Switch to binary mode and get the file "qlook.zip". This compressed file may be decompressed using the publicly available "unzip" program, which may be obtained from many FTP sites, including "oak.oakland.edu". Read the file "WHERE-IS.UNZ" (in the "aces" directory) for further information on where to obtain this program.
You can also access this through the Ada IC server on the World Wide Web. The URL is: http://sw-eng.falls-church.va.us/AdaIC/testing/aces/
When you decompress "qlook.zip" (using "unzip -a qlook"), a subdirectory "aces/qlook" will be created in your current working directory. All the Quick-Look files will be placed in this subdirectory.
To prepare and to use the Setup program, execute the following steps:
a. Obtain the following files by unzipping the file ql_work.zip (found in qlook.zip) into a working directory.
ql_defs.ada zc_delsb.tpl zc_setli.tpl zp_envrn.udf ql_quest.txt zc_dlada.tpl zc_incld.tpl zp_envrn.ugn ql_scrnm.txt zc_dlexc.tpl zp_envrn.def zp_envrn.uns ql_stp00.tpl zc_dlhex.tpl zp_envrn.dgn zp_envrn.uth ql_stp01.tpl zc_holi.tpl zp_envrn.dns zp_envrn.uvr ql_stp02.tpl zc_horun.tpl zp_envrn.dth zp_envrn.vns zc_adaho.tpl zc_linho.tpl zp_envrn.ngn zp_envrn.vva zc_adaop.tpl zc_lnk.tpl zp_envrn.nth zp_setup.ada zc_cpada.tpl zc_run.tpl zp_envrn.nvr
b. Predefined information for your operating system/compiler implementation can be found in the following files. If yours is not listed you can edit one of the existing files for your use. The file zp_envrn.udf contains the symbols, but none of the values for Setup. It is provided for your convenience.
zp_envrn.dgn DOS/GNAT
zp_envrn.dns DOS/non-specified compiler
zp_envrn.dth DOS/Thomsoft (Alsys)
zp_envrn.ngn non-specified operating system/GNAT
zp_envrn.nth non-specified operating system/Thomsoft (Alsys)
zp_envrn.nvr non-specified operating system/Rational (Verdix VADS)
zp_envrn.udf user defined operating system and compiler
zp_envrn.ugn UNIX/GNAT
zp_envrn.uns UNIX/non-specified compiler
zp_envrn.uth UNIX/Thomsoft (Alsys)
zp_envrn.uvr UNIX/Rational (Verdix VADS)
zp_envrn.vns VMS/non-specified compiler
zp_envrn.vva VMS/Vax-Ada
c. Take whatever steps are necessary to compile Ada programs in the working directory. (Create library, Set library, etc.)
d. If your compilation system requires some file suffix other than ".ada", then rename the ".ada" files accordingly.
e. Compile "zp_setup.ada" and "ql_defs.ada", in that order.
f. Link and execute the program "setup".
This program asks the user for certain symbols and commands that are specific to your operating system and your Ada implementation. If you are using one of the operating system/compiler combinations mentioned in (b) the corresponding "zp_envrn.???" file will provide default answers for the operating system and compiler questions. There is also the opportunity to use general defaults (found in "zp_envrn.def"). This is recommended for first time users and eliminates the need to answer 10 additional questions, but you must still answer some questions that are specific to your Ada installation.
To override the default values or to change the values you entered, run "setup" again and choose Option 1 to modify some of your answers. Note that your answers to the cross-compiler question and the Ada 95 question cannot be changed in this way. Your answers to these two questions determine what other questions are asked. Select Option 3 to make those changes.
It is important to use different extensions for both compiled files and command script files in order to eliminate file name conflicts. We recommend that you always use a filename extension for command scripts, even if your operating system does not require it.
g. If the functionality referred to in a question requires more than one command on your system, then prepare a command script file to perform that function. In response to the appropriate question, give the command to invoke that command script. This is especially important for GNAT users, who will generally need to supply a "compile" script that first invokes GNATCHOP and then invokes the script generated by GNATCHOP.
h. If you are working on a UNIX system or another operating system that requires it, change the protection mode of all the command script files so that they are executable.
i. By default, the Quick-Look software measures execution time using calls to "Calendar.Clock". That is, the performance tests report elapsed time. If you wish to measure CPU time instead, then edit the file "ql_clock.ada" (in the "source" directory) and replace the body of the function with a body that returns CPU time.
j. Execute the command script ql_stp0. It will copy the qlook files from the distribution to the source directory. See Section 3 for further explanation.
The following files should be copied into your source directory:
ap_av02_.inc ql_dum02.ada su_pmpkg.ada zc_link2.tpl ap_kf01_.inc ql_dum03.ada su_pp19_.inc zc_lnk.tpl ap_ob01_.inc ql_dum04.ada su_se08_.inc zc_run.tpl ap_obpkg.ada ql_dum05.ada su_si04_.inc zc_setli.tpl ar_io19_.inc ql_dum06.ada su_sl10_.inc zc_time1.tpl ar_mt01_.inc ql_dum07.ada su_slpkg.ada zc_time2.tpl ar_mt02_.inc ql_dum08.ada tk_at01_.inc zg_glob1.ada ar_mtpkg.ada ql_dum09.ada tk_in01_.inc zg_glob2.ada ar_rn01_.inc ql_dum10.ada tk_lf08_.inc zg_glob3.ql cl_dh02_.inc ql_dum11.ada tk_lf26_.inc zg_glob4.ada cl_wh02_.inc ql_dum12.ada tk_lf33_.inc zg_glob5.ada dr_ad01_.inc ql_list.txt tk_lf38_.inc zg_glob6.ada dr_ba11_.inc ql_mn01.inc tk_lf43_.inc zg_incld.ql dr_bapkg.ada ql_mn02.inc tk_rz01_.inc zg_incld.tpl dr_dt01_.inc ql_mn03.inc tk_rz02_.inc zg_iniso.cro dr_dt02_.inc ql_mn04.inc tk_rz03_.inc zg_iniso.sel dr_dt03_.inc ql_mn05.inc tk_rz21_.inc zg_init.ql dt_dp08_.ada ql_mn06.inc tk_rz22_.inc zg_start.ql dt_dp09_.ada ql_mn07.inc tk_rz23_.inc zg_stop0.ql dt_dp10_.ada ql_mn08.inc zc_adack.tpl zg_stop2.ql dt_dp11_.ada ql_mn09.inc zc_adaho.tpl zg_verfy.ql dt_dp12_.ada ql_mn10.inc zc_adano.tpl zm_dblma.a95 dt_dp13_.ada ql_mn11.inc zc_adaop.tpl zm_dblma.por dt_dp14_.ada ql_mn12.inc zc_adasp.tpl zm_depen.por dt_dppkg.inc ql_quest.txt zc_cmpck.tpl zm_genma.ada gn_in10_.inc ql_reprt.ada zc_cmpno.tpl zm_math.a95 gn_su16_.inc ql_rzm01.inc zc_cmpop.tpl zm_math.por gn_su17_.inc ql_rzm02.inc zc_cmpsp.tpl zp_envrn.def oo_dp01_.inc ql_rzm03.inc zc_cpada.tpl zp_envrn.dgn oo_dp02_.inc ql_rzm17.inc zc_cpyck.tpl zp_envrn.dns oo_dppkg.ada ql_scrnm.txt zc_cpyno.tpl zp_envrn.dth oo_tx01_.inc ql_stp00.tpl zc_cpyop.tpl zp_envrn.ngn oo_tx02_.inc ql_stp01.tpl zc_cpyrn.tpl zp_envrn.nth oo_tx03_.inc ql_stp02.tpl zc_cpysp.tpl zp_envrn.nvr oo_tx04_.inc ql_tkdum.ada zc_delbo.tpl zp_envrn.udf oo_tx05_.inc ql_tkm00.inc zc_delsb.tpl zp_envrn.ugn oo_tx06_.inc ql_tkm08.inc zc_delso.tpl zp_envrn.uns oo_tx07_.inc ql_tkm26.inc zc_dlada.tpl zp_envrn.uth pt_po01_.inc ql_tkm33.inc zc_dlexc.tpl zp_envrn.uvr pt_po02_.inc ql_tkm38.inc zc_dlhex.tpl zp_envrn.vns pt_po03_.inc ql_tkm43.inc zc_holi.tpl zp_envrn.vva ql_clock.ada ql_work.zip zc_horun.tpl zp_setup.ada ql_defs.ada su_pa01_.inc zc_linho.tpl ql_dum01.ada su_pm07_.inc zc_link.tpl
k. Execute the command script "ql_stp01" (with whatever command script suffix you specified). Capture the output in a log file.
If you are using a cross-compiler, you will need to edit the log file and copy a section of output into another file. The section of output is bracketed by lines containing "BEGIN QL" and "END QL". Copy the lines between these two to a file named "zg_cpy.ql". If any system messages appear, delete them. The "zg_cpy.ql" file should contain only compilable source lines, including several comments and several assignment statements.
If you are using a self-hosted compiler, the "zg_cpy.ql" file is created automatically.
l. After creating "zg_cpy.ql" (if necessary), execute the command script "ql_stp02" and capture the output in a log file which is needed as input for the report program ql_reprt. It expects a default name of "ql.log", but you can change that while running the report program.
One of the functions performed by the "ql_stp02" script is the compilation and linking of a report program, "ql_reprt". After the "ql_stp02" script has executed all the performance tests, you may use "ql_reprt" to extract the performance data (and some ancillary data) from the log file. Three report files are created, as described in the following paragraphs. (The program provides default names for these report files, as well as for the input (log) file, but gives the user the opportunity to override the defaults.) Note: The input file for this report program is the output log file created by running ql_stp02.
Each performance test executes its test code many times, measuring the execution time at each iteration. After a certain number of iterations, stability statistics are calculated. If these statistics do not indicate that the measurement variability is within preset limits, the number of iterations is increased, and the test loop executes again. The execution time is not reported until a stable sample is found.
The execution time report extracts the reported execution time for each appropriate test and displays it in the following format:
PROBLEM NAME TIMING STANDARD
(Microseconds) DEVIATION
-------------------------------------------------------------------
dr_ad_array_dynamic_01_a 5.1053E-01 2.2%
... (other tests)
The TIMING column gives the mean execution time, expressed in microseconds, for the sample (loop iteration) that showed sufficiently stable results. (Again, the default time measurement represents elapsed time; the user may choose to use CPU time as discussed above.) The STANDARD DEVIATION column gives the standard deviation of that sample, expressed as a percent of the mean. For a stable system with no active daemons or contending processes, you should expect small standard deviations (less than 5%).
For some tests, the report file may contain a message instead of timing results. In most cases, this means that some error has prevented the capture of timing results. Moreover, the "dt" (delay timing) test results are not given in the execution time report. (Since faster is not better when dealing with the performance of the "delay" statement, we don't want to report the execution time for these tests in the same way as the others.)
Among the messages that may appear are the following:
Message Meaning
Failed to compile "Dummy" version executing, suggesting that the actual test did not compile.
System load has changed Null loop timing results out of baseline range.
Error large negative time Observed time is significantly greater than null loop time -- may be a really smart optimizer.
Error at execution time Unexpected exception occurred.
Error_no_data Some serious error prevented even the "dummy" version from running.
Error_packaging An earlier failure in the same executable prevented test execution.
Delay timing test Execution time not reported for "dt" (delay timing) tests. (See ancillary report.)
This report contains the same data as does the text version, but the values are separated by commas. This allows importation of the report file into a common spreadsheet program or other presentation and analysis tool. Error messages are not included; tests with error reports are simply omitted.
This report contains output from selected performance tests, other than the execution time. For example, the Dhrystone test reports the number of Dhrystones per second; certain tasking tests report time per rendezvous; and the "dt" (delay timing) tests report the requested delay, the quantized request, and the actual execution time.
ACEC Ada Compiler Evaluation Capability
ACES Ada Compiler Evaluation System
AES Ada Evaluation System (UK Ministry of Defence)
AJPO Ada Joint Program Office
CA Comparative Analysis
CPU Central Processing Unit
DEC Digital Equipment Corporation
DoD Department of Defense
HOLCF High Order Language Control Facility
LRM (Ada) Language Reference Manual (ANSI/MIL-STD-1815A)
MoD Ministry of Defence
NUMWG Numerics Working Group (ACM SIGAda organization)
PIWG Performance Issues Working Group
SSA Single System Analysis (ACES analysis tool)
VAX Virtual Address eXtension (DEC family of processors)
VMS Virtual Memory System (DEC operating system for VAX processors)
Cross Reference Index for ACES Document Set
Acronyms, Abbreviations
Primer 9
RG 13.1
UG 11
Adding/Modifying Tests
RG 6.8
UG 5.4.3, 9.1.6
Addressing
Primer 3.2.2, 3.2.5
RG 2.4.3, 6.7, 8.4.1, 10
UG 4.3.3.3, 5.1.1, 5.1.2
Analysis, Running
Primer 1.2, 2.1.3, 3.2.4, 3.2.11, 3.1.3.1, 4, 4.1.1, 5, 5.2.1,
5.3.1, 6.4.1, 6.5, 6.5.2, 7.3.3
RG 2.4.2.3, 2.4.2.4, 2.4.2.5, 3.2.7.3, 3.6, 3.6.3, 3.6.4, 5.1
UG 7, 9.1, 9.2, 9.3, 9.4, 9.5, 9.6
Capacity Assessor
Primer 6.1, 6.5, 6.5.1, 7.3.4
RG 2.4.3, 3.6.4, 8.4
UG 4.3.3.2, 8.4, App. F
Code Size
Primer 1.4.1, 1.4.2, 3.2.2, 3.2.5, 5.3.2, 7.2.1, 7.2.2
RG 5.2.2.1.1, 5.2.2.2.1, 5.2.2.3.1, 5.4.2, 5.4.2.2, 5.4.4.1
UG 4.3.3.3, 5.1.1, 5.1.2, 9.3, 9.3.2, 9.4
Comparative Analysis
Primer 3.2.14, 5.2, 5.2.1, 7.1
RG 2.4.2.3, 5.3, 7.2
UG .4
Compatibility of Test Suite
RG 11
UG 4.3.2, 4.3.3.3, 5.2, 5.1.4, 5.1.6.1, 9.4.3, 10.1
Condense
Primer 3.2.13, 5.1
RG 5.2
UG 9.1.7, 9.3
CPU Time
Primer 3.2.2, 3.2.3, 3.2.4, 3.2.5
RG 6.3.2.6, 6.4.2.2, 6.4.2.3
UG 4.3.3.3, 5.1.1, 5.1.2, 5.1.3, 5.4.1
Data Summary Table
Primer 7.1.2
RG 5.3.2.2.3
Decision Issues
Primer 1.4, 1.4.1, 1.4.2, 1.4.3, 3, 3.1, 7.2.2
RG 3.2.6.1, 3.6.2, 5.3, 5.3.2.2.6, 5.3.2.2.7, 5.4.4.3, 6.4.1,
7.2, 7.3
UG 5.1.1, 5.1.6.3, 5.4.3, 6.1, 6.3, 6.9.6, 6.10.7, 9.1.1, 9.3.7,
9.4.3, 9.6
Diagnostic Assessor
Primer 6.3, 7.3.2
RG 2.4.3, 3.6.3, 8.2
UG 2.2, 4.3.3.2, 8.2, App. D
Erroneous Tests
RG 10
UG 6.10
Exceptional Data Report
Primer 3.2.13, 5.1
RG 5.2.2.2
UG 9.3.3
File Name Conventions
Primer 2.1
RG 5.2.2
UG 4.3.3.1.1
Globals
Primer 3.2.1, 3.2.5, 6.5, 6.5.1
RG 3.2.5.2, 8.4.2
UG 4.3.3.3
Harness
Primer 2.1.2, 3.2.11, 4.1, 4.3
RG 5.2
UG 6.0
History (ACES)
Primer 1.1
RG 2, 2.1, 2.2, 3.1, 7.1
UG 2
Include
Primer 2.1, 3.2.5, 3.2.10, 4.3.2
RG 4.3.3.1.1, 5.5.2, 6.5, 6.11, 9.1.6
Interfacing Spreadsheets
Primer 1.1, 5.1, 5.2.4, 7.1.2
RG 2.3, 9.3, 9.3.1, 9.3.2
UG 2.3, 9.3
Interpreting Results
Primer 1.2, 3, 4.2, 7
RG 5.3.2.2.1, 7, 6.1
UG 2, 2.1, 4.2, 9.3.4.2
Level of Effort
Primer 6.5.1.3, 6.5.4
RG 2.4, 7.2, 7.6, 8.1
UG 6.5.1.3, 6.5.4
Math Implementation
Primer 3.2.5
RG 3.2.3
UG 5.1.4, 5.1.6.1
Operating System Commands
See UNIX Commands
Optimization
Primer 5.3.1, 5.3.3, 7.2.2
RG 6.2, 6.3.2, 6.6
UG 5.1.6.2, 9.1.1
Output
Primer 1.2, 1.4.1, 3.1.1, 3.2.1, 4.3.1, 7.1.1
RG 5
UG 7.3, 9.1, 9.2.2, 9.3.3, 9.4.1, 9.5.1
Performance Tests
Primer 1.2, 2.1.3, 4, 4.1
RG 5.4.2.2, 5.4.4.4, 8.3.1
UG 4.3.2, 7.0
Pretest
Primer 3.2, 3.2.1 - 3.2.15
RG 3.2.3, 6.3.1, 6.5, 10
UG 5.2, App. B
Program Library Assessor
Primer 6.3, 6.4, 6.4.1, 7.3.3
RG 2.4.3, 3.6.2, 8.3
UG 4.3.3.2, 8.3, App. E
Referenced Documents
RG 1, 1.1, 1.2
UG 1, 1.1, 1.2
Reports
Primer 4.2, 6.5.2
RG 2.4.2.3, 2.4.2.4, 2.4.2.5, 2.4.3, 3.1, 3.2.4.2, 3.2.7.4, 3.6,
4, 5
UG 4.2.1, 6.5.3.3, 6.10.7, 6.11, 7.1, 7.3, 8
Resources Needed
Primer 3.1.3.3, 5.1, 6.3.1, 6.5.2
RG 3.2.7.3, 3.1.3.2, 3.6.2, 3.6.4, 8.2.1, 8.3.1
UG 4.2, 4.2.1
Setup
See Pretest
Simulators
RG 6.9
UG 5.4.1
Single System Analysis
Primer 3.2.15, 5.3, 7.2
RG 5.4
UG 9.5
Symbolic Debugger Assessor
Primer 6.2, 6.2.1, 6.2.2, 7.3.1
RG 2.4.3, 3.6.1, 8.1
UG 8.1, App. C
Testing Scenarios
Primer 1.4
RG 2.1.3.1, 3.2.1, 3.2.3, 3.6.2, 5.1, 5.4, 6.3.2.6, 8.1, 8.2,
8.4, 10
UG 4.2.2, 6.11
Timing Techniques
Primer 2.1, 2.1.3, 3.1, 3.2.5, 3.2.10, 6.4, 7.1.2
RG 6.0, 6.2, 6.3, 6.3.1, 6.3.2, 6.3.2.5, 6.7, 7.3
UG 7.3
Usability
Primer 1, 1.2
RG 2.1, 2.4.2.2, 3.1, 3.2.1, 3.2.3, 3.6.2, 5.1, 5.4, 6.3.2.6,
8.1, 8.2, 8.4, 10
UG 4.2.2, 5.1.6.1.3, 6.11
User Adaptation
RG 2.4
UG 6.11, 9.6
User Feedback
RG 12
UG 10