INTRODUCTION TO THE ADA COMPILER EVALUATION SYSTEM
The Ada Compiler Evaluation System (ACES) provides
performance tests, test management software, and analysis software for
assessing the performance characteristics of Ada compilation and
execution systems. Functionality/usability assessor tools are also
provided for examining the implementation's diagnostic system, library
management system, and symbolic debugger, as well as for determining
compile-time and run-time capacities of the implementation.
The ACES is a combination of the best features of the Ada Compiler Evaluation Capability (ACEC)
and the Ada Evaluation System
The ACES 2.1 software is contained in a set of compressed files.
These files can be uncompressed with the
unzip utility, installed in a series of subdirectories and be
used to evaluate an Ada Compiler for a specific system. Complete documentation is available. A brief subset of
tests can be found in the Quick Look test
The tests are divided into 21 Performance Test
Groups, each dealing with a specific area for evaluation. For
efficiency, several groups are included in each of the zipped files.
When unzipped they will automatically be stored in 21 individual
subdirectories. The assessors, documents, support and "Quick Look"
files are each in their own zip file and, when unzipped, will be
stored in their own subdirectories.
The ACES is distributed as the following set of 11
compressed files which can be downloaded
from the AdaIC web site:
- contains all of the application, arithmetic and classical tests
- contains all of the data storage, data structures and delays and
- contains all of the generic, interface and input-output tests
- contains all of the miscellaneous, object oriented, and
- program organization, protected types and storage reclamation tests
- contains all of the statement, subprograms and systematic
compile speed tests
- contains all of the tasking, user defined, and exception handling tests
- contains the complete Quick Look subset of
ACES that can be run quickly with limited functionality
- contains all of the support files for the Setup
and Pretest utilities which are used to create Harness to run the
tests and Analysis to view the results
- contains the four ACES usability assessors. These units support
the evaluation of functionality and usability of the diagnostics
system, the symbolic debugger, and the library management system.
They also provide software for determining the compile-time and
run-time capacities of the tested system.
- contains the four basic ACES documents in both ASCII ("*.txt")
and PostScript ("*.ps") form. Also included are document files whose
names include the "x" character to indicate appendices.
In addition, the requirements and design documents are available in both
ASCII and PostScript form in the directory /public/adaic/compilers/aces/info.
Web-browser forms of these documents are also available. All user-interface,
setup, test selection, and analysis software are covered under "Support
Software"; the "Operational Software" consists of performance tests and
2.1. Set the directory
Change your current working (default) directory to the one is
which you want the subdirectory "aces" to be created.
When the files are uncompressed, a subdirectory
"aces" will be created in your current working directory, and the ACES
files will be stored in a directory tree rooted at "aces".
2.2. Download the necessary files
can be downloaded from the Ada IC web site at:
NOTE: the path to this subdirectory may be changed by the server
administrator without notice. Every effort will be maintained to make
the search as logical as possible.
2.3. Unzip the files
For each of the 11 ".zip" files named above,
apply the "unzip" program to that ".zip" file. The order in which you
unzip the files is not significant. Be careful to add the '-a' option
to ensure the proper handling of text files.
The files were compressed with the Info-Zip utility.
The required "unzip" program (for most popular platforms and operating
systems) is available from several FTP sites. For more information,
see the file www.adaic.org/compilers/aces/WHERE-IS.UNZ
If you uncompress all of the files, the following subdirectories containing the indicated files will be
There are two versions of the ACES. Quick Look is a shorter, faster
to use but more limited version. The full ACES includes the Pretest
steps and the Setup, Harness and Analysis utilities.
- (Procedures to determine compile and run time capacities)
- (Procedures to evaluate debugger capabilities)
- (Procedures to evaluate quality of diagnostics system)
- (Procedures to evaluate capabilities of Ada library system)
- (Document files in both PostScript (".ps") and ASCII (".txt") formats)
- (All the files needed to use the ACES Quick-Look facility)
- (Software to set up the ACES for use, manage performance testing,
and analyze performance results)
- aces/tests (Groups of performance tests)
- aces/tests/ap ... Application
- aces/tests/ar ... Arithmetic
- aces/tests/cl ... Classical
- aces/tests/do ... Data Storage
- aces/tests/dr ... Data Structures
- aces/tests/dt ... Delays and Timing
- aces/tests/gn ... Generics
- aces/tests/in ... Interfaces
- aces/tests/io ... Input/Output
- aces/tests/ms ... Miscellaneous
- aces/tests/oo ... Object Oriented
- aces/tests/op ... Optimizations
- aces/tests/pt ... Protected Types
- aces/tests/po ... Program Organization
- aces/tests/sr ... Storage Reclamation
- aces/tests/st ... Statements
- aces/tests/su ... Subprograms
- aces/tests/sy ... Systematic Compile Speed
- aces/tests/tk ... Tasking
- aces/tests/ud ... User Defined
- aces/tests/xh ... Exception Handling
The Setup utility is designed to prompt the user for system and
compiler specific issues and to generate scripts to run the Pretest
steps that will lead to the compilation of both Harness and Analysis.
The Harness utility creates a script which the user can run, sending
the results to a log file. The user then can use that log file to
update the statistics in Harness and for input to Analysis.
The Analysis utility takes the results of the log files produced by
the script that Harness created and outputs reports.
With the Setup, Harness, and Analysis utilities and the Quick Look and
Pretests steps, ACES is virtually automated. After the files are
downloaded and unzipped into a configuration management area, the
initial files needed to compile Setup for Quick Look or Pretest can be
found in the ql_work.zip or zp_work.zip files that are in the
qlook.zip and the support.zip files respectively.
For detailed instructions on using ACES, see the Primer , (file "docs/zd_primr.txt"
or "docs/zd_primr.ps"). Further information is found in the User's Guide, ("docs/zd_us*.*") and
the Reader's Guide
("docs/zd_readg.*"). For questions that cannot be answered from the
documentation, see the points of contact
The Setup facility, introduced in Version 1.1, has been enhanced from
Version 2.0. Setup prompts the user for information about the
operating system and compiler/linker commands, and uses this
information to generate command script files for compiling global
support modules, creating test programs for the support modules,
creating test management tools, and creating analysis tools. The
following improvements are offered in this version:
4.1. Default information for several common operating systems and
compilers is supplied, eliminating approximately 32 of the questions a
user must answer.
4.2. Default information is supplied for the convenience of first time
users. Taking advantage of this feature eliminates and additional 10
questions that the user is otherwise required to answer. More
experienced users may also want ot take advantage of this convenience.
4.3. Once an environment file (zp_envrn.txt) is created, the user may
interactively modify selected information and automatically
re-generate the command scripts unless a change is made in the version
of Ada (83/95) or in the compiler (cross or self hosted). Then the user
must answer all of the questions again since different questions are
presented for each of the four possible combinations.
Version 2.1 is comprised of approximately 2120 tests. There are 144
tests that are totally new to ACES. Most of these were designed to
test the performance of Ada 95 features. However, there are also new
Ada 83 tests for comparisons of Ada 95 and Ada 83 coding styles. They
include tests for six predefined performance
The Quick-Look facility, new to ACES Version 2.0 has been improved for
Version 2.1. It is a stand-alone facility that allows the user to run
a limited set of performance tests and obtain raw execution timings in
less than a day. Quick-Look provides coverage of the important Ada
features, including some of the Ada 95 features, and a selection of
the classical and application tests. It was originally based on the
most useful tests of the Performance Issues Working Group (PIWG)
The Quick-Look software uses a Setup utility similar to the full ACES.
It makes use of the same default files to minimize user input.
However, there are only three steps created for the user to execute.
A report generator extracts the execution time data (and other
ancillary data) from the log file created in step 2 and produces three
reports: a textual report showing the execution times; an ancillary
report showing other useful data output by the tests; and an optional
comma-delimited report of the execution times. The comma-delimited
report can be imported into many popular spreadsheet programs for
By default, Quick-Look reports elapsed (wall-clock) time, using the
Ada Calendar.Clock function. CPU time measurements can be used, but
the user must supply a CPU time function.
The "qlook.zip" file contains all the test files needed for Quick-Look
and all the necessary support software and data for compiling them,
executing them, and generating reports from the results. It also
contains ql_work.zip that has all of the files necessary to compile
Setup. Hence, users
wishing to use Quick-Look alone may download and decompress this
single file. Instructions for using Quick-Look are provided in Section 8 of the Primer, which is
included in the "qlook.zip" file as files "zd_primr.txt" (ASCII) and
An effort has been made to remove all incompatibilities with Ada
95. That is, all software should be compatible with Ada 95
requirements. Code that is not compatible with Ada 83 is isolated so
that the Ada 83 user should never encounter it.
There are now two pages for the Groups Menu which lists all the
performance test groups. The user may select all tests or flip
between the two pages to select individual groups.
Detailed instructions are now included in Section 9.1.1 of the User's
Guide to compile the Analysis Tools separately.
The Menu structure was redefined to allow the user to run any of the
three tools with a predetermined set of defaults, a previously created
request file, or the standard v2.0 individual selection.
8.2. User Defined Benchmarks
There is a template for the user to create new tests. further
instructions can be found in Section 5.5 of the User's Guide. Once
created, these can be selected when running Harness, Condense or
A number of performance Issues were defined for this version of ACES.
In addition to selecting tests individually, by subgroup or by group,
the user can now select associated tests by six predefined performance
- Floating-Point Data/Operations
- Integer Data/Operations
- Fixed-Point Data/Operations
- Character and String Data/Operations
- Representation Clauses and Attributes
8.4. Setup for Pretest and Quick Look
The setup procedure is designed as a tool that will, with minimal user
input, compile the programs that will run and evaluate selected tests.
For this version, it has been enhanced by the addition of a number of
optional files with default answers to free the user from being
required to answer all of the thirty plus questions each time the
setup executable is built. There are default files for a select
number of compiler/ operating system combinations, a template for
creating a default file for a specific system, and a general default
file with answers to other questions that could be used by any system.
To eliminate hand copying of files, an additional step has been added
to automatically copy all of the files needed from a configuration
management area into a user accessible source directory.
A special compressed file within the regular "zipped" file contains
all of the files needed to compile the initial Setup program.
This section contains reports of problems discovered after ACES
Version 2.1 was frozen. It will be updated as new problems are found.
Workarounds are noted where appropriate.
9.1. Template Files
It has been reported that one or more of the template files in "zp_work.zip"
and "ql_work.zip" contain hard-coded '/' characters in path
Workaround: Such characters should be replaced with the symbolic name "path_separator".
9.2. Null Loop Timing
It has been observed that reported times are less than they should be; in
particular, negative times are sometimes reported. The proposed
explanation is this:
When the null loop timing is calculated, the timing test is done with checks
enabled, so that the reported time is inflated. When the actual tests are
run, checks are (generally) suppressed, so that the time attributable to loop
processing is smaller than the time calculated earlier. When the inflated
calculated loop time is subtracted from the total processing time, the result is
too small (and is sometimes negative).
Workaround: If the above explanation is correct, then the null loop timing
should be calculated with checks suppressed. Note
that the proposed explanation has not been verified and that this workaround has
not been tested.
9.3. Performance tests -- Pragma Priority not supported
Many of the tasking tests use Pragma Priority to ensure that rendezvous
occur in a predictable order. The Ada 95 standard does not require that
Pragma Priority be obeyed unless the compiler vendor has implemented
the Real-Time Annex (Annex D). Thus, for an Ada 95 compiler that does
not support Annex D, these tests may not behave as expected.
Workaround: No satisfactory workaround is known at this time. If an
Ada 95 compiler does not support Pragma Priority, then tasking tests
that depend on it may produce misleading results. The following tests
contain Pragma Priority.
- sr_im28_.inc..sr_im33_.inc, sr_im36_.inc..sr_im37_.inc, sr_im49_.inc
- tk_lb01_.inc..tk_lb03_.inc, tk_lb05_.inc
- tk_lf03_.inc..tk_lf04_.inc, tk_lf06_.inc..tk_lf19_.inc
- tk_lf21_.inc..tk_lf31_.inc, tk_lf42_.inc..tk_lf48_.inc
- tk_ls01_.inc..tk_ls21_.inc, tk_ls23_.inc, tk_ls25_.inc..tk_ls33_.inc
9.4. Performance tests -- Priorities in the delays-and-timing tests
The DT series tests count on the "main" for each of the tests to have
a priority higher than that of the tasks in the test. This is
effected by putting Pragma Priority into the test's main subprogram.
However, each test's "main" subprogram is actually a library procedure
that is called from a "main" program generated by Harness. For some
compilers, Pragma Priority is ignored unless it is in the outermost
("main") procedure or in a task. The test then has a default priority
for the environment task, and the test run interprets the resulting
execution sequence to mean that preemptive scheduling is not
supported, resulting in an abort of the test, and a failure result.
Workaround: No satisfactory workaround is known at this time.
None so far.
Electronic Data Systems
Ada Information Clearinghouse