Appendix F: Product Characteristics The following product characteristics are used in Table 5 in the main document. They are defined below [ANSI/IEEE 90] [Lawlis 89], along with an explanation of how each characteristic affects the software development process. A product evaluator must assign a rating for each characteristic of interest. The value assigned to each characteristic is a number between 0 and 10, where 0 means this characteristic of the product is extremely poor or non-existent, and 10 means this characteristic is extremely good or as good as it could possibly be. Product characteristics of interest generally fall into four categories – cost, performance, tool support, and usability. Each of these areas is considered in the following sections. Cost Cost is always a consideration when buying any product. Cost value – the relative value of the total cost of the product as compared with other products considered. The total cost of purchasing a candidate product is calculated in Table 4. In Table 5, how this cost compares with the cost of other products, as well as the relative importance of cost as a product selection factor, can be figured into the final product decision. The product characteristic of cost value indicates how well the cost of this product compares with other products being considered. The cost value should be high if the cost is relatively low, and, conversely, the cost value should be low if the cost is relatively high. The importance of cost as a selection factor is determined by the weight assigned to the cost value characteristic. Performance Various types of performance issues can be important when considering a software development tool set. These issues generally center around the code translator (compiler). Compilation speed – the performance of a translator (compiler or assembler) in the areas of compilation speed, size of object code produced, and ability to produce correct results. This characteristic is an indicator of the performance of the product with respect to compilation. Compilation speed can have an effect on the productivity of those using the development product, but it does not impact those using the software produced by the development team (see Execution speed for that). The size of the executable file produced by the compilation is an issue for executables which must meet size constraints. Another issue is whether the compiler can process large programs, at least as large as expected on the project, and the effect of program size on speed of compilation (see Ability to handle large programs). While larger programs are expected to take longer to compile, the execution speed can be orders of magnitude slower as the compiler reaches its capacity limits. The correctness of the code produced by compilation is of particular concern if optimization techniques have been used in producing the executable code. Execution speed – the performance of object code produced by a translator in terms of execution speed. Execution speed is a particularly tricky characteristic to measure. It is common to run benchmark programs on each candidate product to compare the execution times. However, it should be recognized that each benchmark program has been created to exercise a particular mixture of selected language features, so compiler products may fare differently with different benchmark programs. Benchmark programs which are representative of the mixture of language features expected in programs developed using the compiler should be selected. Suites of tests are also often available for measuring execution speeds of particular compiler features. Ability to handle large programs – the performance of a translator with respect to large programs in terms of limits of capacity and efficiency of performance. It is not uncommon for a product to work very well under expected operating conditions, but to become completely unusable when stressed beyond its expected limits. If a product is to be used in the development of a large software system, it must be stress tested to determine if it can handle the expected system size, number of compilation units, etc., without slowing down to less than acceptable response times (see Compilation speed). Tool Support The type of available tool support determines whether or not a tool set can support development requirements. The most common types of tool support are described in this section. Tool support for requirements specification / risk analysis – the extent of automated support for the preparation of the specification of requirements, including both textual and graphical forms of documentation, and the assessment of risk associated with these requirements. Requirements specification is an important part of the development process, both for capturing the initial system requirements and for evolving those requirements as the development proceeds. Risk assessment is closely tied to the specification and evolution of system requirements. System testing also depends on the specification of system requirements. Without automated support, requirements are often not specified to the desirable level of detail, and they are usually not updated as requirements change, adding to the system risk. Hence, automated support for requirements specification facilitates the process of keeping the development under management control. Tool support for design – the extent of automated support for the process of defining the architecture, components, interfaces, and other characteristics of a system or component. Design is a very creative process, the results of which are difficult to capture under the best of conditions. Without automated support for design, many of the important design decisions (why a process was designed one way as opposed to another, what assumptions were used, what constraints were considered, how problems were resolved, etc.) often go undocumented, and even the design structure is often not properly addressed. Automated support facilitates the creation of graphical diagrams to express the design structure, as well as the capture of design decisions which are important, not only to those developing the original system but also to those who will later maintain it. Tool support for code generation – the extent of automated support for translating a requirements or design specification into source code for a program. If an automated design tool has been used, it is often possible to generate a code template from the design specification. This automates a very straightforward part of the coding process, ensures that the code reflects the design, and permits the programmers to spend their time on the part of the programming process that cannot be automated easily. In some well-defined domains, enough knowledge of software systems for that domain can be captured in automated tools that a large portion of the software can be generated automatically from specifications provided by non-programmers using the tool. These are usually referred to as 4GLs, although those which use knowledge-based systems are sometimes called 5GLs [ANSI/IEEE 90]. A caution applies here, however. A 4GL or 5GL should not be used outside of the domain for which it was intended, for its effectiveness is lost [Appleby 91] [DoD 87] [Glass 91] [Martin 86] [Misra 88] [Wojtkowski 90]. Also, the quality of software produced by a 4GL or 5GL is often lower than the quality of a similar system produced with a 3GL [Misra 88] [Verner 88] [Wojtkowski 90]. Tool support for documentation – the extent of automated support for generating reports, certifying activities, and revising and managing any written or pictorial information describing, defining, or specifying requirements, procedures, or results. This is the type of documentation that is rarely done unless it has automated support. Yet it is also the type of documentation that is important for maintaining management control of a project. Tool support for interfacing with a database – the extent of automated support for creating and managing a database to be used by the developed software. A commercial database product is often used to provide the database in systems requiring one, and the product will usually provide its own 4GL for interfacing with the database. In this case, the ability of the development language to interface with the database product may also become an issue (see the language characteristic of mixed language support as well as the product characteristic of tool support for interfacing with other languages). If a database product does not provide an appropriate interfacing mechanism, or if a special database will be created without using an existing database product, then the interface from the development language to the database must either come from another source or else be created as a part of the system development. Tool support for testing – the extent of automated support for the activities in which a system or component is executed under specified conditions, and the results are recorded and analyzed. It is important that a system provide the correct functionality according to its requirements. The purpose of testing is to exercise the system in such a way that any inconsistencies with the requirements will be found. Automated support for testing can facilitate this process by automatically producing many of the tests from the requirements specification and then documenting the results of tests that are run. Automated support can also ensure proper regression testing, testing which must be repeated after changes have been made to a system. Tool support for static analysis – the extent of automated support for the process of evaluating a system or component based on its form, structure, content, or documentation. Static analysis is an important part of evaluating software quality. It includes everything that can be evaluated about a system without executing it. It is very difficult to do much static analysis without automated support, and many tools are available that do a very thorough static analysis. Tool support for dynamic analysis – the extent of automated support for the process of evaluating a system or component based on its behavior during execution. Evaluating a system as it executes can provide important information about the product under development. Dynamic analysis provides a summary of the flow of execution, finding program segments that are not executed, as well as those parts of the program in which the execution spends most of its time. This can help to locate logic errors. It can also help to optimize program execution by indicating those portions of the code which are bottlenecks and will most benefit from the time and effort of optimization. Tool support for traceability analysis – the extent of automated support for the process of evaluating the degree to which each element in a software development product establishes its reason for existing through a relationship between itself and other products of the development process. Traceability analysis establishes a trace from the program units back through the design to the original system requirements. In this way, the reason for existence of each part of the development can be validated, and management can maintain better control over product development. Tool support for configuration management – the extent of automated support for technical and administrative direction and surveillance to identify and document the functional and physical characteristics of a configuration item, control changes to those characteristics, record and report change processing and implementation status, and verify compliance with specified requirements. Configuration management (CM) is a process that lends itself especially well to automation. People cannot perform the tedious administrative tasks required for CM nearly as quickly or correctly as an automated tool, yet CM is an important management function which facilitates management control over product development. Tool support for interfacing with other languages – the extent of support for interfacing with products or components written in other languages. This is the complement to the mixed language support provided by the development language (described in Appendix C). A tool provides support for interfacing with a development language by providing a specific set of interface calls, called a binding, which permits direct access to the capabilities of the tool from the development language. In this case, the tool could be a development tool, such as a database system, or it could be a COTS product. In either case, the tool will become an integral part of the developed system. When bindings do not already exist for interfacing to a product, they must be developed. Usability A tool or tool set can be a waste of money if it is not usable or if it cannot be set up for use. Usability determines how productive a developer can be while using a tool. If the tool does not enhance developer productivity, it is unlikely that it will be used. Ease of set-up – the extent that setting up a tool set for use in the software development process is straightforward and without difficulty. Set-up includes all aspects of preparing the tools for use in the development process. This includes everything from getting all the correct versions, documentation, etc., to making sure the tools will operate properly for all users, whether over a network or through individual copies of the tools. Difficulties often arise when users cannot get access to a needed tool or a tool cannot be easily set up to work in the desired configuration. Ease of use – the extent that the effort required to learn, operate, prepare inputs for, and interpret outputs of development tools is minimal and without difficulty. The specific areas of interest with respect to usability vary with each product. They include all of the features that make a product difficult or easy to learn and use. A user usually wants a product that is very intuitive and thus easy to learn, has icons or easy menu selections (not many levels of menus) to access the tool’s capabilities, has readily accessible and understandable additional information available when the user wants it, can be tailored to the user’s preferences, etc. A user typically judges a product more by how easy it is to use than by how many functions it can perform, and a product that is judged difficult to use will often not be used.

Sections

1

2

3

4

5

6

7

Appendices

A	B	C	D	E	F	G	H	J
K	L	M	N	P	Q	R	S