2. SURVEY METHOD Several approaches to conducting the survey were initially considered. These approaches are briefly discussed below before describing in detail the selected approach. A comprehensive DoD data call was considered, involving a formal request for specific data elements throughout the DoD. This approach was rejected because it would have encompassed a great deal of effort on the part of operational organizations whose primary mission is readiness. Historically, the response rate has been low to data calls for information that is not directly related to assigned missions. Another approach involved reviewing several automated databases that contain programming language information on DoD systems. Several of these databases were examined as part of this study, but none were able to provide the information required. It was also difficult to determine the lineage and accuracy of the data. Therefore, these databases were not used as part of the present study. The approach that was chosen involved direct contact with the organizations responsible for developing or maintaining systems that contain software. This section provides a detailed description of this approach, including the survey populations and samples, trade-offs made in designing the data collection form, the method used in contacting potential respondents, the methods for handling erroneous response data values, and the methods for analyzing the survey results. 2.1 Population Identification We recognize that a census population of software would include systems that are new or undergoing major modernization and software in a steady state of maintenance. Software being maintained is a collection of applications that are difficult to identify because they are aggregated under operational costs. After a trial effort, we could see clearly that the estimated time and effort to approximate a census population would exceed the targets agreed for this survey effort. Consequently, we identified a judgement population as described in the next sections. 2.1.1 Weapon Systems Population Weapon systems include aircraft, ships, tanks, tactical and strategic missiles, smart munitions, space launch and space-based systems, command and control (C2), and command, control, communications (C3), and intelligence (C3I) systems. For the purposes of this survey, weapon system software is considered to comprise embedded, C3, and C3I systems, as well as any other software that directly supports or is critical to a weapon system's mission [STSC 1994]. Four acquisition categories (ACAT) are defined for weapon systems by DoD Instruction 5000.2 [DoDI 1991, pp. 2-2-2-4]: Acquisition Category I is for major defense acquisition programs with eventual Research, Development, Test and Evaluation (RDT&E) expenditures of more than $300 million and eventual procurement costs of more than $1 billion (in FY90 constant dollars). Acquisition Category II is for major systems with eventual RDT&E expenditures of more than $115 million and eventual procurement costs of more than $540 million (in FY90 constant dollars). Acquisition Categories III and IV are for programs not meeting the criteria for ACAT I and II. These programs do not have specific expenditure profiles and exist to allow different levels of reporting. 2.1.2 Automated Information Systems Population An Automated Information System (AIS) can be functionally described as follows: A combination of computer hardware and computer software, data and/or telecommunications, that performs functions such as collecting, processing, transmitting, and displaying information. Excluded are computer resources, both hardware and software, that are: physically part of, dedicated to, or essential in real time to the mission performance of weapon systems; used for weapon system specialized training, simulation, diagnostic test and maintenance, or calibration; or used for research and development of weapon systems. [DoDI 1993] These systems are often categorized as automatic data processing systems that are designed to meet specific user requirements for business functions (e.g., transaction processing, accounting, statistical analysis, or record keeping) and they are implemented on general purpose computers, including personal computers. An authoritative source for a complete inventory of existing AISs could not be identified. Given the time and effort constraints placed on this study, the list of 53 designated major AISs was used as the AIS survey population [OASD 1994]. A major AIS is defined as one that is not a highly sensitive, classified program (as determined by the Secretary of Defense), and that according to DoDI 8120.1, the instruction on life cycle management of AISs [DoDI 1993], is characterized by the following: Has anticipated program costs, computed in FY 1990 dollars, in excess of $100 million; or Has estimated program costs, computed in FY 1990 dollars, in excess of $25 million in any single year; or Has estimated life-cycle costs, computed in FY 1990 dollars, in excess of $300 million; or Is so designated by the milestone decision authority. 2.2 Sample Selection The approach used in selecting the sample from the population of weapon systems and AISs is described in the next section. 2.2.1 Weapon Systems Sample A close approximation of the population of existing weapon systems was found in a commercially available publication [Carroll 1994]. This publication provided a list of over 1,300 RDT&E and procurement programs for all Services and DoD Agencies. The list, called the Program Management Index (PMI), was based on the President's 1994 budget request and identifies all RDT&E programs with current or future fiscal budgets exceeding $15 million and procurement programs with total budgets of more than $25 million. The PMI contains a number of programs that do not develop or maintain software for a weapon system (e.g., ammunition programs, medical research, biodegradable packaging technology) and lacks some programs that would have been of interest such as intelligence systems, highly classified programs, and programs below the budgetary thresholds cited. The PMI was then reviewed to eliminate programs that were obviously outside of the population of interest. For example, programs such as 25MM Ammunition Development, Health Hazards of Military Material, and Petroleum Distributions were eliminated from the population. Also eliminated were basic and applied research programs that involve technology years away from being fielded. While these programs often involve small amounts of prototype software development, the scope of the survey constrained the size of the survey sample. Each of the programs remaining in the PMI list was briefly examined to characterize the likelihood of being a weapon system. Weapon systems such as aircraft, ships, and tanks were (usually) easily identifiable. However, many of the programs required additional effort to determine their relevance to the population. For example, the AN/BSY-2 is an RDT&E project. Unless one is familiar with the AN/BSY-2 project, it is not immediately clear that it is the combat system for the Seawolf submarine and contains an aggregate of several million lines of software. Of the 423 programs selected from the PMI list to form the survey sample, 142 were eliminated from the sample after we found that they had been cancelled or were combined with another program, or contained no software. The remaining 281 programs included most of the typical weapon platforms (e.g., aircraft, ships, submarines, tanks) and many of the sensors, communication systems, and weapon subsystems. 2.2.2 Automated Information Systems Sample Of the 53 AISs on the original list, 2 have been cancelled, 4 were primarily acquisitions for hardware and commercial off-the-shelf (COTS) software, 5 have not begun to develop software, and 4 programs had no current program manager name and telephone number. The survey sample of AISs for this study, therefore, consists of the remaining 38 major AISs. 2.3 Data Collection Form A data collection form was designed for this survey to reduce respondent error and to present technically accurate language choices. Because data was to be collected on five different programming language generations, definitions of these language generations were adapted from the ANSI/IEEE Standard Glossary of Software Engineering Terminology [ANSI/IEEE 1990] with advice from Ms. Jean Sammet, language historian. These definitions were provided on the form as follows: A first generation language is the same as a machine language, usually consisting of patterns of 1's and 0's with no symbolic naming of operations or addresses. A second generation language is the same as assembly language. A third generation language is a high order language that requires relatively little knowledge of the computer on which a program will run, can be translated into several different machine languages, allows symbolic naming of operations and addresses, provides features designed to facilitate expression of data structures and program logic, and usually results in several machine instructions for each program statement. A special purpose language is used for special-purpose application areas such as robotics, machine tool control, equipment testing, civil engineering, and simulation. Special purpose languages are a subset of third generation languages. A fourth generation language is designed to improve the productivity achieved by high order (third generation) languages and, often, to make computing power available to non-programmers. Features typically include an integrated database management system, query language facility, report generator, screen definition facilities, graphics generators, decision support capabilities, and statistical analysis functions. Fourth generation languages are usually available as components of a COTS software package. A fifth generation language incorporates the concepts of knowledge-based systems, expert systems, inference engines, and natural language processing. Languages were grouped on the data collection form by these generations and listed by name and version within the third generation languages category. We decided not to ask for name and version of first, second, fourth, and fifth generations because supplying that type of data would require an inordinate amount of research effort for respondents to provide and for us to validate. An overriding concern for the data collection form was to keep it as simple as possible. Data collection forms that are lengthy or require a great deal of effort to complete are less likely to be completed and returned. Thus, the following design decisions were made with respect to the data collection form: Survey respondents were allowed to choose the level of abstraction addressed by their response(s). Ideally, we would have liked to obtain a single response covering a single weapon system or AIS. However, many weapon systems are composed of subsystems that are separate procurement programs being developed or maintained concurrently by different contractors. These contractors and their sub-contractors may differ from one another in their choice of programming languages and dialects, depending upon the component(s) being developed or maintained. Because of the likely difficulty in requiring single-system reporting, survey respondents were asked to complete the data collection form at the level of abstraction that was the most convenient for them. Respondents were asked to photocopy the data collection form and return multiple copies if they provided data for more than one system or subsystem. Where possible, a list of allowable values was provided so that the respondent could simply place a check mark by the appropriate value. For example, rather than asking the respondent to write the name of the system life-cycle phase, the allowable values were provided on the data collection form. Such lists also reduced the likelihood of obtaining invalid data responses. Where practical, ranges were used instead of requesting exact values. Ranges were used for the estimation of total source lines of code (SLOC) and for the amount of software developed or maintained per programming language. The use of ranges reduces the precision of the survey results (e.g., the SLOC totals will be partially based on an estimation procedure). However, the reduction in precision was considered justified in terms of the corresponding decrease in effort for filling out the survey form. The temptation to ask for more information than absolutely needed was resisted. A number of interesting data elements were considered for inclusion in the data collection form but rejected because they were not essential and would increase the effort and time needed to complete the form. This concern also led to the decision to ask for the versions of third generation languages only. The key information desired from each survey respondent included the following items: A list of all third generation languages (by version) being used in the development or maintenance of operational and support software for the system of interest. For each programming language listed, an estimate of the percentage that language represents in terms of the total amount of software being developed or maintained. We suggested that the percentage be derived from SLOC since most DoD programs track the amount of software using this measure. However, alternative methods of determining the percentage (e.g., function points) were allowed, as indicated on the questionnaire itself. An estimate of the total amount of software being developed or maintained for the program/system. Again, we suggested using SLOC for this estimate. Secondary information desired from each survey respondent included the following items: The amount of first, second, fourth, and fifth generation software being developed or maintained. The number of distinct assembly languages being used in system development or maintenance. A list of any third generation special purpose languages being used to develop or maintain software (e.g., equipment checkout languages such as ATLAS). The acquisition category assigned to the program/system. The system life-cycle phase of the program/system. A pilot survey was conducted using a preliminary version of the data collection form. Improvements were made according to suggestions made by several respondents as well as by analysis of their responses. Appendix A provides a copy of the final data collection form. 2.4 Contact Process The process for contacting potential survey respondents for weapon systems and AISs differed only in the means by which telephone numbers were obtained. For weapon systems, the PMI list provided the name and telephone number of each weapon system program manager. For AISs, the Office of the Secretary of Defense official responsible for oversight of that AIS was contacted to provide the name and telephone number of the AIS program manager. The purpose of the survey was described upon contacting each potential respondent. Suggestions for filling out the form were provided and the form was then faxed to the potential respondent. If a response was not received after three weeks, a follow-up call was placed. 2.5 Respondent Errors Some data collection forms were not completely or accurately filled out by survey respondents. For example, respondents may have omitted the Acquisition Category because it was not known to the respondent or was overlooked. The most common instance of inaccurate responses was that two different programming languages were listed as being used for over 75% of the system. If the correct data was not immediately obvious, the respondent was either contacted for the correct data or the values reported for the data element were excluded from our analysis and logged as a non-response. Graphic displays of survey results in the next section show these errors as "data not available." 2.6 Analysis Process The process for estimating the total number of SLOC addressed by this survey is now described. As discussed in Section 2.3, respondents were not requested to provide an exact SLOC count for their response. Rather, they were asked to select from a range of "Total Source Lines of Code." A uniform procedure for estimating the SLOC represented by each survey response form was developed. Table 1 provides the Total SLOC ranges on the response form and the corresponding SLOC count assigned to each range. For example, if the "100-500K" range was checked on the response form, 300K was used as the total SLOC covered by the response form. The SLOC sizes in the "Value Assigned" column in Table 1 were subjectively assigned. However, if an exact SLOC count was provided on the response form, that count was used in place of an estimate. The total SLOC addressed by this survey was therefore derived by summing the estimated SLOC (or in some cases the exact SLOC) from each response form. Values assigned in Table 1 were subjectively assigned for the top and bottom ranges; the midpoint was used for other ranges. Table 1. Values Assigned to SLOC Range Estimates "Total SLOC" Range Marked on Response Form Value Assigned 1-100K 75K 100-500K 300K 500-1,000K 750K 1,000-5,000K 3,000K 5,000+K 6,000K Respondents were also requested to provide the percentage of the total system written in each applicable language. Ranges were available to identify this percentage. Table 2 provides the "% of Total" ranges on the response form and the corresponding percentages assigned to each range. For example, if "5-25%" was checked for Jovial 73, 15% was used as the percentage of the total system written in Jovial 73. If an exact percentage was provided on the response form, that percentage was used in place of an estimate. For each response, the SLOC for each language was derived by multiplying the total SLOC count (see Table 1) by the estimated percent of total system written in that language. Table 2. Values Assigned to Language Percentage Estimates "% of Total" System Marked on Response Form Value Assigned <5% 2.5% 5-25% 15.0% 25-50% 37.5% 50-75% 62.5% >75% 87.5% The problems in using SLOC as a means of measuring the amount of software are well publicized [Jones 1991]. It is unlikely that respondents would have provided much data had specific methods for counting SLOC been required. Therefore, survey respondents were allowed to provide SLOC range estimates using their method for counting SLOC. Clearly, non-uniform methods for counting SLOC reduces the precision of the SLOC-related portions of the survey. However, this trade-off does not detract from the primary purpose of the survey (i.e., to produce a count of programming languages being used in the DoD today).

1

2

3

4

5

6

Appendix A	Appendix B	References
Acronyms	Figures	Tables