Lloyd K. Mosemann II
Software Technology Conference
April 29, 2002
In 1994, I closed this annual Software Technology Conference with the observation that the underlying need within the defense community is for
predictability. “From a Pentagon perspective, it is not the fact that software costs are growing annually and consuming more and more of our defense dollars that worries us. Nor is it the fact that our weapon systems and commanders are becoming more and more reliant on software to perform their mission. Our inability to predict how much a software system will cost, when it will be operational, and whether or not it will satisfy user requirements, is the major concern. What our senior managers and DoD leaders want most from us, is to deliver on our promises. They want systems that are on-time, within budget, that satisfy user requirements, and are reliable.”
The Question I should like to briefly explore this morning is: “Where are we today, and where will we be tomorrow?” Did we lose our religion?
Why do I use the metaphor of “religion”? Primarily because “religion” is the traditional example of “faith-based” behavior - that is, behavior that is based on a belief-system rather than on externally imposed rules such as the “law of gravity” or “she that has the gold, rules”. Remember: the emotional discussions regarding whether Ada or C++ should be preferred were frequently described as “religious” arguments based on beliefs rather than facts.
Sadly, I still see the world of software being governed by religious-like belief systems rather than objective appraisal. When I left the Pentagon six years ago I described some of what was happening as “bumper sticker” management, and the situation has not changed for the better. I sometimes have the feeling that the “blind are leading the blind”, or as shown here, the leadership is blissfully ignorant of the direction in which they are headed.
The only meaningful direction from either OSD or the Military Services in the last few years was the Gansler memo that directed the use of CMM Level 3 contractor organizations for ACAT 1 systems. Do you know how many large dollar (by that I mean $50M or more) software intensive acquisitions are
not ACAT 1? Virtually all MIS and C3 systems!
Over the past two years there has been 5.5% annual growth in the cost of ACAT 1 programs due to cost and schedule estimating and engineering changes (sound like software?). Yet these programs have the most experienced DOD/industry managers, and have a requirement for CMM Level 3. About 2/3 of DOD acquisition dollars are for programs below the ACAT 1 threshold for which there is currently no CMM requirement. It’s my guess that these non-ACAT 1 programs are at least twice as bad as ACAT 1 programs - in other words, about $9B per year in cost growth associated with estimating and engineering problems, many of which are likely software related. In my opinion they deserve more software management attention than results from the requirement to use “best commercial practice”.
What’s wrong with “best commercial practice”? Fact is, it just doesn’t exist among DOD contractors. It’s a fantasy created by those who want to streamline acquisition, making it possible to cut the number of oversight personnel by reducing the opportunity for oversight. The best way to justify a “hands off” attitude is to assert that contractors always do everything right!
There are more mature software organizations today. Virtually every large DOD contractor can boast at least one organization at SEI CMM Level 4 or above, and several organizations at SEI CMM Level 3. On the other hand, most DOD software is still being developed in less mature organizations - mainly because the PEO or Program Manager doesn’t demand that the part of the company that will actually build the software be Level 3!
Many people used to tell me that DOD needed to get on the .com bandwagon - ’cause those folks develop and deliver software fast. It is true that the Internet and the availability of web browsers has fundamentally changed the environment within which even mission critical software is being developed. But, instead of adapting proven software methods, the software research community has all but dropped its concerns with formal methods, peer reviews, cleanroom processes and other reliability techniques, including the language known as Ada that was designed to promote reliability. Except for my friend Barry Boehm, much of the academic community has more or less stopped investigating better ways of estimating system complexity and measuring software growth. Instead, invention of new user interfaces, of new distributed computing architectures, of new (more flexible and less reliable) programming languages have been given top priority. The goals of reliable performance and predictable development costs have been largely ignored.
As Wayt Gibbs, the author of the notorious
Scientific American article in 1994 entitled
Software’s Chronic Crisis, told me recently:
It is tempting to argue that the lack of a disciplined approach to software development is the principal reason that so many dot-com ventures failed--and consumed such flabbergasting amounts of money as they failed. That may be stretching it: addle-brained business models clearly played a starring role as well. But it is interesting that, with very few exceptions, the dot-com startups embraced the programmer-as-master-craftsman model of development. That very old-fashioned model is still considered avant garde in the open source movement. How many software startups in the past eight years have submitted to an SEI evaluation or tried to achieve CMM level 3? You won’t need many fingers to count them all.
In addition to abandonment of formal methods, the Internet has also made obsolete the fortress model of security and the notion that an astute administrator can enforce central control of all components of a system. It is no longer realistic to dream of a mathematical certainty that each software component in a system is correct. In a world where the standards on which network communication and user interfaces depend change every 12 to 18 months, it doesn’t make sense to lock down detailed requirements, spend a year proving them self-consistent, and spend two more years building to spec.
As a result, there is a new move toward something called “autonomic computing”. This is a vague term that encompasses several lines of research aimed at taming complexity to achieve reliability. The approach isn’t “formal”, but “organic”: find ways to build systems that can heal, that can achieve their mission despite bugs and equipment failures and even sabotage. In other words, design systems that constantly monitor their own performance, that notice failures and can perform self-maintenance, that do not “crash” but instead degrade gradually. Some folks say these goals can be achieved in 10 to 20 years, but don’t bet your paycheck on it.
Ironically, the problem facing much of industry is exactly the opposite. Wayt Gibbs told me:
Executives who woke up one day in 1997 to discover a time bomb, in the form of Y2K bugs, ticking inside their systems were forced to take a substantial hit to the bottom line. But an ironic consequence of Y2K is that many companies “upgraded” their affected systems in 1999 to work with or through the Internet and various fly-by-night standards and startup-built “solutions” that are now as obsolete as Algol. As a result, their systems are again ticking time bombs. Though they will not all fail at the same time, their failure will be just as unpredictable and much harder to fix.
Let me talk briefly about some subjects that have been notably missing from the plenary sessions of this Conference over the past few years: Software Engineering, Product Line Development, Formal Methods Programming, and Predictability.
Back in 1991 Paul Strassmann, who was then the Director of Defense Information, said:
The #1 priority of DOD, as I see it, is to convert its software technology capability from a cottage industry into a modern industrial method of production. Guess what? That has not happened. Why not? Because this requires “software engineering”. Software engineering encompasses a set of three key elements -- methods, tools, and procedures -- that enable the manager to control the process of software development and provide the practitioner with a foundation for building high quality software in a productive manner.
The fundamental ingredient in a “software engineering” approach is the design of a robust software architecture. By architecture I don’t refer to the grouping and linkage of servers and routers and PCs, but rather to the inherent design of the software itself - the identity of modules and their relationships, including in particular the infrastructure, control and data interfaces that permit software components to operate as a system. I am told by an attendee at a DOD Systems Design Review several months ago that the contractor described his proposed system as “modular”. That’s a good architectural term, and it was accepted at face value. In fact, the system, when looked at by the independent attendee, was found to have only two modules. When this was brought to the Govt Program Manager’s attention, he said, “The contractor says it is modular. He’s smarter than we are.” This little incident underscores two facts: architecture was understood by neither the contractor nor the government PM, and the probability of a successful software delivery is low.
All too often the DOD excuse for not requiring an architectural evaluation is that “requirements change rapidly -- can’t afford to be locked into a specific architecture”. Wrong - that is the very reason that attention should be given to architecture, so that changes to requirements can be accommodated easily.
I am told that SEI is still being asked to do Independent Technical Assessments of “Why a software acquisition has not produced the desired working software system?” Why wait to ask SEI to come in to do a post mortem and tell you how you screwed up? Why not ask them to come in, first, to review the RFP and Statement of Work, and, second, to assist in evaluation of the software engineering practices, software development environment, and proposed architecture proposed in response to the RFP, and then, again, after award to assess the quality of the software engineering and development process? SEI isn’t cheap, but terminating a $100M project for lack of satisfactory performance isn’t cheap either.
Interestingly, when one thinks about “best commercial practice”, there are two very different worlds out there. There is the Govt contractor world; and, there is the real commercial world - banks, insurance companies, UPS and FedEx, Eckerd Drug, and DisneyWorld. These companies develop their own software using the best available software tools such as Rational’s software development environment. They don’t pick the cheapest tools. They don’t rely on COTS or outside software developers - their software is their business, and they consider that it provides them a competitive advantage, and they want to control it, and they use the best tools available regardless of cost.
Product Line Developments are also becoming increasingly commonplace in the true commercial world. In addition to the original example (CelsiusTech, the Swedish firm that was the first to exploit the benefits of a product line architecture approach to software application development back in the late 1980s), there are now a number of firms that you have heard of who are using an architecture-centered approach: for example, Nokia, Motorola, Hewlett-Packard, Philips, Cummins Engine, and (believe it or not) one government application at the National Reconnaissance Office. The NRO is enjoying 50% reductions in overall cost and schedule, and nearly tenfold reductions in defects and development personnel. Let me list for you the most common and relevant-to-DOD reasons that these companies give for adopting the “Architecture-Centered Product Line Approach” to software development:
- Large-scale productivity gains
- Improved time-to-market = field weapons faster
- Improved product quality
- Increased customer satisfaction = Warfighter satisfaction
- Enables mass customization
- Compensates for inability to hire software engineers
These companies and the NRO do have “best practices”, but the “best practices” are not yet widely recognized as such. Frankly, it will take DOD PEOs (not Program Managers) and their overseers, the Service Acquisition Executives, and especially the DOD Comptroller and PA&E folks, to recognize and direct the product line approach to software development and acquisition. (Unfortunately, these folks are not known for their software acumen.) The major impediment to the Product Line Development approach, aside from ignorance of its benefits, are cultural, organizational, and, especially, the DOD’s stovepipe approach to budgeting and funding. DOD has many potential software product lines. None of them have been built, largely for “political” and stovepipe budgeting reasons. As a result development and fielding of defense systems continues to take longer, cost more, and lack predictable quality. Product lines require strategic investment, which appears to be outside the DOD Comptroller and Acquisition communities’ frames-of-reference. Yet, it is the Department of Defense that most often uses the term “strategic”.
Cummins’ Engine Company used to take a year or more to bring software for a new engine to the test lab - now they can do it in less than a week! I strongly recommend that you obtain a copy of a new book,
Software Product Lines, just published in 2002 by Addison-Wesley. The authors are from SEI. Sadly, with the exception of the NRO, it appears that the readers are mainly from commercial organizations. There was a previous version of the book,
Software Architecture in Practice, that sold more than 14,000 copies, but I bet that none of you has ever heard of it.
I work for one, but let me tell you that, although government contractors are commercial organizations, they do not have an identifiable “best commercial practice”. They basically will provide what the Government asks for. The only reason many of them can now boast of having at least a few SEI maturity level organizations is because, starting 10 years ago, many government RFPs required a Software Capability Evaluation as a condition of bidding. In fact, sad to say, many contractors today put into their proposals satisfactory CMM credentials, but then actually perform the work with an organization who couldn’t spell CMM. Why does the Government let this happen? Why aren’t there Software Capability Evaluations anymore? - the word is, they “take too long and cost too much”. Better to make a quick award, and then, down the line, accept an inferior product or terminate for convenience. Too often what the Govt has been asking for is COTS. How many failures of COTS-based acquisitions have there been over the past decade? Too many!
Let me tell you - “best commercial practice” is not eliminating all software smarts in government and relying 100% on contractors to deliver good software. “Best commercial practice” is what real commercial companies are doing. They have in-house software expertise, they use a robust software development environment, and they base their software development on a sound software architecture. It is no secret/no surprise that Rational and their competitors have a growing market in the commercial world and a shrinking market in the government world. I’m not suggesting that the Government can or should develop software in-house. I am strongly suggesting that the Government needs enough in-house software expertise to know what it is buying. It is still true that you get what you pay for”, and that “apples and oranges” are not the same!
Watts Humphrey recently published a new book entitled
Winning with Software - An Executive Strategy. Not surprisingly, the book is directed primarily at executives of commercial companies. But every DOD acquisition executive, PEO and Program Manager needs to real and understand it message. Frankly, its message is pretty simple:
Software projects rarely fail for technical reasons; invariably, the problem is poor management. He poses two interesting questions and buttresses them with numerous examples:
Why do competent software professionals agree to dates when they have no idea how to meet them? Why do executives accept schedule commitments when the engineers offer no evidence that they can meet these commitments? He asserts that Management’s undisciplined approach to commitments contributes to every one of the five most common causes of project failure:
- Unrealistic schedules
- Inappropriate staffing
- Changing requirements
- Poor quality work
- Believing in magic
I don’t have time this morning to amplify these critical management issues - buy and read the book, and make sure that your bosses read it.
What is Formal Methods Programming? Basically, it is all of the above rolled together: sound management, established engineering processes, robust software development environment, model based architecture, and a reliable programming language. I was thrilled to see, in the March edition of
CrossTalk an article by Peter Amey of Praxis Critical Systems, in which he stated:
There is now compelling evidence that development methods that focus on bug prevention rather than bug detection can both raise quality and save time and money. He then goes on to say that a key ingredient is the use of unambiguous programming languages that allow rigorous analysis very early in the development process. As many of you know, I was an early and vocal advocate of Ada, primarily because, unlike other languages, it enforces basic software engineering practice. In that article Peter describes the use of a subset of Ada known as SPARK. He says,
The exact semantics of SPARK require software writers to think carefully and express themselves clearly; any lack of precision is ruthlessly exposed by its support tool, the SPARK Examiner. He indicates that there were significant savings from using SPARK in a critical avionics program, including most notably that formal FAA test costs were reduced by 80%. Unfortunately, this is an isolated DOD example - the same rigor and discipline of formal methods programming should apply for all major software intensive system developments.
Finally, let me say a word about Predictability. Predictability is the only metric that the warfighters care about. Question is - how can we make the warfighters (including the PEOs and PMs who are charged with delivering the capabilities needed by the warfighters) smart enough to know that you can’t just buy software as a product off the showroom floor. There must be understanding of the software engineering paradigm that produces software. In fact, more than this, to be assured of getting software that works on a predictable schedule and at predictable cost requires that someone in government be smart enough to enunciate the basic processes that will be employed by the contractor to produce. This is important because, otherwise, competitors will bid an unrealistically low price and unrealistically fast schedule, and be awarded the contract. To perform at the low cost means no robust software development environment, no time and effort devoted to creating a valid software architecture, and probably means cheap people. If the government wants to “get software right”, sufficient process guidance must be given to assure that the contractors all bid essentially the same level of capability. I believe that Government should be explicit about the need for architecture, a robust software development environment, perhaps even the requirement for a language like SPARK, but, as a minimum, it needs to specify that the performing organization must be at least CMM Level 3. [show slides comparing before and after - In short, at Level 1 virtually all projects have cost and schedule overruns, whereas at Level 3 virtually all projects are on target. As regards Defect Rates and $ per Source Line of Code, there is substantial improvement on the order of 20-50%.] It really is true that
you get what you pay for. If you want it cheap, you’ll get it cheap - but it may not work in the manner envisioned, if it will work at all.
A quick word about CMMI. I believe that CMMI is very important for the embedded world. Software and Systems Engineering need to be treated as a single whole. However, many folks, particularly in the real commercial companies like banks (and large areas of DOD such as MIS systems), believe that the systems engineering aspects of CMMI have little relevance for MIS and most C3 systems. I tend to agree. It will be a gross mistake to discontinue support of the Software CMM, because the CMMI for most non-weapons systems projects is perceived as overkill.
Where have we been? The Government I fear has been wallowing in a slue of self-deceit,
thinking it need not have software expertise, but can simply rely on the so-called
“best commercial practices” of contractors. The problem is that, left to
their own devices, the “best practices” of defense contractors are not very good.
Are there best commercial practices? You bet! The banks, insurance companies and other
truly commercial enterprises have them. But they have them, not because of some automatic
external happenstance, but because their senior managers have had the moxy to realize that it
takes money to make money, and that it takes software expertise to develop or acquire software.
The Government needs to
go and do likewise. Otherwise, the decade of 2000 will likely not show any lessening of the Software Crisis that has carried over from the 1990s.
