Program: Industry Practice Abstracts
Challenges & Solutions for 24/7 System Availability
Abstract: Software reliability is a critical problem for an increasing number of computing users. This is not because computers have become less reliable,but because computers, large and small, have become more pervasive and customers demand greater reliability and availability. As more businesses become increasingly dependent on computing as the backbone of their operation, it is more critical that the computing services provided are available 24 hours a day/7 days a week. Software system failures translate into lost revenues and at times even lost lives. Large enterprises are not the only ones to be adversely affected by poor software quality. The demand for reliable business-critical computing has reached the workgroup, and even the desktop.
Real-time "internet-worked" computing systems are at the core of many enterprise mission-critical applications. In fact, these systems dominate e-commerce, defense, CAD/CAM/CAE, aviation, nuclear power, communications and finance. Due to the complexity of the hardware and software involved and the stringent Quality of Service (QoS) requirements, verifying and guaranteeing the reliability and availability of these systems is a challenging task. "Internet Time" schedules, n-tiered complexities, compatibility and security issues, unpredictable user load, varying runtime configurations and the high cost of failure make the problem even more acute.
In this new era of rapidly changing technology and increasing software system dependence, the software community needs to develop a fresh look at how software development and service organizations, and their management, deal with software quality issues. Based on the practical experiences of the panelists, you will learn what the current challenges and practices are. Results, observations, and recommendations will be highlighted.
A Comprehensive Defect Prevention Program Using the ODC Methodology
Abstract: A key aspect of improving reliability of a software release is to understand the weaknesses in development and testing and implement actions for improvement. The information needed to accomplish this come from the defects escaped to the customers in prior release(s) and from the defects found in real-time during the current development cycle. In this context, we had the following issues to address:
In particular, we have used the ODC methodology to:
Experiences with Orthogonal Defect Classification Technique
Abstract: Orthogonal Defect Classification is a technique to classify defects so that they are meaningful and distinct (orthogonal). ODC defines defect attributes for various stages of software development life cycle and enables in-process and across-process quality assessment and improvement. It provides useful feedback on verification activities as well as development activities and helps in improving product quality at reduced costs.
This paper discusses the application of ODC at Lucent Technologies. The Software Technology Center (STC) in Lucent Technologies Bell Labs, Advance Technologies division, has been working with some of the Lucent corporate entities to introduce ODC in their work. Discussed here are the ODC pilot work done, current implementation, results achieved and experiences gained so far.
A Software Reliability Initiative That Costs Less and Delivers More
Abstract: Is this a typical scenario in your organization? An executive is shown a chart that indicates software quality is not at an acceptable level. The executive calls together the management team, and demands an 'n' percent improvement. The managers bring in their technical leads, and explain that results are needed 'now'. The technical leads, already engaged in trying to meet what they believe to be an impossible schedule, drop everything in order to identify and implement actions, usually based on their 'gut feel' and very little technical decision support. The technical teams and managers close their eyes, and cross their fingers, hoping that at least some of the actions will result in a measurable improvement. The Butterfly Model offers a better, proven approach, that costs less and delivers more as exemplified by this case study. On a regular basis, and at critical checkpoints, defect assessments and analysis were performed at the team level, with ODC attribute information forming an important foundation for the analysis. Complete customer reported defect data for the previous release of the component owned by each technical team, was used to capture and assess customer usage. This enabled a strategy which targeted customer usage as the highest priority in terms of defect prevention and defect removal improvements. By utilizing this approach, the speed and accuracy with which the teams identified meaningful actions were greatly improved, as was the return on investment. In this case study, the resource required to identify, prioritize, and implement actions in the 'top-down' approach for one project was 3.5 times the resource required using the Butterfly Model approach in a subsequent, comparable project. The results, in terms of a reduction in Customer Reported Defects was 4 times better with the Butterfly Model approach than the 'top-down' project. The author will describe the steps of the Butterfly Model approach to software reliability improvement, and will provide a detailed explanation of the benefits to the organization, the project, and the bottom line.
GTE Availability Metrics and Measurements
Abstract: The development and implementation of Availability Engineering to manage GTE's Computer Service Delivery System (SDS) began as a result of the importance of the SDS to the 15,000 GTE customer representatives that provide service to the customer. Unavailable services can lose customers for a lifetime, creating lost revenues of potentially billions of dollars in a competitive environment. The development and implementation of an Availability Engineering Process was required to sustain high service availability and to ensure continuous improvement in these customer services. Central to this Availability Engineering process is a Reliability Modeling System (RMS).
The RMS provides GTE management with a consistent, accurate, quantitative tool to evaluate the impact and risk of proposed system changes to the SDS, as well as consistent data that identifies opportunities for availability improvements. The SDS is a large, integrated set of systems and applications that encompasses hardware, middleware, software applications, networks, standard operating procedures, facilities.
The challenge of building the RMS was to develop an appropriate set of abstract model components along with metrics. Metrics used to describe model component behavior include: MTBF, MTTR, assigned users, redundancy and failover modes. Output metrics from the system include: Failures, MTTR, average users/failure, scored impact hours, risk hours, total user cost and total revenue at risk costs. Model predictions of past performance closely match actuals. Model predictions of future performance are used to assist in business decisions and investment strategies.
This paper will describe the issues encountered in building and implementing the RMS. There will be a brief discussion on future development and uses for the RMS.
Modeling Operational Profiles
Abstract: The importance of an operational profile for conducting reliability-based testing has been long recognized. However, few details have been presented on how to develop operational profiles and use them in generating test cases.
We are currently engaged in a project to set up an Automated Reliability Testing environment using commercially available tools. The purpose of the project is to demonstrate the feasibility of an automated reliability testing process from specifying an operational profile through quantitatively validating the reliability of the product at the completion of test.
A key component of ART is its capability to model operational profiles and to use the models to generate suites of test cases that will support reliability testing. We represent the operational profile graphically using Teradyne Corporation's TestMaster(R) tool and use TestMaster to automatically generate test cases. In this talk, we will briefly describe the ART project and share our experiences in modeling an operational profile for a pilot application that will be tested using ART.
Object Oriented Metrics for Reliability Evaluations
Abstract: Object technology is being used on projects of all applications across industry, and NASA is no exception. The Software Assurance Technology Center (SATC) at NASA has spent over two years identifying appropriate object oriented metrics, then compiling a metrics database from over 20,000 object oriented classes in C++ and Java. A ranking and reporting process was then developed to evaluate the attributes of high quality object oriented design and coding. Based on this work, guidelines for expected reliability have been developed. These guidelines were applied to project data and presented to project personnel at all levels, from top-level officials to project managers, quality assurance professionals, software engineers and experienced and entry level programmers. Post release feedback indicates that SATC's method for ranking object oriented classes does indeed do a good job of predicting the future reliability of the software. This talk will discuss the metrics used, a historical database of metrics information, and the ranking of code for reliability.
Determining the Sampling Adequacy of Software Reliability Measurements
Abstract: A comparison of MTBF numbers for various releases show that in all cases severity 1 (software forced crash) failures are lognormally distributed as was expected based on earlier measures taken in November and December of '98. These measures are calculated using a strict Time Between Failures rule, which requires that a failure interval be bounded at both start and end by an observed failure. To determine the adequacy of the sample, MTBF numbers are compared with Mean Survival Time measures collected through the use of 'censoring'. Censoring is a method developed for use in Survival Analysis, a technique commonly applied in the evaluation of medical protocols. It allows for the use of observations of unterminated individuals, that is observations of subjects that have not failed (died) during the course of observation.
A comparison of censored vs. uncensored measures for several versions of the software reveals that the Mean Survival Time exceeds the Mean Time Between Failures and that the difference is significant at the .0001 level of probability based on a two tailed T test of the lognormal data.
It's proposed that if the sample were adequate, MTBF would closely approximate Mean Survival Time; that is, on average a machine should survive until the MTBF elapsed, after which it should fail with a high probability.
Applying Malaiya et al.’s Method for Estimating the Number of Defects Based on Coverage to Large Datasets
Abstract: At ISSRE’98 Yashwant K. Malaiya, Jason Denton and Michael Naixin Li presented a method for estimating the number of defects based on coverage: "Estimating the Number of Defects: A Simple and Intuitive Approach". Malaiya et al. applied their method to test data of a 6100 line C program and presented the results. To support the presented method I have applied it to test data available from the last releases of the IBM server operating system VSE/ESA with several 100 KLOC of new code. These data also fulfil the request of Malaiya et al. for an evaluation of the method with very low defect densities. In addition I have compared the numbers of actual defects found after shipment of the programs with the numbers predicted by the method.
Software Downtime Predictions using System Test Data
Abstract: We describe the methodology use and the results of our SW reliability estimation of one of our product. By using an Operational Profile (usage of our system in the field with the scale of time compressed from 1 year of operation to 1 week of testing) we use System Test data to predict the SW Downtime at the release date to the customer and the number of faults left to be found in the field by the customer.
Our system has a very high degree of reliability. This is accomplished by having all the HW components redundant and several levels of SW recovery.(from one process recovery to the whole system recovery).
Our SW reliability measurements during our test session consist of the following data:
Keywords: SRE tools, Markov-modeling, Operational-Profile
Y2K Testing and Evaluation
Program-Based Testing, Debugging, and Maintenance
Abstract: This talk addresses the needs of practitioners and introduces techniques and tools for reducing testing cost, improving quality, and achieving a better understanding of code.
The first part of the talk focuses on upstream code coverage testing at the unit level, including the relationship between code inspection, functional testing, and code coverage-based testing. Also considered are how functional and code-based techniques can be used in a compatible way to achieve significant benefits. I will describe how a toolsuite developed at Telcordia Technologies permits the developer to take advantage of sophisticated analysis of the dynamic behavior of the software being tested to generate efficient tests, so that code coverage can be increased quickly with as few tests as possible. The benefits of using code coverage in fault detection by pushing more efficient tests to an earlier point in the development cycle will be examined as well.
The second part of the talk looks at ways of reducing the cost of regression testing. A modification-based test selection technique followed by test set minimization and/or test case prioritization in terms of code coverage and execution cost will be presented.
The focus is on how to identify redundant regression tests by using code coverage, and how to select a small set of effective fault-revealing regression tests to ensure that recent changes made in a software system have not adversely affected its existing features, except where changes are expected.
The third part of the talk considers how information collected at testing can help maintainers in program debugging and understanding. More specifically, I will discuss how analyzing tests can reveal valuable information to
Integrating Requirements Tracking with Testing
Abstract: Many companies have improved their development process to include the explicit definition of product requirements. A valuable benefit of having explicit requirements is implementing a testing phase that can validate that a product conforms to it's requirements. This description, typically an text document, provides the basis for developing tests that will verify that product has been implemented correctly. A process where a product's requirements can be described using a graphical representation of the product's use scenarios is presented. The graphical representation is easy to understand and makes later modifications quick to implement. This graphical model can then be processed by an automated test generation tool to produce not only test plans and scripts but a detailed document describing how each requirement was covered in the resulting set of tests. Tests can be generated for either coverage goals or using probabilities which will allow them to be applied for SRE purposes. The ability to express the requirements in terms of model objects enables each test to be correlated to the requirements. As the tests are executed a team can track the project's completeness and the product's consistency with the requirements. This system provides an organization with an efficient, reusable mechanism to both maintain a suite of tests and respond to feature or requirements changes during the product development cycle.
Generating Event-Driven Tests for Packet-Switched Networks
Abstract: All major telecommunications carriers are rapidly evolving toward packet-switched networks that foster integrated data and voice services. A new generation of network support and operations support systems are being developed to address the unique needs of these new networks. Building new systems which can duplicate the intricate functionality found in today's circuit-switched networks while still maintaining an easy to use and reliable service is a tremendous engineering challenge.
Testing the reliability and availability of these real-time distributed systems has become a time consuming and critical element in the system development lifecycle. A framework approach called the Event Verification Engine was developed by the author to improve the efficiency and effectiveness of testing these systems. The EVE framework incorporates open interfaces for automated construction and execution of multi-threaded test suites. The generated test suites combine two complementary test objectives: functional verification and load simulation. Off-the-shelf state-based test generation tools were plugged into this framework along with a custom-built test execution harness to demonstrate the value of this approach. Lessons learned will be shared in the areas of usability, effectiveness and cost.
Testing as an Application Architecture
Abstract: Testing has always been considered a step in the application development process. As such, many of the formal disciplines that have been associated with testing focus on testing process, testing technique and test coverage. However, maybe we have been applying the wrong disciplines to testing. With the right discipline, testing could become less drudgery and more routine. That pardigm shift could result in more frequent testing with greater coverage.
What if we considered testing an application as opposed to a process? What if we had a development organization dedicated to testing as their 'application'? What if this team was missioned to come up with an 'application architecture' for their application (testing)? Will we find the benefits there to justify its construction? Will it be able to compete for development dollars with strategic business investments? What about improving software quality? What impact does that have on the strategic business? What about reducing business risk? Where are the quantitative and qualitative values in such an architecture?This presentation answers all of these questions. The approach described above was used to successfully implement the merger of the retail banking divisions of two money center banks. For one of the banks, it was their second merger in 3 years. The presentation starts with the business environment for the first merger. It was through this experience that specific business problems were identified and the solution, the testing architecture, was implemented. The project was a complete success and was instrumental in the effective and efficient implementation of the second bank merger.
Systematic and Automated Software Risk Analysis using EMERALD
Abstract: Historically, risk in software projects was associated with schedule risk. Software projects gained/earned a reputation for being late and over budget. Traditional techniques (eg, work-breakdown structures, Gantt charts, critical path analysis), new tools and methods, (eg, collaborative consensus, AI assistants) and better development practices (eg, software development processes, limiting requirements churn) were leveraged to enable greater consistency and accuracy of scheduling, thereby reducing project management variation and risk. However, software quality and reliability risk is still typically managed as a "best effort" activity, implying the supplier will create the "best" product with the effort available and patch the product later. Indeed, patch releases are the accepted norm in many software circles. This is not surprising since achieving high levels of software quality is a difficult task. For example, project and schedule management is a SEI Level 2 activity while quantitative quality and reliability management is considered an SEI Level 3-4 activity. EMERALD, a product developed within Nortel Networks and now commercially available, provides a systematic and automated way of quantitatively assessing quality and reliability risk in a software project. EMERALD assesses software components early during development as to their risk of having software faults yielding failures which occur later during the test cycle as well as after the product is released. This presentation will show how EMERALD is being used in software projects to systematically assess and reduce software fault-proneness, thereby improving reliability and quality.
BEAM: A Software Dependability Analysis Tool
Abstract: Dependability evaluation and prediction is a valuable means to track risky points leading to difficulty during development and maintenance activities. In this paper we present a tool that help software designers and managers in scrutinizing complex structures and in tracking risky components as early as possible in the software life cycle.
Currently, BEAM allows performing complexity and maintainability assessment for imperative as well as Object-Oriented languages. The tool also provides detailed statistical analysis helping in taking the appropriate decisions during design, code inspection, testing and maintenance. The BEAM tool has provided excellent results when applied on several large software written in C, C++ and Java, and is currently used to evaluate large telecommunications software.
Prioritizing Software Test Best Practices
Abstract: In the recent years, we have brought in a significant focus on software testing and along with it the guidance to implement a set of best practices. An original set of twenty-eight best practices which contribute to improved software testing is far too many to uniformly recommend across a large corporation, and therefore we have chosen to prioritize them into three categories. These are the basic, foundational, and incremental practices. The collections represent the experience that several software organizations have gained over time. This talk will discuss how we have arrived at this prioritization of practices and what the practices are. In addition, we will share with you the feedback that we received from the different organizations on how they could use a combination of these best practices.
How "Human Outage Risk Assessment" May Be Able to Improve Software Reliability
Abstract: This talk will discuss an initiative under development for "human outage risk assessment". Our team of researcher/trainers develops training for elite performance enhancement. The development of a battery of assessment tools which could accurately pinpoint individual strengths/weaknesses and their impact on performance, was a critical component in devising appropriate training. In the process of discussing these concepts with several reliability engineers, as well as HR personnel within the tech industries, the very real probability of more accurately approaching "human factoring" has come up, using these same assessment tools. During this talk we will discuss results and possibilities concerning our work.
Building More Reliable Software: Traditional Software Engineering or Formal Methods?
Abstract: During a recent software engineering project development of a real-time software system, strengths and weaknesses of parallel developments were revealed, especially in the determination of the end-product reliability. During the course of software development, process and product metrics were collected and analyzed for a state-of-the-art waterfall method assessed at a CMM Level 4 and a mathematically based formal method which employed automatic code generation. An independent third a party generated and ran operational profile test cases in order to determine the reliability of each software methodology. While neither methodology was able to meet the user’s reliability objectives within cost and schedule, insights and observations point to an improved hybrid methodology where process, people, and technology optimize results. This paper will explore the specification for the system under development, the development processes, the products and metrics captured during development, the analysis tools and testing techniques used by an independent party, and the results of the reliability and process analysis.
Everything you wanted to know about SRE but didn't know who to ask
Recent experience and feedback from panels indicates that what the audience likes best is the chance to ask questions, particularly regarding things that might solve problems on their development projects or in their research studies. The proposed panel would have no presentations, only questions (We will have a few questions in reserve if things start slowly). The questions could be on anything, ranging from theory to details of application. This is a repeat of the panel we had at ISSRE97 and ISSRE98 (of course, the content will be different). From all indications, the panel was well received in both cases. The panelists have been selected to make available wide and extensive experience in the field.
Chair: Anthony T. Rivers; Panel members to be announced
Barbara Hirsh, Robert Bleck, Steven Wood (Motorola)
After exploring various available methodologies, we chose the Orthogonal Defect Classification, ODC, methodology to provide the necessary coherent framework. Over the past three years, we have applied ODC based analysis for a few releases of a communication software. We report the results of this deployment in this presentation.
We discuss our experience of deploying the ODC methodology through some specific examples.
N. B. Sreenivasan (Lucent Technologies, Bell Labs Innovations)
Kathryn Bassin (Center of Software Engineering, IBM T.J. Watson Research Center)
Ed Stoker (GTE Technology), Michael Bitto (GTE National Operations)
W. W. Everett (SPRE, Inc.) & James Widmaier (National Security Agency)
Linda H. Rosenberg and Al Gallo (NASA)
Scott Cherf (Cisco Systems)
Dr. Harald Buczilowski (IBM Germany)
Antonio Rodriguez-Vargas (Siemens, AG, Munich, Germany)
By using a Markov model and a SW reliability model we can estimate the downtime due to the SW and the number of faults left in the SW at the Release date. The SW reliability of a product at the release date is a very important and useful information that the vendors and customers alike would be very interested to know.. We hope that very soon in the future one of the items of the product Release check list is that the vendor provide to the customer the SW reliability estimation of the product and the analysis of it.
Ray Paul (Department Of Defense, US Government)
Bob Horgan (Director & Chief Research Scientist, Telcordia Technologies)
Overall, I will present a set of techniques, supported by tools, which not only emphasize dynamic behavior but also use software visualization and heuristic guidance in the solution of software testing and maintenance.
Larry Apfelbaum (Teradyne Software & Systems Test)
David W. Carman (Telcordia)
Alan Stuart (IBM Year 2000 Test Strategy and Management)
Wendell D. Jones (Nortel Networks Business Unit)
Saida Benlarbi (Director, Research & Development, Cistel Technology)
Ram Chillarege (IBM T.J. Watson Research Center)
Peter Vagg (Dateline Associates)
James Widmaier (NSA)
John D. Musa (Independent Consultant, Chair); Panelists: William W. Everett (SPRE, Inc.), Karama Kanoun (LAAS-CNRS), Alberto Pasquini (ENEA), Norman F. Schneidewind (Naval Postgraduate School), Mladen A. Vouk (North Carolina State University)