Metrics and SPC Analysis Techniques that Really Help in Quantitative Project Management
Dr. Kanhaiya Jethani
Senior Consultant
Tata Consultancy Services
Introduction:
Statistical process control has its origins in manufacturing industry where these techniques are used to monitor and control the performance of single machines or processes. Software industry has also started using SPC techniques for quantitative management of processes. However, there are dissimilarities between the manufacturing and software industry which have a bearing on the applicability of SPC techniques.
Quite often the metrics and SPC analysis technique selection is not accorded due importance. Metrics and SPC analysis techniques are routinely selected to satisfy the requirements of software process improvement frameworks. Considerable time and effort is spent in collecting metrics and doing SPC analysis without realizing the benefits. This may be due to inadequate understanding of the linkages between the selected metrics and the underlying processes.
The SPC analysis techniques may also not be selected correctly due to lack of understanding about the desired process monitoring and control. This paper discusses the pitfalls of incorrect usage and benefits of choosing appropriate metrics and SPC analysis techniques based on observations in several organizations.
Outline:
- Introduction
- The Premise of SPC techniques
- Differences between manufacturing and software environment
- Metrics definitions
- Common Metrics Usage
- Pitfalls in common usage of metrics
- Examples of incorrect and correct usage of metrics
- Precautions while defining metrics for software processes
- Common usage of SPC Analysis
- Examples of correct and incorrect usage of SPC techniques for analysis
- Precautions while selecting SPC techniques for analysis
- Conclusion
Learning Objectives:
- How to avoid incorrect metrics definition
- How to ensure correct metrics definition
- How to avoid incorrect selection of SPC analysis techniques
- How to ensure correct selection of SPC analysis techniques
Biography:
Dr. Kanhaiya Jethani is currently working in the Quality Consulting Service Practice of TCS. He has been involved in Software Process Improvement projects for clients in Taiwan , Singapore , Spain , USA and Japan . He has previously worked as SEPG head in TCS for 2 years. He has been associated with TCS for 15 years and worked on many software projects in the area of Process Control for steel, cement and chemical industries in India . He is a Certified Software Quality Analyst (CSQA) and Project Management Professional (PMP). He is an Assessor for Software CMM®, and a Candidate SCAMPI Appraiser. As an Assessor he has experience in conducting a number of CMM and CMMI assessments and gap analysis internally within TCS and for external customers.
A Doctorate in Chemical Engineering from Oklahoma State University, USA, he was also a Post-Doctoral Research Fellow at Oklahoma State University, USA and University Department of Chemical Technology (UDCT), Mumbai, India. He is a member of PMI.
Software Quality Metric for Complex Systems
Evelyn F. Moritz
Consulting Member of Technical Staff
Avaya
Introduction:
In the System Test and Beta phase of product development, two questions often arise:
- Are we on track to launch the product on the target date?
- How does the quality of this release compare with previous releases?
For complex software products, the answers to these questions are not straightforward. A number of metrics are needed to adequately capture the software quality and often these metrics indicate contradictory information. For example, the defect find rate may be low (good) but a number of critical open (unresolved) defects may still exist; or there may exist a large number of open medium severity defects but system stability is solid. A method was needed to provide a simple representation of product quality to monitor progress during system test and product trials to facilitate business decisions regarding when a product was at an acceptable quality level for general availability.
Once the metric was established, we needed to verify its accuracy with customer data. Is Release X really better (or worse) than Release Y?
A Software Quality composite metric was developed 3 years ago and has been in use for the past 6 releases of AVAYA's largest software product. It has helped us answer the fundamental questions mentioned above. Additionally, data gathered from customer sites has substantiated that the metric is reasonably accurate. This approach is now being used by other projects at AVAYA.
The primary purpose of the metric is to provide a means of distilling a large amount of technical data into a format that can be better understood by business leaders. It is not meant to replace the underlying need for a variety of quality measures.
Outline:
- What is a Software Quality (SQ) composite metric
- Why have a composite metric
- Examples of how quality information was presented to business leaders in the past
- Advantages of a composite metric
- What type of software products would benefit from an SQ composite metric and when would a composite metric not be applicable.
- What measures should be considered as candidate components for an SQ composite
- Specific metrics chosen for the SQ composite metric (there are 10)
- Other candidate metrics
- How are target values (goals) established
- Establishing composite SQ goal for software release based on discussions with Product Managers – Framing the expectations for product quality
- Should a release be targeted to improve quality?
- Maintain existing quality?
- Will degradation in quality be tolerated due to introduction of new technology into the marketplace?
- Establishing component targets based on historic data
- Examples of the metric in use
- Graphs showing actual vs. target progress of a new release
- What an “on target” release looks like
- What happens when quality is not “on track” – adjustments to graph when launch date is slipped
- What happens to graph when content is removed
- Bar chart showing release-to-release quality comparison
- Caveats
- General concerns and challenges regarding any software metric
- Specific concerns regarding a composite metric
- Using customer defect data (software product defects found at customer sites) to determine software release quality
- What data needs to be collected
- Challenges associated with collecting/processing data
- Some simple (though potentially inaccurate) approaches
- A more accurate (and sophisticated) approach
- Graph showing actual release data
Learning Objectives:
- Challenges with interpreting software quality data in the business context
- Metrics/measures to track product quality during the system test interval
- How a software quality composite metric can be used to track product quality and facilitate product launch decisions
- How to create a software quality composite metric
- Examples of how a composite metric is being used on a large commercial software system.
- Challenges and concerns regarding software metrics in general and a composite metric in particular
- Challenges associated with determining if product quality has improved once a new release has launched (general availability)
- Examples of some straightforward methods and their pitfalls
- Example of a more sophisticated approach that is currently in use
Biography:
Evelyn Moritz received an MS in Computer Science from the University of Colorado , Boulder in 1978 and joined AT&T Bell Labs that year, developing software tools for PBX switching systems. She moved to System Test in the 80's with a focus on network and performance testing of large switches. From there, she became a coordinator of System Test activities responsible for quality assurance of Lucent's Enterprise switching products. During this period she participated in a variety of process improvement activities and developed a common system test process that became standard for the Lucent Enterprise product line.
She has had direct experience with the "end-game" of every major release of Lucent/AVAYA Enterprise Communications software products for the past 10 years. These software products are in use at over 50,000 sites world-wide. She has navigated the seas of change in the telecommunications and software development industries, adapting processes to respond to today's demands for rapid time-to-market, while still maintaining a high standard of product quality.
As a Consulting Member of Technical Staff at AVAYA, she has developed a suite of software metrics to measure both product and process quality. She continues to consult with AVAYA projects to develop better test strategies, improve test processes and improve overall product quality.
Thriving in a Diverse Environment Through Technology Change Management
Barbara Dreon
Technology Change Management Lead
Northrop Grumman Corporation
Introduction:
It can be a challenge to envision how to apply Technology Change Management (TCM) in an organization that serves many unrelated customers with a myriad of systems and technologies. Northrop Grumman has adapted its approach to TCM to accommodate this diversity. The presentation highlights the market for and benefits of TCM, touches on a paradigm for success, and couples an industry model used for technology change with mechanisms used by Northrop Grumman to strengthen its technology monitoring capabilities and its ability to more efficiently deploy a technology across multiple projects. Supporting data for the ideas are drawn from industry and from within the company. These ideas can be adapted for use in other organizations. The presentation is accompanied by a detailed white paper.
Outline:
- Brief background: concepts and context for understanding technology change management (TCM) approach
- Examples from industry and within Northrop Grumman showing the usefulness and benefit of technology change management
- Common challenges to deploying TCM in a diverse environment
- Overview of the TCM process in use at Northrop Grumman
- Technology Identification and Monitoring
- Technology Evaluation and Selection
- Technology Planning
- Technology Development and Insertion (including technology replication)
- The importance of people in the success of technology transfer
- Enablers to transfer technology to disparate recipients (organized in terms of a model for learning or technology adoption known as the Patterson-Connor model)
Learning Objectives:
- Be able to define technology change management (TCM) and related terms
- Understand why TCM is important and beneficial
- Be aware of an instantiation of a TCM process that is used successfully in a diverse environment
- See the Patterson-Conner Model applied to change adoption and combine it with ideas to increase success in transitioning technology changes to multiple parts of an organization
- Recognize some opportunities and issues for technology insertion and technology transition
- Based on industry data and company experience presented, be ready to select some ideas to apply in another organization
Biography:
Ms. Dreon is a Technology Change Management Lead in Northrop Grumman Corporation. She has over 10 years of experience in process improvement for software and systems engineering and works for a company that has achieved over 18 CMMI Level 5 ratings. She has an M.S. in Operations Research and enjoys helping projects and groups apply process discipline to creatively suit their goals. She is also an instructor and has provided past presentations at industry forums on such topics as technology change management, process improvement, process asset libraries, and measurement.
Me Facilitate the Session: Fundamentals of Meeting Facilitation
Clyneice Chaney
Quality Manager
Project Performance Corporation
Introduction:
Facilitated sessions are being used in a variety of ways, from gathering requirements, identifying risks, designing processes and performing inspections and reviews. The facilitator's role is to help a group to its best thinking. A good facilitator is helpful when a group is trying to deal with new or difficult issues. In the main, a facilitator helps people persevere as they confront the inevitable confusion and frustration associated with trying to integrate different views and approaches with their own. The more people who learn to facilitate the better. This presentation walks you through the basic steps to become a better facilitator.
Outline:
- Meeting and Facilitation
- Key Facilitator role
- Five Principles of Facilitation
- Meeting Planning and Preparation
- Managing the Session
Learning Objectives:
- Understand the facilitator role in meetings and improvement sessions
- Understand the principles of good facilitation
- Know how to prepare for a meeting as a facilitator
- Know how to manage the facilitation session
- Know how to manage the data and information from a variety of “types” of facilitated sessions
Biography:
Clyneice Chaney, Corporate Quality Manager with Project Performance Corporation brings over 20 years of testing, quality assurance and process improvement experience. Clyneice holds certifications from American Society for Quality as a Certified Quality Manager, Quality Assurance Institute's Certified Quality Analyst, and Project Management Institute's Professional Project Manager. She has served on the Board of Examiners for Virginia and Georgia 's State Quality Awards.
Focusing on process improvement and procedure development in the software testing and quality assurance areas, Clyneice has successfully lead process improvement, methodology development, and reengineering projects for organizations wishing to improve their software development, testing processes, and tool implementation.
Clyneice is currently an instructor for the International Institute for Software Testing and has presented technical papers at the Software Engineering Institute: SEPG Conference, American Society for Quality: Quality Manager's conference, Quality Assurance Institute International Testing conference and STAR East Testing and the Quality Assurance conference.
Wasted Days and Wasted Nights: Leveraging Your Appraisal Teams as a Resource
Tim Davis
Senior Engineer Manager
Raytheon Missile Systems
Introduction:
The scene opens on the program's personnel frantically trying to finish up the task of putting best representative evidence into the PII database before Appraisal Team arrives on Monday for the next readiness review, ok at this point any evidence will do. Frustration is written on these frantic faces, as the appraisal team has been her twice now and the keep saying the same thing, “These rocks are no good, go get different sent of rocks!!” And in the minds of the program folks are the repeated thoughts: “ If there so darn smart how come they don't just come and get the data themselves. Then I would not have to be here ‘til mid-night again.”
Scene two opens and the appraisal mini-teams are grinding on their third late night, trying to dig through the contents of
