CPE 309 Software Quality Assurance Plan
[Test Plan] [Standards]
[Metrics] [Tools] [Integration] [Release]
The table of contents of this SQAP follow that of IEEE standard 730-1989.
1. Purpose
This document describes the plan by which the CPE 309 project
will produce a quality product.
2. Referenced documents
Project
Plan
3. Management
3.1 Organization
The team will designate an individual "quality assurance leader," named
in section 3.3. The course may also have a QA "guru" who
coordinates QA activities between teams.
3.2 Tasks
QA tasks during CPE 309 shall include
- documentation,
- informal, internal QA reviews
- software inspections
- daily build
- defect reporting and tracking
- requirements traceability
- verification (including testing),
- metrics collection and analysis
These tasks are detailed in this document.
3.3 Responsibilities
Each team member is responsible for the quality of his or her
work. Each person should be sure their work conforms to the QA
criteria provided on the course web page.
It is the quality assurance leader's responsibility to see to it
that the tasks in 3.2 are done, and to ensure that the prescriptions in
this document are followed, including scheduling the inspections
specified and the collection of metrics data. The QA leader is
ultimately responsible for the quality of the deliverables.
The Quality Assurance leader for team (insert
name here) _ is (insert
name
here) .
4. Documentation
All the documentation required for this project is described in the
project plan (SPMP).
The
GNOME style guide
is a very good reference for writing tips for user
documentation. You might require your analyst to read this
document. Of particular use to QA people is the section on quality
reviews.
5. Standards, practices, conventions and metrics
5.1 Purpose
This section describes the standards, practices, conventions and
metrics to be used for the CPE 309 project. These are intended not only
to ensure quality for CPE 309 project, but also to obtain quantitative
metric data on the SQA process itself. This data is to be used to help
establish the key practices of a CMM
level four organization.
5.2 Standards
- The IEEE standards, with appropriate modifications, are to be
used for documentation. In most cases teams should follow the
templates Dr. Dalbey has prepared for major documents.
- Quality criteria and checklist are to be used for specification
and design documents.
- UML notation is recommended for Analysis and Design documents.
- Detailed Designs must follow the class pseudocode
standard.
- Java source code must adhere to the class coding
standard.
- E-mail message "subject lines" must follow the guidelines in the
project plan.
- Directory names in the team account must begin with a
capital letter.
5.3 Practices
- The document which describes the quality attributes the finished
software must exhibit is the non-functional
requirements section of the SRS. The QA manager should work
with the Analyst to ensure this document is well written and meets the
customer's needs. Also assist the Test Manager in writing tests
for these requirements.
- From a technical perspective, the success of the software depends
on the detailed designs (pseudocode). It's worth putting extra
effort into making these correct and very detailed. Internal
review is required.
- Because delaying quality is expensive, engineers are strongly
encouraged to apply quality precepts while they work, rather than as an
afterthought. "If you don't have time to do it right in the first
place, when are you going to find time to fix it after it breaks?"
- QA is an important function in CPE 309 as part of the student
course grade depends on the quality of the project deliverables.
A
cooperative, team-oriented attitude is crucial.
- Even though the QA person is ultimately responsible for quality,
the QA person is NOT expected to edit and rework every artifact that
s/he receives. Each individual must produce quality work.
The QA job is to assure that the product has achieved the required
level
of quality before it is delivered.
5.4 Conventions
- HTML (or Wiki) is the preferred format for documents that are to be shared
in computer readable form.
- Guidelines for all written work are given in the syllabus.
- Source code printouts must follow the guidelines given in the
syllabus.
- The web pages on the team web site must be viewable in Internet
Explorer and Firefox.
5.5 Metrics
5.5.1 Individual Metrics
Individuals will track the number of defects found by the inspectors of
their code during formal software
inspections.
(Recommended) Programmers will plan and track their own unit
development work using the Personal Software Process (PSP). Consult the
PSP
script for details on gathering size, cost, and quality data.
During
Stage 1, only size and cost measures will be required. During
Stage 2, quality measures will also be required.
5.5.2 Team Metrics
The QA person is responsible for compiling and reporting the following
metrics, but each person is responsible for gathering their own data.
- Productivity Metrics (Size and Cost)
Actual Lines of Code. Use the LOC procedure. Categorize by
individual and team total. Keep a separate total for the lines of
testing code.
Actual Hours of effort (Summary of Individual hours, categorized by
project phase, as gathered from timecards. Also team total by phase.)
For example:
| Name |
Stage 1 |
Stage 2 |
Total |
| Sue |
55.25 |
47.75 |
113.5 |
| Tom |
63.50 |
54.25 |
128.75 |
| Bill |
50.25 |
56.25 |
115.5 |
| Team Total |
179.00 |
158.25 |
357.75 |
- Defect Metrics
Number of issues (software inspections). Inspections will be
conducted using a formal process and any issues found
will be recorded.
Number of compile defects. That is, when a defect causes
integration failure ("broken build"). This can be during the
official daily build, or unofficially by any developer
who compiles the code from the team repository.
Any time after a module is checked in to the repository,
if the system won't compile it counts as a compile defect.
These defects are not entered into the defect
tracking system, so they must be counted separately.
All developers must gather
this data.
Number of test defects. Count of defects found during integration and
system testing (not unit testing). Just as above,
once a module is checked in to the team repository, if
it causes a defect, it gets counted as a test defect.
Each of these
defects is entered into the team defect reporting system, so this measure
can be obtained from the reporting system itself.
Defects per KLOC. Compute as: total test defects * 1000 / Total LOC.
Total Defect Repair time. Repair times are logged for each defect
in the defect reporting system. The total must be computed by
manually summing the all the individual times.
- Software Design Quality Metrics (recommended)
It is recommended that each team use some objective means
to determine the quality of their design.
The QA person may compute the following metrics to
assess the quality of design and code. These can be collected
using the Together CASE tool available in the lab. The QA person
is responsible for understanding the meaning and how to interpret each
of these metrics. These metrics should be collected every other week
and
a briefing given to the team.
Lines of Code, Comment Ratio, Number of Classes, Lack of Cohesion of
Methods, Cyclomatic Complexity, Response for Class, Weighted Methods
Per
Class, Coupling between Objects, Coupling Factor, Fan Out, Depth of
Inheritance Hierarchy.
- Change Metrics
Number of Formal Change Proposals Submitted.
Number of Formal Change Proposals Accepted.
Number of Uncontrolled changes to SRS. Save a copy of the
original baselined SRS. Perform an automated "diff" against the final
SRS. Count each item appearing in the diff for which there isn't
a change request.
Number of Uncontrolled changes to High Level Design (javadocs).
Save a copy of the original baselined javadocs. Perform an automated
"diff"
against the final javadocs. Count each item appearing in the diff
for
which
there isn't a change request.
5.6 Goals and Reporting
[Note to the student: The quality targets used here are based on
historical data obtained from previous student teams on similar
projects. Based on previous team data, these targets are
aggressive but not unrealistic.]
CPE 309 quality goals for delivered products are as follows.
- Software Inspections: no more than one severe and three minor
issues per inspection.
- Compile: no more than one broken build for the entire quarter.
- Integration and System Test: no more than fifty defects per KLOC
(thousand lines of code).
- Acceptance Test: no defects during acceptance testing.
The actual metric data gathered from this project (all the metrics in 5.5.2) are
to be reported in the
final project submission.
at each release.
6. Demos and Reviews
6.1 Purpose
The purpose of demos and reviews is to provide a means of focusing the
attention of engineers on quality of the application as it develops.
- Internal Reviews are a peer review by the QA leader or
another team member of individual work products. Here is a recommended
QA submittal
form.
- Software
Inspections are peer reviews of source code before unit testing.
- Class Demos are presentations of completed work products
or deliverables in front of the entire class.
- Customer Demos are a formal presentation for approval of
the customer.
6.2 Minimum requirements
6.2.1 Internal Reviews
All new detailed designs in Stage 1 and 2 will
undergo internal review.
6.2.2 Software Inspections
All new source code modules for Stage 1 and 2 require a
formal code inspection.
6.2.3 Post mortem review
The team will conduct a post-mortem review at the end of CPE 309 to
gather the insights and lessons learned during the course.
7. Test Plan
A software project test plan describes the objectives, scope, approach,
activities, and procedures your team will use for software
verification.
It explains who is responsible for each kind of testing, how it is to
be
carried out, and how it will be documented and tracked.
Here is the Test
Plan to be followed for CPE 309, based on the IEEE standard.
8. Defect reporting, correction,
and tracking
This section needs to be updated for the Trac tool!
Defects in the software found during integration and system test (NOT
unit test) are tracked using an automated web-based tool, Elementool. According to
the Integration Procedure (10.1) only unit tested code can be submitted
to the repository. So usually any tests that are run against the
repository codebase are integration or system tests. Any defect
found in such a test must be reported in Elementool.
The instructor will assign you an Elementool "project." When you
receive e-mail from Elementool that your account is setup, the QA
manager should perform these initial steps:
- Login to www.elementool.com
- goto Account Control Panel
- choose Edit User Profiles
- choose Add New User
- Fill in the required fields for each team member.
- Add the instructor as a team member with admin privileges. user:
jdalbey (assign password: John).
- choose Edit Issue Form
- Change "Product" field name to "Release"
- Add values for the drop down box: 0,1,2
- You may make other customizations upon instructor approval.
Make sure the hyperlink "Current Defect List" on the team home page
links to Elementool.
Refer to pg 129-130. Defects are corrected using this Defect
Repair Procedure. Make sure all developers follow this
procedure to the letter!
9. Tools,
techniques and methodologies
SQA techniques include the application of standards,
requirements tracing, FTR's and software verification (testing).
The SQA tools consist of form templates and checklists.
- Form Templates are provided on the course web site.
- QA Criteria Checklists for each major deliverable are provided in
the deliverables
list.
9.1 LOC Counting Procedure
As a measure of product size we will use logical lines of code. Do not
include unit test code as part of the product size measure, only code
in
the product itself. Our procedure requires following the class
coding standard and a tool to count lines of code. We do not
include comment lines in our line count. Here are two suggested
tools for you to consider:
9.2 Requirements
Traceability
See McConnell page 151. Each requirement in the specification is traced
to each module(s) in design and code that implements it. Table
format works fine for this.
9.3 Unit Development
Procedure
This procedure describes every step that a developer undertakes in
producing a single module (unit). The Unit
Development Procedure (aka "PSP script") for CPE 309 is a more
elaborate version of Table 14-1 pg 203. (Note that McConnell calls this
an "integration" procedure, but we will call it a Unit Development
Procedure).
This script has been painstakingly prepared for you. Read the
directions completely, carefully, and be sure you
understand every word. There is also a FAQ.
If there is anything in these directions about which you have the
slightest question, ask for clarification! Completing
the PSP forms is recommended, but not required. You are
expected to complete the PSP forms accurately and precisely.
Be sure you understand this procedure and follow it religiously. If
the procedure doesn't work for you, seek the instructor's permission to
modify it. Note this procedure only applies to new code, not to
maintenance tasks.
9.3.1 PSP Submission
Procedure
The PSP forms for a unit you develop are to be submitted to the QA
manager during the first class session after the unit has been
submitted
to the build. The QA manager gathers all the PSP forms and may
publicize which have been completed (but do not publicize individual
data). The QA manager submits all the PSP forms to the instructor
with the Release. (Completing these forms is an individual choice
as described in the previous section).
9.4 Internal Review
Internal reviews are the best way for a team to ensure quality. The
QA person must write the procedure for internal review and place it
here. At a minimum it requires review of the work product by one
other team member and documentation using a form such as this review
form. The QA leader may create a scoring mechanism,
checklists, or other artifacts that seem helpful for the team.
The
reviews may be staff from a different team. The QA leader may
choose to do all the reviews himself (or herself). Be sure to
work
with the team manager to allocate time for review meetings.
9.5 Software Code Inspections
All new modules for Stage 2 require a code inspection.
Read these guidelines to Software
Inspections for Software Engineering Student Teams.
Here is the required CPE 309 Software
Inspection Process.
9.5.1 Software Inspection
Report
A summary report of results from the software inspections is to be
posted on the team web page under the link "Software Inspection Report"
9.6 Design By Contract
Design by Contract is a module specification technique that intends
to improve design reliability and reduce development costs. After
completing the homework activities and participating in the class
discussion about DBC, each team must decide which specification style
they choose to adopt: Design by Contract or Error Checking. For
the second stage of development, all modules must be specified in
the style the team has chosen. It is the QA manager's job to
enforce conformance to this style.
On (decision date) our
team decided to adopt the following module specification style (check
one):
___ Design by Contract
___ Error Checking
10. Integration Procedure
Integration is a crucial technical area that can make or break a
project. The implementation manager is responsible for having a
detailed integration plan and the QA manager is responsible for
enforcing conformance to the integration procedure. Don't let
people cheat on the integration procedure! Our integration
procedure will include a team "daily build" which helps avoid the "big
bang" syndrome.
10.1 Integration Procedure
Our integration procedure is based on an Incremental Integration
strategy (see notes).
It assumes the design is complete and the interfaces have been
compiled. The implementation manager create the Implementation
Plan which lists the order in which modules are written and integrated.
Developers then write their individual units using the Unit
Developer Procedure. No partial implementations allowed. (See
"Done is Done" pg 203).
10.2 Build Procedure
There must be a regular ("daily") build of the software product. The
build must start at the beginning of the third week of class. The build
must occur at least 4 times a week (suggested SMWF).
The implementation manager is responsible for writing the
specific technical instructions for how to run the build on your
development platform. Include the schedule for when the build
will
be run and procedures for dealing with broken builds..
Here is a ten-step
summary of the steps in the build process.
Here is an example build procedure
from a previous student project.
Here is a REAL
build procedure for Mozilla.
10.3 Daily Build Report
The results from running the daily build are to be posted on the team
web site. Refer to Ch 14. You may want to create a list of
chronological links to a separate page for each build report.
A build report contains
- Date and Time the build was run.
- Optional Build Number.
- Status: Compiled / Broken.
- Which modules were included in the build (Could be CVS log info
for most recent version).
- Computer output from build script.
- A hyperlink to the build itself for testers and developers to
download and run.
Here are example
build reports from a previous student project.
11. Software Release
Release Criteria
These criteria dictate when the software can be released. Pay
attention to these; the software can't be released until these criteria
are met (regardless of the planned schedule). Discussed in Chapter
16. Here are the CPE
309 release criteria. (negotiable until end of week 3).
Release Checklist
This is a list of all tasks that must be completed before the product
can be delivered to the customer. Discussed in Pfleeger Chapter
10
and McConnell Chapter 16. See McConnell's template.
It is the
responsibility of the QA mgr to create the release
checklist and to supervise related tasks.
For example, one release issue is how to package the software and
test artifacts.
Software Deployment Plan
A Deployment Plan describes how you intend to get the software
transformed from source code in your team repository to executing bits
on the user's machine. This Implementation Manager is the
logical person to create the plan
with assistance from the QA manager. Here
are issues and considerations in
writing your plan. Place a link to the plan either here or on the
team home page.
13. Records collection, maintenance & retention
The SQA records collected and archived shall include ...
Individual Status Reports- Individual PSP forms (if collected)
- Internal Review Forms
- Software Inspection Reports
- The Current Defect List
- Release Checklist for each Release (and associated documentation)
14. Training
The only SQA training planned is class lectures. If additional training
seems necessary, describe it here.
Change History
4/23/08
|
JD
|
Changed metrics report (Sec 5.6) delivery from final report to each release.
|
5/18/04
|
JD
|
Added testing code to LOC metrics required.
Added Total Defect Repair Time to metrics required.
|
4/13/04
|
JD
|
Added directions for setting up Elementool to
section 8.
|
3/27/04
|
JD
|
Minor editing and additions for Spring 2004
|
2/18/04
|
JD
|
Added section 9.5 on DBC
|
| 1/12/04 |
JD
|
Revised for Winter 2004. PSP forms made
recommended, not required. Revised quality goals. Added
more
explanation to metrics. |
| 4/8/03 |
JD
|
Changed "teamatic" to "elementool" |
Home