Test Plan Checklist and Standard Format

Testing Plan

Test plan is probably one of the most significant documents for software testing projects. Test plan may contain information related to scope, environment, schedule, risk, resources, execution, reporting, automation, completion criteria etc. Test plan is usually created by Test Manager, Test Lead or senior testers in the team. Before start preparing Test Plan, information should be captured from various stakeholders of the project. Information captured from stake holder is reflected in the Test Plan. Typically, every Test Plan contain information about following activities. Information about these activities can be gathered from various stakeholders by asking questions that are required for your Test Plan. 

Scope Management: 

Before starting Test Planning activity, scope of the test activities should be known. You should have information on what features will be tested and what will not be tested. You should also have information on what areas your team is owning? Are you taking care of all the types of testing that is required for the product including Performance, Security, globalization etc. Defining scope for your testing project is very important for the management as well. If scope is properly defined, every one will have clear understanding about what is tested and what is not. 

Reference: You should clearly define documents you are referring to prepare test plan. Any changes in the documents you are referring should be reflected in your plan. 

Risk Management: Test strategy is derived from the risk analysis. Risk will be different from one project to another and so your Test strategy. Risk associated with a desktop tax calculation software will be different from payment gateway or life support system. In your testing strategy, you need to make sure that all the potential risks are captured and managed by your testing activities. You should, along with the other stake holders define what are the potential risk in the project? What will be the impact if these risks are materialized? What is the mitigation plan for these risks and how your testing activities are making sure that these risks are managed properly. 

Test Environment: You should have information on what will be the environment for the testing. This information is captured from stake holders by asking them, What type of environment will be supported by product? What is the priority for these environment? For example, If product is supported on all the platform and data for user distribution says that 80 percent are on Windows, 15 percent are on Linux and 5 percent are on Mac. From this data you can make out which platform will be tested more. Information captured here will be useful for planning hardware and software requirement for your Test Lab. 

Criteria Definition: Criteria for Entry and Exit should be clearly defined for every activity of your testing project. You should have well defined entry/exit criteria for starting, stopping, suspending and resuming test activities. You should also have criteria defined for specifying when testing is complete. 

Estimation, Scheduling and Resource Management: Mostly, testing project follows development activities. So for estimation and scheduling you should have information on the development plan and milestone. Once you have information on the development plan, you can schedule your testing activities accordingly. Resources in testing projects include hardware, software and people management. 

Testing Tools and Automation: You should have information about what tools you are using to manage your testing activities. You should have information on the configuration management for test artifacts, test case management tool, defect tracking system, tools for automation etc. Ideally, test automation should be treated as separate project and you should have brief information here along with the link to automation plan. 

Execution and Reporting: In this section you should have information on how execution will be managed for the various testing activities. What kind of reports you are planning to generate from the data that you gather from test activities. This should have information on the various matrixes and how they should be interpreted. 

Release Criteria: This should clearly state release criteria for the product. Criteria defined here should be clear and measurable.

                    Test Plan Standard Format

Test Strategy

<Product/Project name>

<Version <here>

<Date Here>

Document Control

Document Detail

Title:	<Formal name for the Strategy with reference to the area it covers>
Version:	<Current Version>
Date:	<Date of issue of current version>
Electronic File Name:	<Document file name.doc>
Electronic File Location:	<SharePoint URL>
Author:	<Your Name>
Contributors:	<Add names of anyone reviewing or providing information>

Change Control

Issue Date	Version	Details	Author
<Issues date>	v0.1d	Working Draft, not yet for review	<Your Name>

Referenced Documentation

Ref	Document Name	Electronic File Location
	STLC	<Your SharePoint, Twiki or Knowledge Tree URL>
	SDLC	<Your SharePoint, Twiki or Knowledge Tree URL>

Table of Contents

1.      Test Strategy Identifier................................................................................................ 4

2.      Introduction...................................................................................................................... 4

2.1.       Purpose....................................................................................................................... 4

3.      Test Items............................................................................................................................. 4

4.      Features to be tested............................................................................................................. 4

5.      Features not to be tested........................................................................................................4

6.      Approach............................................................................................................................ 4

6.1.       Analysis & Planning Phase Entry Criteria.......................................................................... 5

6.2.       Analysis & Planning Phase Exit Criteria.............................................................................5

6.3.       Test Phase Entry Criteria................................................................................................. 5

6.4.       Test Phase Exit Criteria.................................................................................................... 5

6.5.       Change Management....................................................................................................... 5

6.6.       Notification / Escalation Procedures................................................................................. 6

6.7.       Measures and Metrics................................................................................................. 6

7.      ‘Pass/Fail’ Criteria........................................................................................................... 7

8.      Suspension Criteria & Resumption Requirements................................................... 7

9.      Test Deliverables............................................................................................................ 8

10.     Testing Tasks.................................................................................................................... 8

11.     Environmental and Infrastructure Needs................................................................ 8

12.     Responsibility Matrix..................................................................................................... 9

13.     Staffing and Training Needs.......................................................................................... 9

14.     Schedules and Resource Plans................................................................................... 9

15.     Risks and Contingencies................................................................................................. 9

16.     Approvals........................................................................................................................ 10

1.    Test Strategy Identifier

The unique identifier for this Test Strategy is: <Test Strategy ID>

2.    Introduction

<Your audience may not know of the Service that you have put this strategy in place for, provide a brief narrative introduction to the product or service offering. Consider including product history, reasons for introduction or changes, expected outcome of the changes, who might use it and the benefits of them using the new or enhanced product>

2.1.   Purpose

The purpose of this Test Strategy is to define the overall approach that will be taken by the Test Team when delivering testing services to all of the projects within the business.

The document helps to clarify the testing activities, roles and responsibilities, processes and practice to be used across successive projects.

Where a project’s testing needs deviate from what is covered by this Test Strategy the exceptions will be detailed in the Test Plan.

Glossary of Terms

Refer to the Test Department Test Glossary for definitions of company specific terminology.

Refer to the Cyreath Testing Glossary for definitions of general testing terminology.

3.    Test Items:

For each Release the Test Engineer will create a table of Test Items that will be in scope of the testing being planned. These will be identified from the Scope Items in a given Release and include interrelated modules and components of the service that will be affected by the Scope Items.

In addition the Test Engineer will record any Test Items that cannot be tested by the test team. The Test Plan will contain Test Items that are In-Scope and Out-of-Scope.

4.    Features to be tested

The Test Engineer will use the Test Breakdown worksheet (ref#) to record all of the features to be tested for each of the Test Items in scope.

The Test Breakdowns will include details of the Test Scenarios from which the Test Cases will be derived.

5.    Features not to be tested

<What features would we usually not test in a project? Security, Accessibility?>

Where it is not possible for the team to test features of a Test Item that would have been expected or that would fall under the scope of testing shown in section 10. Testing Tasks, it will be recorded in section 5 of the Test Plan.

6.    Approach:

All testing tasks will be conducted in line with the Software Test Life Cycle (STLC) and in support of the Software Development Life Cycle (SDLC). The documents used within the SDLC will be completed both by the Test Team and the project participants that are responsible for providing information and deliverables to the Test Team.

It should be decided at the start of the project if there will be a Post Implementation Review after project delivery and this should be conducted within two weeks of project completion.

<Narrative description of the high level Strategy>

<A description of the test methodology that will be followed>

<Include a diagram where possible – V-Model, Scrum, Waterfall>

<Touch on risks, discuss the use of test equipment or data, etc. – use this paragraph to tone-down the formality of the rest of the document and set the scene for the rest of the plan, look to add value for the audience in the approach and their support of it. Touch on anything that has no clear area for inclusion in the Test Plan>

6.1.   Analysis & Planning Phase Entry Criteria

For all projects the following criteria need to be met before the Test Items are accepted into the Analysis & Planning Phase:

• Release scope item list is locked and prioritised
• Documentation defining the scope items are approved and at release status
• All documents are under change control processes

6.2.   Analysis & Planning Phase Exit Criteria

For the Analysis & Planning phase to be completed and allow items to move into the Test Phase the following criteria need to be achieved:

• Test Breakdowns and Test Cases are written and peer reviewed
• Knowledge Share document has been completed and reviewed by the Test Engineers
• Walkthrough and sign-off completed for the Test Plan and Test Breakdowns
• Defined Test Estimate has been published and agreed
• The list of features in the Test Breakdown have been prioritized.

6.3.   Test Phase Entry Criteria

Before Test Items are made available for the Test Team to test it’s expected that:

• The Test Item Transmittal Report will be completed
• All test tools are available and test infrastructure are available for use during testing
• All Test Items are development complete
• The correct versions of the code have been deployed to the correct test environments
• Sanity and Unit tests have been completed successfully to demonstrate readiness for test

6.4.   Test Phase Exit Criteria

For the Test Items to exit testing the following conditions will have to be met:

• The Test Summary Report will be completed.
• All planned testing activities has been completed to agreed levels.
• All high priority bugs have been fixed, retested and passed.
• No defects must be left in an open unresolved status.

6.5.   Change Management

The Build Manager will ensure that once testing begins no changes or modifications are made to the code used to create the build of the product under test. The Build Manager will inform the Test Team against which version testing will begin and confirm the location within [VSS/Progress/Perforce/Subversion] the build is to be taken from.

If changes or modifications are necessary through bug resolution or for any other reason the Build Manager will inform the Test Team prior to the changes being made.

6.6.   Notification / Escalation Procedures

The following diagram shows the notification and escalation paths to be followed for the duration of the project Test Phase.

         

6.7.   Measures and Metrics

At the Initiation Phase of the project the Test Team will publish a set of measures and metrics related to the test activities of their Planning & Analysis and Execution phases. The Test Plan also defines the milestone dates for key deliverable such as the Test Plan and these are metrics captured for ongoing statistical process analysis across successive projects.

Test Preparation

• Number of Test Scenarios v. Number of Test Cases
• Number of Test Cases Planned v. Ready for Execution
• Total time spent on Preparation v. Planned time

Test Execution and Progress

• Number of Tests Cases Executed v. Test Cases Planned
• Number of Test Cases Passed, Failed and Blocked
• Total Number of Test Cases Passed by Test Item / Test Requirements
• Total Time Spent on Execution vs Planned Time

Bug Analysis

• Total Number of Bugs Raised and Closed per Test Run
• Total Number of Bugs Closed v. Total Number of Bugs Re-Opened
• Bug Distribution Totals by Severity per Test Run
• Bug Distribution Totals by Test Item by Severity per Test Run

7.    ‘Pass/Fail’ Criteria

Each Test Item will be assigned a Pass or Fail state dependant on two criteria:

• Total number and severity of Bugs in an Open & Unresolved state within Bugzilla/Bug Tracker.
• The level of successfully executed test requirements.

The combination of both criteria will be used to recognise the Test Item can be declared Test Complete. However as this is a minimum level of quality that is believed achievable it’s recommended that where project timescales allow further testing and development should be conducted to raise the overall quality level.

Table of Issue Severity

Severity	Definition	Maximum Allowable
S1	Crash/Legal – System crash, data loss, no workaround, legal, Ship Killer	0
S2	Major  – Operational error, wrong result	<Set by PM>
S3	Minor – Minor problems	<Set by PM>
S4	Incidental – Cosmetic problems	<Set by PM>
S5	N/A – Not Applicable; used for feature requests and Development Tasks	Reference Only

The total MAXIMUM number of issues recorded in Bugzilla / Bug Tracker that can remain in an Open & Unresolved state for the Test Item and be acceptable for release.

Table of Test Scenario Priority

Test Scenario	Definition	Minimum Pass Rate
P1 – Critical	Essential to the Product	100%
P2 – Important	Necessary to the Product	<Set by PM>
P3 – Desirable	Preferred, but not essential to the Product	<Set by PM>

The MINIMUM set of Test Scenarios that must pass before the Test Item can be considered for release.

Unforeseen issues arising during the Test Phase may impact the agreed ‘Pass/Fail’ Criteria for the Test Item. Issues can be managed through review with the Test Team and the project authorities.

8.    Suspension Criteria & Resumption Requirements

Testing of Test Items will be suspended if:

1a) Suspension criteria:

A Severity 1 issue is logged and requires fixing before further testing can take place (a Blocking Issue)

1b) Resumption requirement:

The issue will need to be fixed before the Test Item is returned to the Test Team for testing.

2a) Suspension criteria:

Significant differences exist between observed behaviour of the Test Item and that shown in Test Scenario, Test Case or as expected from the previous version of the technology.

2b) Resumption requirement:

Development, the Test Team and PM must come to a conclusion on resolving the issue and agreeing a definition of the expected behaviour.

3a) Suspension criteria:

A Test Item sent for testing fails more than 20% of Developer Unit Tests.

3b) Resumption requirement:

The Test Item must be fixed or Unit Tests re-factored if out of date and then demonstrated to pass with <20% failure rate.

9.    Test Deliverable

The following artifacts will be produced during the testing phase:

·               Test Plan

Used to prescribe the scope, approach, resources, and schedule of the testing activities. To identify the items being tested, the features to be tested, the testing tasks to be performed, the personnel responsible for each task, and the risks associated with this plan.

·               Test Schedule

Which describes the tasks, time, sequence, duration and assigned staff.

·               Test Breakdown

Which includes the Test Scenarios, their priority and related number of Test Cases along with the defined estimates for time to write and execute the Test Cases.

·               Test Cases

Detail the pre-conditions, test steps and expected and actual outcome of the tests. There will be positive and negative test cases.

·               Periodic progress and metric update reports

·               Bug Reporting

·               Test Summary Reports

10.      Testing Tasks

The Testing Tasks that the Test Team will deliver cover the following scope:

·               Fully In Scope: Functional and Regression Testing

·               Partially in Scope: Cross Browser Compatibility, Integration in the Large.

·               Out of Scope: Performance testing, Automated Regression, all forms of Non-Functional, Accessibility Compliance Testing, Security Testing, User Documentation Review.

11.      Environmental and Infrastructure Needs

The following detail the environmental and infrastructure needs required for the testing of last minute.com Test Items and execution of Regression Testing.

Hardware.

• Integration Environment:
• Test-A: http://.....
• Test-B: http://....
• Pre-live Staging:

Software

• <Name of Bug Tracking Tool>: http://...
• <Name of Test Case Management Tool>: http://
• <Name of Automation Tool>: http://

Infrastructure

• Network connections are available on all Test Systems as required.

Test Repository

• http://...

12.      Responsibility Matrix

The table below outlines the main responsibilities in brief for test activities:

Activity	Product Manager	Development Manager	Test Manager	Test Engineer
Provision of Technical Documents	X	X
Test Planning and Estimation			X	X
Review and Sign off Test Plan	X	X	X
Testing Documentation			X	X
Test Preparation and Execution				X
Test Environment Set-up				X
Change Control of Test Environments			X	X
Provision of Unit Tested Test Items		X
Bug fixes and return to the Test Team for re-test		X
Product Change Control	X	X	X
Ongoing Test Reporting			X	X
Test Summary Reporting			X

13.      Staffing and Training Needs

Staffing.

Staffing levels for the test activities will be:

• 1 x Test Manager for the duration of test planning at 50% effort against plan.
• The required number of Test Engineers for the duration of test execution at 100% effort against plan.

Training.

For each project the training needs will be assessed and defined in the Test Plan.

14.      Schedules and Resource Plans

Team Plan.

The Test Team will maintain a Team Plan which records individual assignment to testing tasks against assignable days. This will also record time planned and delivered against the tasks which will be used to update relevant Project Schedules and be used in periodic reporting.

Test Schedule.

The Test Schedule for the Release will be located within <Document Store Name> at: http://

15.      Risks and Contingencies

	Risk	Mitigation Strategy	Impact
1	Delays in delivering completed Test Items from Development would impact test timescales and final Release quality	Product Management and Development to advise of any delays and adjust Release Scope of Resources to allow the test activities to be performed.	High
2	Delays in the turn around time for fixing critical bugs, which would require re-testing, could have an impact on the project dates.	Strong management of bug resolution would be required from Development to ensure bugs are fixed and available for re-testing in the scheduled time.	High
3	The Test Team, Development or PM teams require domain guidance from one or the other and they are not available. This would delay project activities.	The Test Team, Development and PM teams to ensure they are available at critical points or contactable during the project activities.	Medium
4	Features of Test Items will not be testable.	The Test Team will record untested features and request the PM to assess business risk in support of the release of untested features.	Low
5	Unexpected dependencies between Test Items and service components are encountered that require revision of Test Scenarios and related Test Cases.	Information about dependencies is updated and communicated promptly to allow timely revision of Test Scenarios and Test Cases	Low

16.      Approvals

The following people are required to approve the Test Strategy

Approval By	Approval
Test Manager
The Test Department Manager
Product Owner
Development Manager
Project Manager

Test plan template, based on IEEE 829 format

• Test Plan Identifier (TPI).
• References
• Introduction
• Test Items
• Software Risk Issue

1. ·  Features to be Tested
2. ·  Features not to be Tested
3. ·  Approach
4. ·  Item Pass/Fail Criteria
5. ·  Entry & Exit Criteria
6. ·  Suspension Criteria and Resumption Requirements
7. ·  Test Deliverable
8. ·  Remaining Test Tasks
9. ·  Environmental Needs
10. ·  Staffing and Training Needs
11. ·  Responsibilities
12. ·  Planning Risks and Contingencies
13. ·  Approvals
14. ·  Glossary