The Process of Software Testing in Practice

Organizing a software testing process can be quite challenging. We are following the three major steps in the software testing process: planning, execution, and reporting.

Test Planning: Artifacts and Strategy

As any other formal process, testing activities are typically preceded by thorough preparations and planning. The main goal of this stage is to make sure the team understands the customer objectives, the main purpose of the product, the possible risks they need to address, and the outcomes they expect to achieve. One of the documents created at this stage, the mission or assignment of testing, serves to solve this task. It helps align the testing activities with the overall purpose of the product and coordinates the testing effort with the rest of the team’s work.

Roger S. Pressman, a professional software engineer, famous author, and consultant, states: “Strategy for software testing provides a roadmap that describes the steps to be conducted as part of testing, when these steps are planned and then undertaken, and how much effort, time, and resources will be required.”

Also referred to as test approach or architecture, test strategy is another artifact of the planning stage. James Bach, a testing guru who created the Rapid Software Testing course, identifies the purpose of a test strategy as “to clarify the major tasks and challenges of the test project.” A good test strategy, in his opinion, is product specific, practical and justified.

Depending on when exactly in the process they are used, the strategies can be classified as preventive or reactive. In addition to that, there are several types of strategies, that can be used separately or in conjunction:

While a test strategy is a high-level document, test plan has a more hands-on approach, describing in detail what to test, how to test, when to test and who will do the test. Unlike the static strategy document, that refers to a project as a whole, test plan covers every testing phase separately and is frequently updated by the project manager throughout the process.

According to the IEEE standard for software test documentation, a test plan document should contain the following information:

• Test plan identifier
• Introduction
• References (list of related documents)
• Test items (the product and its versions)
• Features to be tested
• Features not to be tested
• Item pass or fail criteria
• Test approach (testing levels, types, techniques)
• Suspension criteria
• Deliverables (Test Plan (this document itself), Test Cases, Test Scripts, Defect/Enhancement Logs, Test Reports)
• Test environment (hardware, software, tools)
• Estimates
• Schedule
• Staffing and training needs
• Responsibilities
• Risks
• Assumptions and Dependencies
• Approvals

Writing a plan, which includes all of the listed information, is a time-consuming task. In agile methodologies, with their focus on the product instead of documents, such a waste of time seems insufficient.

To solve this problem, James Whittaker, a Technical Evangelist at Microsoft and former Engineering Director at Google, introduced the 10 Minute Test Plan approach. The main idea behind the concept is to focus on the essentials first, cutting all the fluff by using simple lists and tables instead of large paragraphs of detailed descriptions. While the 10-minute timebox seems a little bit unrealistic (None of the teams in the original experiment was able to meet this requirement), the idea of reducing and limiting the planning time itself is highly reasonable. As a result, 80 percent of the planning can be finished within only 30 minutes.

Design and Execution

As a starting point for the test execution, we need to define what is subject to testing. In order to answer this question, QA teams develop test cases. In a nutshell, a test case describes the preconditions, desired outcomes, and postconditions of a specific test scenario, aimed at verifying that a feature meets the basic requirements.

The next step in test execution is setting up the testing environment. The main criteria for this part are to make sure that the testing environment is as close to the end user’s actual environment (hardware and software) as possible. For example, a typical test environment for a web application should include Web Server, database, OS, and browser.

The software testing process identifies two broad categories: static testing and dynamic testing.

Static testing initially examines the source code and software project documents to catch and prevent defects early in the software testing life cycle. Also called non-execution technique or verification testing, static testing could be performed as inspections, informal and technical reviews, or reviews during walkthrough meetings. Informal review is a cheap testing variant that a QA analyst can conduct anytime during the project. Inspection, also called a formal review, is planned and controlled by the moderator. During the review meeting, errors found by QA analysts are discussed and documented in the review report.

As soon as the primary preparations are finished, the team proceeds with dynamic testing where software is tested during execution. This whitepaper has the most focus on the dynamic testing process as a practical and most commonly used way to validate code behavior. Dynamic testing can be described by methods, levels, and types of underlying QA activities. Let’s have a closer look at this segment of the dynamic testing process.

Software testing methods are the ways the tests are conducted. They include black box testing, white box testing, grey box testing, and ad hoc testing.

Software testing levels describe stages of software development when testing is conducted. That said, there are four progressive testing levels based on the area they focus on the software development process: unit testing, integration testing, system testing, and user acceptance testing (UAT).

Software testing types are the approaches and techniques that are applied at a given level using an appropriate method to address the test requirements in the most efficient manner. They are vast in number while serving different objectives.

To sum up, you can do use case testing (a type) during system or acceptance testing (a level) using black box testing (a method).

Documentation and Reporting

As there is no perfect software, the testing is never 100 percent complete. It is an ongoing process. However, there exist the so-called “exit criteria”, which define whether there was “enough testing” conducted, based on the risk assessment of the project.

There are common points that are present mostly in exit criteria:

• Test case execution is 100 percent complete.
• A system has no high priority defects.
• Performance of the system is stable regardless of the introduction of new features.
• The software supports all necessary platforms and/or browsers
• User acceptance testing is completed.

As soon as all of these criteria (or any custom criteria that you have set in your project) are met, the testing comes to its closure.

The testing logs and status reports are documented throughout the process of the test execution. Every issue found in the product should be reported and handled accordingly. The test summary and test closure reports are prepared and provided to the stakeholders. The team holds a retrospective meeting in order to define and document the issues that occurred during the development and improve the process.

According to the survey conducted by Practices, an end-to-end QA and test management solution, there is a constant decrease in the amount of formal testing documentation written. This tendency signals the need to streamline testing all across the industry.