Getting familiar with software testing

Testing as an activity is critical to delivering a trustworthy product and forms a strong foundation for building reliable test automation. So, being a good tester makes you a more effective test automation engineer.

Software testing is an indispensable task in any software development project that is mainly done with the goals of validating the specified product requirements and finding bugs. These bugs can be functional or non-functional in nature. Functional bugs include deviations from the specified requirements or product specifications. Usually, non-functional issues are performance-based or security-based. The primary goals of testing are usually interwoven at multiple levels but can be broken down at a high level as follows:

• Functional: Checking the business functionalities of the software system:
  • Compliance: Regulator agencies, government agencies, and more
  • Portability: Cross-browser testing, mobile support, and more
  • Usability: Support for disabled users
  • Maintainability: Vendor support
• Non-functional: Checking the non-functional aspects of the software system, which are never tackled by functional testing:
  • Reliability: Operational up time, failovers, business continuity, and more
  • Security: Vulnerability, penetration testing, and more
  • Performance: Load, stress testing, and more

Now that we have seen a quick introduction to testing, let us understand why testing is so critical.

Knowing the importance of testing

Even though software’s quality is initially laid out by the design and architecture, testing is the core activity that gives stakeholders much-needed confidence in the product. By verifying the behavior of the product against documented test cases, the activity of testing helps uncover bugs and address other design issues promptly. By preventing and identifying bugs early in the software development life cycle, testing helps both the engineering and business teams to increase customer satisfaction and reduce overall operating costs. The valuable insights that the team derives from the repeated testing of the product can be further used to improve the efficiency of the software development process. Therefore, it is imperative to conduct testing for achieving the established goals of a successful product launch.

Tasks involved in testing

Testing is not just executing the documented test cases. There are a multitude of tasks involved in testing, as follows:

• Discussion with product and business about the acceptance criteria
• Creation of a test plan and strategy
• Review of the test plan with the engineering and product teams
• Cross-team collaboration for the test environment setup
• Creation of test cases and test data
• Execution of test cases
• Reporting of test results and associated quality metrics

Depending on the team size, capacity, and structure, some or all of these activities must be performed for a successful release of a software product.

In the following diagram, we get a comprehensive view of all the deliverables involved in software testing:

Figure 1.1 – Testing deliverables

As you can see in the preceding diagram, testing encompasses a wide variety of deliverables resulting from many cross-functional activities. Next, let’s look at some unique demands for testing in the world.

Testing in an world

Unlike a traditional waterfall model, in the Agile world, it is recommended that each user story or feature has the right balance of manual and automated tests before calling it done. This arguably slows down the team, but it also drastically reduces the technical debt and improves code quality. You will notice that scaling the software becomes easy, and there is a significant decrease in reworking as the automated tests increase. This saves huge amounts of time for the engineering team in the long run. Another common occurrence in an Agile setup is developers writing automated tests. Usually, developers own unit tests, and quality engineers write other types of tests. Sometimes, it is productive for the developers to write automated test code while the quality engineers focus on other testing tasks. One of the most important takeaways from the Agile world is that quality is everyone’s responsibility on the team. Test engineers keep the communication and feedback going constantly between the product and the software engineers, which, in turn, results in an excellent team culture. This also opens up early discussions on the necessary tools and infrastructure needed for testing.

Since the Agile environment aims to deliver the simplest possible product to the customer as quickly as possible, the test engineers focus on the most necessary set of test cases for that product to be functionally correct. In each increment, the test engineers create and execute a basic set of test cases for the features being delivered to the customer. Test engineers are constantly working with the product, developers, and in some cases, with customers to keep the stories as simple and concise as feasible. Additionally, the Agile landscape opens doors to automating the entire software development life cycle using the principles of continuous integration, which demands a major shift in the test engineer’s mindset. This demands excellent communication skills and fluent collaboration from the test engineers.

Often, a test engineer’s output is measured by the number and quality of defects they find in the software application. Let us examine some key aspects of defect management next.

Defect management in testing

A defect is primarily a deviation from the business requirement, and it does not necessarily mean there is an error in the code. The analyzing, documenting, and reporting of defects in the product is a core activity for the testing team. It is essential to set and follow standard templates for reporting defects. There are a variety of tools in the market that help with test case and defect management. Maintaining a good rapport with all the engineers on the team goes a long way in making the defect reporting and resolving process much smoother. Usually, testers have to put in as much information as possible in the defect log to assist developers in reproducing the defects in various environments as needed. This also helps in categorizing the defects correctly. There are cases when the defect still cannot be reproduced consistently, and the engineer and tester have to pair up in order to spot the failure in the application precisely. It is important to file high-quality functional defects and follow up when necessary to get them fixed on time. So, the testing team is primarily responsible for the defect management process and keeping the higher management updated on any high-severity defects that might affect the product release.

Defect versus bug

A defect is a deviation from the expected behavior. It can be a missing, incorrect, or extra implementation. In comparison, a bug is a programming error that causes the software to crash, produce incorrect results, or perform badly.

Besides the number of bugs being identified, it is also crucial to know when a defect is identified in the project. Next, let’s explore the effects of shifting testing early or late in the development life cycle.

Shift-Right and Shift-Left propositions

It is evident that there are a lot of benefits that can be reaped by shifting the testing and quality efforts, in general, earlier on in the development life cycle. This is termed the Shift-Left approach. In this approach, quality engineers are involved right from the inception of the project in early discussions about the design and architecture. They start working on deliverables such as the test plan and test cases in parallel with developers. This approach builds a quality-first mindset for the entire team and ensures quality is built right into the vein of every activity in the development life cycle.

The Shift-Right approach is an extension of Shift-Left, where the testing team’s responsibilities are stretched further into the release of the product and maintenance. Test engineers, with their deep knowledge of the product, assist with the implementation and help with testing and monitoring in production. Additionally, the Shift-Right approach has some test engineers support performance, load, and security testing. There is also good scope for test automation to address quality issues under real-world conditions within this approach. Both these approaches are equally important. While the push for Shift-Left has been happening for quite some time now, the Shift-Right approach is relatively new and is extending the tester’s involvement in obtaining and testing actionable production data in safer ways.

This leads us to the question of how to incorporate quality early on, and DevOps processes have helped enormously in that regard. Next, let’s look at how quality and DevOps blend together to deliver organizational value.

Quality and DevOps

The previous section about the Shift-Right approach leads us right into the area of DevOps and how it has accelerated product development and changed the quality process. DevOps strives continuously to align high-performing engineering teams to business value in the most efficient way. Quality is a core component in each of the DevOps processes. DevOps attempts to automate each and every task in the product delivery, right from building the code to deploying the application to production for customers to use. This adds further emphasis on quality. Since the whole process is automated, feedback at every checkpoint is crucial. A predetermined set of units, functional, and integration tests executed at the right times in the deployment pipeline act as a gate to the production deployment. At times, the manual involvement of test engineers will be required for debugging test failures and for specific types of testing.

In the DevOps world, it is essential to maintain nimbleness in testing activities and deploy the right type of testing resources when and where needed. We will dive deep into how test automation helps in continuous integration and continuous delivery in Chapter 9, CI, CD, and Test Automation. For now, it is good to grasp the significant role that testing plays in ensuring a successful DevOps implementation.

So far, we have seen various aspects of testing, and it is undoubtedly a demanding activity. Let’s examine some challenges that test engineers face on a daily basis.

Challenges in testing

Before diving into the world of test automation, it is vital to understand the common challenges faced in the testing world. Building a solid manual testing foundation is paramount to developing a sound test automation strategy. A clear and concise testing process acts as a strong pillar for building reliable automation scripts. Some of the challenges faced regularly by the testing team are outlined as follows:

• The most common challenge faced by Agile teams is the changing requirements. Various reasons such as late feedback from customers or design challenges lead to a change in the requirements and cause major rework for test engineers.
• Test engineers are required to interface constantly with the various teams to get the job done. The lack of proper communication here could lead to major blockers for the project.
• Having stories or requirements with incomplete information is another area that test engineers struggle to cope with. They constantly have to reach out to products for additional information, which causes a lot of back and forth.
• Not having the right technical skills for performing all kinds of testing required for a feature poses a great challenge for test engineers.
• The lack of quality-related metrics hurts fast-growing teams where the velocity is expected to increase with the team size. The engineering team will be imperceptive to any patterns in code issues.
• Inadequate test environments are a major bottleneck to testing efforts and directly affect the confidence in the product delivery. The lack of diverse test data in the test environment leads to an inadequately tested product.
• The absence of standard test processes increases both the development and testing times, thereby delaying the project timelines. It is good to be cautious about partially implemented test processes as they, sometimes, hurt more than they help.

Next, let’s look at how testing early and often helps overcome some of the obstacles in the testing journey.

Test early, test often

Usually, an enterprise software application contains hundreds, if not thousands, of components. Software testing has to ensure the reliability, accuracy, and availability of each of these components. So, it is inevitable to emphasize the importance of the common quality industry term, Test early and test often:

Table 1.1 – Importance of testing early and testing often

The following excerpt highlights the cost of fixing bugs later in the product development cycle. As Kent Beck explains in his book, Extreme Programming Explained, “Most defects end up costing more than it would have cost to prevent them. Defects are expensive when they occur, both the direct costs of fixing the defects and the indirect costs because of damaged relationships, lost business, and lost development time.”

The common arguments for not testing early are “We don’t have the resources” or “We don’t have enough time.” Both of these are results of not examining the risks of the project thoroughly. It is a proven fact that the total amount of effort is greater when testing is introduced later on in the project. On the tester’s part, it is important to reduce redundancy and constantly think about increasing the efficiency of their tests. Test-Driven-Design (TDD) is a common approach to practice Test Early, Test Often. Testing processes have to be fine-tuned and adhered to strictly for this approach to be successful. Testing can have a strategic impact on the quality of the product if introduced early and done often and efficiently. The Agile test pyramid (which is discussed, in detail, in Chapter 2) can act as a guide to strategically categorize and set up different types of tests.

So far, we have dealt with a range of concepts that help us acquire a well-rounded knowledge of testing. We looked at the importance of testing and how crucial it is to integrate it with every facet of the software development process. With this background, let us take on the primary topic of this book, test automation.