Understanding test automation

Imagine an engineering team adding one or two software engineers every quarter. This team wants to delight the customer by delivering more features every sprint. Even though they have one or two quality engineers on the team to test all of the new features, they notice that the faster they try to deliver code, the higher the number of regression bugs introduced. The manual testing of the features just isn’t enough. They want certain core test scenarios to be executed repetitively and, often, when the changes are introduced. This is where test automation helps tremendously. It is always better to get started with test automation before feeling the agony of a high-severity bug in production or a catastrophic incident happening due to the lack of timely testing.

Testing is definitely not a one-time activity and must be done any time and every time a change is introduced into the software application. The longer we go without testing an application, the higher the chances of failure. So, continuous testing is not an option but an absolute necessity in today’s Agile software engineering landscape. Manual testing involves a tester executing the predefined test cases against the system under test as a real user would. Automated testing involves the same tests to be run by a script, thereby saving the tester’s valuable time so that they can focus on usability or exploratory testing. When done right, automated tests are more reliable than a manual tester. More importantly, it provides more time for the tester to draw valuable insights from the results of the automated tests, which further aids in increasing the test coverage of the software application as a whole.

Test automation is one of the chief ways to set up and achieve quality in an orderly fashion. The core benefit of test automation lies in identifying software bugs and fixing them as early and as close as possible to the developer’s coding environment. This helps subdue the damaging effects of late defect detection and also keeps the feedback cycle going between the engineering and product teams. Even though the upfront investment in automated tests might seem large, analysis has shown that, over time, it pays for itself. Test automation enables teams to deliver new features quickly as well as with superior quality. Figure 1.2 shows how various components of a software application can be interdependent, and continuously testing each of them asserts their behavior. It is not only necessary to validate these components as a standalone but also as an integrated system:

Figure 1.2 – Continuous testing

Imagine a simple loan application flow created by a combination of three Application Programming Interfaces (APIs). The individual APIs, in this case, could be as follows:

1. Creating the applicant
2. Creating a loan application
3. Determining loan eligibility

Each of the APIs has to be tested in an isolated manner at first to validate the accuracy of the business functionality. Later when the APIs are integrated, the business workflow should be tested to confirm the system behavior. Here, a test automation suite can be built to automate both the test cases for individual APIs and the whole system behavior. Also, when the applicant creation API is reused in another application, the automation suite can be reused, thus enabling reusability and portability in testing. A similar implementation can be done for user interface components, too.

Test automation is a very collaborative activity involving the commitment of business analysts, software engineers, and quality engineers/software development engineer in test (SDET). It unburdens the whole team from the overwhelming effects of repetitive manual tests, thus enabling the achievement of quality at speed. While peer code reviews and internal design reviews act as supplemental activities to identify defects early, test automation places the team in a fairly good place to start testing the product with the end users. It is a common misconception that automated tests undercut human interaction with the system under test. While it might be true that the tester does not interact with the system as often as they would in manual testing, the very activity of developing and maintaining the automated tests brings together the whole team by commenting on the test code and design. Automated tests open a new way of communication within the team to improve the quality of the system and the prevention of bugs.

Having looked at the basics of test automation, let’s learn some important considerations in the Agile world.

Agile test automation

Having the right selection of automated tests at the right spots in the deployment makes a ton of difference in the quality of the delivered software. In an Agile environment, there has to be a constant discovery of the right tests for the current iteration, and the ratio between the manual and automated tests needs to be tweaked as and when necessary. Since the focus is on delivering a feature as quickly as possible to the customer, it is important that developers, testers, and the product manager are aware of what is being built, tested, and shipped. Collaboration becomes crucial and is the primary driver of the success of test automation in the Agile environment. Some of the important considerations for test automation in an Agile environment are as follows:

• Start small and build iteratively on the automated test scripts. This applies to both functional test coverage and the complexity of the test automation framework.
• Be extremely cautious about what tests are selected for automation. These tests will act as a gate to production deployment. Make every effort to avoid false positives and false negatives.
• Make sure the automated tests are considered when deciding on the acceptance criteria for a feature or a story. It is critical to allocate necessary time and resources for completing and executing the automated tests as part of a feature.
• Get frequent feedback from other engineers on the quality and performance of the automated tests.
• Do not be afraid to adapt and change test automation. Constant innovation and improvement are core activities in an Agile environment.

As we have seen so far, test automation is an intricate pursuit. Like all other complex things in the software world, test automation comes with its own set of challenges. Let’s examine some of them in the next section.

Test automation challenges

There is never enough test automation, and this is a constant challenge faced by test engineers. Test engineers are always under time constraints to finish the manual or automated tests and to get the completed feature out of the door. Just as with manual testing, there are a variety of challenges that test engineers face on a daily basis. Some of them are listed as follows:

• As mentioned earlier, not having enough test automation is always a challenge in the Agile world. Often, testers have to compromise coverage for speed, and at times, the consequence can be adverse.
• Not enough planning before beginning test automation work often leads to duplication and exhaustion.
• Upfront investment in test automation is heavy, and the constant need to convince the stakeholders of the benefits puts a lot of pressure on testing teams.
• The lack of collaboration between developers and test engineers while designing and developing automated tests could lead to poor-quality scripts or complicated frameworks. This affects the quality of the build pipeline.
• The absence of skilled test engineers has a detrimental effect on quality.
• Not aligning the testing processes with the development processes could cause release delays. It is extremely important to time the code completion of the feature with automated test readiness for on-time delivery.
• Sometimes, test engineers do not understand the requirements and assume the expected behavior instead of validating it with the product. It is extremely hard to identify and eliminate such assertions in the test script after they have been built in.
• The test automation framework is not scalable or portable. This hinders the execution speed and results in a lack of test coverage.

Creating and maintaining the test automation infrastructure is a separate project by itself. Every bit of detail should be thought about and discussed in detail with the concerned stakeholders to address these challenges.

Finding and handling regression bugs

One of the chief purposes of test automation is finding regression bugs. Usually, regression means the quality gets worse after a change has been introduced to an already tested product. Well-written automated tests are tremendously helpful in identifying regression bugs. But an important thing to remember about regression issues is that even though more automated tests help find them, the root cause of regression issues still has to be addressed by the management team. Test engineers can provide awareness about the inherent regression issues through automated test results. It is important to educate the team about fixing the underlying lapses. Some of the most common slippages that cause regression bugs include the following:

• Code review standards are below par or non-existent.
• The project schedule is no longer realistic to push good quality code through.
• There is a huge disconnect between developers and the product teams regarding the feature being developed.
• Various integration points are not being considered appropriately when designing the product.
• The product has become too complex over time.

Test engineers can be a key component to break the pattern of regression issues and steer the quality of the product in the right direction by keeping everyone on the team informed. In the next section, let’s survey some of the top metrics used in the test automation world.

Test automation metrics

Why and what to measure in test automation is a constant question lingering in the minds of the testing team. Test engineers are curious to know how effective their scripts are and how they are performing across different conditions. On the other hand, the management team will be interested to know the ROI on the investment made in test automation rather than just hoping that they deliver value in the long run. Let’s look at some key metrics that can help in gauging the value of test automation that is already in place:

• Test automation effectiveness: This is a key metric that provides visibility into how effective the automated tests are in finding bugs. When broken down logically by scripts/test environments, this metric gives direction on where to focus our future efforts:

Test Automation Effectiveness = (# of defects found by automation/Total number of defects found) * 100

• Test automation coverage: This metric provides the test coverage for the automated tests. It is important to be aware of what is not covered by your test automation suite to clearly direct the manual testing efforts:

Test Automation Coverage = (# of test cases automated/Total number of test cases) * 100

• Test automation stability: This metric provides clarity on how well the automated scripts are performing in the long run. This metric, along with test automation effectiveness, is a key indicator of how flaky the automated tests are. It is good to have this metric in an Agile environment to monitor the health of the deployment pipelines:

Test Automation Stability = (# of test failures/Total number of test runs) * 100

• Test automation success or failure rate: This metric provides a quick idea of the health of the current build. In the long term, this metric gives you a good view of how the build qualities have changed:

Test Automation Success Rate = (# of test cases passed/Total number of test cases run) * 100

• Equivalent manual test effort: This metric, usually expressed in man hours, is an indicator of the ROI to determine whether a particular feature can be automated or not:

Equivalent Manual test Effort = Effort to test a specific feature * Number of times a feature is tested in a test cycle

• Test automation execution time: As the name indicates, this metric shows you how long or short your test runs are. Comparing this with the overall deployment times to production gives a good idea of how much time is spent executing the automated tests over time. This is key in the Agile setup where the speed to the customer is considered a key factor:

Test Automation Execution Time = Test Automation End Time - Test Automation Start Time

An important note regarding test automation metrics is that these numbers, by themselves, do not produce much value. A combination of these metrics properly assessed in the context of project delivery provides valuable insights to the product and engineering teams.

So far, we have seen how testing and test automation blend well with every activity in the software development life cycle. It is evident that test automation is a multidimensional undertaking, and the people performing test automation have to meet the needs of this demanding role. In the next section, let’s survey a couple of important roles in the quality engineering space.