The best guide to validation testing

We understand that validation in software testing is critical, and we want to make the process easier for you and your team. Here, we’ll be answering questions about validation testing and provide insights into why it is an essential component of software development.

Whether you are new to validation testing or want to ensure you are on the right track, this section will cover the basics. Later on, we will delve into advanced tips, techniques, strategies, and case studies.

What is validation testing?

Validation testing is the practice of ensuring that software meets the quality standards set by the customer and that the product meets customer requirements. It is one of many types of testing in software.

In certain cases, validation testing is also sometimes called acceptance testing because it is usually performed once the developer has finished writing the code. In this sense, it is a process of checking whether a product can be accepted based on certain criteria or requirements that are defined in advance by the consumer or user.

Businesses often perform validation testing as part of their software development life cycle (SDLC). Both waterfall and agile models require code review to ensure that it works as intended and does not contain defects, bugs, or errors.

Validation testing guarantees that the product is ready for release and meets customer expectations, so it is important to get it right. But how do you know when you’re done? What does validation testing mean? How does this differ from verification testing? And, most importantly, how do you do it?

Let’s break down each element.

What is software testing & why is it important? 

Typically, testing is classified into three categories: functional testing, non-functional testing (such as performance testing), and maintenance (regression and maintenance).

Validation is part of a larger suite of tests known as non-functional testing, which also includes performance and usability testing. Validation testing typically occurs during the final stages of development and prior to releasing a new product or service to customers.

Validation testing can occur alongside other types of testing, such as verification testing, unit testing, and integration testing. Together, these various types of testing seek to ensure that a product is as high-quality and usable as possible while maintaining security and performance standards.

In addition to verifying that a product meets defined requirements, validation also includes ensuring that any appropriate documentation for using the software exists and is both clear and accurate. This includes instructions on installation and configuration, usability guidelines, user support information, and more. Validation should be carried out by users who represent each type of user for the software application. For example, people with different levels of technical ability might test an interface to ensure it is easy enough for a novice user to understand but not so simple that an experienced user finds it boring or repetitive.

What are the different stages of validation testing?

By now, you may be wondering, “How do you perform a validation test?”

In this section, we will guide you through the different stages of development testing. These high-level steps will help you plan your validation testing activities and can serve as a guideline for developing your acceptance testing process. With a well-defined process and strategy, you can save your company or client time and increase their return on investment (ROI).

If you have ever struggled with the validation testing process, this section will be a useful resource.

Stages of validation testing process

Typically, when you conduct validation tests on your software, these are the steps involved:

• Here are the steps typically involved in conducting validation tests on your software:
• Validation planning – This is a project-specific plan that defines the scope and goals of your validation project. Think of this as high-level planning for your product’s validation.
• Defining requirements – Whether you’re just starting the software development process or you’re making changes to an existing product, you’ll need to establish a set of requirements to work toward. These requirements stipulate what features your product should include and how those features should work. This defines the test criteria, which is essential for verification and validation.
• Selecting a team – You’ll need to field an experienced and competent development team (client included).
• Developing documents – Develop a user specification document where you describe the operating conditions.
• Estimation/Evaluation – Next, proceed with the software testing stage, verify that it operates as originally intended, and submit a validation report.Fixing Bugs or
• Incorporating changes – Finally, update the software to remove any bugs or issues found during the evaluation.The validation process may also include functionality testing:
• Unit testing – Black box, White box, and/or Gray box to search for bugs in the product segment.
• Integration testing – Also known as ‘I & T’ (Integration and Testing), ‘String Testing,’ and ‘Thread Testing.’ This is where you test the interfaces and flow of data/information between the modules. Strategies to execute include:
• Big Bang approach
• Incremental approach
• Top-Down approach
• Bottom-Up approach
• Sandwich approach – Combination of Top-Down and Bottom-Up
• System testing – This consists of a series of different tests to evaluate the end-to-end system specifications. May include:
• Sanity Testing – A subset of Regression Testing. This is done to check that the new functionalities/bugs have been fixed in a particular component.
• Smoke Testing – A subset of Acceptance Testing. This is done to ensure the critical functionalities of the entire system from end to end are working as expected.
• Regression Testing
• Performance qualification: UAT (User Acceptance testing) – Alpha and Beta testing.
• Production: This is the part of the software development practice where the software is put into production.

Why Is validation testing important?

In 2020, the global software development market size was valued at $389.86 billion. Experts project that it will exceed $824,84 billion by 2026. It appears that Marc Andreessen, co-founder and general partner of venture capital firm Andreessen Horowitz, was not exaggerating when he said that software is eating the world.

In fact, Herb Krasner —  Consortium for Information & Software Quality™ (CISQ™) Advisory Board Member, retired Professor of Software Engineering at the University of Texas at Austin, and author of The Cost of Poor Software Quality in the U.S.: A 2020 Report — says:

“While software is eating the world more voraciously than ever before, the cost of poor software quality is rising, and mostly still hidden. Organizations spend way too much unquantified time finding and fixing defects in new software and dealing with legacy software that cannot be easily evolved and modified.”

With that, let’s quickly look at some key testing and quality statistics:

• Experts predict by 2026, the global test automation market size will boom to almost $50 billion
• 67% of consumers cite bad experiences as a reason for churn, although only 4% of customers will complain directly to a company about a poor experience
• When Capgemini and Micro Focus surveyed 1,750 senior executives in corporate IT management functions across 32 countries about their top priorities, 63% said enhancing customer experience, 62% said enhancing security, 61% said higher responsiveness to business demands, and 61% said the higher quality of software solutions

In today’s market, consumers have access to a vast array of software. Therefore, it is essential to use software testing to ensure that the final product is of high quality.

Validation testing is a critical part of the software development lifecycle

Because undetected defects and bugs can disappoint and frustrate users, expose your company to data breaches and ransomware attacks, and even trigger critical system outages, it is important to ensure high quality control and testing practices in almost every domain of software development.

Over the last two to three decades, embedded software has become increasingly prevalent in products that were traditionally created using mechanical and electrical disciplines, from cell phones to home appliances, to even satellites. Software has permeated virtually every area of our lives, and in many cases, these products rely heavily on the underlying software to continue operating. There is almost nothing nowadays that doesn’t have some kind of software inside.

Poor quality and failures in embedded software can occasionally lead to death or serious injury, resulting in serious liability issues for devices such as passenger vehicles and medical instruments. Therefore, the highest quality control and testing practices are necessary in almost every domain of software development.

In 2006, NASA’s annual report on independent verification and validation showed that for large software projects the percentage of software rework can approach 50%.

One way to help improve quality and reduce costs is by conducting rigorous testing of your product at various stages of its development. Lowering costs is important because the cost of finding and fixing bugs and defects is the largest single cost driver in the software industry.

In other words, validation testing is important because it helps improve quality and reduce costs. Here are a few more reasons why validation testing is crucial:

• It ensures customer satisfaction
• It allows businesses to be confident about their product quality
• Allows teams to fulfill their client’s requirements efficiently
• Allows for easier software adoption by the end-user

Software testing is important because any software can contain bugs that not only cause monetary loss, but are potentially dangerous to its users. History is full of examples:

• In April 2015, a software glitch caused a Bloomberg terminal in London to crash. This affected more than 300,000 traders in the financial markets. It also forced the government to postpone a 3bn pound debt sale.
• Nissan recalled more than 1 million cars because of a software failure in the airbag sensory detectors. Two reported accidents were attributed to the software failure.
• In response to a software failure in its POS system, Starbucks closed almost 60% of its stores in the U.S and Canada. At one point during the issue, the company served coffee for free because they were unable to process transactions.
• Some of Amazon’s third-party retailers incurred heavy losses when their product prices mysteriously dropped to 1p because of a software glitch.
• In 2015 F-35 fighter planes experienced a software bug that left them unable to correctly detect targets.
• In 1985, Therac-25 radiation therapy machines malfunctioned because of a software bug and delivered lethal radiation doses to patients, causing the deaths of 3 people and critically injuring 3 other people.
• In 1994, a software bug caused the crash of China Airlines Airbus A300, killing 264 people.
• In 1999, a $1.2 billion military satellite launch failed because of a software bug. This is the most expensive accident recorded.
• In 1996, 823 customers of a major U.S. bank were credited with 920 million dollars because of a software bug.

These are only some of the reasons acceptance testing is important to end-users and society as a whole, but what about product developers? What role does validation testing play when it comes to software quality?

Why is validation testing important to product development?

The #1 reason why validation testing is so crucial in product development is because it ensures that the design meets the right set of requirements at each stage. By defining clear exit criteria and deliverables for each phase, you can optimize the use of resources and improve the quality of the product. As a general guideline, the further along the product is in development, the more expensive it is to make changes. For complex products, production investments can run into the millions, with the most ambitious products requiring thousands of workers and billions in fixed production costs. As the development process progresses towards production, the cost of making changes increases exponentially.

Therefore, a fundamental goal of any product development initiative is to identify and fix issues early, reducing the risk of costly modifications at later stages. This is where validation testing comes in. By performing acceptance testing on the product, you can catch any bugs that may have made it through previous phases of the development cycle undetected. This allows you to fix them before production, which ultimately provides your customers with a better experience.

This makes sense, especially since 81% of companies compete on customer experience alone. Validation testing is more than just a testing framework—it’s a customer experience framework. Understanding how to plan, execute, and optimize your validation testing strategy is key to testing from your user’s perspective.

“By applying insights from testing and validation, companies can reduce rework and improve production ramp-up and thereby accelerate time to market. They can also enhance their abilities to integrate features that target specific customer needs early in the design process and improve customers’ perception of quality. The result is greater customer satisfaction and higher market share. High quality is directly linked to increased revenue from repurchases and upgrades, new business gained by word of mouth, and lower costs to serve customers.” – McKinsey and Company

Testing and validation: From hardware focus to full virtualization?

High-quality products lead to happy customers. Effective validation testing can not only help increase product quality but also give your company a competitive advantage and deliver cost savings.

According to McKinsey data, testing and validation processes are an essential part of all steps of the system engineering V-model, which is a framework for testing and validation (illustrated below). Typically, testing and validation account for 20 to 30 percent of development costs.

It is easy to see how software testing is important. A properly tested software product assures reliability, security, and high performance, resulting in saving time, cost-efficiency, and increased customer satisfaction.

On the other hand, poor quality software can result in security and safety issues, loss of revenue and productivity, reputation damage, and increased development costs.

The Consortium for Information & Software Quality™ (CISQ™) defines poor-quality software as “products that don’t provide value to users, fail to meet profit goals, produce serious complaints and problems and fail to contribute in some way to the goals of humanity.”

Testing throughout the development process is one way to increase the quality of the software being developed. Involving testers at the first stages of development has a range of advantages, including helping the team understand customers’ goals, saving time, minimizing expenses, and optimizing the approach to testing.

Therefore, quality becomes increasingly important because traditionally, the costs of finding and fixing bugs have been the largest expense when developing, enhancing, and maintaining large software applications.

Note that software testing is one of the biggest levers to deliver a bug-free product. ITT Corp.’s first Vice President of Quality, Phil Crosby, became famous for the aphorism “quality is free.” He has a good point since software quality is not only free but leads to shorter development schedules, lower development costs, and greatly reduced costs for maintenance and total cost of ownership (TCO).

In their 2012 book, The Economics of Software Quality, Capers Jones and Olivier Bonsignour note,

“From an analysis of about 13,000 software projects between 1973 and today, it is gratifying to observe that high-quality levels are invariably associated with shorter-than-average development schedules and lower-than-average development costs.” Jones and Bonsignour go on to say, “The real economic value of high quality is only partially related to defect repair costs. It is true that high quality leads to fewer defects and, therefore, to lower defect repair costs. But its major economic benefits are due to the fact that high quality:

• Reduces the odds of large-system cancellations
• Reduces the odds of litigation for outsourced projects
• Shortens development schedules
• Lowers development costs
• Lowers maintenance costs
• Reduces warranty costs
• Increases customer satisfaction.”

Validation Testing is an important part of product development since it helps validate if all requirements (functional and nonfunctional) are actually working. Validating these different requirements allows you to verify that the product being developed correctly follows the established guidelines, stakeholder expectations, and quality controls previously specified. On the other hand, Validation Testing also helps make sure that client needs are being taken into account.

What’s the difference between verification and validation in testing?

Now it’s time for the fun stuff: What’s the difference between validation and verification testing in software engineering?

This is a question that is more common than you may think.

Validation and verification testing are important concepts. If you’re a software manager, executive, or quality assurance team member involved in planning, estimating, executing, and maintaining software, you need to understand the difference between verification testing and validation testing.

Why?

Well, it’s no secret that there’s often a lot of confusion around the difference between verification and validation in software testing. Some people think verification and validation are synonymous and use the terms interchangeably. Some claim that they are completely different.

But what’s the truth?

The truth is these two words have distinct meanings and convey different ideas. And product developers need to grasp the importance and difference of these essential tests.

Verification testing ensures that a product meets requirements from an engineering perspective, while validation testing ensures that it meets user needs. Validation testing follows software verification, which takes place earlier in the software development process.

In some cases, these two measures can be combined into one test cycle called “verification and validation” (V&V).

The distinction between validation and verification is a fine one. The two are used interchangeably in some cases, but they do have differences — namely, when they occur during the development process. Software verification occurs first, before the software is released for testing. Validation happens after developers write code and then test it to make sure the product does what it’s intended to do.

For example, part of the verification process includes checking documents, design, code and programs, unlike the validation process which includes testing and validation of the actual product.

Some other differences between verification and validation in software testing include:

• Verification determines whether the software conforms to a specification while validation determines whether the software meets requirements and expectations.
• Verification uses methods like reviews, walkthroughs, inspections and desk-checking while validation uses methods like black-box testing, white-box testing and non-functional testing.
• The goal of verification is to find bugs early in the development cycle while validation is designed to find bugs that verification did not catch.

When comparing validation to verification in software testing, it’s important to remember that the verification process targets software architecture, design, and database, among other things, while the validation process targets the entire software product. One difference between verification and validation in software testing is that the QA team handles verification, while the testing team handles validation with the assistance of the QA team.

Verification comes before validation, and verification does not involve code execution, while validation involves code execution. In software testing, validation and verification are two different approaches to assessing product quality. Verification answers the question, “Do we build it right?” while validation answers the question, “Do we build the right thing?”.

What are the different types of validation testing?

There are several different kinds of validation testing, including:

• Black box/functional validation — Checking how functions perform
• White box/structural validation — Checking code structure through code analysis and automated unit testing
• System validation — Checking how the app works in a system environment
• Smoke testing — Smoke testing is a quick-and-dirty test that the major functions of a piece of software work. It is performed on untested software, either new builds or major changes to existing builds, as a way to quickly reject obviously faulty software.
• Unit testing — Unit testing is the testing of individual units or components of software to determine if they are fit for use. A unit is the smallest testable component of any software. It normally has one or a few inputs and typically a single output.
• Beta/User acceptance testing (UAT) — This is performed by the end-user to ensure that the system meets their business requirements. Business requirements define what an application should do but not how it should be done.
• Operational acceptance testing (OAT) — The OAT test phase is performed at the end of the development project, but before the software is moved into production. This testing is usually performed by the customer and/or a third party on the production-like system.
• Regression testing — Regression validation testing is done to ensure that new changes haven’t introduced any defects in older parts of the software or affected existing functionality or features.

Validation testing types form a V-shaped testing pattern. This pattern includes different variations and testing activities:

• Unit testing – This is a type of validation testing. The goal of unit testing is to find bugs in the product. At the same time, unit testing also significantly improves code quality and provides documentation.
• Integration testing -This is an important part of the validation model where the interaction between the various interfaces of the software are tested. After your developers have performed unit tests, integration testing will ensure that internal modules and components communicate as expected.
• System testing – System testing is done once the entire programming framework is prepared. This is a series of different tests whose chief purpose is to test the entire system. Considered as the black box testing category of software testing, system tests evaluate the end-to-end system specifications of the integrated software product.
• User acceptance testing – Also called beta testing or end-user testing, UAT is where your software is tested by the intended audience or business representative. This is one of the last phases of testing and involves using real-world scenarios to ensure the software is operating according to specifications.

There is an almost endless list of ways to test software for quality. If you are just starting, you should focus on tactics that reduce the number and severity of software bugs, such as prioritizing your efforts on basic and crucial functions. You should think like customers or end-users and keep in mind that no matter how much testing you perform, you cannot guarantee a 100% bug-free application.

We all know that the development world changes quickly. However, if everyone on the team, including Dev, QA, BA, PM, etc., is committed to delivering a high-quality product, embracing testing to achieve product quality and reduce waste should be a shared goal. Let us take a look at some tools and techniques that everyone, from newbies to experienced testing professionals, can use to run validation tests.

What are some techniques and tools used in validation testing?

When you’re ready to perform acceptance testing, you’ll find that there are many tools available for validation. This is the part of your product development where you finally break out your software testing tools. You’ll need to decide when to run automated tests and when to test manually, and select the right tool for the job.

Different techniques and tools can be used for validation testing, which can be either manual or automated and involve different parts of the application or platform. The strategy chosen will depend on business requirements, client specifications, resources, and time availability.

For BackEnd validation, we can implement a Unit Test (an automated test that allows us to check one code’s unit) or Integration Tests (automated tests that check the integration of different methods). The tools utilized will depend on the programming languages that were implemented on the BackEnd. For example, for JavaScript, we can run Jest or Mocha.

We can also test more complex flows, such as e2e, that simulate a real interaction between the BackEnd, while manually testing with Postman or other similar platforms, or automating these interactions with different frameworks such as restAssure. Platforms such as the latter allow us to automate and simulate the requests made by the FrontEnd while automating tests with real data.

Thirdly, we can carry out Validation Testing on the FrontEnd of an application, and, as with BackEnd testing, these tests can be either manual or automated. These tests simulate the real interaction users will have with a system. To automate these tests, there’s a wide variety of frameworks available, and choosing one will depend on business requirements, type of tests that will be performed, resources, and time availability.

If we want to automate mobile tests, different frameworks will be implemented compared to those tests carried out with a browser. Some of the frameworks that we most frequently work with for in-browser testing are Selenium, Playwright, and Nightwatch. On the other hand, to carry out mobile tests, we use Appium, which allows us to test on both Android and iOS.

Validation testing can involve functional and non-functional requirements, which is why we often have to verify our systems’ response times or the number of concurrent users to verify that we are meeting technical requirements. This method is called performance testing, and we generally use Jmeter to implement these tests.

You can also use tools like Cucumber, Appium, Selenium, and HP Quality Center to perform validation tests and store the test results.

Here is an example of a validation tool:

So, maybe you’re ready to perform your own validation tests and wondering “What are the phases of the validation testing process?”

Here’s the answer in a nutshell:

• Define Requirements– During this phase, you’ll be planning the entire process beforehand and establishing the exact requirements that are needed.
• Team Selection– You’ll need to put together a competent, experienced and talented team. The team selection process is where you choose the individuals who will comprise your testing team. They should all have knowledge of what the product should be doing.
• Maintaining Documentation– Any form of testing requires extensive user specification documentation not to mention release cases, test cases, and manuals that have to be compiled. Decide who will be responsible for your project’s validation docs.
• Validation Report– The software is evaluated based on user specifications and a validation report is submitted. Then, your team will cross-check the evaluations and set a target date for the bug removal. The Final Report or Validation Report (VR) usually signals the end of the validation process. Upon approval of the final report, the software system will be released into production.
• Incorporation of changes– Incorporate any necessary changes that were validated in the last stage.

The validation test can be carried out in different ways, depending on the requirements. That said, here’s an example of how it’s done:

• Decide on the business requirements for validation testing with input from stakeholders including the end-user.
• Prepare the validation plan and get it approved by the stakeholders involved.
• Once the validation plan is approved, your team will write the necessary test cases and send them for approval.
• Once the test cases are approved you start testing with the required software, environment and requirements as requested by the client.
• Upon approval of the deliverables, UAT testing is done by the client.
• After that, the software goes into production.

So, how do we run tests at Qubika? 

The testing strategy depends on various factors, such as the client’s requirements, time availability, and resources. Sketching a testing strategy for a long-term project is different from doing so for an MVP that needs to go into production in a short period.

The goal is to implement quality practices from the start of a project to detect important issues early on, allowing us to solve them quickly without incurring extraordinary costs.

We start by analyzing the design and understanding the business requirements and client needs. As the functionalities are implemented, we begin performing manual tests. As soon as the features don’t contain any fixable mistakes, we can start automating them to increase coverage of the system in less time.

To execute manual testing, we decide between traditional tests (in which case we implement all cases) and Agile Tests that implement exploratory (manual) tests. In the case of the former, we analyze all requirements and create a guide to aid the tester.

As tests are executed, we report any bugs or errors to the team to establish their priority and create a strategy to solve them.

At Qubika, we encourage the implementation of Agile Testing as a process. This way, we can automate everything that is automatable and complement those aspects that cannot be automated with manual (exploratory) testing. This strategy allows us to implement Agile sprints in products that need to go into production in a tight timeframe.

With this strategy, we can have test cases that can be executed automatically when a new functionality is created or according to the execution strategy that the client decides to implement. Automated tests are integrated with CI platforms such as Jenkins and run in the cloud without the need for a particular resource.

These tests are complemented with exploratory testing, which performs tests that require human intelligence and cannot be automated. With this combination, the test coverage increases considerably, reducing the probability of finding new issues.

We are still growing, learning, and improving in certain areas, especially with projects that need to go into production very rapidly, which doesn’t allow us to thoroughly carry out validation testing. This is why having enough automated tests helps us to have a complete understanding and regression of the system in case we don’t have enough time to perform manual testing.

Conclusion

From these experiences, we learned that it is vital to communicate the need and importance of implementing a great testing strategy to our team. By explaining and showing them how a new feature can alter other aspects of the system, we can show them the importance of investing time in QA testing, thus guaranteeing that we are building a great product that tests not only isolated parts of the system but also the whole feature or functionalities.

Performing tests appropriately positively impacts the quality of the product that is being developed. We aim to create solutions for our clients that are not impacted by critical issues. Although no application is bug-free, implementing a good testing process can partially guarantee that there are no critical issues and deliver a great user experience.

QA testing can impact the project’s delivery time since testing times must be considered when estimating all services. Nevertheless, QA testing can prevent clients from continually fixing bugs or issues since they can be fixed before the product is launched.