Functional vs non-functional software testing

For development teams, a comprehensive software testing strategy requires both functional and non-functional testing. Together, these tests provide a foundation that is crucial to your team’s ability to deliver successful applications. Why are they so important? Functional testing checks the application’s processes against a set of requirements or specifications. Non-functional testing assesses application properties that aren’t critical to functionality but contribute to the end-user experience, like performance and reliability under load.

Your team’s testing practice should assess the entire application, observe the larger story of how it operates when functioning correctly, and raise alarms when deviations are found.

What is functional testing?

Functional testing compares actual outputs against expected behaviors. This provides a clearer overall picture than testing individual modules in isolation. Interactions between modules are frequently the points where errors occur. Functional testing often includes testing portions of the underlying code.

There are many types of functional testing:

• Unit testing (which can also be used for non-functional testing)
• Integration testing
• User acceptance testing
• Closed-box testing

Unit testing

Unit tests for an API application might make requests against the system deployed in a testing environment and compare the responses against documentation. Unit tests are limited, though. When the application deviates or regresses in functionality, application-wide functional tests often detect changes that unit tests don’t.

Integration testing

Integration testing validates how software modules operate together. When developers write code as loosely-coupled modules — as it usually should be — the components rely on explicit contracts for how they interact. Integration tests validate that each piece of the software lives up to its end of the contract and generate warnings when these interactions introduce regressions.

User acceptance testing

In the user acceptance phase of software testing, developers provide part or all of the application to end users or their representatives to model real-world interactions and functionality. Many healthy engineering cultures avoid relying heavily on user acceptance testing due to its unreliability, cost, and time consumption. Still, some user acceptance testing is a vital part of testing procedures for most applications.

Closed-box testing

Functional testing can also include closed-box testing of the complete application. Closed-box testing treats the application’s outputs holistically without examining its inner workings. Many engineers gravitate toward it when writing functional tests to complement their non-functional tests. The code that interacts with the outermost layers is tested automatically and only the output is evaluated. This process is simple for applications requiring only API testing, as the code simply makes API calls and evaluates the result.

When used for applications with a user interface, closed-box testing becomes increasingly complex. One way to address this complexity is to use a sophisticated testing tool like Selenium. These tools allow code to interact with an application as if it were a user in a web browser. Selenium and tools like it automate user acceptance testing while increasing reliability and scaling up. Still, applying closed-box testing to user interfaces presents a complex problem with unique challenges. It can quickly consume your team’s engineering bandwidth.

What is non-functional testing?

Non-functional testing assesses application properties that aren’t critical to functionality but contribute to the end-user experience. Performance and reliability under load aren’t functional components of a software system but can certainly make or break the user experience. Something that fails a non-functional test doesn’t always cause an issue that users would notice, but it can indicate a problem in the system — especially at scale.

There are many types of non-functional tests:

• Performance testing
• Load testing
• Usability testing
• Security testing

Performance testing

One essential process in the non-functional testing category focuses on performance. Performance tests ensure that a software system responds to requests promptly. Poor latency destroys a user’s experience, and well-written performance tests often catch problems before they become evident to users.

Load testing

Load testing is a related type of non-functional testing. Few systems perform the same in responding to one request per second as they do to 10,000 requests per second. Load testing validates that a system can handle peak loads and fail gracefully when it lacks the resources to handle workload spikes.

Usability testing

Usability testing measures the quality of a user’s experience. For the most part, usability testing is a manual process that does not scale well. Regardless, a lack of usability testing, especially when localizing applications, often leads to confusing and unintuitive interfaces. It is worth spending some time incorporating this type of testing into your software development process.

Security testing

Security testing comprises another set of non-functional tests. Your team should test their applications regularly to make sure they are secure and handle data correctly. Security testing can range from automatic scanning to periodic penetration testing, depending on the application’s level of exposure to potential threats. Many teams don’t view security testing as part of their testing suite. You would be wise to include security testing and take it as seriously as unit testing.

Comparing functional and non-functional tests

Functional tests confirm that the code is doing the right things, while non-functional tests validate that the code is doing things the right way. Both types contain methodologies for validating front-end and back-end elements and behaviors. There is some overlap between the two categories in the kinds of tests a developer might run.

Functional testing suites are the more strictly necessary of the two categories. Software solutions must solve the problems they target. The implementation details and performance metrics that non-functional testing targets are often secondary matters of refinement. A robust testing methodology also accounts for these factors, especially if scaling is a priority.

Functional testing is often expensive to write and maintain and slow to execute. Many developers rely most heavily on unit tests, because they aid test-driven development and often lead to self-documenting code. These tests are quick to execute and easy to modify when the code requirements change. Even with modern tools like Selenium, methods like closed-box testing take a long time to run and write.

The most neutral approach to testing choices includes a comprehensive mix of functional and non-functional testing. Many developers follow the “testing pyramid,” which guides them to write the bulk of their tests as unit tests, since they are quick to write and execute.

As they advance up the pyramid, developers implement lower volumes of methods like integration testing and then progress toward practices like user acceptance testing at the end. Eventually, they top the pyramid more sparingly with testing methods that offer lower returns relative to their implementation costs.

Organizations can reduce the costs associated with implementing and maintaining a testing process by integrating functional and non-functional tests into a continuous integration and continuous deployment (CI/CD) pipeline.

Conclusion

There is no one-size-fits-all solution to testing software applications. It is too rigid to say that the functional method is better than non-functional or the other way around.

Non-functional testing is just as necessary as tests that validate functionality. Many teams consider non-functional testing a lower priority because the improvements it provides are less dramatic. A user may be annoyed if performance degrades, but they may still be able to use the software application.

On the other hand, when functionality is broken, users may not be able to use the features they need at all. This, combined with shorter execution times and lower cost makes functional tests the foundation for many teams’ software testing processes.

Still, non-functional tests play a critical role, and your team should find ways to incorporate them. An excellent testing suite will validate a broad set of non-functional characteristics and components, including performance and usability.

To learn more about testing and how you can further incorporate good testing practices with your continuous integration tools, explore some of the testing tools that integrate with CircleCI.