Integration Vs Contract Tests: Understanding the Differences and When to Use Them

When it comes to automated software testing, there are many types and approaches to do it in a proper way. Every type of testing (unit, functional, integration, performance etc.) serves its own purpose. The main reason for having both integration and contract testing is mainly to do with microservice architecture, the need to test interaction between separated modules. Integration and contract testing propose their own ways to solve this problem, but both have their pros and cons. 

Integration testing is a kind of brute force. It needs all instances up and running in order to execute tests, and to test apps interactions and functionality as is, with or without minimal mocking and stubbing. Integration tests need some preparations before they can be executed. Such an approach is the closest to what happens in the production environment. The drawback of this kind of testing is extremal high weight of tests and long running time. Integration tests are both time- and resource consuming.

Contract testing has recently grown in popularity. It offers a completely different approach to test compatibility between modules, based on strict API definition. The main idea of contract testing is a contract, or pact file, which all participants must follow. A pact file is a file that contains the JSON serialized interactions (requests and responses) or messages. For example, typical pact file of pact.io framework defines a) consumer name; b) provider name; c) a collection of interactions or messages; d) a pact specification version. 

Integration testing overview 

Integration testing verifies compatibility and interoperability between program modules in terms of microservice architecture where each microservice plays role of such module. Any type of interactions between services can be involved in integration testing: API requests, messages to message queues systems, DB calls, UI interactions etc. And such interactions during testing need to be close to reality, as it is the main meaning of interaction testing – to test as full as possible. So, usually we need to run entire copies of apps itself, test database, test message queue etc.  

Integration testing is comprehensive and therefore has its pros and cons. Such fullness allows us to achieve great coverage, since all the app layers are involved: data layer, business logic and API service. The issues that are found at this level are easier to resolve than the ones that are found at later stages of testing, such as acceptance testing. On the other hand, high weight almost always leads to flakiness of such tests. Also, another negative side effect – is the necessity to maintain correct versioning for numerous services during testing.

The diagrams above show how different the coverage for integration and contract testing is. Integration testing covers the application in depth, when coverage for contract testing is shallow – even API layer is not fully covered. It is a feature of contract testing – to verify only interaction logic between apps, and not apps itself. 

Contract testing overview 

Contract testing is based on a contract – shared understanding of how applications must interact. In general, the contract is very similar to an API specification file, for example, OpenAPI specification file, ‘swagger.yml’. Using strict definition as a cornerstone of testing encourages developers to maintain documentation in good state. There are three approaches of contract testing: consumer-driver, provider-driven, bi-directional. All three types have common benefits: they are lightning-fast and easy to create and maintain.

Benefits of contract testing can be noticed at that point, as it wins competition with integration testing in speed, performance, and maintainability. Still, we should mention its disadvantages. There are two main cons. The first drawback is that its community is still relatively young and not so numerous, as well as frameworks for contract tests, so, it may be difficult to get support or answer to your question in time. 

Also, some rare and specific to project features (e.g., implementation for some programming languages) may not be implemented in contract testing frameworks due to relatively small community and less frequent releases than in popular Web or UI-test frameworks. The second issue comes from the pact test design itself, and can be counted as weakness, as well as a simple design specific feature – contract tests cannot fully replace integration tests. When you implement contract testing on your project, you still need to have some amount of integration tests. 

Consumer-driven contracts

Consumer-driven contract testing focuses on the needs of the consumer. Consumer defines rules for interaction between itself and provider. This approach has its benefits, e.g., it helps avoid over-engineering and fix only those rules which are required by consumer to fit.

When consumer creates contract it says, “that’s my expectations about the way in which provider responds to specific types of requests”. And the provider should meet the expectations that contract contains. 

On the diagram above typical flow of consumer-driven approach is shown. The consumer app plays the main role here – it generates contract and publishes it to the broker (broker is a system for storing and maintaining contracts and providing functional for interaction between consumer and provider). The provider can then fetch this contract and execute tests.

It should be mentioned here that consumer-driven tests are the most popular among others. As an example, framework pact.io based on consumer-driven approach.  

Provider-driven contracts 

In some cases, provider-driver contract testing may come in handy. In such approach, the provider defines a set of rules. The provider-driven approach may be preferable when the same provider has multiple consumers, which requirements may differ one from others. So, it’s better to have a centralised provider-driven contracts which all consumers must follow.

Providers create contracts saying, “That’s what I can do”. And a consumer can use provider’s contracts to process responses listed in contract successfully. 

The typical flow for provider-driven approach is shown on diagram 4. Here, the provider leads the entire process – generates contracts, executes contract tests on its side, and sends contract to broker. Then the consumer can fetch a contract and verify it on consumer’s side.  

Bi-directional contracts 

This approach was introduced by Pactflow, a popular pact broker system. In this type of contract testing both the consumer and provider are peers, i.e., they possess equal roles in defining interaction rules between them. Both the provider and consumer generate and send its version of contract to the broker, and the broker compares it and reports verification results. Such an approach requires strong communication between consumer and provider development team. Another feature of bi-directional testing is the possibility to use API specification file instead of generated contract.

Typical flow for bi-directional contract testing is shown in diagram 5.

Comparison 

There are pros and cons for both integration and contract testing in the table below.

Conclusion

Contract tests are faster, less complex, less flaky, easier to maintain, and, on the other hand, less independent than integration tests, but much more far from real apps behaviour.  

So, which test approach is preferable for a particular case?  

Contract tests can be suggested to use every time when you have microservices architecture. It’s worth mentioning that contract tests are not independent. We cannot have only one set of contract tests for all our needs, since contract tests are not about full code coverage. So, we still need to use it in conjunction with other test sets – unit, functional, API tests etc. Remember: contract tests are about complementing other types of testing, but not about replacing them! So, contract tests are applicable everywhere, where integration tests can be applied. Contract tests will help to keep integration tests at reasonable amount and can prevent from using those high weight tests excessively.

Another reason why contract tests are strongly suggested is that they encourage developers to maintain strict API specification and clear versioning. And that’s the opposite to integration tests, which tend to create some mess in testing process. 

What about integration tests? It is suggested to have a limited set of them, especially, it can be useful in role of smoke or regression test suites.