If you’re frequently releasing new products or features, then you’re probably used to implementing a number of tests to verify that they’re working as intended.
There’s no doubt that such tests done in a staging environment are a crucial step prior to deployment.
Pre-production testing, when done right, can significantly improve the performance and time it takes for developers to push new code.
It allows developers to safely develop and test new code and to detect bugs early on before pushing it to production. As part of a shift left friendly practice we strongly encourage you to perform as much of it as possible.
A pre-production or a staging environment is one that exactly resembles a production environment. In other words, it mirrors as closely as possibly an actual production environment.
Your pre-production test suite that you would typically carry out before releasing new features, could include:
While such tests allow developers to verify a product or feature and make sure they’re working as they should on a near-production level, there are still some shortcomings if you’re just relying on staging environments.
While staging environments provide a close copy of the production environment, they cannot fully replicate it.
Additionally, it is not easy to maintain testing over a period of time and usually problems take a while to show up.
Therefore, some problems can only be detected by testing in real-world conditions.
That’s not to say that you shouldn’t be testing using staging environments but that you should combine these tests with those that are run in production. One doesn’t and shouldn’t replace the other.
So what exactly is testing in production? This is a software development practice where you run different tests on your feature or product in a live environment in real time.
Thus, you are essentially testing your new code changes on your real-world users in a live rather than a staging environment.
There are many advantages to this practice, including:
As already mentioned, testing in production isn’t meant to be a replacement for other types of testing. Rather, you can think of it as an extension to your pre-production test suite.
In other words, you should test all systems thoroughly using automated tests before production.
Afterwards, when you test in production, you can then let your users access your new features or products in the real-world to see how they react to it unlike in a staging environment.
Additionally, you need to test in production as well to get accurate data since it’s not easy to replicate production data and traffic. Thus, there are limits as to what can be duplicated in a staging environment.
There are many different tests you can run in production on your users. In this section, we’ll go through some of them.
You could opt for a blue/green deployment. This is when two production environments are used and new code would be deployed into the inactive environment.
Once these changes are ready, traffic is then diverted to the inactive environment which becomes active and users will then be able to access the changes.
Alternatively, you may choose to do a canary deployment. This method allows you to release software gradually to a percentage of your users.
Thus, with a canary test, for example, you can direct 10% of your users to the new version while the remaining 90% remain in the old version. Then, if no errors are detected, you can gradually roll it out to more users.
Another form of testing that you can run in production are experiments or A/B tests. Such experiments allow you to test different variations of your application or feature on users and gather data on which one performs better.
Let’s face it. Many things can go wrong when testing in production from deploying bad code to accidentally leaking sensitive data. The list could go on and on.
However, there is a way in which you can significantly reduce the risk and make it a much safer process.
This is where the real value of feature flags lies.
Feature flags allow you to turn certain functionalities on or off in order to test new features in production.
With feature flags, you can wrap new features in a flag to deploy them to production and make them visible to certain users.
This means that only a small subset of your users will see your new features and if anything goes wrong, you can quickly rollback, thereby limiting the blast radius.
Running tests prior to production is the key to achieving optimal results but as already mentioned, some problems will only come up in real-world conditions so it is vital that you are also testing in production to see how users interact and engage with your new releases.
Therefore, ideally, your test suite will include all the tests listed, both in pre-production and in production.
With feature flags, running tests in production has become easier and safer than ever before.