LaunchDarkly

Test Environments 101: Definition, Types, and Best Practices

LaunchDarkly
By LaunchDarkly   •   April 6, 2021
LaunchDarkly

A test environment is any space in which software undergoes a series of experimental uses. Testing is essential to any software development methodology. And a weak testing strategy can result in final deployments that are rife with bugs and errors. There are many ways to run tests, and most involve test environments.

This post explores test environments from the ground up. Not only will you learn what a test environment is, but you'll also discover how best to set up one in the era of continuous deployment. Then we'll discuss how best to manage them.

What is a test environment?

To fully grasp the definition of a test environment, consider how a typical software application gets deployed and managed. There are times when developers make clones of the main codebase and deploy them separately to iron out bugs and glitches in different virtual machines.

The location and state of these parallel applications are what we're calling test environments. This includes the hardware and operating system on which such versions of software will have to reside for the duration of prescribed test executions. With plenty of applications now hosted in the cloud, we also include the resources allocated for testing as their test environment.

Let's look at a few environment types as we single out and increase our understanding of test environments.

Types of environments

As you may have gathered, an application’s server location is itself an environment. However, it's standard to segment and name subsections based on the state of the application.

We typically have four environments along any software's lifecycle.

  1. Development
  2. Testing 
  3. Staging
  4. Production

Development environment

The development environment is the location of the main branch of a software application. This is where developers spend time writing the first lines of code. From here, an application transforms from concept (on paper) into the MVP (minimum viable product).

Recently, the development stage has been overtaken by version control systems: GitHub and its competitors. They essentially host code in public and private repositories in the cloud. This makes this environment the easiest to set up. With the GitHub case, the free tier has enough features and resources for the starting phase of most applications.

Testing environment

When you set aside an entire server (could be a container) to run tests uninterrupted, you’ll have created a test environment. Typically, a test environment meets the minimum requirements for your application to function. 

Depending on your preferred testing tools and frameworks, you may need more than one test environment for each possible access instance. For example, Selenium tests can’t run for the entire selection of browsers you want your application accessible through at the same time. This means you either run tests one after another, or you create several test environments. 

Staging environment

Going up from the test phase, we have the app staging environment. Staging is when you create an instance of an application that you're confident enough to show the immediate owner (sponsor) but not users. Before exposing to the latter group, you'd want to run more tests. A staging environment is meant to simulate production as much as possible. 

The staging environment is often restricted to a few select people. Whitelisted emails, IPs, and your developer team are the only groups that can use the application in staging. 

It’s worth noting that staging environments are no substitute for a real-world production environment. The surest way to know how new features will perform with real users is to test them in production. At LaunchDarkly, we’re big advocates of testing in production with feature flags.

Test-environment-staging-production-LaunchDarkly

Production environment

As an end-user, when you use a web or mobile application, the application is running on a production server. It has been produced. We mentioned how testing was necessary for every application, however, not every environment is a prerequisite. 

Thanks to feature flags, many development teams are doing away with some staging environments altogether. The first visible benefit of such a move is cost savings. What it actually implies is that the production environment absorbs the other pre-production states.

Tests can be conducted during production, with new features being released safely at the same time. Feature flags make it possible to show a few specific users a future version of an application, while the rest enjoy its current version.

testing-in-production-with-feature-flags-LaunchDarkly

By now, you're well-versed in the types of environments a software application transitions through. Let's revert our focus back to the test environment—particularly, the kinds of tests that take place in this environment.

Types of software tests

While the staging environment typically gives test teams the chance to take an app for "a ride," this kind of testing is far from sufficient for an app to satisfy quality assurance. Beforehand, the app will have undergone test executions looking to break it in a safe environment. System testing gives developers the chance to fix it without the pressure of active users breathing down their necks.

Here's a list of (some of) the software testing routines an application goes through before development. 

  • Performance tests
  • System integration testing
  • Alpha tests
  • Beta tests
  • Unit tests
  • Accessibility testing
  • User acceptance testing (UAT)
  • Usability testing

We'll discuss the first four as we explore an application under testing and its developers practicing DevOps methods through different environments.

Performance test example

Software testing of this kind measures how fast an app responds to an interaction. You'll typically get page load speeds and input processing speed as reports here. You can run performance tests before and after deployment. 

Here's a typical test that developers at Facebook might run, for example:

performance-test-Facebook-example-LaunchDarkly

This particular report rates the performance of a specified application out of 100. It also provides a granular outline of what the score is, making it possible for developers to improve it if desired. Different tools can be used when performance testing, all giving different test results in the end.

System integration testing

It's common practice for teams of developers to assign sections of an app to each developer. In fact, modules made by these team members can continue to run isolated from each other. Inevitably, there comes a time that they absolutely have to integrate. 

This is the stage when integration tests are run. The resulting system testing makes sure all modules can communicate and share the same database. Depending on test results, it may take longer for the system to then come together.

Alpha tests

This is the first end-to-end run conducted on the application when all modules have been integrated successfully. Its main objective is to make sure the application does what the client is expecting.

For example, a payment gateway would undergo payment transactions in sandbox mode as its alpha test. Quality assurance managers typically tick a checklist until the application satisfies the list of functionalities. It's more of a quantitative testing process as it cares more about what’s possible than how it happens.

Beta tests

You may have seen or even used some software applications in beta mode. It's during this phase that real-world users put an application under stress. This load testing serves to unearth and report any bugs the developers may have overlooked. Some bugs actually only appear during production. This means a staging environment with open access is necessary: hence, the existence of beta versions of apps. 

Next time you download beta versions of any apps, make sure to participate in perfecting them. All the while, this should reasonably lower your expectations of perfect functionality as you're part of its load testing process. This kind of testing is classified as manual testing.

What is Test Environment Management (TEM)?

Test environments require deliberate actions to set up correctly. They also need dedicated attention to keep active and functioning the way you'd have pre-set. Let's look at the best way to set up and then maintain test environments—both on-premises and in the cloud.

Best practices when setting up test environments/test beds

When setting up test environments, it's tempting to buy the best-performing servers. However, it's less likely that high-end devices will experience performance bottlenecks. As such, you're better off focusing on securing the application than making it perform fast. 

Think about security before you lose your mind over performance. Since it's cost-effective to run tests in production, your feature flag rules should be well defined. This way, even before the app is performing optimally, new features (the actual test) are visible only to quality control teams and other users you deem necessary.

Be sure to include other departments into the rules of your feature flags. Sales features are best evaluated by employees involved in their intended functionality. This lets you run both alpha and beta tests at the same time. Their nontechnical feedback also helps your developers gain insight into patching the features for perfect functionality.

When building a communication strategy into your test environment, consider channels that all stakeholders use already. This makes it easy for everyone to know where you're at before they send long emails asking for updates.

Use live test cases and test data in a controlled environment. This way, you don't need to have differently prefixed backups of application databases. This makes your test bed the same state and files as your production environment.

Test Environment Management and test beds best practices

Once you’ve set up a test environment following the advice we've discussed above, you still need to maintain it. The following section goes over a few best practices for this.

Say you have different test environments set up. Make sure they’re sanitized from each other. No test environment instance should affect the next instance's resources. This means continuously monitoring each such instance with APMs and log management tools. It’s hectic, but if you don’t take it seriously, you’ll get false reports and eventually carry bugs through to production.

Sanity also means using the same datasets each time you run tests in your test beds. This affords you consistency when analyzing the reports and the resulting data after tests.

If any of your test beds have different app configurations, you're best to avoid configuration management problems by using experimentation management platforms. You get to enhance your tests with A/B tests and conduct broad test cases.

Keep teams on the same page by optimizing communication. Some companies treat tests as stand-alone projects themselves, with QAs at the helm of everything. This gives each variable in the test beds a life that can be traced as it evolves during tests. It becomes easy to assign actions and resources to maintain the environment or even patch test scripts quickly.

If you're testing in production, there's more you'll need to do to hardwire everything into your team. Then, it essentially becomes a standard. 

The art of testing in production

While it may seem scary at first, testing in production is more efficient and effective than soaking your code in staging (note: we're not necessarily encouraging teams to shed their staging environments; we’re just highlighting the benefits of testing in production comparatively). And LaunchDarkly feature flags remove much of the risk typically associated with such a testing approach. With feature flags, you can release a new feature to a small user segment (e.g., a canary group) while concealing it from the rest of the user base. Thus, if the feature throws a bug in production, it will only impact a small portion of users. Moreover, with feature flags, you can disable the buggy feature instantly without having to change a config file and restart your application.

Get started with LaunchDarkly and embrace the philosophy of testing in production to build and deliver better software, faster. 

***

Request a demo of LaunchDarkly’s feature management platform. And don’t miss our Test in Production Meetup (virtual, for now)!

You May Like
  •   BEST PRACTICESTesting in Production to Stay Safe and Sensible
  •   BEST PRACTICESWhat Is Continuous Testing? A Straightforward Introduction
JUNE 17, 2021   •   HOMEPAGE FEATUREDGalaxy Talks: Launching Darkly at Large Organizations
JUNE 15, 2021   •   POPULARWebinar Recap: Safe and Sensible - Deployment and Launch with Reduced Risks
JUNE 10, 2021   •   GALAXYGalaxy Talks: Feature Flags in Your Daily Workflows
JUNE 8, 2021   •   INDUSTRY INSIGHTSAttracting and Retaining Engineering Talent in 2021