Where is Manual Testing in Continuous Deployment?

Transformation from Waterfall to Agile methodology is almost over. Most of the companies are bracing Agile methodology for their release management. Some have perfected and others are catching up. Few have moved forward and took Agile to the next steps following approach of Continuous Delivery / Continuous Deployment. All these changes are suggesting a clear pattern where end users expectations are being valued and served quickly with quality delivery. Days might not be very far, when users can track delivery of their favorite features or changes similar to items being tracked via FedEx (or any other courier service) :). Does it suggest that there won’t be any place for manual testing in future? In this blog, we will try to explore this answer.

Before we delve into details, it important to understand the difference between Continuous Delivery & Continuous Deployment. In Continuous Delivery, decision of releasing to production is manual and release can be delayed depending on the risk or who is taking the decision. In Continuous Deployment, build is automatically deployed to the production once the code is checked-in. In Continuous Delivery, there is still time for manual or exploratory testing that can be done on staging environment(s) before taking build to the production.

How about manual testing in “Continuous Deployment (CD)”? When it is being done or does it happen at all? Is it being done by developer alone or tagging along with tester before code check-in? Are the defects being directly found and raised by users? Will CD approach not affect the brand name of company offering the services? Testing of critical scenarios, scenarios having financial impacts, usability & compliance, few crazy scenarios etc. can’t be left on bots alone and needs to be accompanied by manual testing. I was pursuing these questions, when another question popped up, are there companies who is releasing their code using Continuous Deployment (NOT Continuous Delivery) approach? Below are the list of few (not ALL) companies and their approach of conducting manual testing in CD.

Continuous Deployment at IMVU: Doing the impossible fifty times a day
In praise of continuous deployment: The WordPress.com story
Intercom - Why continuous deployment just keeps on giving
Etsy - Quantum of Deployment
WealthFront - DevOps Cafe on Continuous Deployment and Operations Dashboards

How & when manual testing being conducted?

In the current world, end user applications are getting complex and dependent on many other services or applications. Testing all scenarios manually is not practical but testing of critical scenarios directly on production (TiP) seems feasible. Does this mean, quality of the service or application is being compromised? No, in reality, (in TiP) the new feature or changes are not required to be visible to end users always as soon as the new build is released to the production. There are ways, when some features or changes can be hidden to end users until validated by testers in the production. Initially, the testers in the company validates the changes and later changes are propagated to Beta users. Upon their satisfaction, feature or changes are released slowly to all users. In the meantime, if defects are found, the release can be rolled backed without affecting ALL users.

Does this mean, even after releasing to production, it can take long time before changes are propagated to users?
In CD, the regression testing gets done in almost all the phases. Developer does manual testing before check-in of their code followed by automatic static validation, unit testing and automated regression tests. Build is deployed and validated in multiple environments in an automated fashion prior to deploying in the production environment. In all the phases / environment, monitors are placed, which automatically tests for reliability and other details. There are bots in the test system, which keep on doing the real transactions using real credit cards / debit cards etc. and makes it difficult for defects to get passed to the production. Monitors keep watch on any regression and give alerts in advance. Missed scenarios are added to the automation test suite quickly for the next set of releases.

As most of the testing getting done in an automated way, only few critical scenarios are left for testing manually and that too it depends on the critically of the feature / changes.

Why not all services can be deployed using CD?
In my opinion, CD is comparatively easy to apply for services where defects introduced will not create havoc. There is no loss of life, hazardous to health or directly impact the monetary loss. I see, lot of banks have also started applying the CD but not the features / changes that might involve monetary loss. In all these cases, CD can be adopted but releasing changes to the users can be delayed until confident of the quality. Not all existing design architecture might support the CD but it can be tweaked over a period of time where small changes or new features can coexist with other design elements and can be toggled on-off as per the requirements - like microservices.