human in the loop

Implementing DevOps practices for application development has generated an incredible acceleration in code velocity and reliability, and that is just on the Dev side. As the infrastructure that runs applications is increasingly treated as code, similar benefits are being seen in the operations (Ops) side that deploy and run those applications.  What we would now consider traditional DevOps practices are, however, still largely tied to and limited by human decision-making abilities and associated manual tasks. While automation is increasingly common, it is not required to practice DevOps. Automation, however, and more importantly AI-driven automation is the way forward.  This is why we consider, in the quest to eliminate toil, accelerate and improve development processes, and automate all the things, AIOps is the future of DevOps.

There is a concept – human in the loop or HITL – that serves to transition from a manual or quasi-manual process to one that is truly automated. In the DevOps application development cycle, this is the difference between continuous delivery (production-ready code that needs human approval, a HITL process) and continuous deployment (production-ready code is automatically deployed). It can be considered the training wheels that can be shed when confidence in an automated system’s decision logic is achieved.

When considering a potentially automatable task, there are three steps to using a HITL approach to achieving an AIOps outcome:

  1. Create –A workflow gets architected and created. Manual actions drive the process.
  2. Automate – The workflow is transitioned from manual to automated. Automation models are defined for parts of the workflow. Human oversight, validation, and intervention are expected.
  3. Iterate – The workflow is iteratively improved. Human input continually is reduced but remains required to address low confidence outputs.
  4. Monitor – Automation is fully functional. Human input is not required, except to ensure overall performance meets expectations.

Create Your Manual Workflow

In the initial step, a manual process is developed, and, no matter how complex, it is identified as repetitive and potentially automatable.

Automate Your Workflow

In the second step, the initial move to automation, the process’s decision logic must be clearly understood, key performance indicators (KPIs) for evaluation defined, and tools for automation and their integration paths identified.  Complex processes are broken down into logical subprocesses that can be addressed independently.

The overall process of both decision-making and implementation is put in place. The automation process is implemented and verified as providing the initial function.  This is the stage for an ML model where initial model training would occur.

Work Towards Removing the Human in the Loop

The third, iterative stage, has HITL involvement required to provide sufficiently robust outputs for the entire flow to allow the model’s confident and automated implementation. The overall automation process works, but it does not work well enough to take off the human training wheels. Models’ decisions are still gated and potentially corrected by human gatekeepers.

The challenge in a journey to AIOps is that the process can stall at this point. There are many DevOps solutions, such as the continuous delivery approach that falls short of full automation.  In an AI process, this is equivalent to a state of not trusting the model outputs enough. 

Monitor Your Workflow

If you can achieve confidence in the process, the fourth stage finally takes the human out of the standard decision-making process. The challenge will remain that the environment in which applications run is often dynamic, and changes to the environment may impact the system’s confidence and reliability.  For this reason, appropriate monitoring of the KPIs defined in step remains important.  

If you’ve made it to this point, you will have achieved true AIOps rather than AI-augmented Ops. Removing the human from the day-to-day decision-making loop will allow your developers and engineers to set their sights on higher values tasks.

If you would like to experience AIOps for yourself, see how Opsani’s AI-driven continuous optimization engine improves application performance on Kubernetes infrastructure without a HITL with a free trial.