What distinguishes MLOps from DevOps and makes it a value-adding enterprise in the ML space is the former\u2019s concentration on capabilities specific to the end-to-end ML lifecycle. This includes not only the research and development for the model but also its deployment and post-implementation support.<\/strong> With ML, the latter entails more than making sure your code continues to run smoothly and uninterruptedly but also monitoring and automated retraining.<\/p>\n You may think, \u201cBut my model is pretty good as it is. Why does it need to be monitored and retrained?\u201d<\/p>\n There are all sorts of environmental factors that can cause models to become less effective over time through no fault of their own. When this happens, we call it model drift.<\/strong><\/p>\n When you deploy a model, you implicitly assume that the statistical relationships observed between the features in your data and the outcome variable will consistently hold over time. As circumstances change – as they virtually always do in any complex system – so, too, do these relationships.<\/p>\n In this article, we will walk you through a real-world example of model drift. We will then discuss why drift occurs and some remedies you can explore for how to handle it within the MLOps framework.<\/p>\n We once worked with a client to build and “productionize” an ML model for their sales call center. The goal was to increase the number of good calls their employees experienced, where they defined a good call as meeting one of several criteria, including whether the call led to the prospect becoming a lead or an opportunity.<\/p>\n They could then use the model output to stack-rank their list of prospects at the beginning of each day and prioritize who were the most likely people that sales members would have a good call with. Afterward, we monitored the model performance for several months and observed several interesting findings.<\/strong><\/p>\n First, as expected, the model improved the number of good calls their sales staff experienced by over 50 percent. Second, the model began to drift within a few months of initial deployment. Third, the drift was most notably pronounced downstream when we looked at the sales pipeline to determine what calls ultimately translated into leads and opportunities. In other words, the impact of the drift visibly translated to their bottom line.<\/p>\n There are several reasons why drift can occur:<\/p>\n This could be from an improvement to technology, a desire to drive down inefficiencies, new personnel or a change in strategy. The very use of the model itself could trigger changes to your business\u2019 status quo<\/strong>. For example, we once built an ML model for a client as part of their customer retention initiative, which they would use to help them identify customers proactively and talk them out of leaving. Suffice to say, the circumstances under which we trained the model ended up being different from those under which they used it.<\/p>\n It turns out our call center client above, shortly after their model\u2019s deployment, had adjusted their marketing strategy to focus more on driving traffic to their website via online outlets. This was something the model would not have seen in the training data.<\/p>\n Another one of our previous clients was working to mitigate the shortage of truck drivers in their fleet and had us build an ML model to determine who was at risk of quitting. This model quickly lost a lot of its predictive ability in 2020, however, when the landscape changed with the COVID-19 pandemic.<\/strong> With the closure of restaurants and facilities, a shortage of truck drivers soon became a surplus.<\/p>\n If you are building an ML model to augment your marketing efforts, for example, and 20 percent of your training data comprise prospects from trade shows and 80 percent from purchased leads, much of the model’s performance will be driven by those prospects in the latter group. This can create problems if you primarily apply the model to prospects in trade shows, and those clients typically behave differently than purchased leads.<\/strong><\/p>\n In the case of our call center client\u2019s prospects, one reason for the drift was simply because, by design, it would pick the cream of the crop among the most promising prospects on any given day. Eventually, though, as you go down the list, you start to observe diminishing returns by dialing fewer promising candidates.<\/p>\n If your model depends on select data elements, and you change the way you capture or interpret these elements, these changes can cause drift. Without retraining your model on the newer data format, it will have no way of continuing to pick up those signals.<\/p>\nWhat Does Model Drift Look Like?<\/h2>\n
![\"MLOps](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2021\/12\/ML_Ops_Blog1-1024x327.png\")
<\/a><\/p>\nWhy Does Model Drift Happen?<\/h2>\n
Changes to Business Processes<\/a><\/h3>\n
Changes to Target Behavior<\/h3>\n
Changes to the Composition of the Observations Comprising the Data<\/h3>\n
Changes to How You Collect the Data<\/h3>\n
![\"MLOps\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2021\/12\/ML_Ops_Blog2-1024x715.png\")
<\/a><\/p>\n