Use Machine Learning to predict customers you might lose – Part 3

Cruising through our machine learning journey and starting from where we left in the previous instalment, the next step is to expose our machine learning model as a Web service so that it can be invoked from within Dynamics CRM.

Azure Machine Learning has this fantastic concept of converting an experiment into a trained model. Trained model is like a compiled version of your experiment that can be exposed via a web service, all from the click of just one button i.e. Setup Web Service


Azure ML takes care of the rest by deploying the model. Once deployed you can inspect its configuration by going to the Web Services section as shown here


In order to connect to this web service from within Dynamics CRM, we can use the below JavaScript. We can pass CRM objects to this service in JSON format and get prediction results back

sendRequest: function (avgIncome,overAge,leftOver,houseVal,handsetVal,longCalls) {

var service = AzureScript.getRequestObject();

var url = "";

var jsonObject =


"Inputs": {

"input1": {

"ColumnNames": [














"Values": [


















"GlobalParameters": {}


var dataString = JSON.stringify(jsonObject);

if (service != null) {"POST", url, false);

service.setRequestHeader("X-Requested-Width", "XMLHttpRequest");

service.setRequestHeader("Authorization", "Bearer xxxxxxxxx==");

service.setRequestHeader("Accept", "application/json");

service.setRequestHeader("Content-Type", "application/json; charset=utf-8");

service.setRequestHeader("Content-Length", dataString.length);


//Recieve result

var requestResults = eval('(' + service.responseText + ')');

try {

resultOutput = requestResults.Results.output1.value.Values[0]

return resultOutput;


catch (err) {

console.log('Unable to interpret result');




Let us prepare Dynamics CRM to start consuming this web service. I have created an event onSave() of the Telecom Customer form which passes the relevant data to the Azure Service and gets the score. The Javascript for that is as below

function onFormSave() {

//Prepare data - only fields with high Information gain

var houseValue = Xrm.Page.getAttribute('manny_housevalue').getValue();

var income = Xrm.Page.getAttribute('manny_income').getValue();

var longcalls = Xrm.Page.getAttribute('manny_longcalls').getValue();

var overage = Xrm.Page.getAttribute('manny_overage').getValue();

var phonecost = Xrm.Page.getAttribute('manny_phonecost').getValue();

var leftOver = Xrm.Page.getAttribute('manny_leftover').getValue();

var valOutput = AzureScript.sendRequest(income, overage, leftOver, houseValue, phonecost, longcalls);

if (valOutput != null && valOutput[0]!=null && valOutput[1]!=null) {





var prob = parseFloat(valOutput[1]);

if(prob>=0 && prob<=1.0)




getRequestObject: function () {
 /// Get an instance of XMLHttpRequest for all browsers
 if (XMLHttpRequest) {
 // Chrome, Firefox, IE7+, Opera, Safari
 // ReSharper disable InconsistentNaming
 return new XMLHttpRequest();
 // ReSharper restore InconsistentNaming
 // IE6
 try {
 // The latest stable version. It has the best security, performance,
 // reliability, and W3C conformance. Ships with Vista, and available
 // with other OS's via downloads and updates.
 return new ActiveXObject('MSXML2.XMLHTTP.6.0');
 } catch (e) {
 try {
 // The fallback.
 return new ActiveXObject('MSXML2.XMLHTTP.3.0');
 } catch (e) {
 alertMessage('This browser is not AJAX enabled.');
 return null;


These scripts are trivial and should be self-explanatory. Basically we are passing the highly correlated features to the prediction service and getting two outputs

Prediction score -> assigned to-> manny_predictedchurnstatus

Prediction confidence -> assigned to -> manny_predictionconfidencepercentage

And they are displayed on the form like this, it’s integrated i.e. the moment you change the data the score is updated.


In the next blog post, we will touch upon the Insights that can be gained from a machine learning integration

Use Machine Learning to predict customers you might lose – Part 2

Continuing our journey from the previous post where we defined the issue of churn prediction, in this instalment, let us create the model in Azure Machine Learning. We are trying to predict the likelihood of customer’s churn based on certain features in the profile which are stored in the Telecom Customer entity. We will use a technique called Supervised Learning, where we train the model on our data first and let us understand the trends before it can start giving us some insights.

Obviously you need access to Azure Machine Learning, once you log into it, you can create a new Experiment. That gives you a workspace designer and a toolbox (somewhat like SSIS/Biztalk) where you can drag control and the feed into each other. So it is a flexible model and for most tasks you do not need to write code.

Below is a screenshot of my experiment with toolbox on the left


Now machine learning is something which is slightly atypical for a usual CRM audience, I would not be able to fit full details of each of these tools in this blog but I will touch on each of these steps so that you can understand at high-level that what is going on inside these boxes. Let us address them one by one

Dynamics CRM 2016 Telecom

This module is the input data module where we are reading the CRM customer information in the form of a dataset. At the moment of writing the blog, there is no direct connection available from Azure Machine Learning to CRM online. But where there is a will, there is a way i.e. I discovered that you can connect to CRM using the following

1. You schedule a daily export of CRM data into a location that Azure Machine Learning can read e.g. Azure blob storage, Web Url over Http

2. You can write a small Python based module that connects to Dynamics using Azure Directory Services, the module can the pass the data to the Azure using a DataFrame control

From my experience having an automatic sync is not important from Dynamics to Azure ML but it is important the other way round i.e. Azure ML to Dynamics.

Split Data

This module basically splits your data into a two sets

1. Training dataset – The data based on which the machine learning model will learn

2. Testing dataset – The data based on which the accuracy of the model will be determined

I have chosen stratified split which ensures that the Testing dataset is balanced when it comes to classes being predicted. The split ratio is 80/20 i.e. 80% of the records will be used for training and 20% for testing.

Two-class Decision Forest

This is main classifier i.e. the module that does the grunt of the work. The classifier of choice here is a random forest with bootstrap aggregation. Two-class makes sense for us because our prediction has two outcomes i.e. whether the customer will churn or not.

Random forests are fast classifiers and very difficult to overfit, rather than taking one path they learn your data from different angles (called ensembles). Then in the end the scores of various ensembles are combined to come up with an overall prediction score. You can read more about this classifier here.

Train Model

This module basically connects the classifier to the data. As you can see in the screenshot of the experiment I posted above there are two arrows coming out of Spilt Data, the one of the left is the 80% one i.e. the training dataset. The output of this module is trained model that is ready to make predictions.

Score Model

This step uses the trained model from the previous step and tests the accuracy of the model against our test data. Put simply, here we start feeding the data to the model that it has not seen before and count how many number of times the model gave the correct prediction Vs wrong prediction.

Evaluate Model

The scores (hit vs miss) generated from the previous modules are evaluated in this step. In Data Science there are standard metrics to measure this kind of performance e.g. Confusion Matrices, ROC curves and many more. Below is the screenshot of the Confusion matrix


I know there is a lot of confusing details here (hence the name Confusion Matrix) but as a rule of thumb we need to focus on AUC i.e. area under the curve. As shown in the results above we have a decent 72.9% of the area under the curve (which in layman terms means percentage of correct predictions). Higher percentage does not necessarily equate to a better model, more often than not a higher percentage (e.g. 90%) means overfitting i.e. a state where your model does very well on the sample data but not so well on the real-world data. So our model is good to go.

You can read more about the metrics and terms above here

In the next blog we will deploy and integrate the model with Dynamics CRM.

Use Machine Learning to predict customers you might lose – Part 1

“Customer satisfaction is worthless. Customer loyalty is priceless.”

Jeffrey Gitomer

Business is becoming increasingly competitive these days and getting new customers increasingly difficult. The wisest thing to do in this cut-throat scenario is to hold on to your existing customer base while trying to develop new business. Realistically, no matter how hard it tries, every organisation still loses a percentage of its customers every year to the competition. This process of losing customers is called Churn.

Progressive organisations take churn seriously, they want to know in advance that approximately how many customers they are going to lose this year and what is causing the churn. Having an insight into customer churn at least gives an organisation an opportunity to proactively take measures to control the churn before it is too late and the customer is gone.

Two pieces of information help the most when it comes to minimising the churn

1. Which customers are we going to lose this year

2. What are the biggest drivers of customer churn

The answers to the above questions often are hidden in the customer data itself but revealing these answers out of swathes of data is an art – rather a science called Data Science. With recent advances in some practical Data Science techniques like Machine Learning getting these answers is becoming increasingly feasible even for small scale organisations who do not have the luxury of a Data Science team. Thanks to services like Azure Machine Learning which are trying to democratise these advanced techniques to a level such that even a small scale customer can leverage them to solve their business puzzles.

Let me show you how your Dynamics CRM can leverage the powerful Machine Learning cortex to get some insights into the key drivers of customer churn. In this blog series, we will build a machine learning model that will answer the questions regarding churn. I have divided the series into four parts as below

Part 1 – Introduction

Part 2 – Creating a Machine Learning model

Part 3- Integrate the model with Dynamics CRM

Part 4 – Gaining insights within Dynamics CRM

I will take the example of a Telecom organisation but the model can be extended any kind of organisation in any capacity and from any industry.


Let us say there is a Telecom company called TelcoOrg which uses Microsoft CRM 2016 and they have an entity called Telecom Customer that stores their telco profile. Such profile may include some data regarding a customer mobile plan, phone usage, demography and reported satisfaction.

Understanding the features

In data science projects, it is crucial to understand the data points (called features). You need to carefully select those features that are relevant to the problem at hand, some the features also need to be engineered and normalised before they start generating some information gain. Below are the features that we will be using in this scenario of our Churn problem

Let me quickly explain the features so that we can understand the information contained in them



Has a College degree?

If the customer has a college degree

Cost price of phone

Price of the customer’s phone as per the plan/contract with TelcoOrg

Value of customer’s house

Approximate value of customer’s house based on Property Information websites like RPData, etc.

Average Income

Yearly income as reported by the customer

Leftover minutes per month

Average number of minutes a customer normally does not use from monthly quota

Average call duration

Average duration of calls made based on call history

Usage category

The category customer’s phone usage falls under as compared to other customers e.g. Very High, High, Average, Low or Very Low

Average overcharges

Average number of times a customer is usually overcharged per month

Average long duration calls

Average number of calls a customer usually makes per month that are more than 15 minutes long

Considering change?

How customer responded to TelcoOrg’s survey when asked if they are considering changing to another provider e.g. Yes, considering, Maybe, Not looking, etc.

Reported level of satisfaction

How customer responded to TelcoOrg’s survey when asked if they are satisfied with TelcoOrg’s service e.g. Unsatisfied, Neutral, Satisfied

Account Status

Current Status of the customer (i.e. if they have left or are currently Active)

Predicted Churn Status

This is the predicted status returned by the Azure Machine Learning Web Service

Prediction Confidence Percentage

This field means how confident Azure Machine Learning Web Service is regarding its prediction. A threshold can be set to only consider the predictions above e.g. we can say, take only those predictions where WS is 70% confident.

The screenshot below shows the information from the Telecom Customer entity. The section highlighted in blue are predictions based on Azure Machine Learning web services. Whenever any of the fields on this CRM form changes, the WS updates its prediction scores based on the record’s data. I will provide details later during the series as to how I built this integration.


Below is a screenshot of some of these records


We will achieve the following business benefits using Azure Machine Learning

1. Customers who are predicted to be at a higher risk of leaving (churn) can be flagged, so the customer retention teams can get it touch with them to proactively address their concerns in a bid to retain them

2. Find what factors affect churn the most i.e. out of all these fields we will determine what fields are more likely to make a leave than others

3. We will also get insights into some business rules that dictate churn i.e. the drivers

I hope you understand the problem now and find it interesting so far. Let us meet in next part of the blog where I will show how a machine learning model is created.