Question

Last month your boss sent a mailing to 20,000 of your existing customers with a special offer on a Hoosfoos credeen. The response was exciting: 1% of them responded, which brought in $200,000 in revenue. She has now delegated to you the task of continuing the program, and has given you a budget of $10,000, which will allow you to target another 20,000 customers (out of your customer base of 100,000). You don’t want to just target them randomly, as your boss did. You build a tree model and a logistic regression. Describe how to evaluate them as follows. Describe (a) the confusion matrix and (b) how you will fill it out for one of the models. Describe (c) the cost/benefit matrix for this problem, including the costs and benefits for this case. (d) Show the evaluation function you will use to compare your systems. (e) How do (a) and (c) come into play in this evaluation function?

Answer 1

Once we do the modelling with tree model for the past data on scheme and the response of the customers, the model will be run on the data which were not used for training the model, then the real confusion matrix will be generated as model out put Here is the example of confusion matrix.

1. The confusion matrix is the matrix which shows the classification power of model being employed it compares predicted value along with the actual value of each cases tested.
2. Below is example of how it is field e.g out of 20000 customer base 4000 were kept for model testing purpose. And 16000 customer data were used for training the model. Below is the possible outcome of confusion matrix

Confusion matrix for the model		predicted value
		predicted NO for the offer response	predicted YES for the offer response	Total
Actual Values	Actual NO for the offer response	Count of True Negative- 800	Count of False Positive-100	900
	Actual YES for the offer response	Count of False Negative-100	Count of true positive- 3000	3100
	Total	900	3100

3. The cost benefit matrix for this case would be how much we have spent by sending the scheme communication to right customers and they responded versus customers who did not responded.

we will send scheme to only customers out of 20K customers base who have been predicted positive for the offer. And therefore, we will save scheme money from those customers who would not avail the scheme, but we will incur cost in sending scheme to them as well.

Matrix

# of scheme sent to predicted positive customers	# of customer availed the scheme	percentage conversion
3000	2600	87%

d. evaluation function will be used as to compare the percentage conversion in current system/ earlier system this will give effectiveness.

In order to cost per acquisition, it will be = Cost of targeting the customers/ number of customers converted

This will let us understand which system worked better with tree model targeting or without implementing the tree model and uniformly targeting.

The system which has lower cost of acquisition will be better.

e. a point helps us in selection of model and training the model till the time we get a better model with more true positives outcome in confusion matrix. While C point helps in finding the most cost effective way and justification of new model being implemented.