Question

A large number of insurance records are to be examined to develop a model for predicting fraudulent claims. Of the claims in the historical database, 1% were judged to be fraudulent.

A sample database is taken to develop a model, and oversampling is used to provide a balanced sample in light of the very low response rate. When applied to this sample database (total number of records, N =800), the model ends up correctly classifying 310 frauds, and 270 non-frauds. It misses 90 frauds, and classified 130 records incorrectly as frauds when they were not.

a. Produce the classification matrix for the sample as it stands.

b. Find the adjusted misclassification rate (adjusting for the oversampling).

c. What percentage of new records would you expected to be classified as fraudulent?

Answer 1

a.

A classification matrix sorts all cases from the model into categories, by determining whether the predicted value matched the actual value. All the cases in each category are then counted, and the totals are displayed in the matrix.

The number of frauds in real is mentioned as 1%, which means 8 of the 800 samples were actually fraud

In this case, the classification matrix is given as:

If the model predicts fraud when actually it is fraud, such records are 310.

If the model predicts non fraud when actually it is non-fraud, such records are 270

If the model predicts non-fraud when actually it is fraud, such records are 90

If the model predicts fraud when actually it is non-fraud, such records are 130.

RI 270 | 130 F 90 R-> Non-Fraud F--> Fraud 310

The matrix is 2 X 2 because there are only 2 categories, (fraud and non-fraud) else it would've been n X n matrix if there were n categories for classification.

b.

Misclassification rate:

NOTE: The misclassification rate is the rate of wrong predictions.

• Here, 130 + 90 out of the total 800 predictions were wrong, this means it had a misclassification rate of 220/800 i.e., 27.5% , with 50 % of the sites predicted as a fraud.
• Actual % of fraud= 400/800=50%
• Predicted % of fraud by model= (310+130)/800=55%
• Have to adjust non-fraud's percent in our model.

The historical percent of fraud=1%

• Then non-fraud= 99%,
• Now adjust it in our model
• To reweight to account to the actual number of 0’s and 1’s in the validation set, we need to add enough 1’s to get the original balance (1 : 100), that is
• 440 + 0.99x = x
  x = 44000
• Which means 44000 more non-fraud sites are to be added to the sample? The samples are added in the same ratio of the previous non-fraud sites.

So now, the overweighing is done and the adjusted matrix is:

Therefore, the adjusted misclassification rate is now

(130 +10890)/44000 = 25.045%

c.

Now, with a 25.045% misclassification rate, this model would end up predicting 11200 of the 44000 or 25.4545% of the new records as a fraud.