Coursework – Building and Evaluating a Customer Churn Prediction Model

Introduction

This coursework builds and evaluates a supervised classification model predicting subscriber churn for Coastline Communications, a hypothetical Australian provider serving approximately 180,000 residential mobile and NBN customers across New South Wales, Victoria, Queensland and South Australia. The brief supplies an anonymised extract of 7,000 records and requires data preparation, comparison of three algorithms, and translation of the selected model into a decision rule a retention team could operate. The framing matters because the Australian market is close to saturation: household connectivity has plateaued at near-universal levels (Australian Bureau of Statistics [ABS], 2024), and the Australian Communications and Media Authority (ACMA, 2024) documents a retail layer in which number portability and NBN provider transfers make switching cheap and fast. Where connections are won from competitors rather than first-time users, retained margin becomes the principal growth lever, and the Telecommunications Industry Ombudsman (TIO, 2024) reports that unresolved service and billing complaints remain a leading precursor to disconnection.

The problem is framed as binary supervised classification: the unit of analysis is the account, the snapshot date is 30 June 2025, and the positive class is voluntary disconnection within the following twelve months. Two framing decisions matter more than the choice of algorithm: the label window must be long enough to capture meaningful churn yet short enough that an offer can still change the outcome, and only features observable at the snapshot date may be used, since anything later leaks future information (James et al., 2021). The success criterion is not accuracy but net retained gross margin.

Data Preparation

Source data and cleaning

The raw extract contained 7,180 account rows drawn from the billing, customer relationship management and fault-ticketing systems. Ninety-six exact duplicates, and 84 accounts opened after the snapshot date which carry zero tenure and cannot logically exhibit the twelve-month outcome, were removed, leaving 7,000 analysable records. Lifetime charges were missing for 213 accounts (3.0 per cent), of which 198 were reconstructed from tenure multiplied by the monthly charge and 15 were median-imputed by plan tier.

The cleaned file records 1,820 churn events against 5,180 retained accounts, a base rate of 26.0 per cent. Stratified sampling produced a 4,900-record training set and a 2,100-record holdout preserving that rate, so the test set holds 546 churners and 1,554 retained accounts, quarantined until final evaluation.

Feature engineering and leakage control

Eighteen source fields were expanded into 32 modelling features, with seven categorical variables one-hot encoded and derived variables capturing behaviour rather than status: a bill-shock flag triggered when the monthly charge exceeds the trailing six-month average by more than 15 per cent, a data-allowance utilisation ratio, a twelve-month fault-ticket count and remaining handset subsidy. Two candidates were excluded on leakage grounds, since a retention-call indicator is recorded only once a customer telephones to cancel and a final-invoice flag is generated by disconnection itself; confusing predictive with consequential variables is the commonest source of over-optimistic churn results (James et al., 2021).

Class imbalance

At 26.0 per cent positives the imbalance is moderate but sufficient to pull an unadjusted classifier towards the majority class, so three treatments were compared under five-fold stratified cross-validation on the training set only. Unadjusted, gradient boosting achieved precision of .731 and recall of .641 (F1 = .683), identifying fewer than two-thirds of churners because predicting retention is the safer default. Class weights inversely proportional to class frequency lifted recall to .720 while precision fell to .678, improving F1 to .698. Synthetic minority over-sampling (Chawla et al., 2002), applied strictly inside each training fold, produced indistinguishable results (precision .673, recall .716, F1 .694) with greater variance, so class weighting was retained.

Imbalance also disqualifies accuracy as a headline metric, since a classifier predicting retention for every account would score 1,554 / 2,100 = .740 while identifying no churners at all; that baseline appears in Table 1 for exactly this reason. Where the positive class is the class of interest, precision-recall behaviour is more informative than the receiver operating characteristic curve alone (Saito & Rehmsmeier, 2015).

Modelling Pipeline

Figure 1 sets out the pipeline implemented in Python 3.11 with scikit-learn 1.4, running from acquisition to deployment with a feedback loop that returns scoring outcomes to the preprocessing stage so drift can be detected and the model retrained.

Three algorithms of increasing flexibility were trained: L2-penalised logistic regression as an interpretable additive benchmark; a random forest of 500 unpruned trees grown on bootstrap samples, considering the square root of the feature count at each split, which reduces variance through decorrelation (Breiman, 2001); and gradient boosting, fitting trees sequentially to the current ensemble’s residuals with 400 trees, a learning rate of .05, maximum depth of 3 and a subsample fraction of .8, selected by randomised search over 60 draws under five-fold stratified cross-validation (Chen & Guestrin, 2016).

Model Comparison

Table 1 reports performance on the quarantined test set, with the majority-class baseline included for reference. All models were evaluated at the conventional threshold of .50 so that the comparison is like for like.

Table 1: Test-set performance of candidate churn classifiers (n = 2,100; base churn rate 26.0 per cent; threshold = .50).

Every model exceeds the .740 baseline, but the informative comparison concerns the trade-off structure. Logistic regression is weakest on all discriminatory measures, since the strongest predictors interact and an additive model cannot represent that without manually specified interaction terms. Random forest and gradient boosting are close, the .009 AUC gap being smaller than the standard deviation of the cross-validated estimates. The material distinction is that gradient boosting recovers 4.1 percentage points more recall for a .005 loss in precision, which matters because a missed churner is far more expensive than an unnecessary offer. It is therefore selected; the closeness of its cross-validated AUC of .856 to its test AUC of .861 indicates that tuning did not overfit.

Confusion Matrix and Worked Evaluation Metrics

Table 2 decomposes the gradient boosting result at the .50 threshold into its four constituent cells.

Table 2: Confusion matrix for the gradient boosting model, test set (n = 2,100), threshold = .50.

The metrics in Table 1 follow from these counts. Precision asks how many flagged accounts were genuinely at risk:

Precision = TP / (TP + FP) = 393 / (393 + 187) = 393 / 580 = .678

Recall, or sensitivity, asks how many genuine churners were captured:

Recall = TP / (TP + FN) = 393 / (393 + 153) = 393 / 546 = .720

F1 is their harmonic mean, which penalises imbalance between them more heavily than an arithmetic mean:

F1 = 2 x (precision x recall) / (precision + recall) = 2 x (.678 x .720) / (.678 + .720) = .976 / 1.398 = .698

Accuracy is (TP + TN) / n = (393 + 1,367) / 2,100 = .838 and specificity is TN / (TN + FP) = 1,367 / 1,554 = .880, so balanced accuracy of (.720 + .880) / 2 = .800 sits 3.8 points below raw accuracy, the gap being the flattery the majority class supplies. The commercially meaningful summary is lift, precision divided by the base rate, .678 / .260 = 2.61, so a flagged account is roughly 2.6 times more likely to churn than one drawn at random. The 153 false negatives nonetheless represent 28.0 per cent of churners leaving without intervention, which motivates the threshold analysis.

Feature Importance

Permutation importance was computed on the test set by shuffling each feature in turn and recording the mean decrease in AUC over 30 repetitions, normalised across the 32 features. Table 3 lists the ten most influential predictors, accounting for 92.8 per cent of measured importance; the remaining 22 contribute 7.2 per cent.

Table 3: Top ten predictors by normalised permutation importance, gradient boosting model.

The ranking is behaviourally coherent rather than merely statistical. Contractual lock-in and accumulated tenure dominate, consistent with switching-cost reasoning: customers facing no exit penalty and holding no habitual attachment are the cheapest to lose. The third and fourth ranked features are the actionable ones, because bill shock and repeated faults are service failures the provider creates and can therefore prevent, and their prominence supports the TIO (2024) observation that complaint volume tracks disconnection intent. The U-shaped utilisation effect, invisible to the logistic regression, points the same way: both allowance-exhausting and light users leave, the first underserved and the second overcharged, so a plan review rather than a discount addresses them. Because permutation importance is global, Shapley additive explanations were generated for the flagged cohort so a consultant sees why a particular customer was scored at risk (Lundberg & Lee, 2017).

Threshold Selection and the Business Cost Trade-off

The .50 threshold is a convention with no commercial standing; the operating point depends on the cost of each error type. Finance supplied three parameters: a retention offer costs A$45 per contacted account, the margin forgone when a subscriber leaves is A$620 over twelve months, and the historical acceptance rate among customers who genuinely intended to leave is 30 per cent. Correctly identifying one churner is therefore worth .30 x A$620 = A$186, while every false positive costs A$45 with no return. Table 4 evaluates four thresholds.

Table 4: Threshold sensitivity and expected net benefit, gradient boosting model, test set (n = 2,100).

The calculation at the optimum is set out in full. At a threshold of .35 the model flags 779 accounts, of which 448 are true churners:

Campaign cost = 779 x A$45 = A$35,055

Expected margin retained = 448 x .30 x A$620 = 448 x A$186 = A$83,328

Net benefit = A$83,328 – A$35,055 = A$48,273, or A$22.99 per subscriber in the test cohort

Whether to extend the campaign further is a marginal rather than an average question. Moving from .50 to .35 contacts an additional 779 – 580 = 199 accounts and converts an additional 448 – 393 = 55 true churners, a marginal precision of 55 / 199 = .276 worth .276 x A$186 = A$51.34 per contact, comfortably above the A$45 cost. Moving from .35 to .25 contacts a further 1,027 – 779 = 248 accounts for only 481 – 448 = 33 additional churners, a marginal precision of 33 / 248 = .133 worth .133 x A$186 = A$24.74, well below cost. The campaign should therefore stop at .35, notwithstanding that F1 peaks at .50. This divergence between the statistically and commercially optimal operating point is the central practical finding.

One caveat qualifies the arithmetic. The theoretical break-even probability is p* = A$45 / A$186 = .242, which would recommend operating at .25, yet empirical marginal precision in the .25 to .35 band is only .133, far below the .30 that calibrated scores would imply. The model therefore over-states risk at the low end, and isotonic calibration is required before the theoretical rule can be trusted. Extrapolated across the 180,000-subscriber base at A$22.99 per account, the program is worth approximately A$4.1 million a year, a figure to read alongside the limitations.

Ethical and Privacy Considerations

The modelling extract consists of personal information as defined by the Privacy Act 1988 (Cth), and three Australian Privacy Principles bear directly on the project. The clearest issue arises under APP 6: billing, fault and usage records are collected to supply a service, whereas churn scoring is a secondary purpose, permitted only with consent or as a reasonably expected related use. The collection notice must therefore describe retention analytics explicitly rather than rely on a generic clause (Office of the Australian Information Commissioner [OAIC], 2023). APP 1 imposes the duty of open and transparent management, and APP 11 the safeguards adopted here: identifiers were salted and hashed, name, address and date of birth were excluded from the modelling table, and access to scored output was restricted to the retention team. Amendments passed in 2024 additionally require disclosure of substantially automated decisions significantly affecting an individual’s rights or interests. A discretionary offer is not adverse in that sense, but the boundary would be crossed if the same scores rationed service or set differential pricing, so a governance rule prohibiting those uses should be documented.

Fairness was assessed rather than assumed. Postcode was excluded because it proxies for socio-economic status and, in some regions, for Indigenous status, a form of indirect discrimination the Australian Human Rights Commission (2021) identifies as characteristic of systems trained on administrative data. A disparate-impact check found contact rates of 27.8 per cent for metropolitan and 29.1 per cent for regional accounts, a gap worth monitoring since regional customers face fewer competing providers. The design was also tested against Australia’s AI Ethics Principles (Department of Industry, Science and Resources, 2024): scores inform a consultant’s conversation rather than replacing it, and any customer may seek review of an offer decision.

Limitations

Six limitations constrain these conclusions. First, the analysis rests on a single snapshot of one hypothetical provider, so the coefficients and the .35 threshold are specific to this dataset and cost structure. Second, the label conflates voluntary switching with involuntary disconnection for non-payment, which have different causes and remedies; separating them would likely improve both accuracy and intervention design.

Third, and most seriously, prediction is not persuasion. The model identifies customers likely to leave, not customers whose behaviour an offer would change, and Ascarza (2018) demonstrates that the highest-risk group is often the least responsive because the decision is already effectively made. Some of the 448 correctly flagged churners would have stayed regardless, making the offer deadweight cost. An uplift design estimated against a randomised holdout control is the correct successor to this work.

Fourth, the economic parameters are point estimates, and sensitivity analysis shows how consequential that is. If the true save rate is 20 per cent rather than 30 per cent, a true positive is worth .20 x A$620 = A$124, so net benefit at the .35 threshold becomes 448 x A$124 – A$35,055 = A$55,552 – A$35,055 = A$20,497, less than half the headline figure, and the optimal threshold reverts to .50 because the marginal band would then return only .276 x A$124 = A$34.22 against a A$45 cost. Fifth, the calibration deficiency identified above means raw scores should not be reported as probabilities until isotonic or Platt scaling has been validated. Sixth, concept drift is certain where competitive offers, NBN wholesale pricing and handset cycles all change within a modelling year, so deployment requires monthly monitoring of feature distributions, a population stability index above .20 triggering investigation, and quarterly retraining (Figure 1).

Conclusion

This coursework has built, compared and operationalised a churn classifier for a hypothetical Australian telecommunications provider. Gradient boosting proved strongest of three candidates, achieving an AUC of .861, recall of .720 and an F1 of .698 on a quarantined test set of 2,100 accounts, against a baseline scoring .740 accuracy while identifying no churners whatsoever. Feature importance points at contractual lock-in, tenure and service failure, and only the last, expressed as bill shock and repeated fault tickets, can be remedied at source rather than discounted away. The decisive step was translating classification output into an economic decision: the marginal value test identifies .35 as the operating threshold, delivering an estimated A$48,273 of net retained margin even though F1 peaks at the conventional .50 cut. The principal reservations are that the model predicts risk rather than persuadability, that its probabilities are not yet calibrated at the low end, and that the economic case is highly sensitive to a save-rate assumption only a randomised control test can establish. Those three items, rather than any further gain in AUC, define the next iteration.

References

Ascarza, E. (2018). Retention futility: Targeting high-risk customers might be ineffective. Journal of Marketing Research, 55(1), 80-98.

Australian Bureau of Statistics. (2024). Household use of information technology (Cat. No. 8146.0).

Australian Communications and Media Authority. (2024). Communications and media in Australia: Trends and developments in telecommunications 2023-24.

Australian Human Rights Commission. (2021). Human rights and technology: Final report.

Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32.

Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321-357.

Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785-794). Association for Computing Machinery.

Department of Industry, Science and Resources. (2024). Australia’s artificial intelligence ethics principles. Australian Government.

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An introduction to statistical learning with applications in R (2nd ed.). Springer.

Lundberg, S. M., & Lee, S. (2017). A unified approach to interpreting model predictions. In Advances in neural information processing systems 30 (pp. 4765-4774). Curran Associates.

Office of the Australian Information Commissioner. (2023). Australian Privacy Principles guidelines.

Privacy Act 1988 (Cth).

Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), Article e0118432.

Telecommunications Industry Ombudsman. (2024). Annual report 2023-24.