Build a baseline before choosing a complex model

A baseline establishes what a simple, reproducible method achieves under the same target and evaluation conditions as a candidate model. Without that comparison, a standalone score does not show whether additional modelling complexity has earned its place.

Our synthetic inactivity project starts with a constant probability: the positive-label fraction in the training set. There are 69 positives among 160 training snapshots, so every baseline prediction is 0.43125. This baseline uses no feature signal and no future prevalence estimate.

Keep the comparison fair

Both the baseline and logistic model use January training data, February validation data and March test data. They predict the same seven-day inactivity outcome for the same rows. The candidate model's preprocessing is fitted only on training data.

Scikit-learn's DummyClassifier with strategy='prior' provides the training-class probability baseline used here. A different task may need a stronger simple reference, such as the current business rule, a seasonal forecast or a prior-period estimate.

Do not calculate the baseline from test-label prevalence. That would give it information unavailable at the prediction time and answer a different comparison question.

Report the actual experiment

Three logistic regularization values were declared in advance: C=0.1, 1 and 10. Minimum validation log loss selects C=0.1. The selected model remains fitted on January data for this reference; there is no train-plus-validation refit.

Lower log loss is better. The model improves on this baseline in the recorded synthetic comparison. That does not establish a universal improvement, a calibrated probability model or a commercially useful intervention.

Reproduce instead of copying the score

import numpy as np
from evaluation_core import run

report,predictions,model = run()
assert np.isclose(report['training_prevalence'],69/160)
assert report['selected_C']==.1
assert all(np.isclose(row['baseline_probability'],.43125) for row in predictions)
test = report['results']['test']
assert np.isclose(test['baseline']['log_loss'],.7545819975366955)
assert np.isclose(test['model']['log_loss'],.7027491222927091)
assert test['model']['log_loss'] < test['baseline']['log_loss']
assert test['model']['recall']==.4
print(test)

Run in the included lab. The input CSV, contract, row-level predictions, versions and hashes support reproduction. The snapshots are an authored simulation rather than continuous real account histories.

Inspect the cost of the operating threshold

At the preset threshold of 0.5, the baseline predicts no positive cases because 0.43125 is below the threshold. Its recall is zero; precision is undefined because it makes no positive predictions.

The selected model identifies 22 of 55 test positives and misses 33. Its precision is about 95.65%, but recall is only 40%. A headline highlighting precision alone would conceal most of the missed inactive accounts.

The data also show changing prevalence across periods. That deserves investigation before using the model operationally. Repeatedly changing the threshold after reading test outcomes would turn the test set into another tuning set.

Exercise: add a documented business-rule baseline using only allowed historical features. Select any rule parameters on training or validation data, preserve the original test comparison, and report probability and threshold metrics separately. State the decision cost that would justify choosing the more complex model.

NeuraPath's Data Science course connects algorithms with defensible evaluation. A credible project demonstrates what improved over a meaningful baseline and what still prevents a confident operational recommendation.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Define a machine learning prediction target without future leakage.
• Continue with Train, validation and test sets: assign each a separate job.