Logistic Regression

Upon completing this topic

You will be able to express a binary outcome as a relationship between probability and log-odds, and distinguish between coefficients, odds ratios, predicted probabilities, and calibration. You will not overstate the success of logistic model fitting as predictive performance or causal effect.

Structure of the binary outcome

When the outcome is recorded in two categories, reading the linear predictor directly as a probability can yield values smaller than 0 or larger than 1. Logistic regression uses a structure that places the linear predictor on the log-odds scale and converts it to a probability.

Reading coefficients

Coefficients represent the change on the log-odds scale when other inputs are held constant. Exponentiating the coefficient yields the odds ratio, but odds and probability are not the same value. Because the difference in probability can vary depending on the baseline probability, one should not explain research results using a single coefficient alone.

Python fitting

import numpy as np
import statsmodels.api as sm

X = sm.add_constant(data[["exposure", "covariate"]])
model = sm.Logit(data["outcome"], X).fit(disp=False)
probability = model.predict(X)
odds_ratio = model.params.apply(lambda value: np.exp(value))
print(probability.head())
print(odds_ratio)

This code assumes that the outcome is binary, the variables have appropriate data types, and missing data have been handled. Verify the NumPy import and the locked environment before using np.exp. Distinguish the fact that probability predictions were generated from the judgment that those predictions are well calibrated.

Diagnostics and prediction

Discrimination asks how well the model ranks or separates cases with different outcomes, while calibration asks how well the predicted probabilities match the observed frequencies. A single metric does not answer both questions simultaneously. When creating class predictions by applying a threshold, record the purpose and cost of that threshold.

Returning to the research question for interpretation

Logistic coefficients represent conditional relationships allowed by the design and model. One should not read odds ratios from observational data as if they were causal risk ratios, and should note that interpretation may vary depending on prevalence, baseline, and sample composition.

Common failures

• Do not use odds and probability interchangeably.
• Do not treat coefficients as established biological effects.
• Do not trust predicted probabilities without calibration.
• Do not report only accuracy in the presence of class imbalance.

Key takeaways

• Logistic regression models the relationship between a binary outcome and log-odds.
• Coefficients, odds ratios, and probabilities are on different scales.
• Discrimination and calibration must be separated.

Next topic

The next section covers count, rate, offset, and overdispersion.

References

• statsmodels API: https://www.statsmodels.org/stable/api.html
• scikit-learn calibration: https://scikit-learn.org/stable/modules/calibration.html
• scikit-learn metrics: https://scikit-learn.org/stable/modules/model_evaluation.html

The explanation and data structure in this section were written independently by BioStatPy.