Binary Classifier with Model Improvement

Problem Overview

This is an open-ended, hands-on machine learning coding exercise. You will train a binary classifier using scikit-learn on a toy dataset and then iteratively improve the model's evaluation metrics.

The interviewer expects you to:

1. Choose an appropriate toy dataset from scikit-learn
2. Select and train a binary classifier
3. Evaluate the model using appropriate metrics
4. Identify and implement strategies to improve performance
5. Explain your reasoning at each step

This question tests your practical knowledge of the ML workflow, understanding of evaluation metrics, and ability to iterate on model improvements.

Part 1: Dataset Selection and Initial Model

Available Toy Datasets

You can choose from any of these binary-friendly datasets:

from sklearn.datasets import (
    load_breast_cancer,  # Binary: malignant/benign, 569 samples, 30 features
    load_iris,           # Multi-class, but take 2 classes for binary
    make_classification, # Synthetic with controllable difficulty
    make_moons,          # Non-linearly separable, good for testing
    make_circles         # Concentric circles, tests non-linear models
)

Basic Setup

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Load dataset
data = load_breast_cancer()
X, y = data.data, data.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Scale features (important for many classifiers)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Part 2: Model Selection

Choose a binary classifier and fit it to the data. Common choices:

Model	Strengths	When to Use
Logistic Regression	Fast, interpretable, good baseline	Linear relationships
Random Forest	Handles non-linearity, feature interactions	Mixed feature types, no scaling needed
SVM	Works well in high dimensions	Clean data, smaller datasets
Gradient Boosting	High accuracy, handles imbalance	When accuracy is critical

Initial Training

from sklearn.linear_model import LogisticRegression

# Train initial model
model = LogisticRegression(random_state=42, max_iter=1000)
model.fit(X_train_scaled, y_train)

# Evaluate
y_pred = model.predict(X_test_scaled)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall: {recall_score(y_test, y_pred):.4f}")
print(f"F1 Score: {f1_score(y_test, y_pred):.4f}")

Part 3: Improvement Strategies

After establishing a baseline, the interviewer will ask you to improve the metrics. Here are common strategies:

Strategy 1: Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = make_pipeline(
    StandardScaler(),
    LogisticRegression(random_state=42, max_iter=1000)
)

param_grid = {
    'logisticregression__C': [0.01, 0.1, 1, 10, 100],
    'logisticregression__penalty': ['l1', 'l2'],
    'logisticregression__solver': ['liblinear', 'saga']
}

grid_search = GridSearchCV(
    pipeline,
    param_grid,
    cv=5,
    scoring='f1',  # Optimize for F1
    n_jobs=-1
)
grid_search.fit(X_train, y_train)

print(f"Best params: {grid_search.best_params_}")
print(f"Best CV F1: {grid_search.best_score_:.4f}")

Strategy 2: Try Different Models

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.svm import SVC

models = {
    'Logistic Regression': LogisticRegression(random_state=42, max_iter=1000),
    'Random Forest': RandomForestClassifier(random_state=42),
    'Gradient Boosting': GradientBoostingClassifier(random_state=42),
    'SVM': SVC(random_state=42)
}

for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    y_pred = model.predict(X_test_scaled)
    print(f"{name}: F1={f1_score(y_test, y_pred):.4f}")

Strategy 3: Feature Engineering

from sklearn.preprocessing import PolynomialFeatures
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

# Add polynomial features
poly = PolynomialFeatures(degree=2, include_bias=False)
X_train_poly = poly.fit_transform(X_train_scaled)
X_test_poly = poly.transform(X_test_scaled)  # Apply same transform to test

# Select best features
selector = SelectKBest(f_classif, k=20)
X_train_selected = selector.fit_transform(X_train_poly, y_train)
X_test_selected = selector.transform(X_test_poly)  # Apply same selection to test

# Retrain and evaluate on engineered features
fe_model = LogisticRegression(random_state=42, max_iter=1000)
fe_model.fit(X_train_selected, y_train)
y_pred_fe = fe_model.predict(X_test_selected)
print(f"F1 after feature engineering: {f1_score(y_test, y_pred_fe):.4f}")

Strategy 4: Handle Class Imbalance

from sklearn.utils.class_weight import compute_class_weight
import numpy as np

# Compute class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
weights_dict = dict(zip(np.unique(y_train), class_weights))

# Train with class weights
model = LogisticRegression(class_weight=weights_dict, random_state=42, max_iter=1000)
model.fit(X_train_scaled, y_train)

Strategy 5: Threshold Tuning

from sklearn.metrics import precision_recall_curve
from sklearn.model_selection import train_test_split
from sklearn.base import clone
import numpy as np

# Split training data into fit/validation (keep test untouched)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train_scaled, y_train, test_size=0.2, random_state=42, stratify=y_train
)

threshold_model = clone(model)
threshold_model.fit(X_fit, y_fit)

# If you calibrate, use calibrated probabilities here instead of raw scores
y_val_proba = threshold_model.predict_proba(X_val)[:, 1]

# Find optimal threshold for F1
precisions, recalls, thresholds = precision_recall_curve(y_val, y_val_proba)
f1_scores = 2 * (precisions * recalls) / (precisions + recalls + 1e-8)
best_threshold = thresholds[np.argmax(f1_scores[:-1])]

# Apply chosen threshold to test set once
y_test_proba = threshold_model.predict_proba(X_test_scaled)[:, 1]
y_pred_optimal = (y_test_proba >= best_threshold).astype(int)
print(f"Optimal threshold: {best_threshold:.3f}")
print(f"F1 with optimal threshold: {f1_score(y_test, y_pred_optimal):.4f}")

Strategy 6: Cross-Validation for Robust Evaluation

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Put scaling inside CV pipeline to avoid leakage
cv_pipeline = make_pipeline(
    StandardScaler(),
    LogisticRegression(random_state=42, max_iter=1000)
)
cv_scores = cross_val_score(cv_pipeline, X_train, y_train, cv=5, scoring='f1')
print(f"CV F1: {cv_scores.mean():.4f} (+/- {cv_scores.std()*2:.4f})")

Strategy 7: Probability Calibration

Threshold tuning assumes your probabilities are meaningful. Calibration checks whether predicted probabilities match actual event frequency (for example, among predictions near 0.8, roughly 80% should be positive).

from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.metrics import brier_score_loss
import matplotlib.pyplot as plt

# Train an uncalibrated model
base_model = RandomForestClassifier(random_state=42)
base_model.fit(X_train, y_train)
y_proba_uncal = base_model.predict_proba(X_test)[:, 1]

# Calibrate probabilities using CV to reduce leakage risk
calibrated_model = CalibratedClassifierCV(
    estimator=RandomForestClassifier(random_state=42),
    method='isotonic',  # 'sigmoid' (Platt scaling) is another common choice
    cv=5
)
calibrated_model.fit(X_train, y_train)
y_proba_cal = calibrated_model.predict_proba(X_test)[:, 1]

print(f"Brier (uncalibrated): {brier_score_loss(y_test, y_proba_uncal):.4f}")
print(f"Brier (calibrated): {brier_score_loss(y_test, y_proba_cal):.4f}")

# Reliability diagram
prob_true_uncal, prob_pred_uncal = calibration_curve(y_test, y_proba_uncal, n_bins=10)
prob_true_cal, prob_pred_cal = calibration_curve(y_test, y_proba_cal, n_bins=10)

plt.plot(prob_pred_uncal, prob_true_uncal, marker='o', label='Uncalibrated')
plt.plot(prob_pred_cal, prob_true_cal, marker='o', label='Calibrated')
plt.plot([0, 1], [0, 1], '--', color='gray', label='Perfect calibration')
plt.xlabel('Predicted probability')
plt.ylabel('Observed frequency')
plt.legend()
plt.show()

Common techniques:

• Platt scaling / sigmoid: Fits a logistic mapping from score to probability; often strong on smaller datasets.
• Isotonic regression: Non-parametric monotonic mapping; flexible but can overfit when data is limited.
• Temperature scaling: Common in deep learning; rescales logits with one temperature parameter.

Key Discussion Points

1. Choosing Evaluation Metrics

Question: "Why did you choose this particular metric?"

Answer: It depends on the problem:

• Accuracy: Good for balanced classes, but misleading for imbalanced data
• Precision: Important when false positives are costly (e.g., spam detection)
• Recall: Important when false negatives are costly (e.g., cancer detection)
• F1 Score: Balanced metric when both precision and recall matter
• AUC-ROC: Good for ranking and threshold-agnostic evaluation

2. Overfitting vs. Underfitting

Question: "How do you detect if your model is overfitting?"

Answer:

• Compare training score vs. validation/test score
• Use learning curves to visualize
• Apply cross-validation for more robust estimates

from sklearn.model_selection import learning_curve
import matplotlib.pyplot as plt
import numpy as np

train_sizes, train_scores, val_scores = learning_curve(
    model, X_train_scaled, y_train, cv=5, scoring='f1',
    train_sizes=np.linspace(0.1, 1.0, 10)
)
# Plot to diagnose overfitting/underfitting

3. Feature Importance

Question: "Which features are most important for your model?"

Answer:

import numpy as np

if hasattr(model, 'feature_importances_'):
    # Tree-based models (Random Forest, Gradient Boosting)
    importances = model.feature_importances_
    indices = np.argsort(importances)[::-1]
    for i in indices[:10]:
        print(f"{data.feature_names[i]}: {importances[i]:.4f}")
elif hasattr(model, 'coef_'):
    # Linear models (Logistic Regression)
    coefficients = model.coef_[0]
    ranked = sorted(
        zip(data.feature_names, coefficients),
        key=lambda x: abs(x[1]),
        reverse=True
    )
    for name, coef in ranked[:10]:
        print(f"{name}: {coef:.4f}")
else:
    print("Use model-agnostic methods like permutation importance or SHAP.")

4. Calibration Follow-Up

Question: "How does calibration work, and when should you use it?"

Answer:

• Calibration maps raw model scores into probabilities that better reflect real-world frequencies.
• It matters when probabilities drive decisions (risk thresholds, triage queues, expected-cost optimization), not just class labels.
• Diagnose with reliability diagrams and Brier score.
• Common methods:
  • Platt scaling (sigmoid): Parametric and data-efficient.
  • Isotonic regression: Flexible monotonic mapping, better with more calibration data.
  • Temperature scaling: Common for neural network logits.
• To avoid leakage, fit calibration on validation/CV folds (CalibratedClassifierCV) or a separate holdout split.

5. Deployment Considerations

Question: "What would you consider before deploying this model?"

Answer:

• Model serialization (pickle, joblib)
• Inference latency requirements
• Feature pipeline consistency (same preprocessing as training)
• Monitoring for data drift
• A/B testing framework
• Fallback strategy