Deep Learning And ML Systems

Parameters | Parameter | Type | Required | Description | |---|---|---:|---| | task_type | {"classification","regression"} | Yes | Select target modeling type using CRISP-Goal Fit Matrix | | loss | str | Yes | Classification: binary_crossentropy, categorical_crossentropy, log_loss; Regression: mse, mae, huber_loss | | metric | str | Yes | Classification: f1, roc_auc, pr_auc; Regression: mae, rmse, r2 | | threshold | float | No | Decision boundary for classification (e.g., 0.5 default); tune on validation for desired recall/precision | | class_weight | dict[int,float] | No | Balances imbalanced classes (e.g., churn vs non-churn) for classification loss weighting | | label_encoding | {"binary","one_hot","ordinal"} | Yes (classification) | Defines label format expected by the loss | | output_activation | str | Yes | Classification: sigmoid (binary) / softmax (multi-class); Regression: linear | | target_transform | {"none","log1p","standardize"} | No | Regression-only transforms; use for heavy-tailed targets to stabilize gradients | | calibration | {"none","platt","isotonic"} | No | Post-hoc probability calibration when thresholds must be reliable |

Code Example `python import numpy as np from sklearn.model_selection import train_test_split from sklearn.metrics import f1_score, roc_auc_score, mean_absolute_error from sklearn.linear_model import LogisticRegression, Ridge

Assumption: churn labels are discrete {0,1}; churn score is not a continuous quantity.

If your label is discrete -> classification; if continuous -> regression.

---------- Classification (binary) ---------- X = np.random.randn(5000, 20) y = np.random.binomial(1, 0.18, size=5000) # churned vs retained

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

clf = LogisticRegression(max_iter=1000, class_weight="balanced") clf.fit(X_train, y_train)

p_val = clf.predict_proba(X_val)[:, 1]

churn probability threshold = 0.35

tune on validation to meet recall/precision constraints y_pred = (p_val >= threshold).astype(int)

print("ROC-AUC:", roc_auc_score(y_val, p_val)) print("F1 @ threshold:", f1_score(y_val, y_pred))

---------- Regression (poor fit example guard) ----------

If you mistakenly model discrete y with regression, you must threshold anyway.

Ridge regression outputs numeric values; you still decide churn via threshold. reg = Ridge(alpha=1.0) reg.fit(X_train, y_train) y_score = reg.predict(X_val)

print("MAE (regression loss on discrete labels):", mean_absolute_error(y_val, y_score)) `

Response Format json { "task_decision": { "task_type": "classification|regression", "justification": { "target_semantics": "discrete|continuous", "decision_cost_alignment": "thresholded_decision|magnitude_error", "poor_fit_indicator": "quantization_or_threshold_instability|misleading_magnitude_metrics" } }, "model_config": { "loss": "binary_crossentropy|mse|mae|huber_loss|log_loss", "metric": "f1|roc_auc|mae|rmse|r2", "threshold": 0.5 }, "evaluation": { "primary_metric": 0.0, "secondary_metrics": { "roc_auc": 0.0, "pr_auc": 0.0, "mae": 0.0 } } }

Notes & Best Practices - Thresholding is not optional for classification: report the metric at the tuned threshold used for decisions (especially for churn). - Avoid metric mismatch: do not use RMSE as the primary metric when your business objective is correct classification under class imbalance; use F1/PR-AUC and threshold selection. - Handle imbalance explicitly: set class_weight (or resampling) and validate with PR-AUC; accuracy can remain high while churn recall collapses. - Use calibration when probabilities drive thresholds: if you rely on p(churn) values across segments, apply Platt or isotonic calibration before

segment-level targeting. Without calibration, a single global threshold can create segment-specific bias (e.g., over-targeting low-risk cohorts).

CRISP-Goal Fit Matrix (Decision Limits) Use this matrix to decide whether classification or regression is a poor fit.

| CRISP-Goal pattern | Label form | Primary decision you must optimize | Choose | Why the other is a poor fit | |---|---|---|---|---| | Risk flag | Discrete (e.g., churn ∈ {0,1}) | “Send offer only if risk exceeds limit” | Classification | Regression magnitude (e.g., 0.18 vs 0.22) has no guaranteed meaning; you still threshold, so magnitude loss metrics mislead | | Rank ordering | Discrete | “Sort customers by likelihood” | Classification | Regression can rank, but loss like MAE/RMSE does not directly optimize ranking/threshold behavior | | Quantity estimate | Continuous (e.g., monthly spend) | “Minimize absolute error in amount” | Regression | Classification forces discretization; bin edges create artificial decision limits | | Mixed objective | Continuous target but needs “bad/good” action | “Control both action rate and error in units” | Regression + thresholded action | Pure classification loses unit-level signal; pure regression still needs an explicit threshold for actions |

Label & Loss Coupling (what to check) | Check | If it fails | Impact on decision limits | |---|---|---| | Target is discrete but treated as continuous | You measure RMSE/MAE on numeric scores | Score differences become arbitrary; thresholds dominate outcomes | | Target is continuous but converted to bins | You pick bin widths/edges | Boundary artifacts; small feature shifts flip class | | Class imbalance not addressed | Majority class dominates gradients | Decision threshold becomes unstable; recall collapses at fixed precision |

Thresholding API Contract (for churn-like flags) | Field | Type | Default | Role | |---|---|---|---| | threshold | float | 0.5 | Converts p(churn) into y_hat for cost-controlled decisions | | decision_mode | enum ("hard"|"soft") | "hard" | "soft" returns probabilities for downstream thresholding | | target_rate | float | null | Alternative selector: choose threshold to hit a desired positive rate on validation |

Link to next chapter This decision-limit framing directly feeds the next topic on overfitting vs underfitting: the same “wrong objective” (classification/regression mismatch) often looks like model error, but it is actually a loss/label design error that will persist across architectures and regularization settings.

End of chapter one. 4 more chapters in the full book.