Debugging · Medium
Detect Null Drift
Flag suspicious changes in feature null rates.
Task
Implement detect_null_drift(reference_missing: np.ndarray, current_missing: np.ndarray, threshold: float) -> np.ndarray. Repair the buggy null-drift detector so it flags feature columns whose null rate changed by more than an absolute threshold between a reference dataset and a current dataset.
Requirements
- reference_missing and current_missing are rectangular row-major boolean masks with the same number of feature columns whenever both are non-empty.
- True means the corresponding value is null and False means it is observed.
- For each feature, compute its null rate independently in each mask as true_count / row_count.
- Define every feature's null rate in an empty dataset as 0.0; infer the feature count from the non-empty mask. If both masks are empty, return an empty output.
- Flag feature j only when abs(current_rate[j] - reference_rate[j]) > threshold; equality with the threshold is not flagged.
- threshold is between 0.0 and 1.0 inclusive.
- Return flagged zero-based feature indices in increasing order, and do not mutate either input.
Example
reference_missing = np.array([
[False, True],
[False, False],
[False, False],
[False, False],
])
current_missing = np.array([
[True, True],
[True, False],
[False, True],
[True, False],
])
detect_null_drift(reference_missing, current_missing, 0.25)
# array([0])