Feature Distribution Comparison

Task

Implement feature_distribution_comparison(reference: np.ndarray, current: np.ndarray, bin_edges: np.ndarray) -> tuple[float, str]. Debug the provided feature-distribution comparison so it correctly computes Population Stability Index (PSI) over fixed bins and returns the required drift category.

Requirements

• reference and current contain finite numeric feature values; bin_edges contains finite values in strictly increasing order.
• The fixed bins are (-infinity, bin_edges[0]], (bin_edges[0], bin_edges[1]], ..., (bin_edges[-1], infinity). A value exactly equal to an edge belongs to the bin on its left.
• Compute each raw bin proportion independently as its count divided by the size of that dataset.
• Before taking logarithms, replace each raw proportion p with max(p, 1e-6). Do not renormalize the clipped proportions.
• For every bin, add (current_proportion - reference_proportion) * ln(current_proportion / reference_proportion), using the clipped proportions and the natural logarithm.
• Return (psi, "stable") when psi < 0.1, (psi, "moderate_drift") when 0.1 <= psi < 0.25, and (psi, "significant_drift") when psi >= 0.25.
• If either dataset is empty, return (0.0, "insufficient_data"). This is deliberately different from the zero-PSI result (0.0, "stable") for two identical non-empty distributions.
• An empty bin_edges list is valid and creates one bin containing every value.
• Do not mutate any input.

Example

feature_distribution_comparison(
    np.array([-2.0, -1.0, 0.0, 1.0, 2.0]),
    np.array([-2.0, -1.0, -0.5, 0.0, 0.5]),
    np.array([-1.0, 1.0]),
)
# (2.5222953447, "significant_drift")