Skip to content

MRMR research synthesis

Three parallel research agents produced MRMR / feature-selection findings. Below: clean markdown extracts of each agent's final report, with priorities + cross-references. Kept as a literature survey + FS competition matrix; the actionable proposals have largely shipped.

Shipping status (verified against feature_selection/filters/). ✅ DONE: JMIM (_jmim_scorer.py, _orthogonal_jmim_fe.py), RelaxMRMR / 3-way redundancy (_relaxmrmr_3d.py), MRwMR-BUR (_bur_term.py), Chao-Shen entropy (_chao_shen.py), KSG estimator (_ksg.py, mi_estimator="mixed_ksg"), CMI-permutation stop + UAED elbow (_cmi_perm_stop.py), Conditional Permutation Test (_conditional_permutation.py), PID decomposition (_pid_decomposition.py), adaptive bin-count + Knuth / Bayesian-Blocks binning (_adaptive_nbins.py, discretization/), interaction-information / JMI term (_interaction_information.py), Cluster Stability + Complementary-Pairs SS (_stability_cluster.py, stability_selection_method='complementary_pairs'), Sotoca-Pla hierarchical CMI clustering (_dynamic_cluster_discovery/, dcd_distance='sotoca_pla'), and kernel_tuning_cache'd cluster thresholds (DCD layer's dcd_tau_cluster, and _cluster_aggregate.py's corr_threshold/homogeneity_tau). Adaptive nbins is complete: Sturges/rice/sqrt (discretization/_discretization_edges.py:histogram, numpy schemes), Knuth + Bayesian Blocks (same module), and MDLP/Fayyad-Irani/optimal_joint (supervised nbins_strategy options in discretization/_discretization_dataset.py). Inf-FS eigenvector-style centrality re-rank shipped 2026-07-20: friend_graph.py's _inf_fs_centrality (Roffo et al. 2017 infinite-walk score on the MI-weighted adjacency), exposed as FriendGraphNode.centrality + FriendGraph.low_centrality. Matrix-based Rényi α-entropy CMI (Yu et al. 2020) shipped 2026-07-20 as _renyi_alpha.py, opt-in via estimator='renyi_alpha' in _mi_dispatch.py; benchmarked against plug-in/mixed-KSG on a closed-form bivariate-Gaussian small-n fixture (bench_renyi_alpha_small_n.py) — it is NOT a general win (mixed-KSG dominates at low-to-moderate correlation, and Rényi's O(n^3) eigendecomposition is ~100-1000x slower than KSG at n=500-1000), but it is materially more accurate than both plug-in and KSG at high dependency (rho=0.9: mean_abs_err 0.04-0.07 vs 0.28-0.42) — a validated narrow-niche opt-in, same pattern as Chao-Shen, not a default-flip candidate. The six FS methods mapped in Agent C's competition matrix all exist (boruta_shap, wrappers/_knockoffs, wrappers/_univariate_ht, shap_proxied_fs, wrappers/rfecv, MRMR filters); the synthetic bench scenarios are partially realized under _benchmarks/.

Still forward research (NOT started): Quadratic-MI / Cauchy-Schwarz plugin-free estimator (#4, QMIFS) — deliberately not built alongside Rényi: both target the same "plug-in-free continuous-MI" gap, Rényi (Yu et al. 2020) is the more general / more cited of the two (also covers the CMI form MRMR actually needs), and building both would duplicate validation effort for no distinct win over the single estimator already shipped. The full 6-method x 15-relationship empirical competition matrix (Agent C's benchmark scenarios) remains UNBUILT — the 15 dataset generators are described in _benchmarks/_datasets.py-style detail in this doc's Agent C section below, but wiring all 6 selectors (RFECV / ShapProxiedFS / BorutaShap / Knockoffs / UnivHT / MRMR, each with a materially different fit API) through a uniform empirical harness is a multi-day integration effort the doc itself frames as "foundation for auto-tune" future work, not a correctness gap in any shipped selector. FCBF-style ordered pruning of _cluster_aggregate shipped 2026-07-18 (rejects chain-transitivity artifacts in _discover_clusters; see audits/mrmr_audit_2026-07-16/06_docs_backlog_drift.md). Everything else below has landed — the surveys, weakness analyses, and the 6-method × 15-relationship competition matrix are kept for reference and future estimator work.

This file consolidates the final reports for easier reading.

Agent A — MRMR simple wins (14 findings, 2020-2025)

Context: mrmr.py is 961 LOC, at split threshold from MEMORY.md — every new item must land in a new sibling file, never inside mrmr.py.

A. Better scoring criteria (alternatives to Fleuret CMI)

1. JMIM — Bennasar, Hua, Setchi 2015 ("Feature selection using Joint Mutual Information Maximisation", Expert Systems with Applications 42(22), ~1500+ citations). Replaces JMI's sum of pairwise joint MI with the minimum: J_JMIM(X_k) = min_{X_j in S} I(X_k, X_j ; Y). Brown 2012's framework shows JMI/CMIM/DISR are all special cases of conditional likelihood maximisation; JMIM reduces relative classification error by ~6% vs the next best in the original paper. - Why us: plugs directly alongside existing fleuret.py; the min aggregator is more robust than Fleuret's sum when one already-selected feature is strongly correlated. - LoC: ~80, sibling _jmim_scorer.py. Risk: changes selected order → repro break behind opt-in redundancy_aggregator="jmim". Priority: HIGH.

2. RelaxMRMR / FJMI — Vinh, Zhou, Chan, Bailey 2016 (Pattern Recognition 53, ~300 citations). Adds a 3-D MI redundancy term I(X_k;X_j;X_i|Y) — relaxes the conditional-independence assumption Fleuret uses. - Why us: mlframe already computes 2-D joint histograms via batch_pair_mi_gpu.py; extending to a 3-D batched joint hist on GPU is incremental and would catch higher-order redundancy that single-pair Fleuret misses. - LoC: ~250 (sibling _relaxmrmr_3d.py + a 3-D CUDA kernel under the existing dispatcher). Risk: memory blowup for 3-D hist; needs guard by min(n_bins**3, threshold). Priority: MEDIUM (gain only when 3-way redundancy exists).

3. MRwMR-BUR — Gao et al. 2022 (arXiv:2212.06143; cs.LG; cited ~30, preprint). Adds an explicit "unique relevance" term boosting features whose contribution cannot be explained by any already-selected feature. Reports +2-3% accuracy with 25-30% fewer features selected. - Why us: complements SU normalisation directly; the BUR term is computed from quantities mlframe already has (per-feature I(X;Y) and joint I(X,S;Y) from Fleuret). - LoC: ~120 (_bur_term.py). Risk: repro-break behind flag; preprint citation count borderline — verify after final submit. Priority: MEDIUM-HIGH.

4. Quadratic MI / Cauchy-Schwarz (QMIFS) — Sluga & Lotrič 2017 (Entropy 19(4):157, ~150 citations). Plugin-free MI estimator via Gaussian kernels + Cauchy-Schwarz divergence; handles discrete and continuous uniformly without discretisation. - Why us: mlframe currently must discretise via discretization.py even for continuous targets; QMIFS would let regression problems skip binning entirely. - LoC: ~150 sibling _qmi_cs.py. Risk: new dep (none — pure numpy/numba), but O(n²) without Nyström approx; gate by n<20k. Priority: MEDIUM.

5. Matrix-based Rényi α-entropy CMI — Yu, Giraldo, Jenssen, Príncipe 2020 (IEEE TPAMI, ~250 citations). Estimates joint/conditional entropy from eigen-spectrum of a kernel Gram matrix; no histogram, no plug-in bias, scales to multi-variate naturally. - Why us: mlframe's current MI estimation chain (info_theory.py + Miller-Madow) is plug-in; α-entropy beats plug-in on small samples (n<500) which is where Besag-Clifford early-stop is least reliable. - LoC: ~180 sibling _renyi_alpha.py. Risk: O(n²) Gram matrix; new dep on scipy.linalg.eigh (already in deps). Priority: MEDIUM (paired with finding #7 below). - Status (2026-07-20): ✅ Shipped: _renyi_alpha.py's renyi_alpha_mi/renyi_alpha_cmi, opt-in via estimator='renyi_alpha' in _mi_dispatch.py. Benchmarked (bench_renyi_alpha_small_n.py) against plug-in/mixed-KSG on a closed-form bivariate-Gaussian fixture: NOT a general win (mixed-KSG dominates at low/moderate correlation; ~100-1000x slower than KSG at n=500-1000 from the O(n^3) eigendecomposition), but materially more accurate at high dependency (rho=0.9: mean_abs_err 0.04-0.07 vs plug-in/KSG's 0.28-0.42) — a validated narrow-niche opt-in, same disposition pattern as Chao-Shen (#13 above), not a default-flip candidate.

B. Better MI estimation for small samples

6. KSG / Kraskov estimator (NeurIPS 2023 evaluation) — Czyż et al., "Beyond Normal: On the Evaluation of MI Estimators". Confirms KSG has the lowest sample requirements among non-neural estimators on smooth continuous distributions. - Why us: mlframe's MI is histogram-only (quantile/uniform binning); KSG removes binning bias on continuous features and is already shipped in sklearn (feature_selection.mutual_info_regression). - LoC: ~60 (wrapper in _ksg_estimator.py calling sklearn). Risk: zero new dep (sklearn already in deps); ~5-20× slower per pair so only use for screened shortlist, not the full O(p²) pass. Priority: HIGH (cheap and orthogonal to the SU win). - Status (2026-07): ✅ Shipped as _ksg.py's mixed_ksg_mi/ksg_lnc_mi, opt-in via mi_estimator="mixed_ksg"/"ksg_lnc" in _mi_dispatch.py.

7. Bayesian-block / Knuth adaptive binning — Scargle Bayesian Blocks (cited 1000+), Knuth 2019 rule. Used in astropy. Optimises bin edges to maximise a marginal likelihood, handles skewed/multimodal data where Freedman-Diaconis fails. - Why us: mlframe's discretization.py ships only quantile + uniform; for heavy-tailed regression targets both produce singleton-mass bins that inflate MI estimates. - Status (2026-05-28): ✅ Native Knuth + Bayesian Blocks shipped at discretization.py:histogram().

C. Better stopping criteria

8. CMI-permutation + CMI-heuristic — Yu & Príncipe 2019 ("Simple stopping criteria for information theoretic feature selection", Entropy 21(1):99, ~80 citations). Stops when I(X_candidate; Y | S_selected) — estimated via matrix-based Rényi — is not significantly larger than a permutation null. - Why us: mlframe already runs Besag-Clifford; this work fuses CMI estimation + permutation into one step, giving an automatic stop without a threshold * H(y) knob. Eliminates one hyperparameter. - LoC: ~90 in permutation.py (sibling exists, just add cmi_stop_mode='renyi'). Risk: none if Rényi is opt-in. Priority: HIGH.

9. Spectral Information Criterion / UAED elbow detection — Llorente, Martino et al. 2023 (arXiv:2308.09102 / Signal Processing 2024). Universal automatic elbow detector that generalises AIC/BIC for arbitrary error curves. - Why us: mlframe's min_features_fallback is a hard floor; UAED would auto-pick subset size from the CMI-gain curve when the user provides n_features=None. - LoC: ~70 in stability.py. Risk: none (pure numpy). Priority: MEDIUM.

D. Better permutation test methodology

10. Conditional Permutation Test (CPT) — Berrett, Wang, Barber, Samworth 2020 (JRSS-B 82(1), ~400 citations). Permutes X conditional on Z, preserving the X|Z distribution; gives valid p-values under arbitrary confounding. - Why us: mlframe's Besag-Clifford permutes the candidate column unconditionally, which inflates Type-I error when the candidate is correlated with already-selected features. CPT is the principled fix — directly relevant to the redundancy-controlled selection MRMR is designed for. - LoC: ~160 sibling _conditional_permutation.py. Risk: needs an MCMC-style permutation; compute cost +3-5× per test; gate to last-stage confirmation only. Priority: HIGH (methodologically correct test for MRMR's redundancy claim).

E. Group / structured feature selection

11. Cluster Stability Selection — Faletto & Bien 2022 (arXiv:2201.00494, JMLR 2024). Pre-clusters highly-correlated features then applies stability selection at the cluster level; gives selection-frequency error bounds (Shah-Samworth 2013). - Why us: mlframe has _cluster_aggregate.py and friend_graph.py already — the infrastructure is there; cluster-level stability bounds would make the existing cluster scoring statistically rigorous. - LoC: ~150 wiring in stability.py. Risk: none. Priority: HIGH — leverages existing mlframe primitives.

12. Complementary Pairs Stability Selection — Shah & Samworth 2013 (JRSS-B 75(1), ~1500 citations). Run selection on B random half-splits + complements; derive a tight error bound on falsely selected features without exchangeability assumptions. - Why us: mlframe's Besag-Clifford controls per-feature Type-I but not family-wise error across all candidates; Shah-Samworth gives a defensible FWER bound. - LoC: ~100 in stability.py. Risk: doubles permutation budget. Priority: MEDIUM.

F. Estimator-choice and unique-vs-synergistic decomposition

13. Mutual Information Estimator empirical study — Pawluszek-Filipiak et al. 2025 (MDPI Information 16(9):724). Benchmarks Miller-Madow, Chao-Shen, Shrinkage, Jackknife on mRMR; concludes corrected estimators (esp. Chao-Shen and Shrinkage / James-Stein) outperform Miller-Madow on small/sparse contingency tables. - Why us: mlframe currently exposes only Miller-Madow as the opt-in correction; Chao-Shen handles unseen-symbol bias that MM doesn't. - LoC: ~80 added to info_theory.py (sibling _chao_shen_entropy.py). Risk: none, gated by entropy_correction='chao_shen'. Priority: HIGH (direct extension of the just-shipped SU work, same shape of win). - Status (2026-07-16): ✅ Shipped: info_theory/_entropy_kernels.py's entropy_chao_shen/mi_chao_shen + _class_mi_kernels.py's classes-based path, opt-in via set_mi_chao_shen(True) (thread-local, mirrors use_mi_miller_madow). Benchmarked against Miller-Madow and plug-in (_benchmarks/bench_mi_correction_chao_shen_vs_miller_madow.py, see audits/mrmr_audit_2026-07-16/12_chao_shen_benchmark_results.md): no measurable accuracy difference on the tested scenarios, so the package default stays "none" — this is a validated opt-in, not a default-flip candidate. Independent of _chao_shen.py's pre-existing chao_shen_entropy_from_counts/_joint_chao_shen_mi_njit, which back a separate joint-MI path.

14. PID-based redundancy/relevance — Wollstadt, Schmitt, Wibral 2023 ("A rigorous information-theoretic definition of redundancy and relevancy in feature selection based on (partial) information decomposition", JMLR 24(131):1-44, ~50 citations). Decomposes I(X_set;Y) into unique / redundant / synergistic components via Williams-Beer PID. - Why us: mlframe's cat_interactions step finds synergistic pairs after the fact; PID provides a principled criterion to detect synergy during selection so synergistic features aren't filtered out. - LoC: ~250 sibling _pid_decomposition.py. Risk: PID definition not unique (BROJA, Iccs, MMI variants); pick one and document. Priority: MEDIUM (high theoretical value, larger effort).

Findings #13 (Chao-Shen), #1 (JMIM), #6 (KSG wrapper), #8 (CMI-permutation stop), and

11 (Cluster Stability Selection) have all shipped — see the per-finding ✅ status notes above.

All can land as sibling files under feature_selection/filters/ per the monolith-split rule.

Sources

Agent B — Auto-clustering critique

Current state

The MRMR redundancy logic is split across four independent layers with non-overlapping cluster semantics, which is the headline structural finding:

  1. Greedy Fleuret per-Z redundancy (filters/evaluation.py:230-339, filters/fleuret.py:184-257). For each candidate X, gain is min over Z in selected_vars of I(X; Y | Z). This is the canonical Fleuret 2004 / Brown 2012 CMIM criterion. There is no clustering here at all — the redundancy lookup is one-feature-at-a-time. A genuine collinear cluster {Z1, Z2, Z3} triggers three identical vetoes (no extra suppression beyond the min).

  2. Friend graph post-hoc diagnosis (filters/friend_graph.py, 480 LOC). Built after the greedy loop on the already-selected set. Edge gate at pairwise_mi_edge:165-188: pairwise MI must clear max(mi_eps=1e-6, edge_significance=3.0 * (na-1)(nb-1)/(2n)) (a G-test bias floor). Node classification at lines 332-353. No connected-component analysis, no transitive closure, no graph partitioning. prune_by_friend_graph:368-409 is worst-first node removal protecting the single "justifier" neighbor — it is not a cluster operation.

  3. Cluster-aggregate FE step (filters/_cluster_aggregate.py, 315 LOC). The only place that actually clusters: _discover_clusters:143-222 builds an edge list gated by |Pearson corr| >= corr_threshold=0.6 AND pairwise_mi_edge, then runs naive union-find connected components (_connected_components:123-140). A component is accepted only if PC1 explains >= homogeneity_tau=0.6 of standardized variance. Members are aggregated (mean_z / PCA-PC1 / Bartlett) and the aggregate is augmented onto the selected set, not used to deduplicate it.

  4. Operator-supplied feature_groups in RFECV (wrappers/rfecv/_finalize.py / _validate.py / _configs.py). All-or-nothing post-fit support expansion over user-declared disjoint groups. Completely orthogonal to MRMR; there is no data-driven equivalent inside MRMR.

The friend graph and cluster-aggregate paths share the pairwise_mi_edge primitive but use different thresholds and different decision criteria for the same underlying question "are these two features redundant?".

Critical gaps

  • ~~The Fleuret per-Z min I(X;Y|Z) criterion is provably brittle on multi-collinear groups.~~ — shipped: MRMR(redundancy_aggregator='jmim') (Bennasar 2015) and the data-dependent redundancy_aggregator='auto' gate (_synergy_detector.detect_synergy) address this; see filters/fleuret.py:4-9 and the benchmarked-decision note there. When {Z1, Z2, Z3} are noisy reflections of one latent z, each gives I(X;Y|Z_k) ≈ I(X;Y|z), so plain-Fleuret's min_k underestimates the true conditional MI exactly when there is most redundancy to clean up — Brown et al. 2012 (JMLR) show CMIM (min_k) is dominated by JMI (sum_k I(X, Z_k; Y)) on small samples, motivating the JMIM fix.

  • Two incompatible redundancy thresholds for the same edge. Friend graph uses an MI significance floor only. Cluster-aggregate requires both |corr| >= 0.6 and the same MI floor. A non-linear functional dependency Y=f(X) with corr ~ 0 passes friend-graph but is rejected by cluster-aggregate. The "shared primitive" claim hides a semantic split.

  • No transitive / hierarchical cluster structure anywhere. Cluster-aggregate uses single-linkage connected components on a hard-thresholded graph. Single-linkage on continuous similarity is known to "chain" — one weak edge merges two unrelated cliques. Sotoca & Pla 2010 and FCBF (Yu & Liu 2003) both use rank-then-prune or agglomerative complete-linkage on a CMI-based distance precisely to avoid this.

  • Threshold values are dev-machine constants. corr_threshold=0.6, homogeneity_tau=0.6, garbage_min_degree=3, unique_ratio=1.0 are hardcoded defaults with no data-driven calibration. Per the project memory's kernel_tuning_cache rule, these should route through a calibration cache.

  • ~~No protection against synergy destruction.~~ — shipped: plain-Fleuret's min criterion does reject synergistic pairs on its own (acknowledged in fleuret.py:4-7), but redundancy_aggregator='jmim'/'auto' (same fix as above) keeps synergies because JMI/JMIM sum the joint I(X,Z;Y) rather than minimize a conditional (Brown 2012), and the benchmarked auto gate routes to it exactly when synergy is detected.

  • Friend-graph "sink" detection scales O(k²) and skips when k > 200. For typical MRMR runs that select 100-300 features this is just barely on — it silently degrades to "node stats only" with no clustering output. There is no fallback to a sparse k-NN graph (Roffo 2017 ICCV recommends keeping only top-k edges per node).

  • ~~No graph-Laplacian / eigenvector-centrality global ranking.~~ — shipped 2026-07-20: friend_graph.py's _inf_fs_centrality computes Roffo's Inf-FS infinite-walk score (S = sum_r alpha^r A^r on the MI-weighted adjacency) and flags the bottom centrality_percentile (default 5%) as FriendGraph.low_centrality. Diagnostic only — does not change selected_vars.

Recommendations

  • ~~Add JMI / JMIM as a mrmr_redundancy_algo option~~ — shipped as redundancy_aggregator='jmim' (see the JMIM section above; the constructor knob is redundancy_aggregator, not mrmr_redundancy_algo).

  • ~~Replace single-linkage components with FCBF-style ordered pruning~~ — shipped 2026-07-18 in _cluster_aggregate.py / _discover_clusters (see audits/mrmr_audit_2026-07-16/06_docs_backlog_drift.md).

  • ~~Add Sotoca-Pla hierarchical CMI clustering as an alternative discovery backend~~ — shipped as dcd_distance='sotoca_pla' in _dynamic_cluster_discovery/.

  • ~~Add Inf-FS eigenvector-centrality re-ranking pass on the friend graph~~ — shipped 2026-07-20, see above.

  • ~~Route all four thresholds through pyutilz.system.kernel_tuning_cache~~ — shipped: DCD layer's dcd_tau_cluster and _cluster_aggregate.py's corr_threshold/homogeneity_tau are kernel_tuning_cache'd.

Sources

Agent C — FS competition map (6 methods × 15 relationship types)

Per-method mathematical view + strengths/weaknesses

A. MRMR (Fleuret variant) — filters/mrmr.py, filters/fleuret.py, filters/evaluation.py

Math. Greedy forward selector on discrete (binned) X, y. Per candidate computes gain = I(X;Y) - max_{k in S} I(X;Y | S_k) (Fleuret/CMIM formulation), validated via permutation null. Assumes informative interactions are at most pairwise with already-selected variables (max_veteranes_interactions_order defaults to 1) and that MI estimated on binned data approximates continuous MI well.

Strengths. 1. Captures monotone-or-not nonlinear marginals (MI is invariant to monotone transforms) 2. Removes pure redundancy (a perfect duplicate gets gain≈0) 3. Handles multiclass y natively 4. Permutation-confidence step gives p-value-like stopping 5. SU-normalized variant balances entropy of high-cardinality predictors

Weaknesses. 1. Explicitly rejects pure synergy / XOR — file docstring already names this 2. Binning destroys very-fine threshold signal at default nbins 3. max_k is sensitive — one already-selected near-duplicate kills the candidate (CMIM pessimism, see Brown 2012) 4. Group-additive cluster effects (need majority vote across cluster) get suppressed once one cluster member is in 5. Pure linear-Gaussian signal in high-D gets out-ranked by lower-MI but discrete signals because of binning bias

B. RFECV wrapper — wrappers/rfecv/

Math. Model-in-the-loop backward (or MBH heuristic-search) elimination, ranks features by an importance signal from the trained estimator (gain / coef / permutation), CV-votes across folds, picks subset size by averaged CV score. Assumes model importance is monotone in true relevance.

Strengths. 1. Native to whatever model will ship (tree gain ≈ what CatBoost actually uses) 2. Picks up multivariate signals the model can fit (tree-friendly thresholds, sigmoidal, additive) 3. CV votes give noise robustness 4. MBH supports non-greedy steps so it can escape one-feature-at-a-time traps 5. must_include lets domain priors carry through

Weaknesses. 1. Splits importance across correlated copies — both look unimportant individually, both eliminated (NCBI RF-RFE study) 2. Unstable across folds when many correlated features (sklearn docs) 3. Inherits the base model's blind spots (linear model can't catch XOR; tree can't catch smooth high-order interactions efficiently) 4. Cost = O(p · CV · model_fit) 5. No FDR guarantee

C. ShapProxiedFS — shap_proxied_fs.py

Math. Fit one big model on all features, compute OOF SHAP phi_ij. Approximate the score of subset S by base + sum_{j∈S} phi_ij (Shapley additive coalition proxy à la Mazzanti). Optimize subset via brute force / beam / GA / annealing, then honestly re-validate the top-N on a held-out split.

Strengths. 1. Searches in subset space, not feature space — natively handles compensation (drop A, keep B that captures same signal) 2. Cheap because no retrain per subset 3. Disjoint re-validation guards against proxy over-fit 4. Supports interaction-aware mode via treeshap_interactions 5. Active learning / uncertainty penalty for proxy quality

Weaknesses. 1. Additive coalition assumption breaks under strong interactions — SHAP linearity axiom is exactly the failure mode (Kumar 2020) 2. Proxy mis-credits features whose correlated survivors would compensate (<50% coverage wall, called out in docstring) 3. Requires a tree model that supports tree-SHAP 4. High-dim n_features > 22 falls back to heuristic search — no exactness guarantee 5. Useless on data where SHAP itself misranks (e.g. categorical with many rare levels under TreeSHAP path-dependent attribution)

D. BorutaShap — boruta_shap.py

Math. Permute each feature to make "shadow" features, fit RF/GBM, retain originals whose importance is statistically greater (binomial-test over rounds) than the max shadow. Importance can be Gini or mean-|SHAP|. Assumes "max-shadow" is a valid null for a relevant feature.

Strengths. 1. "All-relevant" set, not "minimal" — keeps backups for downstream stability 2. Captures nonlinear + tree-fittable interactions because the underlying RF/GBM does 3. FWER-style control via binomial p-values + BH/BY 4. Robust to monotone transforms 5. SHAP mode reduces bias toward high-cardinality features that plagues Gini

Weaknesses. 1. Keeps both members of a redundant pair — kxy.ai: "considered both highly correlated variables important" 2. Blind to XOR if RF depth/n_trees insufficient 3. Shadow-max null is loose at high p → low power 4. Cost = many RF refits 5. No subset-quality metric — only per-feature accept/reject

E. Knockoffs (Barber-Candès) — wrappers/_knockoffs.py

Math. Build X̃ with same corr(X̃_j, X_{-j}) = corr(X_j, X_{-j}) but X̃ ⫫ y | X. For any importance Z_j, Z̃_j, statistic W_j = Z_j - Z̃_j has IID-sign null. Threshold gives finite-sample FDR ≤ q. mlframe uses equicorrelated fixed-design Gaussian knockoffs.

Strengths. 1. Provable finite-sample FDR — unique in this lineup 2. Couples with any importance W (gain, coef, SHAP) 3. No multiple-testing pessimism vs BY-FDR 4. Detects features whose marginal MI is near zero but joint signal is strong 5. Works for any p < n (fixed-design path)

Weaknesses. 1. Equicorrelated construction only valid for ~Gaussian X — RANK/JASA 2020 explicitly addresses this 2. Degrades severely under heavy multicollinearity (lambda_min(Σ) → 0 → s → 0 → W → 0, no power; module raises) 3. Requires numeric X 4. Power depends entirely on the importance statistic — bad W kills FDR power 5. n < p requires model-X variant (not yet wired here)

F. Univariate HT prescreen — wrappers/_univariate_ht.py

Math. Per-feature test: Mann-Whitney U (binary y, numeric x), Kruskal-Wallis (multiclass y, numeric x), Kendall τ (continuous y, numeric x), χ² (categorical x, discrete y). BY-FDR across p tests. Assumes the marginal (X_j, y) distribution carries the signal.

Strengths. 1. Cheap — O(n p log n) 2. Distribution-free (rank-based) 3. Correct FDR control under arbitrary dependency (BY) 4. Strong on linear-monotone and unimodal-class-separable signals 5. Great as a prescreen upstream of expensive methods

Weaknesses. 1. Completely blind to XOR / pure interaction (univariate) 2. Blind to V-shaped / non-monotone signals (Kendall = 0) 3. Cannot rank redundant features against each other 4. Misses sigmoid/threshold where median split doesn't separate 5. No coverage of group-additive effects where only the SUM is informative

Competition matrix

W = wins (rank-1) / T = ties (detects but doesn't dominate) / L = misses

# Relationship type MRMR RFECV ShapProxied Boruta Knockoffs UnivHT
1 Linear additive marginal, continuous y T W W W W W
2 Nonlinear monotone marginal (log/sqrt) W W W W T W
3 Non-monotone unimodal (V/U-shape) W W T W T L
4 Threshold step / piecewise constant W W T W T T
5 Sigmoid / tanh marginal T W W W W W
6 2-way XOR (pure synergy, no marginal) L T T T L L
7 3-way XOR / parity L L L L L L
8 Multiplicative interaction (x1·x2) T W T W T T
9 Two perfect duplicates (redundancy) W L W L L L
10 Block of 5 correlated copies, one signal W L W L L L
11 Group-additive (sum of 10 weak features) L W W W W T
12 High-cardinality categorical, target-mean signal T W W T L T
13 Heavy-tail/skew predictor, linear y T T T T L W
14 Many noise + 1 strong signal (n_inf=1, p=1000) W T W W W W
15 Mixed-relevance under known FDR budget T L T T W T

Synthetic benchmark scenarios

Each discriminates ≥ 2 methods orthogonally. Pseudocode in original report; condensed here:

# Name n / p / n_inf y formula Discriminates
1 bench_dataset_xor_2way 8000 / 50 / 2 y = x1 XOR x2 RFECV/Boruta/Shap vs MRMR/Univ/Knockoffs
2 bench_dataset_xor_3way_parity 20000 / 100 / 3 y = x1⊕x2⊕x3 All fail except RFECV+deep
3 bench_dataset_threshold_step 5000 / 200 / 1 y = 1[x_5 > 1.3] UnivHT loses (weak τ)
4 bench_dataset_v_shape 5000 / 200 / 1 y = x_5² UnivHT loses (τ=0)
5 bench_dataset_redundant_duplicates 4000 / 50 / 1+5dup y = x_1; x_2..x_6 = x_1 MRMR/Shap win, Boruta/RFECV lose
6 bench_dataset_correlated_block 4000 / 50 / 1+10corr y = x_1; block corr=0.95 Knockoffs fails (Σ near-singular)
7 bench_dataset_group_additive 8000 / 200 / 10 y = sum(x_1..x_10) MRMR loses (per-marginal too weak)
8 bench_dataset_sigmoid_marginal 5000 / 100 / 3 y ~ Bernoulli(σ(...)) Calibration check
9 bench_dataset_high_card_cat 10000 / 30 / 1 cat 200 levels; y = target_mean[cat] Knockoffs fails (numeric only)
10 bench_dataset_heavy_tail 4000 / 100 / 3 x ~ t(2); y = sum(x) Knockoffs fails (Gaussian assumed)
11 bench_dataset_fdr_under_budget 2000 / 500 / 20 Linear with FDR target Only Knockoffs guarantees
12 bench_dataset_p_gg_n_sparse 300 / 5000 / 5 Linear, sparse Knockoffs (model-X) wins
13 bench_dataset_compensable_pair 5000 / 100 / 2 y=x_1+x_2; corr(x_1,x_3)=0.9 ShapProxied uniquely best
14 bench_dataset_pure_noise 2000 / 200 / 0 Random y Type-I check
15 bench_dataset_mixed_realistic 10000 / 300 / 20 mixed Combined types Profile-level

Sources

Cross-agent prioritised backlog

Items ranked by cost/benefit, combining all 3 agents.

Pri Item Source LOC Expected biz value
✅ done JMIM redundancy as redundancy_aggregator='jmim' Agent A #1 + Agent B critic ~80 Fixes Fleuret-CMIM brittleness on multi-collinear (acknowledged TODO)
✅ done Chao-Shen entropy correction (opt-in, no measured default-flip win — see finding #13 above) Agent A #13 ~80 Same impact-shape as SU; sparse-contingency bias fix
✅ done FCBF ordered pruning in _cluster_aggregate.py Agent B #2 ~40 Replaces chaining-prone single-linkage
✅ done Adaptive nbins (Sturges / Freedman-Diaconis / Knuth / Bayesian Blocks / Fayyad-Irani / OptimalJoint) User Q1 + Agent A #7 ~250 Eliminates fixed-10-bin compromise (SU normalization default is mi_normalization="none", not forced-on)
🟡 deferred 15 synthetic bench scenarios from Agent C Agent C ~600 Foundation for auto-tune (DataFingerprint → best method); UNBUILT — wiring 6 selectors with heterogeneous fit APIs through one harness is a multi-day integration effort, not a correctness gap (see note above)
✅ done kernel_tuning_cache for cluster thresholds Agent B #5 ~50 Per memory rule; eliminates hardcoded magic
✅ done Conditional Permutation Test (_conditional_permutation.py) Agent A #10 (Berrett 2020) ~160 Methodologically correct test for MRMR's redundancy
✅ done Inf-FS centrality re-rank (friend_graph.py's _inf_fs_centrality) Agent B #4 ~60 Global structural score replaces local degree heuristic
✅ done KSG estimator wrapper (mi_estimator="mixed_ksg"/"ksg_lnc") Agent A #6 ~60 Bypasses binning for continuous-y
✅ done Cluster Stability Selection (stability_selection_method='cluster') Agent A #11 ~150 Leverages existing mlframe cluster infra
✅ done Sotoca-Pla hierarchical CMI clustering (dcd_distance='sotoca_pla') Agent B #3 ~150 Alternative discovery backend, complete-linkage
✅ done RelaxMRMR (3D MI) (_relaxmrmr_3d.py) Agent A #2 ~250 3-way redundancy detection on GPU
✅ done PID-based redundancy/relevance (_pid_decomposition.py) Agent A #14 ~250 Explicit synergy detection
✅ done MRwMR-BUR (_bur_term.py) Agent A #3 ~120 Boost features with unique signal
✅ done Matrix-based Rényi α-entropy CMI (_renyi_alpha.py, opt-in estimator='renyi_alpha') Agent A #5 ~180 Plug-in-free estimator; validated narrow-niche win at high dependency, not a general/default win (see finding #5 status note above)
🚫 not planned Quadratic-MI (QMIFS) Agent A #4 ~150 Redundant with the shipped Rényi estimator (same "plug-in-free continuous MI" gap); see note above