Composite-target discovery: TVT walkthrough¶
This tutorial walks through the composite-target discovery feature on a synthetic dataset that mirrors the canonical autoregressive-lag scenario: a target TVT whose dominant predictor is its own previous-period value TVT_prev. Without composite-target discovery the resulting model puts ~95% of feature importance on TVT_prev and the remaining structural signal in x1/x2/x3 stays buried. With composite-target discovery the suite finds and trains models on transforms like T = y - alpha*TVT_prev, exposing the residual structure to the rest of the feature set.
What this notebook covers:
- Build the synthetic dataset and inspect the dominance.
- Run a baseline
train_mlframe_models_suitecall without composite discovery and read the FI chart. - Enable composite discovery and show how
metadata['composite_target_specs']surfaces the discovered transforms. - Compare baseline vs composite-ensemble RMSE on a held-out test set.
- Render the stakeholder Markdown report from
report_to_markdown.
Prerequisites¶
- mlframe installed from a source checkout (
pip install -e .from the repo root). - LightGBM and scikit-learn (transitive deps).
- Matplotlib for the FI chart.
Estimated runtime on a laptop: 1-2 minutes.
1. Build the synthetic TVT dataset¶
We synthesise 1500 rows where y = 0.95 * TVT_prev + 0.5 * x1 - 0.3 * x2 + noise. The 0.95 coefficient is the dominant lag; x1 and x2 carry small structural signal; x3 is pure noise. A baseline GBM will find the 0.95-relationship instantly and fit y_hat ~ TVT_prev, leaving the structural part in noise.
import numpy as np
import pandas as pd
rng = np.random.default_rng(42)
n = 1500
base = rng.normal(loc=10.0, scale=3.0, size=n)
x1 = rng.normal(size=n)
x2 = rng.normal(size=n)
x3 = rng.normal(size=n)
y = 0.95 * base + 0.5 * x1 - 0.3 * x2 + rng.normal(scale=0.3, size=n)
df = pd.DataFrame({
'TVT_prev': base, 'x1': x1, 'x2': x2, 'x3': x3, 'TVT': y,
})
print(df.describe().round(2))
print('\ncorr(TVT, TVT_prev) =', round(df[['TVT', 'TVT_prev']].corr().iloc[0, 1], 4))
2. Baseline run (no composite discovery)¶
Use train_mlframe_models_suite with the suite's standard config. We pick a single lightweight model (linear) for runtime. With composite discovery off (the default), the suite fits a single model on the raw target and reports T-scale RMSE.
import tempfile
from mlframe.training.core import train_mlframe_models_suite
from mlframe.training.extractors import SimpleFeaturesAndTargetsExtractor
df_for_suite = df.rename(columns={'TVT': 'target'}) # use a single 'target' regression column
with tempfile.TemporaryDirectory() as work_dir:
baseline_models, baseline_metadata = train_mlframe_models_suite(
df=df_for_suite,
target_name='target',
model_name='tutorial_baseline',
features_and_targets_extractor=SimpleFeaturesAndTargetsExtractor(
regression_targets=['target']),
mlframe_models=['linear'],
output_config={'data_dir': work_dir, 'models_dir': 'models'},
verbose=0,
)
print('Models keys:', list(baseline_models['regression'].keys()))
print('Baseline diagnostics recommendation:',
baseline_metadata['baseline_diagnostics']['regression']['target']['composite_recommendation'])
3. Read the baseline diagnostics¶
Even without composite discovery the suite runs BaselineDiagnostics by default. Its recommendation tells us whether composite-mode is likely to help: high_potential means a dominant feature exists and the init_score baseline does NOT yet capture all of it. Let's inspect the ablation table.
diag = baseline_metadata['baseline_diagnostics']['regression']['target']
print('Headline metric:', diag['headline_metric'])
print('\nAblation:')
for entry in diag['ablation']:
print(f" rank {entry['rank']}: drop {entry['feature']!r} -> RMSE_after_drop={entry['metric_after_drop']:.3f} (delta%={entry['delta_pct']:+.2f})")
print('\ninit_score baseline:', diag['init_score_baseline'])
print('\nrecommendation:', diag['composite_recommendation'])
print('reason:', diag['composite_recommendation_reason'])
Expected output: TVT_prev lands at rank 1 with delta% > 20, init_score(TVT_prev) shows a small RMSE delta vs raw (init_score nearly captures the dominance), and the recommendation is high_potential (residual structure still unrecovered).
4. Enable composite-target discovery¶
Now we opt in to discovery. Configure:
enabled=Truebase_candidates='auto'to let MI-screening pick the dominant feature.- All four core transforms:
diff,ratio,logratio,linear_residual. screening='hybrid'to follow MI-pre-screening with a tiny-model CV-RMSE rerank.cross_target_ensemble_strategy='oof_weighted'to combine the discovered composites + raw into one final predictor.
from mlframe.training.configs import CompositeTargetDiscoveryConfig
cfg = CompositeTargetDiscoveryConfig(
enabled=True,
base_candidates='auto',
transforms=['diff', 'ratio', 'logratio', 'linear_residual'],
mi_sample_n=1000,
top_k_after_mi=4,
eps_mi_gain=0.01,
screening='hybrid',
tiny_model_n_estimators=40,
tiny_model_sample_n=800,
top_m_after_tiny=3,
cross_target_ensemble_strategy='oof_weighted',
)
with tempfile.TemporaryDirectory() as work_dir:
composite_models, composite_metadata = train_mlframe_models_suite(
df=df_for_suite,
target_name='target',
model_name='tutorial_composite',
features_and_targets_extractor=SimpleFeaturesAndTargetsExtractor(
regression_targets=['target']),
mlframe_models=['linear'],
output_config={'data_dir': work_dir, 'models_dir': 'models'},
verbose=0,
composite_target_discovery_config=cfg,
)
5. Inspect the discovered specs¶
metadata['composite_target_specs'] carries the discovered (target, transform, base) triples with their MI gains and fitted parameters. The cross-target ensemble metadata is at composite_target_ensemble.
specs = composite_metadata['composite_target_specs']['regression']['target']
print(f'{len(specs)} composite spec(s) discovered:')
for s in specs:
print(f" {s['name']!r}: mi_gain={s['mi_gain']:+.4f} base={s['base_column']!r} transform={s['transform_name']!r}")
print(f" fitted_params: {s['fitted_params']}")
ens_meta = composite_metadata.get('composite_target_ensemble', {}).get('regression', {}).get('target')
if ens_meta:
print(f"\nCross-target ensemble strategy: {ens_meta['strategy']}")
for nm, w in zip(ens_meta['component_names'], ens_meta['weights']):
print(f" weight {w:.4f} -> {nm!r}")
6. Compare y-scale RMSE: baseline vs composite ensemble¶
metadata['composite_target_y_scale_metrics'] carries train/val/test RMSE on the y-scale (after the wrapper inverse transforms T -> y). Compare the baseline raw model's test RMSE against the composite ensemble's test RMSE.
y_metrics = composite_metadata['composite_target_y_scale_metrics']['regression']
print('Composite-target y-scale metrics:')
for cname, entries in y_metrics.items():
for e in entries:
test = e.get('metrics', {}).get('test', {})
if test:
print(f" {cname!r}: test_RMSE={test.get('RMSE', float('nan')):.4f} test_MAE={test.get('MAE', float('nan')):.4f}")
7. Stakeholder Markdown report¶
report_to_markdown renders a one-page summary suitable for a code-review comment or Slack message. It lists every discovered spec with its provenance paragraph, the rejected candidates with reasons, and the cross-target ensemble's per-component weights.
from mlframe.training.composite import (
CompositeSpec, report_to_markdown,
)
spec_objs = [CompositeSpec(**{k: v for k, v in s.items()
if k in {'name', 'target_col', 'transform_name', 'base_column',
'fitted_params', 'mi_gain', 'mi_y', 'mi_t',
'valid_domain_frac', 'n_train_rows'}})
for s in specs]
failures = composite_metadata.get('composite_target_failures', {}).get('regression', {}).get('target', [])
ens_meta = composite_metadata.get('composite_target_ensemble', {}).get('regression', {}).get('target')
md = report_to_markdown(
target_col='target',
specs=spec_objs,
failures=failures,
ensemble_metadata=ens_meta,
)
from IPython.display import Markdown
Markdown(md)
8. When NOT to use composite-target discovery¶
If BaselineDiagnostics recommendation is unlikely_to_help, composite-mode adds compute cost without payoff: no feature dominates strongly enough that residualisation exposes structure. Examples:
- Features have roughly equal contribution (no "lag" or "dominant" feature).
- The
init_score(top_feature)baseline already nearly matches the raw RMSE: native residual learning has captured the structure. - Tiny datasets where the K composite-target re-fits would dominate runtime.
When unlikely_to_help, set composite_target_discovery_config=None (the default) and ship the raw model.
9. Production rollback / kill switch¶
If composite-mode causes a production regression, set the env var MLFRAME_DISABLE_COMPOSITE=1 before the suite call. The suite reads the env var and forces enabled=False regardless of the config the caller passed -- rollback in seconds without a code change.
10. Where to learn more¶
docs/baseline_diagnostics_guide.md-- the diagnostic that runs by default.mlframe/training/composite.py-- the registry, wrapper, ensemble.CHANGELOG.md-- full provenance of the feature roll-out.benchmarks/composite_target_benchmark.py-- runnable synthetic benchmark across the S1-S16 scenarios.