Causal Forge

Verified project card

# Causal Forge

Causal Forge is a ground-truth causal inference laboratory. Its structural causal
model creates confounding, heterogeneous treatment effects, nonlinear outcomes, and
known counterfactuals. The benchmark measures whether estimators recover the
population average treatment effect rather than merely predicting observed outcomes.

The evaluation compares:

- the unadjusted difference in observed group means;
- inverse-propensity weighting (IPW);
- outcome regression with separate treatment/control response surfaces;
- the augmented IPW (AIPW) doubly robust estimator;
- four nuisance-model regimes where the propensity model, outcome model, both, or
  neither receive the correct nonlinear feature basis.

The exact potential outcomes and treatment probabilities are retained only because
this is a synthetic benchmark. They make estimator bias directly measurable.

## Verified results

The benchmark ran 100 independent replications with 3,000 observations and five-fold
cross-fitting in each replication.

| Nuisance-model regime | Naive MAE | IPW MAE | Outcome MAE | AIPW MAE | AIPW bias |
| --- | ---: | ---: | ---: | ---: | ---: |
| Both correct | 1.1109 | 0.0644 | 0.0388 | 0.0426 | 0.0004 |
| Propensity misspecified | 1.1109 | 0.1342 | 0.0388 | 0.0414 | 0.0008 |
| Outcome misspecified | 1.1109 | 0.0644 | 0.2688 | 0.0481 | 0.0078 |
| Both misspecified | 1.1109 | 0.1342 | 0.2688 | 0.3953 | 0.3953 |

This demonstrates the intended double-robustness boundary: AIPW remains accurate
when either the treatment or outcome nuisance model is correct, but not when both
are wrong. Results are Monte Carlo measurements on this synthetic SCM, not claims
about arbitrary real-world observational data.

## Reproduce

```powershell
uv run python projects/causal-forge/train.py
```

Evaluation snapshot

{
  "benchmark": "Causal Forge",
  "replications": 100,
  "samples_per_replication": 3000,
  "cross_fitting_folds": 5,
  "structural_truth": "known heterogeneous individual treatment effects",
  "summary": {
    "both_correct": {
      "naive": {
        "mean_estimate": 3.1123144027184075,
        "bias": 1.110939263534936,
        "mean_absolute_error": 1.110939263534936,
        "rmse": 1.1132682867094192
      },
      "ipw": {
        "mean_estimate": 2.005159416540855,
        "bias": 0.0037842773573832744,
        "mean_absolute_error": 0.0643549025548575,
        "rmse": 0.07985124612336089
      },
      "outcome_regression": {
        "mean_estimate": 2.0031127881794304,
        "bias": 0.0017376489959579633,
        "mean_absolute_error": 0.038795035337436425,
        "rmse": 0.04801133697032812
      },
      "aipw": {
        "mean_estimate": 2.0017294974257873,
        "bias": 0.00035435824231527135,
        "mean_absolute_error": 0.04261334125461211,
        "rmse": 0.05444966094807796,
        "confidence_interval_coverage": 0.89
      }
    },
    "propensity_misspecified": {
      "naive": {
        "mean_estimate": 3.1123144027184075,
        "bias": 1.110939263534936,
        "mean_absolute_error": 1.110939263534936,
        "rmse": 1.1132682867094192
      },
      "ipw": {
        "mean_estimate": 2.1339948243616265,
        "bias": 0.13261968517815434,
        "mean_absolute_error": 0.1341636821632256,
        "rmse": 0.14885056735045618
      },
      "outcome_regression": {
        "mean_estimate": 2.0031127881794304,
        "bias": 0.0017376489959579633,
        "mean_absolute_error": 0.038795035337436425,
        "rmse": 0.04801133697032812
      },
      "aipw": {
        "mean_estimate": 2.002138366914246,
        "bias": 0.0007632277307738455,
        "mean_absolute_error": 0.04142272629021512,
        "rmse": 0.05271409176654249,
        "confidence_interval_coverage": 0.92
      }
    },
    "outcome_misspecified": {
      "naive": {
        "mean_estimate": 3.1123144027184075,
        "bias": 1.110939263534936,
        "mean_absolute_error": 1.110939263534936,
        "rmse": 1.1132682867094192
      },
      "ipw": {
        "mean_estimate": 2.005159416540855,
        "bias": 0.0037842773573832744,
        "mean_absolute_error": 0.0643549025548575,
        "rmse": 0.07985124612336089
      },
      "outcome_regression": {
        "mean_estimate": 2.2701977886081592,
        "bias": 0.26882264942468725,
        "mean_absolute_error": 0.26882264942468725,
        "rmse": 0.2753485792353652
      },
      "aipw": {
        "mean_estimate": 2.009154042503896,
        "bias": 0.007778903320424111,
        "mean_absolute_error": 0.048137374861682396,
        "rmse": 0.06166591119979539,
        "confidence_interval_coverage": 0.99
      }
    },
    "both_misspecified": {
      "naive": {
        "mean_estimate": 3.1123144027184075,
        "bias": 1.110939263534936,
        "mean_absolute_error": 1.110939263534936,
        "rmse": 1.1132682867094192
      },
      "ipw": {
        "mean_estimate": 2.1339948243616265,
        "bias": 0.13261968517815434,
        "mean_absolute_error": 0.1341636821632256,
        "rmse": 0.14885056735045618
      },
      "outcome_regression": {
        "mean_estimate": 2.2701977886081592,
        "bias": 0.26882264942468725,
        "mean_absolute_error": 0.26882264942468725,
        "rmse": 0.2753485792353652
      },
      "aipw": {
        "mean_estimate": 2.396631864687349,
        "bias": 0.395256725503877,
        "mean_absolute_error": 0.395256725503877,
        "rmse": 0.4048953844650616,
        "confidence_interval_coverage": 0.01
      }
    }
  }
}

Backed-up artifact tree

• README.md
• __pycache__/app.cpython-311.pyc
• __pycache__/causal.cpython-311.pyc
• app.py
• artifacts/causal-forge/evaluation.json
• artifacts/causal-forge/nuisance_models.joblib
• artifacts/causal-forge/replication_estimates.parquet
• causal.py
• data/causal_benchmark.parquet
• requirements.txt
• train.py