---
title: "An End-to-End MFF Workflow"
author: "Nihat Tak and Sadık Çoban"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{An End-to-End MFF Workflow}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 4
)
```

## Overview

The MFF package combines candidate regression predictions using weights obtained
from clustering the candidates' validation prediction profiles. Candidate models
may be fitted with `model.train()`, generated by bootstrap resampling with
`boot.train()`, or trained externally. In every case, MFF needs:

* a validation prediction matrix, with observations in rows and candidate models
  in columns;
* the validation response vector;
* a test prediction matrix whose columns represent the same candidates in the
  same order; and
* the test response only for final, unbiased performance reporting.

This vignette illustrates the complete bootstrap workflow. It deliberately
selects the MFF configuration and cluster using validation data before inspecting
test performance.

## Generate candidate predictions

We use the Boston housing data only as a compact reproducible illustration.
`boot.train()` splits the data into training, validation, and test subsets, fits
one linear regression to each bootstrap sample of the training subset, and
predicts the same validation and test observations with every fitted model.

```{r bootstrap}
library(MFF)

set.seed(123)
boot_fit <- boot.train(
  target = "medv",
  data = MASS::Boston,
  ntest = 50,
  nvalid = 50,
  B = 20,
  seed = 123,
  parallel = FALSE
)

dim(boot_fit$pred_matrix_valid)
dim(boot_fit$pred_matrix_test)
boot_fit$metadata
```

Each column is now a candidate learner. The two matrices are aligned: column
`j` contains predictions from the same bootstrap model in both matrices.

## Select an MFF on validation data

`tune.mff()` searches over candidate numbers of clusters. Here k-means is used
for a fast example; `"fcm"`, `"pfcm"`, and `"gk"` provide the alternative
membership-generation geometries. The `eval.method` value determines which
validation loss is minimized.

```{r tune}
tuned <- tune.mff(
  x = boot_fit$pred_matrix_valid,
  y = boot_fit$y_valid,
  max_c = 4,
  mff.method = "kmeans",
  eval.method = "RMSE",
  nstart = 20,
  seed = 123,
  parallel = FALSE,
  logging = FALSE
)

tuned$best_params
tuned$best_cluster
tuned$best_scores
```

The value in `best_cluster` is determined exclusively from validation responses.
Consequently, the selected cluster is fixed before test responses or test error
metrics are examined. The fact that a particular cluster later gives the best
test result must not be used retrospectively to choose that cluster.

## Predict and evaluate once on the test set

The validation-selected weights are transferred unchanged to the aligned test
prediction matrix.

```{r predict}
test_prediction <- predict(
  tuned,
  pred_matrix = boot_fit$pred_matrix_test,
  type = "best"
)

head(test_prediction$mff_preds)
test_prediction$mff_weights
```

Only after selection is complete do we use `y_test` for final evaluation.

```{r evaluate}
evaluate(test_prediction$mff_preds, boot_fit$y_test)
```

The `evaluate()` helper is optional. Predictions can instead be assessed with
other R packages or user-defined metrics, provided that model selection remains
confined to the validation data.

## Supplying predictions from other R workflows

Neither `boot.train()` nor `model.train()` is required. Learners may be trained
with packages such as `tidymodels`, `caret`, `mlr3`, or forecasting packages.
For externally generated inputs, users must ensure that:

1. validation predictions are genuinely out of sample for model selection;
2. validation and test matrices have the same candidate columns in the same
   order;
3. rows align exactly with the corresponding response vectors; and
4. for forecasting applications, train-validation-test splits respect temporal
   order and do not leak future information.

Once these conditions are met, the downstream interface is unchanged:

```{r external-inputs, eval = FALSE}
tuned <- tune.mff(
  x = validation_predictions,
  y = validation_response,
  max_c = 4,
  mff.method = "gk",
  eval.method = "RMSE"
)

final_prediction <- predict(
  tuned,
  pred_matrix = test_predictions,
  type = "best"
)
```

Thus MFF can combine heterogeneous regression or forecasting candidates without
requiring those candidates to have been trained inside the package.