---
title: "Automated Machine Learning with tidylearn"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Automated Machine Learning with tidylearn}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5
)
```

```{r setup}
library(tidylearn)
library(dplyr)
library(ggplot2)
```

## Introduction

Automated Machine Learning (AutoML) streamlines the model development process by
automatically trying multiple approaches and selecting the best one. tidylearn's
`tl_auto_ml()` function explores various modeling strategies including
dimensionality reduction, clustering, and different supervised methods.

**Note:** AutoML orchestrates the wrapped packages (glmnet, randomForest,
xgboost, etc.) rather than implementing new algorithms. Each model in the
leaderboard wraps an established package, and you can access the raw model
objects via `model$fit`.

## Basic Usage

### Classification Task

```{r, eval=FALSE}
# Run AutoML on iris dataset
result <- tl_auto_ml(iris, Species ~ .,
                     task = "classification",
                     time_budget = 60)

# View best model
print(result$best_model)
```

```{r, eval=FALSE}
# View all models tried
names(result$models)
```

```{r, eval=FALSE}
# View leaderboard
result$leaderboard
```

### Regression Task

```{r, eval=FALSE}
# Run AutoML on regression problem
result_reg <- tl_auto_ml(mtcars, mpg ~ .,
                         task = "regression",
                         time_budget = 60)

# Best model
print(result_reg$best_model)
```

## How AutoML Works

The `tl_auto_ml()` function follows a four-phase pipeline. Which phases
actually run -- and how thoroughly -- depends on the `time_budget` and the
toggle parameters `use_reduction` and `use_clustering`.

| Phase | What it does | Models added (classification) | Models added (regression) |
|-------|-------------|-------------------------------|---------------------------|
| 1. Baselines | Trains standard models | tree, logistic\*, forest | tree, linear, forest |
| 2. PCA variants | PCA preprocessing + baseline methods | pca\_tree, pca\_logistic\*, pca\_forest | pca\_tree, pca\_linear, pca\_forest |
| 3. Cluster variants | Adds cluster assignments as features | clustered\_tree, clustered\_logistic\*, clustered\_forest | clustered\_tree, clustered\_linear, clustered\_forest |
| 4. Advanced | Tries heavier methods | svm, xgboost | ridge, lasso |

\* Logistic regression is binary only, so it is included only when the
response has exactly two classes. On a multiclass problem such as `iris`
the baselines are tree and forest, and the `_logistic` variants are absent.

Each model is first fit on the full training data, then (if budget allows)
evaluated with k-fold cross-validation. When time is tight, models fall back
to training-set metrics instead of CV.

The `evaluation` column on the leaderboard records which of the two produced
each score — `"cv"` or `"train"`. Training metrics are optimistically biased,
so a leaderboard mixing the two is not ranking like with like. If you see
both, raise the budget.

## Understanding the Time Budget

The `time_budget` parameter (in seconds) is the most important knob for
controlling the speed/thoroughness trade-off. It is checked **between** model
fits, not during them. Once a model starts training it runs to completion,
because many of the wrapped packages (randomForest, xgboost, e1071) execute
C-level code that R cannot safely interrupt mid-execution. This means the
actual wall-clock time may modestly exceed the budget by the duration of the
last model that started before the budget expired.

### Budget tiers at a glance

| Budget | Baselines | CV | PCA/Cluster variants | Advanced models | Typical models | Use case |
|--------|-----------|-----|----------------------|-----------------|----------------|----------|
| < 30 s | tree + logistic/linear only | No (training metrics) | No | No | 1--2 | Quick sanity check, interactive use |
| 30--120 s | tree + logistic/linear + forest | When time remains | If enabled and > 10 % budget left | If > 40 % budget left | 3--7 | Development iteration, notebook exploration |
| 120 s+ | All | Yes | Yes (if enabled) | Yes | 8--11 | Thorough comparison, final model selection |

Model counts are for a binary response. A multiclass problem drops the
logistic variants, so expect one fewer per phase.

The "forest" baseline and all advanced models (SVM, XGBoost, ridge, lasso)
involve C-level code that typically takes 3--15 seconds per fit depending on
data size. They are only attempted when `time_budget >= 30`.

### Why CV is the expensive step

A single `tl_model()` call fits one model. Cross-validation
(`tl_cv(folds = 5)`) fits **five** models on subsets, so it
costs roughly 5x the time. The function checks the remaining
budget after each model fit and skips CV when it would likely
exceed the budget, falling back to training-set evaluation instead.
Reducing `cv_folds` (e.g. from 5 to 2) is the most effective way to
stay closer to the budget while still getting out-of-sample estimates.

### Practical examples

```{r, eval=FALSE}
# Quick sanity check -- 2 fast models, no CV, done in ~1s
quick <- tl_auto_ml(iris, Species ~ .,
                    time_budget = 10,
                    use_reduction = FALSE,
                    use_clustering = FALSE)
quick$leaderboard
#> baseline_tree only -- iris is multiclass, so logistic is skipped.
#> On a two-class response this would be baseline_tree, baseline_logistic.

# Development iteration -- baselines + forest, some CV
medium <- tl_auto_ml(iris, Species ~ .,
                     time_budget = 60,
                     cv_folds = 3)
medium$leaderboard
#> 4--7 models depending on data size and number of classes

# Thorough search -- all phases, full CV
thorough <- tl_auto_ml(iris, Species ~ .,
                       time_budget = 300,
                       cv_folds = 5)
thorough$leaderboard
#> 8--11 models with cross-validated scores (iris is multiclass,
#> so the logistic variants are skipped)
```

## Task Type Detection

AutoML automatically detects the task type from the response variable:

```{r, eval=FALSE}
# Factor/character response -> classification
result_class <- tl_auto_ml(iris, Species ~ ., task = "auto")

# Numeric response -> regression
result_reg <- tl_auto_ml(mtcars, mpg ~ ., task = "auto")
```

## Controlling the Search

### Feature Engineering Options

```{r, eval=FALSE}
# Disable dimensionality reduction
no_reduction <- tl_auto_ml(iris, Species ~ .,
                           use_reduction = FALSE,
                           time_budget = 60)

# Disable cluster features
no_clustering <- tl_auto_ml(iris, Species ~ .,
                            use_clustering = FALSE,
                            time_budget = 60)

# Baseline models only
baseline_only <- tl_auto_ml(iris, Species ~ .,
                            use_reduction = FALSE,
                            use_clustering = FALSE,
                            time_budget = 30)
```

### Cross-Validation Settings

```{r, eval=FALSE}
# Adjust cross-validation folds
result_cv <- tl_auto_ml(iris, Species ~ .,
                        cv_folds = 10,
                        time_budget = 120)

# Fewer folds for faster evaluation
result_fast <- tl_auto_ml(iris, Species ~ .,
                          cv_folds = 3,
                          time_budget = 60)
```

## Understanding Results

### Accessing Models

```{r, eval=FALSE}
result <- tl_auto_ml(iris, Species ~ ., time_budget = 60)

# Best performing model
best_model <- result$best_model

# All models trained
all_models <- result$models

# Specific model
baseline_logistic <- result$models$baseline_logistic
pca_forest <- result$models$pca_forest
```

### Leaderboard

```{r, eval=FALSE}
# View performance comparison. Columns are model, score and evaluation
# ("cv" or "train" -- see "How AutoML Works" above).
leaderboard <- result$leaderboard

# Already sorted best-first, but to re-sort explicitly:
# higher is better for accuracy, lower for RMSE
leaderboard <- leaderboard %>%
  arrange(desc(score))

print(leaderboard)
```

### Making Predictions

```{r, eval=FALSE}
# Use best model for predictions
predictions <- predict(result$best_model, new_data = new_data)

# Or use a specific model
predictions_pca <- predict(result$models$pca_forest, new_data = new_data)
```

## Practical Examples

### Example 1: Iris Classification

```{r, eval=FALSE}
# Split data for evaluation
split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 123)

# Run AutoML on training data
automl_iris <- tl_auto_ml(split$train, Species ~ .,
                          time_budget = 90,
                          cv_folds = 5)

# Evaluate on test set
test_preds <- predict(automl_iris$best_model, new_data = split$test)
test_accuracy <- mean(test_preds$.pred == split$test$Species)

cat("AutoML Test Accuracy:", round(test_accuracy * 100, 1), "%\n")
```

```{r, eval=FALSE}
# Compare models
for (model_name in names(automl_iris$models)) {
  model <- automl_iris$models[[model_name]]
  preds <- predict(model, new_data = split$test)
  acc <- mean(preds$.pred == split$test$Species)
  cat(model_name, ":", round(acc * 100, 1), "%\n")
}
```

### Example 2: MPG Prediction

```{r, eval=FALSE}
# Split mtcars data
split_mtcars <- tl_split(mtcars, prop = 0.7, seed = 42)

# Run AutoML
automl_mpg <- tl_auto_ml(split_mtcars$train, mpg ~ .,
                         task = "regression",
                         time_budget = 90)

# Evaluate
test_preds_mpg <- predict(automl_mpg$best_model, new_data = split_mtcars$test)
rmse <- sqrt(mean((test_preds_mpg$.pred - split_mtcars$test$mpg)^2))

cat("AutoML Test RMSE:", round(rmse, 2), "\n")
```

### Example 3: Custom Preprocessing + AutoML

```{r, eval=FALSE}
# Preprocess data first
processed <- tl_prepare_data(
  split$train,
  Species ~ .,
  scale_method = "standardize",
  remove_correlated = TRUE
)

# Run AutoML on preprocessed data
automl_processed <- tl_auto_ml(processed$data, Species ~ .,
                               time_budget = 60)

# Note: Need to apply same preprocessing to test data
test_processed <- tl_prepare_data(
  split$test,
  Species ~ .,
  scale_method = "standardize"
)

test_preds_proc <- predict(
  automl_processed$best_model,
  new_data = test_processed$data
)
```

## Comparing AutoML with Manual Selection

```{r, eval=FALSE}
# Manual approach: choose one model
manual_model <- tl_model(split$train, Species ~ ., method = "forest")
manual_preds <- predict(manual_model, new_data = split$test)
manual_acc <- mean(manual_preds$.pred == split$test$Species)

# AutoML approach
automl_model <- tl_auto_ml(split$train, Species ~ ., time_budget = 60)
automl_preds <- predict(automl_model$best_model, new_data = split$test)
automl_acc <- mean(automl_preds$.pred == split$test$Species)

cat("Manual Selection:", round(manual_acc * 100, 1), "%\n")
cat("AutoML:", round(automl_acc * 100, 1), "%\n")
```

## Advanced AutoML Strategies

### Strategy 1: Iterative AutoML

```{r, eval=FALSE}
# First pass: quick exploration
quick_automl <- tl_auto_ml(split$train, Species ~ .,
                           time_budget = 30,
                           use_reduction = TRUE,
                           use_clustering = FALSE)

# Analyze what worked — best model name is in the leaderboard
best_name <- quick_automl$leaderboard$model[1]
best_method <- quick_automl$best_model$spec$method
cat("Best model:", best_name, "(method:", best_method, ")\n")

# Second pass: if a PCA variant won, invest more in reduction
if (grepl("^pca_", best_name)) {
  refined_automl <- tl_auto_ml(split$train, Species ~ .,
                               time_budget = 60,
                               use_reduction = TRUE,
                               use_clustering = TRUE)
}
```

### Strategy 2: Ensemble of AutoML Models

```{r, eval=FALSE}
# Get top 3 models
top_models <- automl_iris$leaderboard %>%
  arrange(desc(score)) %>%
  head(3)

# Make predictions with each
ensemble_preds <- list()
for (i in seq_len(nrow(top_models))) {
  model_name <- top_models$model[i]
  model <- automl_iris$models[[model_name]]
  ensemble_preds[[i]] <- predict(model, new_data = split$test)$.pred
}

# Majority vote for classification
final_pred <- apply(do.call(cbind, ensemble_preds), 1, function(x) {
  names(which.max(table(x)))
})

ensemble_acc <- mean(final_pred == split$test$Species)
cat("Ensemble Accuracy:", round(ensemble_acc * 100, 1), "%\n")
```

## Performance Metrics

### Classification Metrics

```{r, eval=FALSE}
# AutoML automatically uses accuracy for classification
result_class <- tl_auto_ml(iris, Species ~ .,
                           metric = "accuracy",
                           time_budget = 60)
```

### Regression Metrics

```{r, eval=FALSE}
# AutoML automatically uses RMSE for regression
result_reg <- tl_auto_ml(mtcars, mpg ~ .,
                         metric = "rmse",
                         time_budget = 60)
```

## Best Practices

1. **Start fast, then expand**: Use `time_budget = 10` to verify the pipeline
   runs, then increase to 60--120s for real evaluation, and 300s for final
   model selection.
2. **Reduce `cv_folds` before reducing `time_budget`**: Going from 5-fold to
   2-fold CV cuts evaluation time by ~60% while still providing out-of-sample
   estimates. A 30s budget with `cv_folds = 2` is often more useful than a
   30s budget with the default 5 folds (which will skip CV entirely).
3. **Preprocess when needed**: Handle missing values before AutoML.
4. **Split your data**: Always evaluate on held-out test data.
5. **Examine multiple models**: The "best" model may not always be robust.
6. **Consider ensemble approaches**: Combine top models for better performance.
7. **Understand training-set metrics**: When CV is skipped (short budgets),
   the leaderboard uses training-set metrics which are optimistically biased.
   These are useful for ranking but not for reporting final performance.

## When to Use AutoML

**Good use cases:**

- Quick prototyping and baseline establishment
- When you're unsure which algorithm to use
- Feature engineering exploration
- Benchmark for manual approaches
- Limited ML expertise

**Consider manual selection when:**

- You have domain knowledge about the best approach
- Interpretability is critical
- You need fine-grained control over hyperparameters
- Computational resources are very limited

## Troubleshooting

### AutoML takes longer than `time_budget`

The budget is checked **between** model fits. A single random forest or
XGBoost fit can take 5--30 seconds depending on data size, and R cannot
safely interrupt C-level code mid-execution. To stay closer to the budget:

```{r, eval=FALSE}
# 1. Reduce CV folds (biggest impact)
fast_result <- tl_auto_ml(data, formula,
                          cv_folds = 2,
                          time_budget = 30)

# 2. Disable slow phases
baseline_result <- tl_auto_ml(data, formula,
                              use_reduction = FALSE,
                              use_clustering = FALSE,
                              time_budget = 30)

# 3. Use a budget under 30s to skip forest/SVM/XGBoost entirely
quick_result <- tl_auto_ml(data, formula, time_budget = 10)
```

### A model is missing, or its score is NA

A model that fails to fit, or that errors during evaluation, is dropped
with a message explaining why — it never reaches the leaderboard. Run with
messages enabled to see which ones and why.

A model that evaluates successfully but produces no value for the chosen
metric appears with an `NA` score. That usually means the metric name is
not one `tl_evaluate()` recognises for the task: classification supports
`accuracy`, `precision`, `recall`, `sensitivity`, `specificity`, `f1`,
`auc` and `pr_auc`; regression supports `rmse`, `mse`, `mae`, `mape` and
`rsq`. If every score is `NA`, `tl_auto_ml()` warns and returns the first
model trained rather than pretending to have ranked them.

### Not enough models tried

```{r, eval=FALSE}
# Increase time budget to unlock all phases
thorough_result <- tl_auto_ml(data, formula, time_budget = 300)

# Ensure feature engineering is enabled
full_result <- tl_auto_ml(data, formula,
                          use_reduction = TRUE,
                          use_clustering = TRUE,
                          time_budget = 300)
```

## Summary

tidylearn's AutoML provides:

- **Automated model selection** across multiple algorithms
- **Feature engineering** with PCA and clustering
- **Cross-validation** for robust performance estimates
- **Easy comparison** through leaderboard
- **Flexible configuration** for different scenarios
- **Integration workflows** combining supervised and unsupervised learning

```{r, eval=FALSE}
# Complete AutoML workflow
workflow_split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 123)

automl_result <- tl_auto_ml(
  data = workflow_split$train,
  formula = Species ~ .,
  task = "auto",
  use_reduction = TRUE,
  use_clustering = TRUE,
  time_budget = 120,
  cv_folds = 5
)

# Evaluate best model
final_preds <- predict(automl_result$best_model, new_data = workflow_split$test)
final_accuracy <- mean(final_preds$.pred == workflow_split$test$Species)

cat("Final AutoML Accuracy:", round(final_accuracy * 100, 1), "%\n")
cat("Best approach:", automl_result$best_model$spec$method, "\n")
```

AutoML makes machine learning accessible and efficient, allowing you to quickly find good solutions while learning which approaches work best for your data.
