--- title: "Automated Machine Learning with tidylearn" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Automated Machine Learning with tidylearn} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 5 ) ``` ```{r setup} library(tidylearn) library(dplyr) library(ggplot2) ``` ## Introduction Automated Machine Learning (AutoML) streamlines the model development process by automatically trying multiple approaches and selecting the best one. tidylearn's `tl_auto_ml()` function explores various modeling strategies including dimensionality reduction, clustering, and different supervised methods. **Note:** AutoML orchestrates the wrapped packages (glmnet, randomForest, xgboost, etc.) rather than implementing new algorithms. Each model in the leaderboard wraps an established package, and you can access the raw model objects via `model$fit`. ## Basic Usage ### Classification Task ```{r, eval=FALSE} # Run AutoML on iris dataset result <- tl_auto_ml(iris, Species ~ ., task = "classification", time_budget = 60) # View best model print(result$best_model) ``` ```{r, eval=FALSE} # View all models tried names(result$models) ``` ```{r, eval=FALSE} # View leaderboard result$leaderboard ``` ### Regression Task ```{r, eval=FALSE} # Run AutoML on regression problem result_reg <- tl_auto_ml(mtcars, mpg ~ ., task = "regression", time_budget = 60) # Best model print(result_reg$best_model) ``` ## How AutoML Works The `tl_auto_ml()` function follows a four-phase pipeline. Which phases actually run -- and how thoroughly -- depends on the `time_budget` and the toggle parameters `use_reduction` and `use_clustering`. | Phase | What it does | Models added (classification) | Models added (regression) | |-------|-------------|-------------------------------|---------------------------| | 1. Baselines | Trains standard models | tree, logistic\*, forest | tree, linear, forest | | 2. PCA variants | PCA preprocessing + baseline methods | pca\_tree, pca\_logistic\*, pca\_forest | pca\_tree, pca\_linear, pca\_forest | | 3. Cluster variants | Adds cluster assignments as features | clustered\_tree, clustered\_logistic\*, clustered\_forest | clustered\_tree, clustered\_linear, clustered\_forest | | 4. Advanced | Tries heavier methods | svm, xgboost | ridge, lasso | \* Logistic regression is binary only, so it is included only when the response has exactly two classes. On a multiclass problem such as `iris` the baselines are tree and forest, and the `_logistic` variants are absent. Each model is first fit on the full training data, then (if budget allows) evaluated with k-fold cross-validation. When time is tight, models fall back to training-set metrics instead of CV. The `evaluation` column on the leaderboard records which of the two produced each score — `"cv"` or `"train"`. Training metrics are optimistically biased, so a leaderboard mixing the two is not ranking like with like. If you see both, raise the budget. ## Understanding the Time Budget The `time_budget` parameter (in seconds) is the most important knob for controlling the speed/thoroughness trade-off. It is checked **between** model fits, not during them. Once a model starts training it runs to completion, because many of the wrapped packages (randomForest, xgboost, e1071) execute C-level code that R cannot safely interrupt mid-execution. This means the actual wall-clock time may modestly exceed the budget by the duration of the last model that started before the budget expired. ### Budget tiers at a glance | Budget | Baselines | CV | PCA/Cluster variants | Advanced models | Typical models | Use case | |--------|-----------|-----|----------------------|-----------------|----------------|----------| | < 30 s | tree + logistic/linear only | No (training metrics) | No | No | 1--2 | Quick sanity check, interactive use | | 30--120 s | tree + logistic/linear + forest | When time remains | If enabled and > 10 % budget left | If > 40 % budget left | 3--7 | Development iteration, notebook exploration | | 120 s+ | All | Yes | Yes (if enabled) | Yes | 8--11 | Thorough comparison, final model selection | Model counts are for a binary response. A multiclass problem drops the logistic variants, so expect one fewer per phase. The "forest" baseline and all advanced models (SVM, XGBoost, ridge, lasso) involve C-level code that typically takes 3--15 seconds per fit depending on data size. They are only attempted when `time_budget >= 30`. ### Why CV is the expensive step A single `tl_model()` call fits one model. Cross-validation (`tl_cv(folds = 5)`) fits **five** models on subsets, so it costs roughly 5x the time. The function checks the remaining budget after each model fit and skips CV when it would likely exceed the budget, falling back to training-set evaluation instead. Reducing `cv_folds` (e.g. from 5 to 2) is the most effective way to stay closer to the budget while still getting out-of-sample estimates. ### Practical examples ```{r, eval=FALSE} # Quick sanity check -- 2 fast models, no CV, done in ~1s quick <- tl_auto_ml(iris, Species ~ ., time_budget = 10, use_reduction = FALSE, use_clustering = FALSE) quick$leaderboard #> baseline_tree only -- iris is multiclass, so logistic is skipped. #> On a two-class response this would be baseline_tree, baseline_logistic. # Development iteration -- baselines + forest, some CV medium <- tl_auto_ml(iris, Species ~ ., time_budget = 60, cv_folds = 3) medium$leaderboard #> 4--7 models depending on data size and number of classes # Thorough search -- all phases, full CV thorough <- tl_auto_ml(iris, Species ~ ., time_budget = 300, cv_folds = 5) thorough$leaderboard #> 8--11 models with cross-validated scores (iris is multiclass, #> so the logistic variants are skipped) ``` ## Task Type Detection AutoML automatically detects the task type from the response variable: ```{r, eval=FALSE} # Factor/character response -> classification result_class <- tl_auto_ml(iris, Species ~ ., task = "auto") # Numeric response -> regression result_reg <- tl_auto_ml(mtcars, mpg ~ ., task = "auto") ``` ## Controlling the Search ### Feature Engineering Options ```{r, eval=FALSE} # Disable dimensionality reduction no_reduction <- tl_auto_ml(iris, Species ~ ., use_reduction = FALSE, time_budget = 60) # Disable cluster features no_clustering <- tl_auto_ml(iris, Species ~ ., use_clustering = FALSE, time_budget = 60) # Baseline models only baseline_only <- tl_auto_ml(iris, Species ~ ., use_reduction = FALSE, use_clustering = FALSE, time_budget = 30) ``` ### Cross-Validation Settings ```{r, eval=FALSE} # Adjust cross-validation folds result_cv <- tl_auto_ml(iris, Species ~ ., cv_folds = 10, time_budget = 120) # Fewer folds for faster evaluation result_fast <- tl_auto_ml(iris, Species ~ ., cv_folds = 3, time_budget = 60) ``` ## Understanding Results ### Accessing Models ```{r, eval=FALSE} result <- tl_auto_ml(iris, Species ~ ., time_budget = 60) # Best performing model best_model <- result$best_model # All models trained all_models <- result$models # Specific model baseline_logistic <- result$models$baseline_logistic pca_forest <- result$models$pca_forest ``` ### Leaderboard ```{r, eval=FALSE} # View performance comparison. Columns are model, score and evaluation # ("cv" or "train" -- see "How AutoML Works" above). leaderboard <- result$leaderboard # Already sorted best-first, but to re-sort explicitly: # higher is better for accuracy, lower for RMSE leaderboard <- leaderboard %>% arrange(desc(score)) print(leaderboard) ``` ### Making Predictions ```{r, eval=FALSE} # Use best model for predictions predictions <- predict(result$best_model, new_data = new_data) # Or use a specific model predictions_pca <- predict(result$models$pca_forest, new_data = new_data) ``` ## Practical Examples ### Example 1: Iris Classification ```{r, eval=FALSE} # Split data for evaluation split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 123) # Run AutoML on training data automl_iris <- tl_auto_ml(split$train, Species ~ ., time_budget = 90, cv_folds = 5) # Evaluate on test set test_preds <- predict(automl_iris$best_model, new_data = split$test) test_accuracy <- mean(test_preds$.pred == split$test$Species) cat("AutoML Test Accuracy:", round(test_accuracy * 100, 1), "%\n") ``` ```{r, eval=FALSE} # Compare models for (model_name in names(automl_iris$models)) { model <- automl_iris$models[[model_name]] preds <- predict(model, new_data = split$test) acc <- mean(preds$.pred == split$test$Species) cat(model_name, ":", round(acc * 100, 1), "%\n") } ``` ### Example 2: MPG Prediction ```{r, eval=FALSE} # Split mtcars data split_mtcars <- tl_split(mtcars, prop = 0.7, seed = 42) # Run AutoML automl_mpg <- tl_auto_ml(split_mtcars$train, mpg ~ ., task = "regression", time_budget = 90) # Evaluate test_preds_mpg <- predict(automl_mpg$best_model, new_data = split_mtcars$test) rmse <- sqrt(mean((test_preds_mpg$.pred - split_mtcars$test$mpg)^2)) cat("AutoML Test RMSE:", round(rmse, 2), "\n") ``` ### Example 3: Custom Preprocessing + AutoML ```{r, eval=FALSE} # Preprocess data first processed <- tl_prepare_data( split$train, Species ~ ., scale_method = "standardize", remove_correlated = TRUE ) # Run AutoML on preprocessed data automl_processed <- tl_auto_ml(processed$data, Species ~ ., time_budget = 60) # Note: Need to apply same preprocessing to test data test_processed <- tl_prepare_data( split$test, Species ~ ., scale_method = "standardize" ) test_preds_proc <- predict( automl_processed$best_model, new_data = test_processed$data ) ``` ## Comparing AutoML with Manual Selection ```{r, eval=FALSE} # Manual approach: choose one model manual_model <- tl_model(split$train, Species ~ ., method = "forest") manual_preds <- predict(manual_model, new_data = split$test) manual_acc <- mean(manual_preds$.pred == split$test$Species) # AutoML approach automl_model <- tl_auto_ml(split$train, Species ~ ., time_budget = 60) automl_preds <- predict(automl_model$best_model, new_data = split$test) automl_acc <- mean(automl_preds$.pred == split$test$Species) cat("Manual Selection:", round(manual_acc * 100, 1), "%\n") cat("AutoML:", round(automl_acc * 100, 1), "%\n") ``` ## Advanced AutoML Strategies ### Strategy 1: Iterative AutoML ```{r, eval=FALSE} # First pass: quick exploration quick_automl <- tl_auto_ml(split$train, Species ~ ., time_budget = 30, use_reduction = TRUE, use_clustering = FALSE) # Analyze what worked — best model name is in the leaderboard best_name <- quick_automl$leaderboard$model[1] best_method <- quick_automl$best_model$spec$method cat("Best model:", best_name, "(method:", best_method, ")\n") # Second pass: if a PCA variant won, invest more in reduction if (grepl("^pca_", best_name)) { refined_automl <- tl_auto_ml(split$train, Species ~ ., time_budget = 60, use_reduction = TRUE, use_clustering = TRUE) } ``` ### Strategy 2: Ensemble of AutoML Models ```{r, eval=FALSE} # Get top 3 models top_models <- automl_iris$leaderboard %>% arrange(desc(score)) %>% head(3) # Make predictions with each ensemble_preds <- list() for (i in seq_len(nrow(top_models))) { model_name <- top_models$model[i] model <- automl_iris$models[[model_name]] ensemble_preds[[i]] <- predict(model, new_data = split$test)$.pred } # Majority vote for classification final_pred <- apply(do.call(cbind, ensemble_preds), 1, function(x) { names(which.max(table(x))) }) ensemble_acc <- mean(final_pred == split$test$Species) cat("Ensemble Accuracy:", round(ensemble_acc * 100, 1), "%\n") ``` ## Performance Metrics ### Classification Metrics ```{r, eval=FALSE} # AutoML automatically uses accuracy for classification result_class <- tl_auto_ml(iris, Species ~ ., metric = "accuracy", time_budget = 60) ``` ### Regression Metrics ```{r, eval=FALSE} # AutoML automatically uses RMSE for regression result_reg <- tl_auto_ml(mtcars, mpg ~ ., metric = "rmse", time_budget = 60) ``` ## Best Practices 1. **Start fast, then expand**: Use `time_budget = 10` to verify the pipeline runs, then increase to 60--120s for real evaluation, and 300s for final model selection. 2. **Reduce `cv_folds` before reducing `time_budget`**: Going from 5-fold to 2-fold CV cuts evaluation time by ~60% while still providing out-of-sample estimates. A 30s budget with `cv_folds = 2` is often more useful than a 30s budget with the default 5 folds (which will skip CV entirely). 3. **Preprocess when needed**: Handle missing values before AutoML. 4. **Split your data**: Always evaluate on held-out test data. 5. **Examine multiple models**: The "best" model may not always be robust. 6. **Consider ensemble approaches**: Combine top models for better performance. 7. **Understand training-set metrics**: When CV is skipped (short budgets), the leaderboard uses training-set metrics which are optimistically biased. These are useful for ranking but not for reporting final performance. ## When to Use AutoML **Good use cases:** - Quick prototyping and baseline establishment - When you're unsure which algorithm to use - Feature engineering exploration - Benchmark for manual approaches - Limited ML expertise **Consider manual selection when:** - You have domain knowledge about the best approach - Interpretability is critical - You need fine-grained control over hyperparameters - Computational resources are very limited ## Troubleshooting ### AutoML takes longer than `time_budget` The budget is checked **between** model fits. A single random forest or XGBoost fit can take 5--30 seconds depending on data size, and R cannot safely interrupt C-level code mid-execution. To stay closer to the budget: ```{r, eval=FALSE} # 1. Reduce CV folds (biggest impact) fast_result <- tl_auto_ml(data, formula, cv_folds = 2, time_budget = 30) # 2. Disable slow phases baseline_result <- tl_auto_ml(data, formula, use_reduction = FALSE, use_clustering = FALSE, time_budget = 30) # 3. Use a budget under 30s to skip forest/SVM/XGBoost entirely quick_result <- tl_auto_ml(data, formula, time_budget = 10) ``` ### A model is missing, or its score is NA A model that fails to fit, or that errors during evaluation, is dropped with a message explaining why — it never reaches the leaderboard. Run with messages enabled to see which ones and why. A model that evaluates successfully but produces no value for the chosen metric appears with an `NA` score. That usually means the metric name is not one `tl_evaluate()` recognises for the task: classification supports `accuracy`, `precision`, `recall`, `sensitivity`, `specificity`, `f1`, `auc` and `pr_auc`; regression supports `rmse`, `mse`, `mae`, `mape` and `rsq`. If every score is `NA`, `tl_auto_ml()` warns and returns the first model trained rather than pretending to have ranked them. ### Not enough models tried ```{r, eval=FALSE} # Increase time budget to unlock all phases thorough_result <- tl_auto_ml(data, formula, time_budget = 300) # Ensure feature engineering is enabled full_result <- tl_auto_ml(data, formula, use_reduction = TRUE, use_clustering = TRUE, time_budget = 300) ``` ## Summary tidylearn's AutoML provides: - **Automated model selection** across multiple algorithms - **Feature engineering** with PCA and clustering - **Cross-validation** for robust performance estimates - **Easy comparison** through leaderboard - **Flexible configuration** for different scenarios - **Integration workflows** combining supervised and unsupervised learning ```{r, eval=FALSE} # Complete AutoML workflow workflow_split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 123) automl_result <- tl_auto_ml( data = workflow_split$train, formula = Species ~ ., task = "auto", use_reduction = TRUE, use_clustering = TRUE, time_budget = 120, cv_folds = 5 ) # Evaluate best model final_preds <- predict(automl_result$best_model, new_data = workflow_split$test) final_accuracy <- mean(final_preds$.pred == workflow_split$test$Species) cat("Final AutoML Accuracy:", round(final_accuracy * 100, 1), "%\n") cat("Best approach:", automl_result$best_model$spec$method, "\n") ``` AutoML makes machine learning accessible and efficient, allowing you to quickly find good solutions while learning which approaches work best for your data.