v0.2.15

Documentation for regression (24*.ipynb)

Author: bartzbeielstein

notebooks/14_spot_ray_hpt_torch_cifar10.ipynb

@@ -4183,7 +4183,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.11"
+   "version": "3.10.10"
   },
   "vscode": {
    "interpreter": {

notebooks/16_spot_hpt_sklearn_multiclass_classification_randomforest.ipynb

@@ -1,126 +1,13 @@
 {
   "cells": [
-    {
-      "cell_type": "raw",
-      "metadata": {},
-      "source": [
-        "---\n",
-        "title: PyTorch Hyperparameter Tuning --- A Tutorial for spotPython\n",
-        "subtitle: Version 0.2.14\n",
-        "format:\n",
-        "  pdf:\n",
-        "    template: bart23e_template.tex\n",
-        "    fig-width: 7\n",
-        "    fig-height: 5\n",
-        "    keep-tex: true\n",
-        "    linenumbers: false\n",
-        "    doublespacing: false\n",
-        "    number-sections: true\n",
-        "    runninghead: PyTorch Hyperparameter Tuning With spotPython\n",
-        "  html:\n",
-        "    fig-width: 7\n",
-        "    fig-height: 5\n",
-        "author:\n",
-        "  - name: Thomas Bartz-Beielstein\n",
-        "    affiliations:\n",
-        "      - name: SpotSeven Lab\n",
-        "      - city: Gummersbach\n",
-        "        country: Germany\n",
-        "        postal-code: 51643\n",
-        "    orcid: 0000-0002-5938-5158\n",
-        "    email: bartzbeielstein@gmail.com\n",
-        "    url: 'https://www.spotseven.de'\n",
-        "abstract: |\n",
-        "  The goal of hyperparameter tuning (or hyperparameter optimization) is to optimize the hyperparameters to improve the performance of the machine or deep learning model. spotPython (\"Sequential Parameter Optimization Toolbox in Python\") is the Python version of the well-known hyperparameter tuner SPOT, which has been developed in the R programming environment for statistical analysis for over a decade. PyTorch is an optimized tensor library for deep learning using GPUs and CPUs. This document shows how to integrate the spotPython hyperparameter tuner into the PyTorch training workflow.  As an example, the results of the CIFAR10 image classifier are used. In addition to an introduction to spotPython, this tutorial also includes a brief comparison with Ray Tune, a Python library for running experiments and tuning hyperparameters. This comparison is based on the PyTorch hyperparameter tuning tutorial. The advantages and disadvantages of both approaches are discussed. We show that spotPython achieves similar or even better results while being more flexible and transparent than Ray Tune.\n",
-        "keywords:\n",
-        "  - hyperparameter tuning\n",
-        "  - hyperparameter optimization\n",
-        "  - spotPython\n",
-        "  - PyTorch\n",
-        "  - CIFAR10\n",
-        "  - optimization\n",
-        "  - deep learning\n",
-        "  - machine learning\n",
-        "bibliography: bart23e.bib\n",
-        "execute:\n",
-        "  cache: false\n",
-        "  eval: false\n",
-        "  echo: true\n",
-        "  warning: false\n",
-        "---"
-      ]
-    },
-    {
-      "attachments": {},
-      "cell_type": "markdown",
-      "metadata": {},
-      "source": [
-        "# Hyperparameter Tuning {#sec-hyperparameter-tuning}\n",
-        "\n",
-        "Hyperparameter tuning is an important, but often difficult and computationally intensive task.\n",
-        "Changing the architecture of a neural network or the learning rate of an optimizer can have a significant impact on the performance.\n",
-        "\n",
-        "The goal of hyperparameter tuning is to optimize the hyperparameters in a way that improves the performance of the machine learning or deep learning model.\n",
-        "The simplest, but also most computationally expensive, approach uses manual search (or trial-and-error [@Meignan:2015vp]).\n",
-        "Commonly encountered is simple random search, i.e., random and repeated selection of hyperparameters for evaluation, and lattice search (\"grid search\").\n",
-        "In addition, methods that perform directed search  and other model-free algorithms, i.e., algorithms that do not explicitly rely on a model, e.g., evolution strategies [@Bart13j] or pattern search [@Torczon00] play an important role.\n",
-        "Also, \"hyperband\", i.e., a multi-armed bandit strategy that dynamically allocates resources to a set of random configurations and uses successive bisections to stop configurations with poor performance [@Li16a], is very common in hyperparameter tuning.\n",
-        "The most sophisticated and efficient approaches are the Bayesian optimization and surrogate model based optimization methods, which are based on the optimization of cost functions determined by simulations or experiments.\n",
-        "\n",
-        "We consider below a surrogate model based optimization-based hyperparameter tuning approach based on the Python version of the SPOT (\"Sequential Parameter Optimization Toolbox\") [@BLP05], which is suitable for situations where only limited resources are available. This may be due to limited availability and cost of hardware, or due to the fact that confidential data may only be processed locally, e.g., due to legal requirements.\n",
-        "Furthermore, in our approach, the understanding of algorithms is seen as a key tool for enabling transparency and explainability. This can be enabled, for example, by quantifying the contribution of machine learning and deep learning components (nodes, layers, split decisions, activation functions, etc.).\n",
-        "Understanding the importance of hyperparameters and the interactions between multiple hyperparameters plays a major role in the interpretability and explainability of machine learning models.\n",
-        "SPOT provides statistical tools for understanding hyperparameters and their interactions. Last but not least, it should be noted that the SPOT software code is available in the open source `spotPython` package on github^[[https://github.com/sequential-parameter-optimization](https://github.com/sequential-parameter-optimization)], allowing replicability of the results.\n",
-        "This tutorial descries the Python variant of SPOT, which is called `spotPython`. The R implementation is described in @bart21i.\n",
-        "SPOT is an established open source software that has been maintained for more than 15 years [@BLP05] [@bart21i].\n",
-        "\n",
-        "This tutorial is structured as follows. The concept of the hyperparameter tuning software `spotPython` is described in @sec-spot. \n",
-        "@sec-quickstart (\"Quickstart\") describes the execution of the example from the tutorial \"Hyperparameter Tuning with Ray Tune\" [@pyto23a].\n",
-        "@sec-hyperparameter-tuning-for-pytorch describes the integration of `spotPython` into the ``PyTorch`` training workflow in detail and presents the results. Finally, @sec-summary presents a summary and an outlook.\n",
-        "\n",
-        "::: {.callout-note}\n",
-        "The corresponding ` .ipynb` notebook [@bart23e] is updated regularly and reflects updates and changes in the `spotPython` package.\n",
-        "It can be downloaded from [https://github.com/sequential-parameter-optimization/spotPython/blob/main/notebooks/14_spot_ray_hpt_torch_cifar10.ipynb](https://github.com/sequential-parameter-optimization/spotPython/blob/main/notebooks/14_spot_ray_hpt_torch_cifar10.ipynb).\n",
-        ":::\n",
-        "\n",
-        "\n",
-        "# The Hyperparameter Tuning Software SPOT {#sec-spot}\n",
-        "\n",
-        "Surrogate model based optimization methods are common approaches in simulation and optimization. SPOT was developed because there is a great need for sound statistical analysis of simulation and optimization algorithms. SPOT includes methods for tuning based on classical regression and analysis of variance techniques.\n",
-        "It presents tree-based models such as classification and regression trees and random forests as well as Bayesian optimization (Gaussian process models, also known as Kriging). Combinations of different meta-modeling approaches are possible. SPOT comes with a sophisticated surrogate model based optimization method, that can handle discrete and continuous inputs. Furthermore, any model implemented in `scikit-learn` can be used out-of-the-box as a surrogate in `spotPython`.\n",
-        "\n",
-        "SPOT implements key techniques such as exploratory fitness landscape analysis and sensitivity analysis. It can be used to understand the performance of various algorithms, while simultaneously giving insights into their algorithmic behavior.\n",
-        "In addition, SPOT can be used as an optimizer and for automatic and interactive tuning. Details on SPOT and its use in practice are given by @bart21i.\n",
-        "\n",
-        "A typical hyperparameter tuning process with `spotPython` consists of the following steps:\n",
-        "\n",
-        "1. Loading the data (training and test datasets), see @sec-data-loading.\n",
-        "2. Specification of the preprocessing model, see @sec-specification-of-preprocessing-model. This model is called `prep_model` (\"preparation\" or pre-processing).\n",
-        "The information required for the hyperparameter tuning is stored in the dictionary `fun_control`. Thus, the information needed for the execution of the hyperparameter tuning is available in a readable form.\n",
-        "3. Selection of the machine learning or deep learning model to be tuned, see @sec-selection-of-the-algorithm. This is called the `core_model`. Once the `core_model` is defined, then the associated hyperparameters are stored in the `fun_control` dictionary. First, the hyperparameters of the `core_model` are initialized with the default values of the `core_model`.\n",
-        "As default values we use the default values contained in the `spotPython` package for the algorithms of the `torch` package.\n",
-        "4. Modification of the default values for the hyperparameters used in `core_model`, see @sec-modification-of-default-values. This step is optional.\n",
-        "   1. numeric parameters are modified by changing the bounds.\n",
-        "   2. categorical parameters are modified by changing the categories (\"levels\").\n",
-        "5. Selection of target function (loss function) for the optimizer, see @sec-selection-of-target-function.\n",
-        "6. Calling SPOT with the corresponding parameters, see @sec-call-the-hyperparameter-tuner. The results are stored in a dictionary and are available for further analysis.\n",
-        "7. Presentation, visualization and interpretation of the results, see @sec-results-tuning.\n",
-        "\n",
-        "\n",
-        "# Quickstart {#sec-quickstart}"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": 1,
       "metadata": {},
       "outputs": [],
       "source": [
-        "#| echo: true\n",
-        "#| eval: false\n",
         "import numpy as np\n",
         "import pandas as pd\n",
-        "import itertools\n",
         "from math import inf\n",
         "import torch\n",
         "import torchmetrics\n",
@@ -132,7 +19,6 @@
         "\n",
         "from spotPython.spot import spot\n",
         "from spotPython.utils.init import fun_control_init\n",
-        "from spotPython.data.torchdata import load_data_cifar10\n",
         "from spotPython.hyperparameters.values import (\n",
         "    add_core_model_to_fun_control,\n",
         "    modify_hyper_parameter_levels,\n",
@@ -146,7 +32,6 @@
         "from spotPython.data.torch_hyper_dict import TorchHyperDict\n",
         "from spotPython.fun.hypertorch import HyperTorch\n",
         "from spotPython.torch.netvbdp import Net_vbdp\n",
-        "# from spotPython.torch.netcifar10 import Net_CIFAR10\n",
         "from spotPython.torch.traintest import (\n",
         "    train_tuned,\n",
         "    test_tuned,\n",
@@ -170,9 +55,6 @@
       "metadata": {},
       "outputs": [],
       "source": [
-        "\n",
-        "#| echo: true\n",
-        "#| eval: false\n",
         "fun_control = fun_control_init(task=\"classification\", tensorboard_path=\"runs/25_spot_torch_vbdp\")\n",
         "fun_control.update({\"show_batch_interval\": 100_000_000})\n",
         "# load data\n",
@@ -200,7 +82,7 @@
           "name": "stderr",
           "output_type": "stream",
           "text": [
-            "/var/folders/hf/rnzlxrlx0fq3fr59krvfr1800000gn/T/ipykernel_46069/3336454340.py:8: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
+            "/var/folders/dw/pvtj6mt91znd0hftcztqb0k00000gn/T/ipykernel_12542/3336454340.py:8: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
             "  train_df[target_column] = target\n"
           ]
         },
@@ -496,10 +378,30 @@
             "\n",
             "config: {'_L0': 6112, 'l1': 32768, 'dropout_prob': 0.7103122166156, 'lr_mult': 0.001, 'batch_size': 4, 'epochs': 64, 'k_folds': 1, 'patience': 64, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}\n",
             "Epoch: 1\n",
-            "Loss on hold-out set: 2.39726079185054\n",
-            "Accuracy on hold-out set: 0.10377358490566038\n",
-            "MAPK value on hold-out data: 0.19732704402515722\n",
-            "Epoch: 2\n"
+            "Loss on hold-out set: 2.397013398836244\n",
+            "Accuracy on hold-out set: 0.14622641509433962\n",
+            "MAPK value on hold-out data: 0.23506289308176098\n",
+            "Epoch: 2\n",
+            "Loss on hold-out set: 2.3962632890017526\n",
+            "Accuracy on hold-out set: 0.12735849056603774\n",
+            "MAPK value on hold-out data: 0.2272012578616352\n",
+            "Epoch: 3\n",
+            "Loss on hold-out set: 2.3953690034038617\n",
+            "Accuracy on hold-out set: 0.14622641509433962\n",
+            "MAPK value on hold-out data: 0.23977987421383645\n",
+            "Epoch: 4\n",
+            "Loss on hold-out set: 2.394227927585818\n",
+            "Accuracy on hold-out set: 0.13679245283018868\n",
+            "MAPK value on hold-out data: 0.2366352201257861\n",
+            "Epoch: 5\n",
+            "Loss on hold-out set: 2.3928299624964877\n",
+            "Accuracy on hold-out set: 0.13679245283018868\n",
+            "MAPK value on hold-out data: 0.23663522012578614\n",
+            "Epoch: 6\n",
+            "Loss on hold-out set: 2.3910861375196926\n",
+            "Accuracy on hold-out set: 0.13679245283018868\n",
+            "MAPK value on hold-out data: 0.23506289308176098\n",
+            "Epoch: 7\n"
           ]
         }
       ],
@@ -723,7 +625,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.10.11"
+      "version": "3.10.10"
     }
   },
   "nbformat": 4,