diff --git a/.gitignore b/.gitignore index a20707a..ab1d214 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ /docs/build/ Manifest.toml build/ +.DS_Store diff --git a/Project.toml b/Project.toml index 86ea0a5..427e8aa 100644 --- a/Project.toml +++ b/Project.toml @@ -1,7 +1,15 @@ name = "RidgeRegression" uuid = "739161c8-60e1-4c49-8f89-ff30998444b1" -authors = ["Vivak Patel "] version = "0.1.0" +authors = ["Eton Tackett ", "Vivak Patel "] + +[deps] +CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b" +DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" +Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6" [compat] +CSV = "0.10.15" +DataFrames = "1.8.1" +Downloads = "1.7.0" julia = "1.12.4" diff --git a/README.md b/README.md index 542e8fe..f08f8ec 100644 --- a/README.md +++ b/README.md @@ -5,3 +5,180 @@ [![Build Status](https://github.com/vp314/RidgeRegression.jl/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/vp314/RidgeRegression.jl/actions/workflows/CI.yml?query=branch%3Amain) [![Coverage](https://codecov.io/gh/vp314/RidgeRegression.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/vp314/RidgeRegression.jl) [![Code Style: Blue](https://img.shields.io/badge/code%20style-blue-4495d1.svg)](https://github.com/invenia/BlueStyle) + +# Project Overview + +This project investigates the performance of numerical algorithms for solving the ridge regression problem under varying dimension regimes and conditioning levels. + +# Directory Structure + +The source-code layout will be structured as follows: +```text +. +├── Project.toml +│ +├── src +│ ├── RidgeRegression.jl +│ ├── units.jl + ├── treatments.jl + ├── measurements.jl +│ └── algorithms +│ ├── closed_form.jl +│ ├── gradient_descent.jl +│ ├── stochastic_gradient_descent.jl +│ └── bidiagonalization.jl +│ +├── test +│ ├── Project.toml +│ ├── runtests.jl +│ └── src +│ ├── RidgeRegression_test.jl +│ ├── units +│ │ ├── units_dataset_tests.jl +│ │ ├── units_one_hot_encode_tests.jl + ├── units_end_to_end_tests.jl +│ │ └── units_load_csv_dataset_tests.jl +│ ├── treatments +│ │ └── treatments_test.jl +│ ├── measurements +│ │ └── measurements_test.jl +│ └── algorithms +│ ├── closed_form_test.jl +│ ├── gradient_descent_test.jl +│ ├── stochastic_gradient_descent_test.jl +│ └── bidiagonalization +│ ├── bidiagonalization_compute_givens_tests.jl +│ ├── bidiagonalization_rotate_rows_tests.jl +│ ├── bidiagonalization_rotate_cols_tests.jl +│ ├── bidiagonalization_apply_Ht_to_b_tests.jl +│ └── bidiagonalization_with_H_tests.jl +│ +└── docs + ├── make.jl + └── src + ├── design.md + ├── getting_started_guide.md + ├── experimental_pipeline.md + ├── algorithm_explanations.md + ├── dataset.md + ├── measurements.md + ├── output.md + └── index.md +``` +# PR Schedule and Roadmap + +## PR 1: Experimental Design +**Expected Date:** June 19, 2026 + +### [WHY] +A rigorous experimental framework ensures that the ridge regression algorithms are evaluated under identical conditions, enabling fair and reproducible comparisons across methods. + +### [WHAT] +This PR introduces the experimental design. The primary deliverable is a design document that specifies the structure of the study, including the experimental units, treatments, blocking factors, measurements, and observations that will be used to evaluate ridge regression algorithms. + +### [HOW] +This PR accomplishes the experimental design by creating a formal design document in `docs/src/design.md`. The document will define the experimental units, treatments, blocking factors, measurements, and observations used throughout the project. It will specify the problem dimensions, conditioning regimes, noise levels, regularization parameters, and benchmark metrics that will be used to evaluate ridge regression algorithms. + +### [SO WHAT] +This PR establishes the foundation for all algorithm comparisons and ensures that experimental results are reproducible and meaningful. + +## PR 2: Units.jl and Corresponding Tests +**Expected Date:** June 30, 2026 + +### [WHY] +This project requires our experimental units to be defined in accordance with the experimental design. The experimental units need to have certain properties and be consistent and reproducible throughout the experiment. + +### [WHAT] +This PR introduces `units.jl` and the corresponding tests. The module will provide a framework for generating and managing experimental units that conform to the specifications established in the experimental design, including factors such as problem dimension, conditioning, among others. It will also ensure that all generated units are reproducible, consistent across experimental conditions, and validated through unit testing. + +### [HOW] +This PR ensures that `units.jl` and the corresponding tests satisfy the experimental design requirements through a combination of testing, code coverage, and end-to-end pipeline validation. Unit tests will verify that the generated experimental units possess certain properties specified in the design. Code coverage will be run to ensure sufficient coverage and appropriate handling of edge cases. In addition, end-to-end pipeline checks will be performed to verify that the experimental units are compatible with future algorithms and measurements/observations needed. + +### [SO WHAT] +This PR ensures that we have generated experimental units consistent with our design and it allows us to apply treatments (Ridge Regression Algorithms) so that we can collect measurements and observations to analyze and compare the performance of these algorithms. + +### [FILES and Functions] +| File | Structure / Function | Purpose | +|------|----------------------|---------| +| `src/units.jl` | `Dataset{TX<:AbstractMatrix, TY<:AbstractVector}` | Defines a dataset as an experimental unit for ridge regression experiments. Stores the design matrix `X`, response vector `y`, and dataset `name` while allowing dense or sparse matrix types. | +| `src/units.jl` | `Dataset(name::String, X::AbstractMatrix, y::AbstractVector)` | Constructs a `Dataset` object and validates that the number of rows in `X` matches the length of `y`. | +| `src/units.jl` | `one_hot_encode(Xdf::DataFrame; cols_to_encode, drop_first=true)` | Converts selected categorical columns in a feature `DataFrame` into numeric dummy variables while leaving numeric columns unchanged. | +| `src/units.jl` | `load_csv_dataset(path_or_url::String; target_col, cols_to_encode=Symbol[], name="csv_dataset")` | Loads a dataset from a local CSV file or URL, removes missing observations, separates features from the target column, applies one-hot encoding, and returns a `Dataset` object. | +| `test/src/units/units_dataset_tests.jl` | `Dataset` constructor tests | Verify that valid matrices and response vectors produce a `Dataset`, and that mismatched dimensions throw an `ArgumentError`. | +| `test/src/units/units_one_hot_encode_test.jl` | Encoding tests | Verify that categorical variables are correctly one-hot encoded, that numeric columns are preserved, and that invalid nonnumeric columns trigger appropriate errors. | +| `test/src/units/units_load_csv_dataset_tests.jl` | CSV-loading tests | Verify that CSV data can be loaded, cleaned, encoded, and converted into a valid `Dataset` object. | +| `test/src/units/units_end_to_end_tests.jl` | End-to-end dataset pipeline tests | Verify that raw tabular data can move through the full pipeline: CSV loading, preprocessing, encoding, dataset construction, and compatibility with downstream ridge regression routines. | + +## PR 3: Golub Kahan Bidiagonalization and Corresponding Tests +**Expected Date:** June 30, 2026 + +### [WHY] +This project requires efficient and stable methods for solving ridge regression problems. Direct methods are an important baseline against which iterative and stochastic approaches can be compared. Golub-Kahan bidiagonalization is a direct method that transforms the data matrix into a bidiagonal form through a series of orthogonal transformations, yielding a simpler problem that can be solved more efficiently. +### [WHAT] +This PR introduces `bidiagonalization.jl` and the corresponding tests. The module implements Golub-Kahan bidiagonalization using a sequence of Givens rotations to get the matrix into upper bidiagonal form. The implementation includes routines for computing Givens rotation coefficients, applying orthogonal transformations to matrix rows and columns, accumulating the orthogonal matrices (H and K), and applying the resulting transformations to the constant vector. The module serves as the project's first direct method for solving ridge regression problems. + +### [HOW] +This PR ensures correctness and reliability through a combination of unit testing, code coverage analysis, structural validation, and end-to-end pipeline checks. Unit tests will verify the correctness of Givens rotation coefficients, row and column transformations, and bidiagonalization procedures on square and rectangular matrices. Structural tests will confirm that the computed matrices satisfy the expected properties, including orthogonality of H and K, preservation of matrix dimensions, and the relation (H^T A K = B), where B is upper bidiagonal. Code coverage analysis will be used to ensure that core computational paths and edge cases are exercised, while end-to-end tests will verify compatibility with downstream ridge regression solvers and benchmarking routines. + +### [SO WHAT] +This PR is the first algorithm (treatment) in the project and establishes a direct method baseline for comparison with future gradient-based and stochastic approaches. + +### [FILES and Functions] +| File | Structure / Function | Purpose | +|------|----------------------|---------| +| `src/algorithms/bidiagonalization.jl` | `compute_givens(...)` | Computes the cosine and sine coefficients defining a Givens rotation used to eliminate selected matrix entries. | +| `src/algorithms/bidiagonalization.jl` | `rotate_rows!(...)` | Applies a Givens rotation to two rows of a matrix during the left-transformation stage of the bidiagonalization procedure. | +| `src/algorithms/bidiagonalization.jl` | `rotate_cols!(...)` | Applies a Givens rotation to two columns of a matrix during the right-transformation stage of the bidiagonalization procedure. | +| `src/algorithms/bidiagonalization.jl` | `apply_Ht_to_b(...)` | Applies the accumulated left orthogonal transformations to the constant vector, producing the transformed right-hand side of the reduced problem. | +| `src/algorithms/bidiagonalization.jl` | `bidiagonalize_with_H(...)` | Performs Golub–Kahan Bidiagonalization using Givens rotations and accumulates the orthogonal transformations required to reduce a matrix to upper bidiagonal form. | +| `test/src/algorithms/bidiagonalization/bidiagonalization_compute_givens_test.jl` | Givens rotation tests | Verify that the computed rotation coefficients satisfy the expected numerical and trigonometric properties. | +| `test/src/algorithms/bidiagonalization/bidiagonalization_rotate_rows_test.jl` | Row rotation tests | Verify that row rotations correctly eliminate targeted entries while preserving orthogonality. | +| `test/src/algorithms/bidiagonalization/bidiagonalization_rotate_cols_test.jl` | Column rotation tests | Verify that column rotations correctly eliminate targeted entries while preserving orthogonality. | +| `test/src/algorithms/bidiagonalization/bidiagonalization_apply_Ht_to_b_test.jl` | Transformation tests | Verify that accumulated orthogonal transformations are correctly applied to the constant vector. | +| `test/src/algorithms/bidiagonalization/bidiagonalization_with_H_test.jl` | Bidiagonalization tests | Verify that the resulting matrix is upper bidiagonal and that the accumulated orthogonal matrices satisfy the expected structural properties. | + + +## PR 4: Gradient Based Optimization and Corresponding Tests +**Expected Date:** July 6, 2026 + +### [WHY] +This project requires efficient and stable methods for solving ridge regression problems. Iterative optimization methods such as gradient descent provide an alternative approach to direct methods, which may become prohibitive or too costly. +### [WHAT] +This PR introduces `gradient_descent.jl` and the corresponding tests. The module will implement gradient-based optimization methods for solving ridge regression problems, including routines for evaluating objective functions, computing gradients, and performing iterative updates. + +### [HOW] +This PR ensures correctness and reliability through a combination of unit testing, code coverage analysis, convergence validation, and end-to-end pipeline checks. + +### [SO WHAT] +This PR establishes the project's first iterative optimization baseline and provides a foundation for comparing direct and optimization-based approaches to ridge regression. The resulting implementation will enable experiments evaluating convergence behavior, computational efficiency, solution accuracy, and robustness across a variety of problem settings. + +### [FILES AND FUNCTIONS] +The exact function names may evolve, but this PR is expected to include the following core components: +| File | Structure / Function | Purpose | +|------|----------------------|---------| +| `src/algorithms/gradient_descent.jl` | `ridge_objective_evaluation(...)` | Evaluate the ridge regression objective for a given coefficient vector. | +| `src/algorithms/gradient_descent.jl` | `ridge_gradient_calculation(...)` | Compute the gradient of the ridge regression objective. | +| `src/algorithms/gradient_descent.jl` | `gradient_descent(...)` | Implement the main iterative update procedure for solving ridge regression problems. | +| `src/algorithms/gradient_descent.jl` | `stopping_criterion(...)` | Determine when the iterative method should terminate based on tolerance, maximum iterations, or convergence behavior. | +| `src/algorithms/gradient_descent.jl` | `gradient_descent_results(...)` | Store or return solution information such as coefficients, objective values, iteration count, and convergence status. | +| `test/src/algorithms/gradient_descent_test/gradient_descent_objective_gradient_test.jl` | Objective and gradient tests | Verify that objective values and gradients are computed correctly. | +| `test/src/algorithms/gradient_descent_test/gradient_descent_update_rule_test.jl` | Update rule tests | Verify that gradient descent updates move in the expected direction and reduce the objective under appropriate conditions. | +| `test/src/algorithms/gradient_descent_test/gradient_descent_convergence_test.jl` | Convergence tests | Verify that the method approaches known ridge regression solutions on small benchmark problems. | +| `test/src/algorithms/gradient_descent_test/gradient_descent_pipeline_test.jl` | End-to-end pipeline tests | Verify compatibility with experimental units, benchmarking routines, and downstream measurement collection. | + +## PR 5: Stochastic Optimization and Corresponding Tests +**Expected Date:** July 30, 2026 + +### [WHY] +This project requires solving the ridge regression problem in settings where traditional methods may become infeasible. In large-scale settings, stochastic optimization methods provide an alternative by approximating the optimization problem using random samples rather than processing the entire dataset at every iteration. These methods are particularly when the dimensions of the problem become too large. + +### [WHAT] +This PR introduces stochastic optimization methods and the corresponding tests. The module will implement stochastic optimization methods for solving ridge regression problems, including stochastic gradient descent and related variants. + +### [HOW] +This PR ensures correctness and reliability through a combination of unit testing, code coverage analysis, convergence validation, and end-to-end pipeline checks. + +### [SO WHAT] +This PR extends the project's ability to solve ridge regression problems beyond the regimes where direct and traditional iterative methods are practical. By introducing stochastic optimization methods, we can apply treatments to larger experimental units and collect the measurements and observations necessary to compare algorithm performance across a broader range of problem dimensions and computational settings. + +### [TO DO] \ No newline at end of file diff --git a/docs/make.jl b/docs/make.jl index a1097bb..d42cfbe 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -14,6 +14,7 @@ makedocs(; ), pages=[ "Home" => "index.md", + "Design" => "design.md", ], ) diff --git a/docs/src/design.md b/docs/src/design.md new file mode 100644 index 0000000..db8541e --- /dev/null +++ b/docs/src/design.md @@ -0,0 +1,182 @@ +# Motivation and Background +Many modern science problems involve regression problems with extremely large numbers of +predictors. +[Genome-wide association studies (GWAS)](https://doi.org/10.1371/journal.pone.0245376), +for example, try to identify genetic +variants associated with a disease phenotype using hundreds of thousands or millions of +genomic features. In such settings, traditional least squares methods fail. +Penalized Least Squares (PLS) extends ordinary least squares (OLS) +regression by adding a penalty term to shrink parameter estimates. Ridge regression, an +approach within PLS, adds a regularization term, producing a regularized estimator. + +Mathematically, ridge regression estimates the regression coefficients by solving the +penalized least squares problem +```math +\hat{\boldsymbol{\beta}} = +\arg\min_{\boldsymbol{\beta}} +\left( +\| \mathbf{y} - X\boldsymbol{\beta} \|_2^2 ++ +\lambda \| \boldsymbol{\beta} \|_2^2 +\right) +``` +where $\lambda > 0$ is a regularization parameter that controls the strength of the penalty. + +The purpose of ridge regression is to stabilize regression estimates when the predictors are +highly correlated or the design matrix $X$ is nearly singular. Ridge regression modifies the +least squares objective by adding a penalty on the squared $\ell_2$-norm of the coefficient +vector. The estimator is obtained by minimizing a penalized least squares objective in which +large coefficient values are discouraged through the penalty term $\lambda +\|\boldsymbol{\beta}\|_2^2$. This penalty shrinks the estimated coefficients toward the +origin, which reduces the variance of the estimator and mitigates the effects of +multicollinearity. + +There are many numerical algorithms available to compute ridge regression estimates +including direct methods, Krylov subspace methods, gradient-based optimization, coordinate +descent, and stochastic gradient descent. These algorithms differ in their computational +costs and numerical stability. + +The goal of this experiment is to investigate the performance of these algorithms when we +vary the structure and scale of the regression problem. To do this, we consider the linear +model $\mathbf{y} = X\boldsymbol{\beta} + \boldsymbol{\varepsilon}$ where the matrix ${X}$ +may be constructed with varying dimensions, sparsity patterns, and conditioning properties. +# Questions +The primary goal of this experiment is to compare numerical algorithms for computing ridge +regression estimates under various conditions. In particular, we aim to address the +following questions: + +1. How does the performance of ridge regression algorithms change as the structural + and numerical properties of the regression problem vary? + +2. Which ridge regression algorithm provides the best balance between numerical + stability and computational cost across these problem regimes? + +# Experimental Units +An experimental unit is defined by a triplet: a design matrix, $X$; a response vector, $y$; +and a non-negative regularization parameter $\lambda$. + +Datasets will be generated either fully artificially or semi-artificially. In the artifical setting, both the design matrix (X) and the response vector (y) are generated. In the semi-artifical setting, the design matrix is fixed from a real dataset (e.g. GWAS) and the response vector is generated conditional on X. This is done so that we can compute solution error. + +Experimental units are selected based on factors we believe will potentially impact +the performance of a specific algorithm. +These factors, or blocks, correspond to: +- The dimensional regime of $X \in \mathbb{R}^{n \times p}$: + $p \ll n$, $p ≈ n$, and $p \gg n$. +- The magnitude of the regularization parameter, $\lambda$, as described below. + +Matrix sparsity is a factor that can influence algorithm performance. However, we will not consider it in this experiment because none of the algorithms under consideration are implemented in a manner that exploits sparse matrix structure. + +The regularization parameter is often selected using one of the following strategies, +depending on application domain: + +- [Generalized Cross Validation](https://doi.org/10.1080/00401706.1979.10489751) +- [The L-Curve Method](https://doi.org/10.1137/1034115) +- [Information Criteria](https://doi.org/10.1002/wics.1607) +- [Morozov's Discrepancy Principle](https://doi.org/10.1088/0266-5611/26/2/025001) + +Excluding Morozov's discrepancy principle, the remaining methods require +computing $\hat{\beta}_\lambda = (X^\top X + \lambda I)^{-1}X^\top y$ or +$\Vert X\hat{\beta}_\lambda - y \Vert_2$ over a grid of values for $\lambda$. +Given our assumption that problem conditioning will impact algorithm behavior, +we will choose this grid to modify the problem's condition number systematically. + +Recall, the ridge estimator's closed form is given by solving +```math +\min_{\beta} \left\Vert \begin{bmatrix} X \\ \lambda I \end{bmatrix}\beta - +y \right\Vert_2^2, +``` +or (equivalently, from a mathematical perspective) solving +$(X^\top X + \lambda I)\hat{\beta}_\lambda = X^\top y$. +Let $\sigma_{\max}$ denote the largest singular value +of $X$, and let $\sigma_{\min}$ denote the smallest singular value of +$X$ (note, $0$ is allowed). +Then the condition number for the normal systems approach is +```math +\kappa(X^\top X+\lambda I) += +\frac{\sigma_{\max}^2+\lambda}{\sigma_{\min}^2+\lambda}. +``` + +There are two cases we consider. +- Case 1: $\sigma_{\min}=0$. In this case, the least squares problem is ill-posed. We choose + the smallest value of $\lambda$ in the grid to be + $\sigma_{\max}^2 / (10^{12} - 1)$, which ensures + $\kappa(X^\top X + \lambda I) = 10^{12}$. +- Case 2: $\sigma_{\min}>0$. In this case, we choose the smallest value of $\lambda$ in the + grid to be $\sigma_{\max}\sigma_{\min}$, which ensures + $\kappa(X^\top X + \lambda I) = \sigma_{\max}/\sigma_{\min}$. + That is, it sets the regularized normal system's condition number to that of the original + least squares problem. + +In both cases, assuming $\sigma_{\max}^2 > 2 \sigma_{\min}$, we choose the larges value of +$\lambda$ to be $\sigma_{\max}^2 - 2 \sigma_{\min}$. The condition number for the +regularized normal equations is $\kappa(X^\top X + \lambda I)=2$. + +For each experimental unit, a grid of 15 regularization parameters will be constructed between $\lambda _{min}$ and $\lambda _{max}$. The endpoints of the grid are chosen to correspond to the range of condition numbers of the normal equations. Intermediate values will be selected using a log-scale. We will use logarithmic scaling because $\kappa(X^\top X+\lambda I)$ changes rapidly as $\lambda \rightarrow 0$, making linear spacing insufficient for capturing performance. + +Suppose we have m values, $\lambda _1\:,...,\:\lambda _m$ where $\lambda _1=\lambda_{min}$ and $\lambda _m=\lambda _{max}$. Logarithmic scaling tells us we want the ratio between these values to be constant. To ensure this, we consider a geometric series of the form: +```math +\lambda _i=\lambda _{min}\left(r\right)^{i-1}, i = 1, 2, ..., 15 +``` +Our goal here is to derive the common ratio r. Relating $\lambda _{min}$ and $\lambda _{max}$ gives us +```math +\lambda _{max}=\lambda _{min}\left(r\right)^{m-1} +``` +Using algebra to isolate r +```math +\frac{\lambda_{\text{max}}}{\lambda_{\text{min}}} = r^{m-1} +``` +Applying logarithms and rearranging we obtain +```math +r=e^{\frac{log\left(\frac{\lambda _{max}}{\lambda _{min}}\right)}{m-1}}=\left(\frac{\lambda _{max}}{\lambda _{min}}\right)^{\frac{1}{m-1}} +``` +In our case $m=15$ + +The total number of block combinations is determined by the product of the number of levels +in each factor, denoted b. We will also denote r to be the number of replicated +datasets in each block. Here, we mean the number datasets within a block. The total number +of experimental units is then ${b * r}$. + + +| Blocking System | Factor | Blocks | +|:----------------|:-------|:-------| +| Dataset | Dimensional regime | $(p \ll n)$, $(p \approx n)$, $(p \gg n)$| +| Conditioning | Condition number of $X^\top X + \lambda I$ | 15 logartihmically spaced $\lambda$ values so that $\kappa(X^\top X+\lambda I)$ ranges from poorly conditioned to conditioned. | + +# Treatments + +The treatments are the ridge regression solution methods: + +- Gradient-based optimization +- Stochastic gradient descent +- Direct Methods + - Golub Kahan Bidiagonalization + + Since each experimental unit will recieves all t treatments, the total number of algorithm + runs in the experiment is ${t * b * r}$. For this experiment, ${t=3}$. To ensure fair + comparison between algorithms, each treatment will be applied under a fixed wall time + constraint. Each algorithm will be run for a maximum of two hours per experimental unit. +# Observational Units and Measurements + +The observational units are each algorithm-dataset pair. For each combination we will observe the following + +| Column Name | Data Type | Description | +|:---|:---|:---| +| `dataset_id` | Positive Integer | Identifier for the generated dataset (experimental unit). | +| `dimensional_regime` | String | Relationship between predictors and observations: `p << n`, `p ≈ n`, or `p >> n`. | +| `lambda` | Positive Floating-point| Regularization parameter used. | +| `condition_number` | Positive Floating-point| $\kappa(X^\top X+\lambda I)$ corresponding to the selected $\lambda$ | +| `algorithm` | String | Ridge regression solution method used: `GradientDescent`, `SGD`, or `DirectMethod`. | +| `runtime_seconds` | Positive Floating-point | Time required for the algorithm to compute a solution. | +| `iterations` | Positive Integer | Number of iterations performed by the algorithm (`NA` for direct methods). | +| `step_size` | Positive Floating-point | Step size used in gradient descent or SGD (`NA` for direct methods). | +| `batch_size` | Positive Integer | Number of samples used per SGD update (`NA` for direct methods and gradient descent). | +| `number_of_epochs` | Positive Integer | Number epochs per observation (`NA` for direct methods). | + + + +The collected measurements will be written to a CSV file. Each row in the file corresponds +to a single algorithm–dataset pair, which forms the observational unit of the experiment. +The columns represent the recorded measurements. After the experiment, the resulting CSV +file should contain ${Algorithms∗Datasets}$ number of rows and each row will contain exactly +10 columns. \ No newline at end of file diff --git a/test/Project.toml b/test/Project.toml index 0c36332..73141b0 100644 --- a/test/Project.toml +++ b/test/Project.toml @@ -1,2 +1,5 @@ [deps] +CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b" +DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" +LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" \ No newline at end of file