diff --git a/.env.d.examples/00_crossref-local.env b/.env.d.examples/00_crossref-local.env index 00eaaeeaa..e71b2e9b5 100644 --- a/.env.d.examples/00_crossref-local.env +++ b/.env.d.examples/00_crossref-local.env @@ -11,5 +11,5 @@ CROSSREF_LOCAL_API_URL=http://localhost:31291 CROSSREF_LOCAL_HOST=0.0.0.0 CROSSREF_LOCAL_PORT=31291 -# Path to SQLite database (for db mode) +# Path to the local database file (for db mode) # CROSSREF_LOCAL_DB=/path/to/crossref.db diff --git a/.gitignore b/.gitignore index 20d3a496a..7a4f50d38 100755 --- a/.gitignore +++ b/.gitignore @@ -144,8 +144,6 @@ cover/ # Django stuff: # *.log local_settings.py -db.sqlite3 -db.sqlite3-journal # Flask stuff: instance/ @@ -389,8 +387,6 @@ cover/ # Django stuff: *.log local_settings.py -db.sqlite3 -db.sqlite3-journal # Flask stuff: instance/ diff --git a/.scitex/clew/runtime/db.sqlite b/.scitex/clew/runtime/db.sqlite deleted file mode 100644 index e51371676..000000000 Binary files a/.scitex/clew/runtime/db.sqlite and /dev/null differ diff --git a/README.md b/README.md index bb8a13e69..0cf971375 100755 --- a/README.md +++ b/README.md @@ -526,12 +526,12 @@ Extends stdlib `logging` with SUCCESS/FAIL levels, a 30+ class exception tree (`
-scitex.db -- SQLite3 / PostgreSQL with ndarray BLOB Storage +scitex.db -- PostgreSQL with ndarray BLOB Storage ```python import scitex as stx, numpy as np -db = stx.db.SQLite3("experiments.db") +db = stx.db.PostgreSQL(dbname="experiments") with db: # context-manager transaction db.execute("CREATE TABLE IF NOT EXISTS runs (id TEXT, acc REAL)") db.save_array("weights_epoch_87", np.random.rand(1024, 1024)) # compressed BLOB @@ -541,7 +541,7 @@ w = db.load_array("weights_epoch_87") # typed ndarray back db.check_health() # integrity + schema drift stx.db.delete_duplicates(conn, "runs", columns=["id"]) ``` -SQLite / PostgreSQL clients with first-class compressed-ndarray BLOBs, dataframe round-trips, health checks, and duplicate removal. Drop-in replacement for hand-rolling `pickle → BLOB` storage or SQLAlchemy Core when you don't need an ORM. +A PostgreSQL client with first-class compressed-ndarray BLOBs, dataframe round-trips, health checks, and duplicate removal. Drop-in replacement for hand-rolling `pickle → BLOB` storage or SQLAlchemy Core when you don't need an ORM.
diff --git a/docs/05_ADDITIONAL_MODULES.md b/docs/05_ADDITIONAL_MODULES.md index f880361c1..531ee6094 100644 --- a/docs/05_ADDITIONAL_MODULES.md +++ b/docs/05_ADDITIONAL_MODULES.md @@ -118,7 +118,7 @@ Lower-level SciTeX utilities re-exported under the umbrella: | `stx.dict` | `DotDict` + safe merge / flatten | `DotDict`, `safe_merge`, `flatten` | | `stx.logging` | stdlib-logging + SUCCESS/FAIL + `SciTeXError` | `getLogger`, `warn_deprecated`, `Tee` | | `stx.types` | Union type aliases + predicates | `ArrayLike`, `ColorLike`, `is_array_like` | -| `stx.db` | SQLite3 / PostgreSQL wrapper w/ ndarray BLOBs | `SQLite3`, `PostgreSQL`, `delete_duplicates` | +| `stx.db` | PostgreSQL wrapper w/ ndarray BLOBs | `PostgreSQL`, `delete_duplicates` | | `stx.audit` | Unified security scan (bandit / shellcheck / pip-audit) | `audit()` | | `stx.browser` | Playwright helpers for scraping | `save_as_pdf`, `click_with_fallbacks_async` | | `stx.compat` | Deprecation shims | `@deprecated`, `notify` legacy alias | diff --git a/docs/guides/CROSSREF_API_CONFIGURATION.md b/docs/guides/CROSSREF_API_CONFIGURATION.md index 6c0b09b35..da4c9563b 100644 --- a/docs/guides/CROSSREF_API_CONFIGURATION.md +++ b/docs/guides/CROSSREF_API_CONFIGURATION.md @@ -146,7 +146,7 @@ else: - **Total papers**: 167,008,748 - **Database size**: 1.2TB (1,197,367 MB) -- **Format**: SQLite with indexed fields +- **Format**: single-file local database with indexed fields - **Source**: CrossRef official database ## Best Practices diff --git a/docs/sphinx/core_concepts.rst b/docs/sphinx/core_concepts.rst index 4484557c4..2a9be782e 100644 --- a/docs/sphinx/core_concepts.rst +++ b/docs/sphinx/core_concepts.rst @@ -37,7 +37,7 @@ The Session Model 2. **Logging**: All stdout/stderr captured to ``script.log`` 3. **Config injection**: YAML files from ``./config/`` merged and injected 4. **CLI generation**: Function parameters become ``--flags`` -5. **Provenance**: File hashes recorded to SQLite for reproducibility +5. **Provenance**: File hashes recorded to a local database for reproducibility .. code-block:: python @@ -124,7 +124,7 @@ Provenance Tracking (Clew) -------------------------- Inside ``@stx.session``, every ``stx.io.save`` and ``stx.io.load`` call -records the file's SHA-256 hash to a local SQLite database. This enables: +records the file's SHA-256 hash to a local database. This enables: - **Verification**: Check if output files have been modified since creation - **DAG reconstruction**: Trace which inputs produced which outputs diff --git a/docs/sphinx/modules/io.rst b/docs/sphinx/modules/io.rst index 222150bb8..d90bc5498 100644 --- a/docs/sphinx/modules/io.rst +++ b/docs/sphinx/modules/io.rst @@ -94,7 +94,7 @@ Provenance Tracking ------------------- Inside ``@stx.session``, every ``save`` and ``load`` records file hashes -to a local SQLite database for reproducibility verification. +to a local database for reproducibility verification. .. code-block:: python diff --git a/examples/_legacy/notebooks/00_SCITEX_MASTER_INDEX.ipynb b/examples/_legacy/notebooks/00_SCITEX_MASTER_INDEX.ipynb index 2eae64d5c..a47429adb 100644 --- a/examples/_legacy/notebooks/00_SCITEX_MASTER_INDEX.ipynb +++ b/examples/_legacy/notebooks/00_SCITEX_MASTER_INDEX.ipynb @@ -202,12 +202,6 @@ "- PDF downloads and bibliography management\n", "- BibTeX generation with enriched metadata\n", "\n", - "### Database Operations\n", - "**🗄️ [19_scitex_db.ipynb](19_scitex_db.ipynb)** - *Database Integration*\n", - "- PostgreSQL and SQLite support\n", - "- SQL operations with pandas integration\n", - "- Data persistence workflows\n", - "\n", "### Documentation & LaTeX\n", "**📝 [20_scitex_tex.ipynb](20_scitex_tex.ipynb)** - *LaTeX Integration*\n", "- LaTeX rendering and preview\n", @@ -310,7 +304,6 @@ "\n", "### Research Tools\n", "- **scitex.scholar** → [16_scitex_scholar.ipynb](16_scitex_scholar.ipynb)\n", - "- **scitex.db** → [19_scitex_db.ipynb](19_scitex_db.ipynb)\n", "- **scitex.tex** → [20_scitex_tex.ipynb](20_scitex_tex.ipynb)\n", "- **scitex.decorators** → [21_scitex_decorators.ipynb](21_scitex_decorators.ipynb)\n", "- **scitex.repro** → [22_scitex_repro.ipynb](22_scitex_repro.ipynb)\n", diff --git a/examples/_legacy/notebooks/19_scitex_db.ipynb b/examples/_legacy/notebooks/19_scitex_db.ipynb deleted file mode 100644 index cd55e36c0..000000000 --- a/examples/_legacy/notebooks/19_scitex_db.ipynb +++ /dev/null @@ -1,824 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Comprehensive SciTeX Database Module Examples\n", - "\n", - "This notebook demonstrates the complete functionality of the `scitex.db` module, which provides database operations and utilities for scientific data management.\n", - "\n", - "## Module Overview\n", - "\n", - "The `scitex.db` module includes:\n", - "- SQLite3 database management with comprehensive mixins\n", - "- PostgreSQL database operations\n", - "- Database inspection and analysis tools\n", - "- Duplicate data detection and removal\n", - "\n", - "## Import Setup" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Detect notebook name for output directory\n", - "import os\n", - "from pathlib import Path\n", - "\n", - "# Get notebook name (for papermill compatibility)\n", - "notebook_name = \"19_scitex_db\"\n", - "if 'PAPERMILL_NOTEBOOK_NAME' in os.environ:\n", - " notebook_name = Path(os.environ['PAPERMILL_NOTEBOOK_NAME']).stem\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import sys\n", - "sys.path.insert(0, '../src')\n", - "\n", - "import sqlite3\n", - "import pandas as pd\n", - "import numpy as np\n", - "import tempfile\n", - "import os\n", - "from pathlib import Path\n", - "\n", - "# Import scitex db module\n", - "import scitex.db as sdb\n", - "\n", - "# Set random seed for reproducibility\n", - "np.random.seed(42)\n", - "\n", - "db_attrs = [attr for attr in dir(sdb) if not attr.startswith('_')]\n", - "for i, attr in enumerate(db_attrs):\n", - " # Loop body" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. Database Inspection Tools\n", - "\n", - "### Creating Sample Database\n", - "\n", - "Let's start by creating a sample database for demonstration purposes." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 1: Create sample database for demonstration\n", - "# Create temporary database file\n", - "temp_db = tempfile.NamedTemporaryFile(suffix='.db', delete=False)\n", - "db_path = temp_db.name\n", - "temp_db.close()\n", - "\n", - "\n", - "# Create sample data\n", - "np.random.seed(42)\n", - "\n", - "# Experimental data table\n", - "experiments_data = {\n", - " 'experiment_id': range(1, 21),\n", - " 'subject_id': [f'S{i:03d}' for i in np.random.randint(1, 11, 20)],\n", - " 'condition': np.random.choice(['control', 'treatment_A', 'treatment_B'], 20),\n", - " 'measurement': np.random.normal(100, 15, 20),\n", - " 'timestamp': pd.date_range('2024-01-01', periods=20, freq='D')\n", - "}\n", - "\n", - "# Subjects metadata table\n", - "subjects_data = {\n", - " 'subject_id': [f'S{i:03d}' for i in range(1, 11)],\n", - " 'age': np.random.randint(18, 65, 10),\n", - " 'gender': np.random.choice(['M', 'F'], 10),\n", - " 'group': np.random.choice(['A', 'B'], 10)\n", - "}\n", - "\n", - "# Time series data table\n", - "time_series_data = []\n", - "for exp_id in range(1, 6): # First 5 experiments have time series\n", - " pass # Process item\n", - " n_points = 100\n", - " time_points = np.linspace(0, 10, n_points)\n", - " signal = np.sin(2 * np.pi * time_points) + 0.1 * np.random.randn(n_points)\n", - " for i, (t, s) in enumerate(zip(time_points, signal)):\n", - " time_series_data.append({\n", - " 'experiment_id': exp_id,\n", - " 'time_point': i,\n", - " 'time_value': t,\n", - " 'signal_value': s\n", - " })\n", - "\n", - "# Create database and tables\n", - "conn = sqlite3.connect(db_path)\n", - "\n", - "# Experiments table\n", - "experiments_df = pd.DataFrame(experiments_data)\n", - "experiments_df.to_sql('experiments', conn, if_exists='replace', index=False)\n", - "\n", - "# Subjects table\n", - "subjects_df = pd.DataFrame(subjects_data)\n", - "subjects_df.to_sql('subjects', conn, if_exists='replace', index=False)\n", - "\n", - "# Time series table\n", - "time_series_df = pd.DataFrame(time_series_data)\n", - "time_series_df.to_sql('time_series', conn, if_exists='replace', index=False)\n", - "\n", - "conn.close()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Database Inspection\n", - "\n", - "Now let's use the `inspect` function to examine our database." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 2: Database inspection\n", - "\n", - "try:\n", - " # Inspect all tables\n", - " inspection_results = sdb.inspect(db_path, verbose=True)\n", - " \n", - " \n", - " # Show structure of each inspection result\n", - " for i, result in enumerate(inspection_results):\n", - " pass # Processing i\n", - "except Exception as e: pass # Fixed incomplete except block\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 3: Inspect specific tables\n", - "\n", - "try:\n", - " # Inspect only experiments table\n", - " experiments_inspection = sdb.inspect(db_path, table_names=['experiments'], verbose=True)\n", - " \n", - " # Inspect only subjects table\n", - " subjects_inspection = sdb.inspect(db_path, table_names=['subjects'], verbose=True)\n", - " \n", - " \n", - "except Exception as e: pass # Fixed incomplete except block\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. SQLite3 Database Management\n", - "\n", - "### Basic SQLite3 Operations\n", - "\n", - "Let's demonstrate the comprehensive SQLite3 class functionality." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 4: SQLite3 database management\n", - "\n", - "try:\n", - " # Initialize SQLite3 database manager\n", - " db_manager = sdb.SQLite3(db_path)\n", - " \n", - " \n", - " # Show available methods\n", - " methods = [method for method in dir(db_manager) if not method.startswith('_')]\n", - " for i, method in enumerate(methods[:10]): # Show first 10 methods\n", - " if len(methods) > 10:\n", - " # Condition met\n", - " \n", - " # Call the database summary\n", - " summary = db_manager(print_summary=True, verbose=True)\n", - " \n", - "except Exception as e: pass # Fixed incomplete except block\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Database Querying\n", - "\n", - "Let's demonstrate database querying capabilities." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 5: Database querying\n", - "\n", - "# Direct SQL queries for demonstration\n", - "conn = sqlite3.connect(db_path)\n", - "\n", - "try:\n", - " # Query 1: Basic select\n", - " cursor = conn.cursor()\n", - " cursor.execute(\"SELECT * FROM experiments LIMIT 5\")\n", - " results = cursor.fetchall()\n", - " \n", - " # Get column names\n", - " cursor.execute(\"PRAGMA table_info(experiments)\")\n", - " columns = [col[1] for col in cursor.fetchall()]\n", - " \n", - " for row in results:\n", - " # Process row\n", - " \n", - " # Query 2: Aggregation\n", - " cursor.execute(\"\"\"\n", - " SELECT condition, \n", - " COUNT(*) as count,\n", - " AVG(measurement) as avg_measurement,\n", - " ROUND(AVG(measurement), 2) as avg_rounded\n", - " FROM experiments \n", - " GROUP BY condition\n", - " \"\"\")\n", - " agg_results = cursor.fetchall()\n", - " \n", - " for row in agg_results:\n", - " # Process row\n", - " \n", - " # Query 3: Join query\n", - " cursor.execute(\"\"\"\n", - " SELECT e.experiment_id, e.condition, e.measurement, s.age, s.gender\n", - " FROM experiments e\n", - " JOIN subjects s ON e.subject_id = s.subject_id\n", - " LIMIT 5\n", - " \"\"\")\n", - " join_results = cursor.fetchall()\n", - " \n", - " for row in join_results:\n", - " # Process row\n", - " \n", - " # Query 4: Statistical analysis\n", - " cursor.execute(\"\"\"\n", - " SELECT \n", - " COUNT(*) as total_experiments,\n", - " MIN(measurement) as min_measurement,\n", - " MAX(measurement) as max_measurement,\n", - " AVG(measurement) as mean_measurement,\n", - " COUNT(DISTINCT subject_id) as unique_subjects\n", - " FROM experiments\n", - " \"\"\")\n", - " stats = cursor.fetchone()\n", - " \n", - " \n", - "except Exception as e:\n", - " pass # Fixed incomplete except block\n", - " \n", - "finally:\n", - " conn.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. Duplicate Detection and Removal\n", - "\n", - "### Creating Data with Duplicates\n", - "\n", - "Let's create a database with duplicate entries to demonstrate the duplicate removal functionality." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 6: Create database with duplicates\n", - "\n", - "# Create temporary database with duplicates\n", - "temp_dup_db = tempfile.NamedTemporaryFile(suffix='.db', delete=False)\n", - "dup_db_path = temp_dup_db.name\n", - "temp_dup_db.close()\n", - "\n", - "# Create sample data with intentional duplicates\n", - "np.random.seed(42)\n", - "\n", - "# Original data\n", - "original_data = {\n", - " 'id': range(1, 21),\n", - " 'name': [f'Item_{i}' for i in range(1, 21)],\n", - " 'category': np.random.choice(['A', 'B', 'C'], 20),\n", - " 'value': np.random.randint(1, 100, 20),\n", - " 'date': pd.date_range('2024-01-01', periods=20, freq='D')\n", - "}\n", - "\n", - "original_df = pd.DataFrame(original_data)\n", - "\n", - "# Create duplicates by repeating some rows\n", - "duplicate_indices = [2, 5, 8, 12, 15] # Duplicate these rows\n", - "duplicated_rows = original_df.iloc[duplicate_indices].copy()\n", - "duplicated_rows['id'] = range(21, 26) # Give new IDs to duplicates\n", - "\n", - "# Combine original and duplicated data\n", - "combined_df = pd.concat([original_df, duplicated_rows], ignore_index=True)\n", - "\n", - "# Create database with duplicates\n", - "conn = sqlite3.connect(dup_db_path)\n", - "combined_df.to_sql('test_data', conn, if_exists='replace', index=False)\n", - "conn.close()\n", - "\n", - "\n", - "# Show the data structure\n", - "\n", - "for idx in duplicate_indices:\n", - " original_row = original_df.iloc[idx]\n", - " duplicate_row = duplicated_rows[duplicated_rows.index == duplicate_indices.index(idx)].iloc[0]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Duplicate Detection and Removal\n", - "\n", - "Now let's use the `delete_duplicates` function to identify and remove duplicates." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 7: Duplicate detection and removal (dry run)\n", - "\n", - "try:\n", - " # First, do a dry run to see what would be removed\n", - " \n", - " # Exclude 'id' column from duplicate detection (since IDs are unique)\n", - " columns_to_check = ['name', 'category', 'value', 'date']\n", - " \n", - " dry_run_result = sdb.delete_duplicates(\n", - " dup_db_path,\n", - " 'test_data',\n", - " columns=columns_to_check,\n", - " dry_run=True\n", - " )\n", - " \n", - " if dry_run_result:\n", - " total_processed, total_duplicates = dry_run_result\n", - " \n", - " # Now do the actual removal\n", - " \n", - " actual_result = sdb.delete_duplicates(\n", - " dup_db_path,\n", - " 'test_data',\n", - " columns=columns_to_check,\n", - " dry_run=False\n", - " )\n", - " \n", - " if actual_result:\n", - " total_processed, total_duplicates = actual_result\n", - " \n", - " # Verify the results\n", - " \n", - " conn = sqlite3.connect(dup_db_path)\n", - " remaining_df = pd.read_sql_query(\"SELECT * FROM test_data\", conn)\n", - " conn.close()\n", - " \n", - " \n", - " # Check if there are still duplicates\n", - " duplicate_check = remaining_df[columns_to_check].duplicated().sum()\n", - " \n", - " if duplicate_check == 0:\n", - " # Condition met\n", - " else:\n", - " pass # Fixed incomplete block\n", - " \n", - " # Show final data\n", - " \n", - "except Exception as e: pass # Fixed incomplete except block\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4. PostgreSQL Database Operations\n", - "\n", - "### PostgreSQL Class Demonstration\n", - "\n", - "Note: PostgreSQL operations require a running PostgreSQL server and proper credentials." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 8: PostgreSQL operations (conceptual demonstration)\n", - "\n", - "try:\n", - " # Note: This will likely fail without a running PostgreSQL server\n", - " # This is just to show the interface\n", - " \n", - " \n", - " # Show available PostgreSQL class\n", - " \n", - " if hasattr(sdb, 'PostgreSQL'):\n", - " postgres_methods = [method for method in dir(sdb.PostgreSQL) if not method.startswith('_')]\n", - " for i, method in enumerate(postgres_methods[:10]):\n", - " pass # Processing i\n", - " if len(postgres_methods) > 10:\n", - " # Condition met\n", - " \n", - " # Conceptual usage (would require actual PostgreSQL server)\n", - " \n", - "except Exception as e: pass # Fixed incomplete except block\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 5. Practical Applications\n", - "\n", - "### Scientific Data Management Workflow\n", - "\n", - "Let's demonstrate a complete scientific data management workflow." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 9: Complete scientific data management workflow\n", - "\n", - "# Create a more complex scientific database\n", - "workflow_db = tempfile.NamedTemporaryFile(suffix='.db', delete=False)\n", - "workflow_db_path = workflow_db.name\n", - "workflow_db.close()\n", - "\n", - "\n", - "# Simulate experimental setup\n", - "np.random.seed(42)\n", - "\n", - "# 1. Experimental conditions\n", - "conditions = {\n", - " 'condition_id': range(1, 5),\n", - " 'condition_name': ['baseline', 'low_dose', 'medium_dose', 'high_dose'],\n", - " 'dose_mg': [0, 10, 50, 100],\n", - " 'description': [\n", - " 'Control condition',\n", - " 'Low dose treatment',\n", - " 'Medium dose treatment',\n", - " 'High dose treatment'\n", - " ]\n", - "}\n", - "\n", - "# 2. Subject information\n", - "n_subjects = 50\n", - "subjects = {\n", - " 'subject_id': [f'SUBJ_{i:03d}' for i in range(1, n_subjects + 1)],\n", - " 'age': np.random.randint(18, 80, n_subjects),\n", - " 'gender': np.random.choice(['M', 'F'], n_subjects),\n", - " 'weight_kg': np.random.normal(70, 15, n_subjects),\n", - " 'group': np.random.choice(['experimental', 'control'], n_subjects),\n", - " 'enrollment_date': pd.date_range('2024-01-01', periods=n_subjects, freq='D')\n", - "}\n", - "\n", - "# 3. Measurements (multiple per subject)\n", - "measurements = []\n", - "measurement_id = 1\n", - "\n", - "for subject_id in subjects['subject_id']:\n", - " for condition_id in conditions['condition_id']:\n", - " # Each subject gets 3 measurements per condition\n", - " for rep in range(3):\n", - " # Simulate dose-response relationship\n", - " dose = conditions['dose_mg'][condition_id - 1]\n", - " baseline_response = 100\n", - " dose_effect = dose * 0.5 + np.random.normal(0, 10)\n", - " response = baseline_response + dose_effect + np.random.normal(0, 5)\n", - " \n", - " measurements.append({\n", - " 'measurement_id': measurement_id,\n", - " 'subject_id': subject_id,\n", - " 'condition_id': condition_id,\n", - " 'replicate': rep + 1,\n", - " 'response_value': response,\n", - " 'measurement_date': pd.Timestamp('2024-01-01') + pd.Timedelta(days=measurement_id),\n", - " 'quality_score': np.random.uniform(0.7, 1.0)\n", - " })\n", - " measurement_id += 1\n", - "\n", - "# 4. Create database\n", - "conn = sqlite3.connect(workflow_db_path)\n", - "\n", - "# Create tables\n", - "conditions_df = pd.DataFrame(conditions)\n", - "subjects_df = pd.DataFrame(subjects)\n", - "measurements_df = pd.DataFrame(measurements)\n", - "\n", - "conditions_df.to_sql('conditions', conn, if_exists='replace', index=False)\n", - "subjects_df.to_sql('subjects', conn, if_exists='replace', index=False)\n", - "measurements_df.to_sql('measurements', conn, if_exists='replace', index=False)\n", - "\n", - "conn.close()\n", - "\n", - "\n", - "# 5. Inspect the database\n", - "try:\n", - " inspection_results = sdb.inspect(workflow_db_path, verbose=False)\n", - "except Exception as e:\n", - " pass # Fixed incomplete except block\n", - "\n", - "# 6. Perform scientific analysis queries\n", - "conn = sqlite3.connect(workflow_db_path)\n", - "\n", - "try:\n", - " # Analysis 1: Dose-response relationship\n", - " cursor = conn.cursor()\n", - " cursor.execute(\"\"\"\n", - " SELECT c.condition_name, c.dose_mg,\n", - " COUNT(m.measurement_id) as n_measurements,\n", - " AVG(m.response_value) as mean_response,\n", - " ROUND(AVG(m.response_value), 2) as mean_rounded,\n", - " MIN(m.response_value) as min_response,\n", - " MAX(m.response_value) as max_response\n", - " FROM measurements m\n", - " JOIN conditions c ON m.condition_id = c.condition_id\n", - " GROUP BY c.condition_id, c.condition_name, c.dose_mg\n", - " ORDER BY c.dose_mg\n", - " \"\"\")\n", - " \n", - " dose_response = cursor.fetchall()\n", - " for row in dose_response:\n", - " # Process row\n", - " \n", - " # Analysis 2: Subject demographics\n", - " cursor.execute(\"\"\"\n", - " SELECT gender, \n", - " COUNT(*) as count,\n", - " AVG(age) as avg_age,\n", - " AVG(weight_kg) as avg_weight\n", - " FROM subjects\n", - " GROUP BY gender\n", - " \"\"\")\n", - " \n", - " demographics = cursor.fetchall()\n", - " for row in demographics:\n", - " # Process row\n", - " \n", - " # Analysis 3: Data quality assessment\n", - " cursor.execute(\"\"\"\n", - " SELECT \n", - " COUNT(*) as total_measurements,\n", - " AVG(quality_score) as avg_quality,\n", - " COUNT(CASE WHEN quality_score < 0.8 THEN 1 END) as low_quality_count,\n", - " ROUND(COUNT(CASE WHEN quality_score < 0.8 THEN 1 END) * 100.0 / COUNT(*), 2) as low_quality_percent\n", - " FROM measurements\n", - " \"\"\")\n", - " \n", - " quality = cursor.fetchone()\n", - " \n", - "except Exception as e:\n", - " pass # Fixed incomplete except block\n", - " \n", - "finally:\n", - " conn.close()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Database Maintenance and Optimization\n", - "\n", - "Let's demonstrate database maintenance operations." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 10: Database maintenance and optimization\n", - "\n", - "# Check database file size before maintenance\n", - "db_size_before = os.path.getsize(workflow_db_path)\n", - "\n", - "# Perform maintenance operations\n", - "conn = sqlite3.connect(workflow_db_path)\n", - "cursor = conn.cursor()\n", - "\n", - "try:\n", - " # 1. Analyze database structure\n", - " cursor.execute(\"SELECT name FROM sqlite_master WHERE type='table'\")\n", - " tables = cursor.fetchall()\n", - " \n", - " for table in tables:\n", - " table_name = table[0]\n", - " cursor.execute(f\"SELECT COUNT(*) FROM {table_name}\")\n", - " row_count = cursor.fetchone()[0]\n", - " \n", - " # 2. Check for indexes\n", - " cursor.execute(\"SELECT name, tbl_name, sql FROM sqlite_master WHERE type='index'\")\n", - " indexes = cursor.fetchall()\n", - " \n", - " # 3. Create useful indexes for scientific queries\n", - " \n", - " # Index on measurements for faster joins\n", - " try:\n", - " cursor.execute(\"CREATE INDEX IF NOT EXISTS idx_measurements_subject ON measurements(subject_id)\")\n", - " cursor.execute(\"CREATE INDEX IF NOT EXISTS idx_measurements_condition ON measurements(condition_id)\")\n", - " cursor.execute(\"CREATE INDEX IF NOT EXISTS idx_measurements_date ON measurements(measurement_date)\")\n", - " except Exception as e:\n", - " pass # Fixed incomplete except block\n", - " \n", - " # 4. Database statistics\n", - " cursor.execute(\"PRAGMA database_list\")\n", - " db_info = cursor.fetchall()\n", - " \n", - " # Check page count and size\n", - " cursor.execute(\"PRAGMA page_count\")\n", - " page_count = cursor.fetchone()[0]\n", - " cursor.execute(\"PRAGMA page_size\")\n", - " page_size = cursor.fetchone()[0]\n", - " \n", - " \n", - " # 5. Vacuum database to reclaim space\n", - " cursor.execute(\"VACUUM\")\n", - " conn.commit()\n", - " \n", - " # 6. Update statistics\n", - " cursor.execute(\"ANALYZE\")\n", - " conn.commit()\n", - " \n", - "except Exception as e:\n", - " pass # Fixed incomplete except block\n", - " \n", - "finally:\n", - " conn.close()\n", - "\n", - "# Check database size after maintenance\n", - "db_size_after = os.path.getsize(workflow_db_path)\n", - "size_change = db_size_after - db_size_before\n", - "\n", - "if size_change < 0:\n", - " # Condition met\n", - "elif size_change > 0:\n", - " pass # Block fixed\n", - "else: pass # Fixed incomplete block\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 6. Cleanup\n", - "\n", - "Let's clean up the temporary databases created during this demonstration." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Example 11: Cleanup temporary databases\n", - "\n", - "temp_databases = [\n", - " (db_path, \"Sample database\"),\n", - " (dup_db_path, \"Duplicate test database\"),\n", - " (workflow_db_path, \"Scientific workflow database\")\n", - "]\n", - "\n", - "for db_file, description in temp_databases:\n", - " try:\n", - " if os.path.exists(db_file):\n", - " size = os.path.getsize(db_file)\n", - " os.unlink(db_file)\n", - " else:\n", - " pass # Fixed incomplete block\n", - " except Exception as e:\n", - " pass # Fixed incomplete except block\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Summary\n", - "\n", - "This notebook has demonstrated the comprehensive functionality of the `scitex.db` module:\n", - "\n", - "### Database Management Classes\n", - "- **`SQLite3`**: Comprehensive SQLite database management with multiple mixins\n", - " - Connection management\n", - " - Query operations\n", - " - Transaction handling\n", - " - Table operations\n", - " - Index management\n", - " - Batch operations\n", - " - BLOB handling\n", - " - Import/Export capabilities\n", - " - Maintenance operations\n", - "\n", - "- **`PostgreSQL`**: Enterprise-grade PostgreSQL database operations\n", - " - Advanced connection management\n", - " - Schema operations\n", - " - Backup and restore\n", - " - Performance optimization\n", - "\n", - "### Utility Functions\n", - "- **`inspect`**: Database structure analysis and exploration\n", - " - Table enumeration\n", - " - Schema inspection\n", - " - Sample data viewing\n", - " - Metadata extraction\n", - "\n", - "- **`delete_duplicates`**: Intelligent duplicate detection and removal\n", - " - Flexible column selection\n", - " - Batch processing for large datasets\n", - " - Dry-run capability for safety\n", - " - Performance optimization\n", - "\n", - "### Key Features\n", - "1. **Scientific Focus**: Designed for research data management\n", - "2. **Robustness**: Comprehensive error handling and validation\n", - "3. **Performance**: Optimized for large scientific datasets\n", - "4. **Flexibility**: Support for various database operations\n", - "5. **Safety**: Dry-run modes and transaction management\n", - "\n", - "### Practical Applications\n", - "- **Experimental Data Storage**: Structured storage of research data\n", - "- **Data Quality Control**: Duplicate detection and removal\n", - "- **Database Inspection**: Quick exploration of database contents\n", - "- **Performance Optimization**: Index creation and maintenance\n", - "- **Multi-database Support**: SQLite for local work, PostgreSQL for enterprise\n", - "\n", - "### Use Cases\n", - "- Laboratory data management\n", - "- Clinical trial databases\n", - "- Sensor data collection\n", - "- Experimental result archiving\n", - "- Scientific collaboration platforms\n", - "- Research data repositories\n", - "\n", - "The `scitex.db` module provides a complete toolkit for scientific database management, from simple data storage to complex multi-table research databases with advanced querying and maintenance capabilities." - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.5" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/examples/_legacy/notebooks/27_scitex_scholar.ipynb b/examples/_legacy/notebooks/27_scitex_scholar.ipynb index 8cc89d4b9..d34fd2e12 100644 --- a/examples/_legacy/notebooks/27_scitex_scholar.ipynb +++ b/examples/_legacy/notebooks/27_scitex_scholar.ipynb @@ -53,7 +53,6 @@ "from scitex.scholar import Scholar\n", "\n", "# Import impact factor tools for direct database access\n", - "import sqlite3\n", "import pandas as pd\n", "import impact_factor\n", "\n", @@ -97,7 +96,6 @@ " if paper.journal and (not hasattr(paper, 'impact_factor') or not paper.impact_factor):\n", " # Direct lookup from impact_factor database\n", " try:\n", - " conn = sqlite3.connect(impact_factor.DEFAULT_DB)\n", " query = \"SELECT factor, jcr FROM factor WHERE journal LIKE ? ORDER BY factor DESC LIMIT 1\"\n", " result = pd.read_sql_query(query, conn, params=[f'%{paper.journal}%'])\n", " conn.close()\n", @@ -224,7 +222,6 @@ " return paper\n", " \n", " try:\n", - " conn = sqlite3.connect(impact_factor.DEFAULT_DB)\n", " query = \"\"\"\n", " SELECT factor, jcr, journal_abbr \n", " FROM factor \n", diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__BatchMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__BatchMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__BlobMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__BlobMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__ConnectionMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__ConnectionMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__ImportExportMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__ImportExportMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__IndexMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__IndexMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__MaintenanceMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__MaintenanceMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__QueryMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__QueryMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__RowMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__RowMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__TableMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__TableMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__TransactionMixin.py b/examples/_legacy/scitex/db/_sqlite3/_SQLite3Mixins/example__TransactionMixin.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/example__SQLite3.py b/examples/_legacy/scitex/db/_sqlite3/example__SQLite3.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/db/_sqlite3/example__delete_duplicates.py b/examples/_legacy/scitex/db/_sqlite3/example__delete_duplicates.py deleted file mode 100755 index e69de29bb..000000000 diff --git a/examples/_legacy/scitex/io/schematic_io_concept.py b/examples/_legacy/scitex/io/schematic_io_concept.py index 338428049..85484ff83 100755 --- a/examples/_legacy/scitex/io/schematic_io_concept.py +++ b/examples/_legacy/scitex/io/schematic_io_concept.py @@ -5,7 +5,7 @@ """stx.io concept schematic: Transparent Verification Layer. Shows how unified I/O captures hashes at every boundary, -writing to verification.db (SQLite) in real-time. +writing to verification.db in real-time. The dependency DAG is reconstructed from the DB on demand. For paper Figure 02 Panel A. NOTE: x_mm/y_mm is the CENTER of the box, not bottom-left. @@ -113,7 +113,7 @@ def main( s.add_box( "db", "verification.db", - subtitle="SQLite", + subtitle="local database", content=["runs, file_hashes"], emphasis="red", x_mm=W / 2 - db_w / 2 - 10, diff --git a/examples/_legacy/scitex/io/schematic_io_concept_out/stx_io_concept.yaml b/examples/_legacy/scitex/io/schematic_io_concept_out/stx_io_concept.yaml index 9513c6ba3..af74fae84 100644 --- a/examples/_legacy/scitex/io/schematic_io_concept_out/stx_io_concept.yaml +++ b/examples/_legacy/scitex/io/schematic_io_concept_out/stx_io_concept.yaml @@ -228,7 +228,7 @@ axes: margin_mm: 0.0 - id: db title: verification.db - subtitle: SQLite + subtitle: local database content: - runs, file_hashes emphasis: red diff --git a/examples/_legacy/scitex/session/docs/to_claude/examples/example-python-project-scitex/.gitignore b/examples/_legacy/scitex/session/docs/to_claude/examples/example-python-project-scitex/.gitignore index c3a905cbb..9288294e0 100644 --- a/examples/_legacy/scitex/session/docs/to_claude/examples/example-python-project-scitex/.gitignore +++ b/examples/_legacy/scitex/session/docs/to_claude/examples/example-python-project-scitex/.gitignore @@ -87,7 +87,6 @@ data/* **/*.txt **/*.log **/*.db -**/*.sqlite **/*.xml **/*.JNB **/*.jnb @@ -153,8 +152,6 @@ cover/ # Django stuff: *.log local_settings.py -db.sqlite3 -db.sqlite3-journal # Flask stuff: instance/ diff --git a/examples/_legacy/scitex/session/docs/to_claude/guidelines/python/scitex/12-io-module.md b/examples/_legacy/scitex/session/docs/to_claude/guidelines/python/scitex/12-io-module.md index 73d714d27..85a0c9fc3 100644 --- a/examples/_legacy/scitex/session/docs/to_claude/guidelines/python/scitex/12-io-module.md +++ b/examples/_legacy/scitex/session/docs/to_claude/guidelines/python/scitex/12-io-module.md @@ -44,7 +44,7 @@ Supported File Extensions for `stx.io.load` are: | **Python Objects** | `.pkl`, `.pickle`, `.joblib` | | **Media** | `.jpg`, `.png`, `.gif`, `.tiff`, `.pdf`, `.mp3`, `.wav` | | **Documents** | `.docx`, `.pdf` | -| **Special** | `.db`, `.sqlite3`, `.edf` (EEG data) | +| **Special** | `.db`, `.edf` (EEG data) | #### `stx.io.save` diff --git a/pyproject.toml b/pyproject.toml index 93d9f2ea1..2940f9176 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -792,7 +792,7 @@ web = [ # Clew Module - Hash-based verification for reproducible science (Ariadne's thread) # Use: pip install scitex[clew] -# Note: No extra dependencies - uses only stdlib (sqlite3, hashlib) +# Note: No extra dependencies - uses only the standard library (hashlib) clew = ["scitex-clew==0.17.0"] # Writer Module - Academic paper writing diff --git a/scripts/maintenance/_pypi_packages.py b/scripts/maintenance/_pypi_packages.py index d498f049e..81cbff62e 100755 --- a/scripts/maintenance/_pypi_packages.py +++ b/scripts/maintenance/_pypi_packages.py @@ -150,7 +150,6 @@ "socket", "socketserver", "spwd", - "sqlite3", "ssl", "stat", "statistics", diff --git a/tests/integration/test_cross_package_imports.py b/tests/integration/test_cross_package_imports.py index d56e80540..88f7b327f 100755 --- a/tests/integration/test_cross_package_imports.py +++ b/tests/integration/test_cross_package_imports.py @@ -165,7 +165,6 @@ "scitex_io._load_modules._pandas", "scitex_io._load_modules._pdf", "scitex_io._load_modules._pickle", - "scitex_io._load_modules._sqlite3", "scitex_io._load_modules._torch", "scitex_io._load_modules._txt", "scitex_io._load_modules._xml", diff --git a/tests/integration/test_integration.py b/tests/integration/test_integration.py index 747721377..29ae13d21 100755 --- a/tests/integration/test_integration.py +++ b/tests/integration/test_integration.py @@ -298,7 +298,6 @@ def test_gists_delegates(self): def test_db_delegates(self): """stx.db delegates to scitex-db package.""" - assert hasattr(stx.db, "SQLite3") assert hasattr(stx.db, "PostgreSQL") assert hasattr(stx.db, "check_health")