UniProt Database Creator unidb

Table of Contents

• Purpose
• Licenses
• Introduction
• Installation
• Usage
  • Syntax
  • Required Parameters
  • Optional Parameters
  • Basic Example
  • Advanced Example with Optimization
  • Using Pre-downloaded And Extracted Decompressed Files
  • Notes
• System Requirements and Prerequisites
• Main Terms (briefly)
• Database
  • Database Structure
  • Database Sources
  • Taxonomy
  • Isoforms
• Technical Details
  • Supported Databases
  • Performance
• Troubleshooting
  • Common Issues

Purpose

This application is designed to create and configure a UniProt database with parallel processing capabilities and flexible logging configuration.

As an alternative to setting up your own UniProt database, you may use existing web interfaces:

• UniProt peptide search: https://www.uniprot.org/peptide-search - searches for sequences ≥7 amino acids.
• pepstring: http://pepstring.eimb.ru/ - searches for sequences ≥3 amino acids.

Licenses

This program uses and downloads data from UniProt and NCBI FTP sites.

All data from UniProt is distributed under license: Creative Commons Attribution 4.0 International (CC BY 4.0).

License Terms:

• You may freely copy, distribute, and adapt the data
• You must provide appropriate attribution

Attribution:

• Data Source: UniProt Knowledgebase (via FTP)
• License: CC BY 4.0
• License Link: https://creativecommons.org/licenses/by/4.0/
• Source Link: https://www.uniprot.org/help/license

All data from NCBI FTP is public, non-sensitive, unrestricted scientific data sharing among scientific communities.

Terms of Use:

• You may freely copy, distribute, and adapt the data for scientific use.
• You must provide appropriate attribution to NCBI and the original data sources.
• You should not present NCBI data as your own.

Attribution:

• Data Source: National Center for Biotechnology Information (NCBI)
• Terms of Use: https://www.ncbi.nlm.nih.gov/home/about/policies/
• Source Link: https://ftp.ncbi.nih.gov/pub/README.ftp
• Data accessed via: https://ftp.ncbi.nlm.nih.gov/

This program is distributed under the MIT license.

• License: MIT
• License Link: https://mit-license.org/

This software creates a derivative database based on the original UniProt and NCBI data. The derivative database is distributed under the same terms of the CC BY 4.0 license. You should not present NCBI data as your own.

Introduction

This util will create UniProt database using official data from UniProt FTP site and NCBI FTP site.

Installation

Download the unidb.pex file from Releases.

Usage

Syntax

./unidb.pex [OPTIONS]

Or

python unidb.pex [OPTIONS]

Required Parameters

--dbname, -d

• Description: PostgreSQL database name
• Type: string
• Example: --dbname uniprot_db or -d uniprot_db

--dbuser, -U

• Description: Database username
• Type: string
• Example: --dbuser postgres or -U postgres

Optional Parameters

--password, -p

• Description: Database password
• Type: string
• Example: --password mypassword

--port, -P

• Description: PostgreSQL connection port
• Type: positive integer
• Default: 5432
• Example: --port 5433

--host, -u

• Description: Database host
• Type: string
• Default: localhost
• Example: --host 192.168.1.100

--processes, -j

• Description: Number of processes for work distribution
• Type: positive integer
• Default: 1
• Example: --processes 4 (to use 4 CPU cores)

--path-to-source-files, -k

• Description: Path to pre-downloaded unpacked source files
• Type: directory path
• Example: --path-to-source-files /path/to/uniprot/files

--path-to-source-archives, -z

• Description: Path to pre-downloaded archives
• Type: directory path
• Example: --path-to-source-archives /path/to/uniprot/archives

--accept-all, -y

• Description: Automatically accept all conditions and setup
• Type: flag (no value required)
• Example: -y

--trgm, -i

• Description: Build trigram index on sequence column in uniprot_kb table
• Type: flag
• Example: --trgm (improves sequence search performance)

--verbose, -v

• Description: Enable detailed logging
• Type: flag
• Example: --verbose

--no-clean-up-on-failure, -g

• Description: Prevent source files and database clean up on failure
• Type: flag
• Example: --no-clean-up-on-failure

--logtype, -t

• Description: Log output destination
• Type: console | file
• Default: console
• Example: --logtype file

--loglevel, -l

• Description: Log detail level
• Type: string (debug, info, warning, error, critical)
• Default: info
• Example: --loglevel debug

--logpath, -L

• Description: Directory path for log files
• Type: directory path
• Default: uniprotdb/logs
• Example: --logpath /var/log/uniprot

Basic Example

./unidb.pex -d uniprot_database -U postgres -p mypassword

Advanced Example with Optimization

./unidb.pex \
  -d uniprot_prod \
  -U dbadmin \
  -p securepass \
  -j 8 \
  -i \
  -y \
  -v \
  -t file \
  -L /var/log/uniprot

Using Pre-downloaded And Extracted Decompressed Files

./unidb.pex \
  -d uniprot_test \
  -U tester \
  -k /home/user/uniprot_files \
  -j 4 \
  -i

Notes

The first setup will throw error and warning when utility will try to truncate non-existing tables. Ignore these messages. When you try to update database - the old data will be erased first.

It is recommended to use -j with number of CPU cores. The -i (--trgm) flag improves sequence search performance but increases database creation time tremendously and doubles the database size. When using -y, all confirmations are accepted automatically. If no source files provided - they will be downloaded automatically.

Required source files in case you use --path-to-source-files, -k option:

• names.dmp
• nodes.dmp
• delnodes.dmp
• merged.dmp
• taxidlineage.dmp
• uniprot_sprot.fasta
• uniprot_sprot_varsplic.fasta
• uniprot_treml.fasta

Required archives in case you use --path-to-source-archives, -z option:

• new_taxdump.tar.gz
• uniprot_sprot.fasta.gz
• uniprot_sprot_varsplic.fasta.gz
• uniprot_trembl.fasta.gz

System Requirements And Prerequisites

• Python 3.12+
• PostgreSQL 9+
• *nix system (Linux, MacOS*)
• cat utility
• Internet connection (unless using pre-downloaded files)
• RAM 4 GB
• Disk space 200 GB (minimum)

*Should work as well but not tested.

To create database without trigram index and download all the files you will need approximately 205 GB free space. The database itself after setup will weigh approximately 108 GB.

To create database with trgm index and download all the files you will need approximately 215 GB free space.

Main Terms (briefly)

• UniProtKB - is a collection of peptide sequences which in place consists of two main partitions - TrEMBL and Swiss-Prot.

• TrEMBL - peptide sequences that were computationally analyzed but not reviewed by humans.

• Swiss-Prot - peptide sequences that were manually reviewed by humans.

• Accession - protein sequence unique identifier (ID) from UniProtKB.

• Entry name - protein sequence former ID with biological info from UniProtKB.

• Taxonomy - is the study that name, group, classify organisms and put them into hierarchy. This hierarchy is called taxonomic lineage. (Do not mix organism names (from peptide table) and taxons (from taxonomy table). Taxons include all the organism names but not vice versa.)

• Lineage - taxonomic lineage is a hierarchy which illustrates relationships between organisms. This hierarchy is divided by rank (domain, kingdom, phylum, class, order, family, genus, species, etc.). What is important - different organisms may share the same taxons and that is where we can trace their relations. (For example humans and rats have some common taxons. One of them is Vertebrata(vertebrates)).

• NCBI ID - ID of taxons from the NCBI Taxonomy database.

Database

Database Structure

The database consists of four tables:

uniprot_kb - protein sequence database from UniProtKB (Swiss-Prot and Isoforms + TrEMBL).

• source: source database - 'sp' (Swiss-Prot), 'tr' (TrEMBL), 'sp_iso' (reviewed isoforms).
• is_reviewed: whether the sequence was manually reviewed/curated.
• accession: unique protein accession ID.
• entry_name: protein entry name containing biological information.
• peptide_name: full name/description of the protein.
• ncbi_organism_id: NCBI taxon ID of the source organism.
• organism_name: scientific name of the source organism.
• sequence: protein amino acid sequence (single letter codes).

taxonomy - NCBI Taxonomy database information:

• rank: taxonomic rank (e.g., species, genus, family).
• ncbi_taxon_id: unique NCBI taxonomy identifier.
• tax_name: scientific name of the taxon.

lineage - taxonomic lineage relationships:

• ncbi_taxon_id: organism taxon ID. Organism is the last taxon in the lineage.
• ncbi_lineage_id: ancestor taxon ID in the lineage. This table stores the complete taxonomic hierarchy for each organism, linking organisms to all their ancestral taxa.

metadata - contains licensing and attribution information for the database sources:

• data_source: Name of the data source (e.g., "UniProt Knowledgebase FTP", "NCBI FTP").
• data_license: License type for the data (e.g., "CC BY 4.0", "Public unrestricted scientific data").
• license_url: URL to the full license text.
• attribution_required: Whether attribution is required ("Yes"/"No"). This table stores the legal and attribution requirements for using the database content.

Database Sources

The database is made on the basis of two knowledgebases - UniProtKB and NCBI Taxonomy

Taxonomy

The taxonomy is implemented identical to the NCBI taxonomy. Note that NCBI IDs and organism names may not coincide with data from UniProt knowledgebase. The reason is that UniProt updates its taxonomy once every 8 weeks. However, current database uses NCBI Taxonomy which receives new updates daily so it might happen that version the database uses for update differs from version UniProt did. If you see 'deleted' in taxonomy list - it means there is no lineage for this taxon in the database. For more information about this taxon you should move to UniProt official site or NCBI Taxonomy Browser.

Isoforms

Only additional manually curated isoform sequences that are described in UniProtKB/Swiss-Prot from uniprot_sprot_varsplic.fasta.gz are available.

Technical Details

Supported Databases

Currently only PostgreSQL database is supported.

Performance

Using -j with CPU cores significantly improves performance. Trigram index improves search speed but increases build time tremendously and doubles the storage size.

Test environment

• OS: Ubuntu 24.04
• Database: PostgreSQL 16
• Python: 3.13
• CPU: 12 cores
• RAM: 8 GB
• Storage: SSD 235 GB NVMe
• Internet: 250 Mb/s

Standard Setup:

./unidb.pex \
  --dbname database_name \
  --dbuser database_user \
  --password 'password' \
  -j 12 \
  --logtype file \
  --loglevel DEBUG \
  --verbose \
  -y

Estimated time: ~1 hour.

With Trigram Index:

./unidb.pex \
  --dbname database_name \
  --dbuser database_user \
  --password 'password' \
  -j 12 \
  --logtype file \
  --loglevel DEBUG \
  --verbose \
  --trgm \
  -y

Estimated time: ~6 hours 30 minutes.

Troubleshooting

Common Issues

Connection Issues:

Ensure PostgreSQL is running and accessible. Verify connection parameters (host, port, credentials). Check user permissions for database creation.

Disk Space:

Monitor free space during database creation. Use --path-to-source-files or --path-to-source-archives for external storage.

Memory Issues:

Reduce parallel processes with -j parameter. Increase system swap space if needed.

Build Failures:

Check PostgreSQL version. Verify all prerequisites are met.