diff --git a/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md b/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md index 47d38c9f6..01d9eb9b7 100644 --- a/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md @@ -1,7 +1,11 @@ # Data Export ## 1. Function Overview -The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows: + +IoTDB supports two methods for data export: + +* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory. It can export the query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) files. +* PIPE Framework-based TsFileBackup: `tsfile-backup.sh/bat` is located in the `tools` directory. It can export specified data files into TsFile format using the PIPE framework. @@ -19,15 +23,18 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an - + + + + +
File containing custom SQL statements.
TsFileTsFile Open-source time-series file format.
tsfile-backup.sh/batAn open-source time-series data file format,and this script supports the Object data type.
-- Does not support the Object data type. -## 2. Detailed Features +## 2. Data Export Tool ### 2.1 Common Parameters | Short | Full Parameter | Description | Required | Default | |----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |-----------------------------------------------| @@ -91,7 +98,7 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an > export-data.sh -ft csv -sql_dialect table -t /path/export/dir -q "select * from table1" Parse error: Missing required option: db ``` -## 2.3 SQL Format +### 2.3 SQL Format #### 2.3.1 Command ```Shell # Unix/OS X @@ -168,3 +175,72 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + + +## 3. TsFileBackup Based on PIPE Framework +Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This script can automatically generate and send the `CREATE PIPE` SQL command to the server, exporting specified data files to TsFile format. + +**Notes:** +1. **To use this script, contact the Timecho Team to obtain the customized installation package `timechodb--extension`.** +2. **This script supports exporting Object-type data to TsFile files.** + + +### 3.1 Execution Commands +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] + +# Windows +> tools\windows\tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + + +### 3.2 Script Parameters +| Abbreviation | Full Name | Description | Required | Default | +| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- | +| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - | +| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` | +| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` | +| `-u` | `--user` | Username for IoTDB authentication. | No | `root` | +| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` | +| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - | +| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` | +| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` | +| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - | +| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - | +| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - | +| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - | +| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - | +| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` | +| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - | +| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - | +| `-help` | `--help` | Show help information. | No | - | + + +### 3.3 Execution Examples + +Example 1: SCP Remote Export (Send Data to Another Server) + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +Example 2: Remote Object Data Export with Rate Limiting + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +Example 3: Specify Pipe Plugin JAR Directory + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` diff --git a/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md b/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md index 7c18a1e54..b2c8cfe81 100644 --- a/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md @@ -37,7 +37,8 @@ IoTDB supports three methods for data import:
-- Does not support the Object data type. +- The table model TsFile import currently only supports local import. +- Since version V2.0.9.2, the import-data.sh/bat script supports the Object data type when importing TsFile files. ## 2. Data Import Tool ### 2.1 Common Parameters @@ -184,7 +185,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -197,18 +198,19 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x and later versions > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 TsFile-Specific Parameters -| Short | Full Parameter | Description | Required | Default | -| ----------- | ----------------------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- | --------------------------- | -| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | -| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | -| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | -| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | -| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| Short | Full Parameter | Description | Required | Default | +|---------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|-----------------------| +| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | +| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | +| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | +| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | +| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| `-o` | `--object-file-paths` | Storage path for Object files.
Default mode: If this parameter is not specified, the script automatically identifies and imports Object files located in the subdirectory with the same name as the TsFile.
Absolute path mode: Explicitly specifies the external storage root directory for Object files; the tool creates an associated data index based on this path.
Note: This parameter is supported since V2.0.9.2 | No | | #### 2.4.3 Examples @@ -221,6 +223,54 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` + +**Object Type Import** + +1. Import Directory Structure + +* Default Mode + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TsFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* Specified Object Directory + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. Command Line Examples + +* Basic Import (automatically identifies Object files in the TsFile-named directory) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* Batch Directory Import (specify concurrent threads and post-success action) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* Table Model Associated Import (specify external Object storage path and target database) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile Auto-Loading This feature enables IoTDB to automatically monitor a specified directory for new TsFiles and load them into the database without manual intervention. @@ -281,7 +331,7 @@ load '' with ( |--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|--------------------------------|-------------------|----------------------------| | `database-level` | When the database corresponding to the TsFile does not exist, the database hierarchy level can be specified via the ` database-level` parameter. The default is the level set in `iotdb-common.properties`. For example, setting level=1 means the prefix path of level 1 in all time series in the TsFile will be used as the database. | Integer | `[1: Integer.MAX_VALUE]` | No | 1 | | `on-success` | Action for successfully loaded TsFiles: `delete` (delete the TsFile after successful import) or `none` (retain the TsFile in the source folder). | String | `delete / none` | No | delete | -| `model` | Specifies whether the TsFile uses the `table` model or `tree` model. This parameter becomes invalid starting from V2.0.2.1. The system automatically identifies whether the data model is tree-based or table-based. | String | `tree / table` | No | Aligns with `-sql_dialect` | +| `model` | Specifies whether the TsFile uses the `table` model or `tree` model. This parameter becomes invalid starting from V2.0.2.1. The system automatically identifies whether the data model is tree-based or table-based. | String | `tree / table` | No | Aligns with `-sql_dialect` | | `database-name` | Table model only: Target database for import. Automatically created if it does not exist. The database-name must not include the `root.` prefix (an error will occur if included). | String | `-` | No | null | | `convert-on-type-mismatch` | Whether to perform type conversion during loading if data types in the TsFile mismatch the target schema. | Boolean | `true / false` | No | true | | `verify` | Whether to validate the schema before loading the TsFile. | Boolean | `true / false` | No | true | diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 17774e0ed..9628b6d30 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - Data synchronization between IoTDB of 1. x series version and IoTDB of 2. x and above series versions is not supported. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. -- Does not support the Object data type. +- Object-type data export is supported since version V2.0.9.2. ## 2. Usage Instructions @@ -227,17 +227,18 @@ Example Output: ```SQL IoTDB> SHOW PIPEPLUGINS -**+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): @@ -263,8 +264,8 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param - - + + @@ -288,6 +289,14 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param + + + + + + + +
Default processor plugin that does not process incoming data.
sink PluginSupportedsink PluginSupported do-nothing-sink Does not process outgoing data.
opc-ua-sink An OPC UA protocol data transfer plugin for IoTDB (V2.0.2 and above), supporting both Client/Server and Pub/Sub communication modes.
tsfile-local-sinkUsed in IoTDB (V2.0.9.2 and later) to support exporting Object data to the local file system where the IoTDB server resides.
tsfile-remote-sinkUsed in IoTDB (V2.0.9.2 and later) to support sending Object data to a remote server via the SSH/SCP protocol.
@@ -514,6 +523,77 @@ WITH SINK ( ) ``` +### 3.9 Object-Type Data Export +Since version V2.0.9.2, IoTDB supports exporting Object-type data. The following two methods are supported by configuring sink parameters: + +* **Local Mode**: Exports data to the local file system where the IoTDB server resides. +* **SCP Mode**: Sends data to a remote server via the SSH/SCP protocol. + +**Example 1: Local Export** + +You can directly use the built-in `tsfile-local-sink` plugin to create a PIPE statement for data export. For example: + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- Required, specifies the Sink type + 'sink.local.target-path' = '/data/backup/export_2024' -- Target export path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Example 2: Remote Transfer** + +1. Contact the Timecho Team to obtain the JAR package related to the `tsfile-remote-sink` plugin, such as `tsfile-remote-sink--jar-with-dependencies.jar`, and place it in a path accessible to IoTDB (e.g., all Data Node hosts). +2. Register the plugin using the following statement: + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink--jar-with-dependencies.jar'; +``` + +3. Create the PIPE statement: + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- Specifies SCP mode + 'sink.scp.host' = '192.168.1.100', -- Remote host IP + 'sink.scp.port' = '22', -- SSH port + 'sink.scp.user' = 'backup_user', -- SSH username + 'sink.scp.password' = 'ComplexPass123!', -- SSH password + 'sink.scp.remote-path' = '/remote/archive/', -- Remote storage path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Sink Exported TSFile and Object Format:** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TSFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + ## Reference: Notes You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: @@ -689,3 +769,23 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.user | User for OPC UA, specified in the configuration | String | No | root | | sink.password | Password for OPC UA, specified in the configuration | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | | sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" | + + +#### tsfile-local-sink +| Parameter | Description | Value Range | Required | Default | +|-----------------------------------|-----------------------------------------------------------------------------|------------------------|----------|---------| +| sink | Component name | String: tsfile-local-sink | Yes | - | +| sink.local.target-path | Local target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Rate limit threshold (unit: bytes/second). Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + +#### tsfile-remote-sink +| Parameter | Description | Value Range | Required | Default | +|------------------------------------|-----------------------------------------------------------------------------|-------------------------|----------|---------| +| sink | Component name | String: tsfile-remote-sink | Yes | - | +| sink.scp.host | Remote host IP | String | Yes | - | +| sink.scp.port | Remote SSH port | Long | No | 22 | +| sink.scp.user | Remote SSH user | String | Yes | - | +| sink.scp.password | Remote SSH password | String | Yes | - | +| sink.scp.remote-path | Remote target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Unit: bytes/second. Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + diff --git a/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md b/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md index 47d38c9f6..01d9eb9b7 100644 --- a/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md @@ -1,7 +1,11 @@ # Data Export ## 1. Function Overview -The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows: + +IoTDB supports two methods for data export: + +* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory. It can export the query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) files. +* PIPE Framework-based TsFileBackup: `tsfile-backup.sh/bat` is located in the `tools` directory. It can export specified data files into TsFile format using the PIPE framework. @@ -19,15 +23,18 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an - + + + + +
File containing custom SQL statements.
TsFileTsFile Open-source time-series file format.
tsfile-backup.sh/batAn open-source time-series data file format,and this script supports the Object data type.
-- Does not support the Object data type. -## 2. Detailed Features +## 2. Data Export Tool ### 2.1 Common Parameters | Short | Full Parameter | Description | Required | Default | |----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |-----------------------------------------------| @@ -91,7 +98,7 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an > export-data.sh -ft csv -sql_dialect table -t /path/export/dir -q "select * from table1" Parse error: Missing required option: db ``` -## 2.3 SQL Format +### 2.3 SQL Format #### 2.3.1 Command ```Shell # Unix/OS X @@ -168,3 +175,72 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + + +## 3. TsFileBackup Based on PIPE Framework +Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This script can automatically generate and send the `CREATE PIPE` SQL command to the server, exporting specified data files to TsFile format. + +**Notes:** +1. **To use this script, contact the Timecho Team to obtain the customized installation package `timechodb--extension`.** +2. **This script supports exporting Object-type data to TsFile files.** + + +### 3.1 Execution Commands +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] + +# Windows +> tools\windows\tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + + +### 3.2 Script Parameters +| Abbreviation | Full Name | Description | Required | Default | +| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- | +| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - | +| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` | +| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` | +| `-u` | `--user` | Username for IoTDB authentication. | No | `root` | +| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` | +| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - | +| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` | +| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` | +| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - | +| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - | +| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - | +| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - | +| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - | +| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` | +| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - | +| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - | +| `-help` | `--help` | Show help information. | No | - | + + +### 3.3 Execution Examples + +Example 1: SCP Remote Export (Send Data to Another Server) + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +Example 2: Remote Object Data Export with Rate Limiting + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +Example 3: Specify Pipe Plugin JAR Directory + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` diff --git a/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md b/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md index 6f8263103..b2c8cfe81 100644 --- a/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md @@ -37,7 +37,8 @@ IoTDB supports three methods for data import:
-- Does not support the Object data type. +- The table model TsFile import currently only supports local import. +- Since version V2.0.9.2, the import-data.sh/bat script supports the Object data type when importing TsFile files. ## 2. Data Import Tool ### 2.1 Common Parameters @@ -184,7 +185,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -197,18 +198,19 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x and later versions > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 TsFile-Specific Parameters -| Short | Full Parameter | Description | Required | Default | -| ----------- | ----------------------------- |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- | --------------------------- | -| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | -| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | -| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | -| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | -| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| Short | Full Parameter | Description | Required | Default | +|---------|---------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|-----------------------| +| `-os` | `--on_success` | Action for successful files:
`none`: Do not delete the file.
`mv`: Move the successful file to the target directory.
`cp`:Create a hard link (copy) of the successful file to the target directory.
`delete`:Delete the file. | ​**Yes** | - | +| `-sd` | `--success_dir` | Target directory for `mv`/`cp` actions on success. Required if `-os` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/success` | +| `-of` | `--on_fail` | Action for failed files:
`none`:Skip the file.
`mv`:Move the failed file to the target directory.
`cp`:Create a hard link (copy) of the failed file to the target directory.
`delete`:Delete the file.. | ​**Yes** | - | +| `-fd` | `--fail_dir` | Target directory for `mv`/`cp` actions on failure. Required if `-of` is `mv`/`cp`. The file name will be flattened and concatenated with the original file name. | Conditional | `${EXEC_DIR}/fail` | +| `-tp` | `--timestamp_precision` | TsFile timestamp precision: `ms`, `us`, `ns`.
For non-remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The system will manually verify if the timestamp precision matches the server. If it does not match, an error will be returned.
​For remote TsFile imports: Use -tp to specify the timestamp precision of the TsFile. The Pipe system will automatically verify if the timestamp precision matches. If it does not match, a Pipe error will be returned. | No | `ms` | +| `-o` | `--object-file-paths` | Storage path for Object files.
Default mode: If this parameter is not specified, the script automatically identifies and imports Object files located in the subdirectory with the same name as the TsFile.
Absolute path mode: Explicitly specifies the external storage root directory for Object files; the tool creates an associated data index based on this path.
Note: This parameter is supported since V2.0.9.2 | No | | #### 2.4.3 Examples @@ -221,6 +223,54 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` + +**Object Type Import** + +1. Import Directory Structure + +* Default Mode + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TsFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* Specified Object Directory + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. Command Line Examples + +* Basic Import (automatically identifies Object files in the TsFile-named directory) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* Batch Directory Import (specify concurrent threads and post-success action) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* Table Model Associated Import (specify external Object storage path and target database) + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile Auto-Loading This feature enables IoTDB to automatically monitor a specified directory for new TsFiles and load them into the database without manual intervention. diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 17774e0ed..9628b6d30 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - Data synchronization between IoTDB of 1. x series version and IoTDB of 2. x and above series versions is not supported. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. -- Does not support the Object data type. +- Object-type data export is supported since version V2.0.9.2. ## 2. Usage Instructions @@ -227,17 +227,18 @@ Example Output: ```SQL IoTDB> SHOW PIPEPLUGINS -**+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` Detailed introduction of pre-installed plugins is as follows (for detailed parameters of each plugin, please refer to the [Parameter Description](#reference-parameter-description): @@ -263,8 +264,8 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param - - + + @@ -288,6 +289,14 @@ Detailed introduction of pre-installed plugins is as follows (for detailed param + + + + + + + +
Default processor plugin that does not process incoming data.
sink PluginSupportedsink PluginSupported do-nothing-sink Does not process outgoing data.
opc-ua-sink An OPC UA protocol data transfer plugin for IoTDB (V2.0.2 and above), supporting both Client/Server and Pub/Sub communication modes.
tsfile-local-sinkUsed in IoTDB (V2.0.9.2 and later) to support exporting Object data to the local file system where the IoTDB server resides.
tsfile-remote-sinkUsed in IoTDB (V2.0.9.2 and later) to support sending Object data to a remote server via the SSH/SCP protocol.
@@ -514,6 +523,77 @@ WITH SINK ( ) ``` +### 3.9 Object-Type Data Export +Since version V2.0.9.2, IoTDB supports exporting Object-type data. The following two methods are supported by configuring sink parameters: + +* **Local Mode**: Exports data to the local file system where the IoTDB server resides. +* **SCP Mode**: Sends data to a remote server via the SSH/SCP protocol. + +**Example 1: Local Export** + +You can directly use the built-in `tsfile-local-sink` plugin to create a PIPE statement for data export. For example: + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- Required, specifies the Sink type + 'sink.local.target-path' = '/data/backup/export_2024' -- Target export path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Example 2: Remote Transfer** + +1. Contact the Timecho Team to obtain the JAR package related to the `tsfile-remote-sink` plugin, such as `tsfile-remote-sink--jar-with-dependencies.jar`, and place it in a path accessible to IoTDB (e.g., all Data Node hosts). +2. Register the plugin using the following statement: + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink--jar-with-dependencies.jar'; +``` + +3. Create the PIPE statement: + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- Specifies SCP mode + 'sink.scp.host' = '192.168.1.100', -- Remote host IP + 'sink.scp.port' = '22', -- SSH port + 'sink.scp.user' = 'backup_user', -- SSH username + 'sink.scp.password' = 'ComplexPass123!', -- SSH password + 'sink.scp.remote-path' = '/remote/archive/', -- Remote storage path + 'sink.rate-limit-bytes-per-second' = '10485760' -- Rate limit: 10MB/s +); +``` + +**Sink Exported TSFile and Object Format:** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (matches the TSFile name) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + ## Reference: Notes You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows: @@ -689,3 +769,23 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.user | User for OPC UA, specified in the configuration | String | No | root | | sink.password | Password for OPC UA, specified in the configuration | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | | sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" | + + +#### tsfile-local-sink +| Parameter | Description | Value Range | Required | Default | +|-----------------------------------|-----------------------------------------------------------------------------|------------------------|----------|---------| +| sink | Component name | String: tsfile-local-sink | Yes | - | +| sink.local.target-path | Local target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Rate limit threshold (unit: bytes/second). Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + +#### tsfile-remote-sink +| Parameter | Description | Value Range | Required | Default | +|------------------------------------|-----------------------------------------------------------------------------|-------------------------|----------|---------| +| sink | Component name | String: tsfile-remote-sink | Yes | - | +| sink.scp.host | Remote host IP | String | Yes | - | +| sink.scp.port | Remote SSH port | Long | No | 22 | +| sink.scp.user | Remote SSH user | String | Yes | - | +| sink.scp.password | Remote SSH password | String | Yes | - | +| sink.scp.remote-path | Remote target directory | String | Yes | - | +| sink.rate-limit-bytes-per-second | Unit: bytes/second. Takes effect when enabled. No limit if rate-limit <= 0 | Long | No | 0 | + diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md b/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md index c5f6499a8..3d3056dd7 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md @@ -2,7 +2,10 @@ ## 1. 功能概述 -数据导出工具 `export-data.sh/bat` 位于 `tools` 目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile（开源时间序列文件格式）格式。具体功能如下： +IoTDB 支持两种方式进行数据导出： + +* 数据导出工具 ：`export-data.sh/bat` 位于 `tools `目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile （开源时间序列文件格式）格式。 +* 基于 PIPE 框架的 TsFileBackup：`tsfile-backup.sh/bat`位于 `tools `目录下，能够使用 PIPE 将指定的数据文件导出为 TsFile 格式。 @@ -20,15 +23,18 @@ - + + + + +
包含自定义 SQL 语句的文件
TsFileTsFile 开源时序数据文件格式
tsfile-backup.sh/bat开源时序数据文件格式，支持 Object 数据类型
-- 不支持 Object 数据类型。 -## 2. 功能详解 +## 2. 数据导出工具 ### 2.1 公共参数 @@ -178,3 +184,71 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. 基于 PIPE 框架的 TsFileBackup + +IoTDB 自 **V2.0.9.2** 版本起支持 `tsfile-backup.sh/bat` 脚本，该脚本能够自动生成并向服务端发送 `CREATE PIPE` SQL 指令，将指定的数据文件导出为 TsFile 格式。 + +**注意：** + +1. ​**使用该脚本需联系天谋团队获取定制安装包 ​`timechodb--extension`**​**。** +2. **该脚本支持 Object 类型数据导出为 TsFile 文件。** + +### 3.1 运行命令 + +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +# Windows +> tools\windows>tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + +### 3.2 脚本参数 + +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------- | -------------------- | +| `-sql_dialect` | `--sql_dialect` | 指定数据模型类型，可选值：`tree`(树模型) 或`table`(表模型)。 | 是 | - | +| `-h` | `--host` | 本地主机地址。指当前数据所在的 IoTDB 实例 IP。 | 否 | `127.0.0.1` | +| `-p` | `--port` | 端口号，IoTDB RPC 服务端口。 | 否 | `6667` | +| `-u` | `--user` | 用户名，用于登录 IoTDB 验证。 | 否 | `root` | +| `-pw` | `--password` | 密码，对应用户的IoTDB密码，支持隐藏输入。 | 否 | `root` | +| `-t` | `--target` | 导出目标目录。在 SCP 模式下，此路径指远程服务器上的绝对物理路径。TsFile 和关联的 Object 目录将导出至此。 | 是 | - | +| `-db` | `--database` | 数据库名称 (表模型可选) | 否 | `.*` | +| `-table` | `--table` | 表名 (表模型可选) | 否 | `.*` | +| `-s` | `--start_time` | 起始时间。支持 ISO8601 格式（如 2026-01-01T00:00:00）或毫秒时间戳。仅导出该时间点及之后的数据。 | 否 | - | +| `-e` | `--end_time` | 截止时间。格式同上。仅导出该时间点之前的数据。 | 否 | - | +| `-th` | `--target_host` | 远程目标主机 IP，默认自动识别启动脚本的IP。指定此参数后，脚本将自动配置 Pipe 使用 SCP 模式进行数据传输。 | 否 | - | +| `-tu` | `--target_host_user` | 远程主机用户名。用于 SSH/SCP 登录目标服务器。 | 否 | - | +| `-tpw` | `--target_host_pw` | 远程主机密码。用于远程身份验证，支持隐藏输入。 | 否 | - | +| `-tp` | `--target_host_port` | 远程 SSH 端口。 | 否 | `22` | +| `--rate_limit` | `--rate_limit` | 发送速率限制。单位：字节/秒 (Bytes/s)。防止导出任务占用过多网络带宽。 | 否 | - | +| `--plugin_jar` | `--plugin_jar` | 指定 Pipe 插件的Jar包路径 | 否 | - | +| `-help` | `--help` | 查看帮助 | 否 | - | + +### 3.3 运行示例 + +示例一：SCP 远程导出（将数据发送到另一台服务器） + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +示例二：带限速的远程 Object 数据导出 + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +示例三：指定 Pipe jar 目录 + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` \ No newline at end of file diff --git a/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md b/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md index 6341f0649..bc94826d1 100644 --- a/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/zh/UserGuide/Master/Table/Tools-System/Data-Import-Tool_timecho.md @@ -38,7 +38,7 @@ IoTDB 支持三种方式进行数据导入：
- **表模型 TsFile 导入暂时只支持本地导入。** -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起，import-data.sh/bat 脚本导入 tsfile 文件时支持 Object 数据类型。 ## 2. 数据导入工具 @@ -189,7 +189,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -202,19 +202,20 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x 版本及之后 > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 私有参数 -| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | -| ---------- | ------------------------ |----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | -| -os| --on\_succcess| 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || -| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| -| -of| --on\_fail| 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || -| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | -| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +|------|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | +| -os | --on\_succcess | 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || +| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| +| -of | --on\_fail | 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || +| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | +| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| -o | --object-file-paths | Object 文件存储路径。
默认模式：若不指定此参数，脚本将自动识别并导入位于 `/` 同名子目录下的 Object 文件。
绝对路径模式：显式指定 Object 文件的外部存储根目录，工具将基于此路径建立数据的关联索引。
注意：该参数自 V2.0.9.2 版本起支持 | 否 | | #### 2.4.3 运行示例 @@ -228,6 +229,53 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` +**Object 类型导入** + +1. 导入格式 + +* 默认 + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* 指定 Object 目录 + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. 命令行示例 + +* 基础导入（自动识别 TsFile 同名目录下的 Object 文件） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* 批量导入目录（指定并发线程数与成功后的处理动作） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* 表模型关联导入（指定外部 Object 存储路径与目标数据库） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile 自动加载功能 本功能允许 IoTDB 主动监听指定目录下的新增 TsFile，并将 TsFile 自动加载至 IoTDB 中。通过此功能，IoTDB 能自动检测并加载 TsFile，无需手动执行任何额外的加载操作。 diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index 2968a8301..99d523cdc 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ - 不支持 1.x 系列版本 IoTDB 与 2.x 以及以上系列版本的 IoTDB 之间进行数据同步。 - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起支持 Object 类型数据导出。 ## 2. 使用说明 @@ -223,17 +223,18 @@ SHOW PIPEPLUGINS ```SQL IoTDB> SHOW PIPEPLUGINS -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` 预置插件详细介绍如下(各插件的详细参数可参考本文[参数说明](#参考参数说明))： @@ -260,8 +261,8 @@ IoTDB> SHOW PIPEPLUGINS - - + + @@ -285,6 +286,14 @@ IoTDB> SHOW PIPEPLUGINS + + + + + + + +
默认的 processor 插件，不对传入的数据做任何的处理
sink 插件支持sink 插件支持 do-nothing-sink 不对发送出的数据做任何的处理
opc-ua-sink 用于 IoTDB （V2.0.2 及以上）支持OPC UA协议的数据传输插件，支持Client/Server 和 Pub/Sub 两种通信模式。
tsfile-local-sink用于 IoTDB (V2.0.9.2及以上）支持将 Object 数据导出到 IoTDB 服务器所在的本地文件系统。
tsfile-remote-sink用于 IoTDB (V2.0.9.2及以上）支持通过 SSH/SCP 协议将 Object 数据发送到远程服务器。
@@ -507,6 +516,79 @@ with sink ( ) ``` +### 3.9 Object 类型数据导出 + +IoTDB 自 V2.0.9.2 版本起支持导出 Object 类型数据，通过配置 sink 参数支持如下两种方式： + +* Local 模式（本地导出）：将数据导出到 IoTDB 服务器所在的本地文件系统。 +* SCP 模式（远程传输）：通过 SSH/SCP 协议将数据发送到远程服务器。 + +**示例一：本地导出** + +可直接使用系统内置的 `tsfile-local-sink `插件创建 PIPE 语句导出数据，例如： + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- 必填，指定 Sink 类型 + 'sink.local.target-path' = '/data/backup/export_2024' -- 导出目标路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**示例二：远程传输** + +1. 联系天谋团队获取 `tsfile-remote-sink` 插件相关的 jar 包，如 `tsfile-remote-sink--jar-with-dependencies.jar`，并放至 IoTDB 可访问的路径（例如所有数据节点主机）。 +2. 使用如下语句注册插件 + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink-<版本号>-jar-with-dependencies.jar'; +``` + +3. 创建 PIPE 语句 + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- 指定为 SCP 模式 + 'sink.scp.host' = '192.168.1.100', -- 远程主机 IP + 'sink.scp.port' = '22', -- SSH 端口 + 'sink.scp.user' = 'backup_user', -- SSH 用户名 + 'sink.scp.password' = 'ComplexPass123!', -- SSH 密码 + 'sink.scp.remote-path' = '/remote/archive/', -- 远程存放路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**Sink 导出 TSFile 与 Object 格式：** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + + ## 参考：注意事项 可通过修改 IoTDB 配置文件（`iotdb-system.properties`）以调整数据同步的参数，如同步数据存储目录等。完整配置如下：： @@ -683,3 +765,22 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.password | 密码，这里指 OPC UA 的允许密码 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | | sink.opcua.placeholder | 当ID列的值出现null时，用于替代null映射路径的占位字符串 | String | 选填 | "null" | +#### tsfile-local-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|-----------------------------------|-----------------------------------------|---------------------------|------|---------| +| sink | 组件名称 | String: tsfile-local-sink | 是 | - | +| sink.local.target-path | 本地目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 限速阈值。单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 | + +#### tsfile-remote-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|------------------------------------|-----------------------------------|--------------------------|----------|---------| +| sink | 组件名称 | String: tsfile-remote-sink | 是 | - | +| sink.scp.host | 远程主机 IP | String | 是 | - | +| sink.scp.port | 远程 SSH 端口 | Long | 否 | 22 | +| sink.scp.user | 远程 SSH 用户 | String | 是 | - | +| sink.scp.password | 远程 SSH 密码 | String | 是 | - | +| sink.scp.remote-path | 远程目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 | diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md b/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md index c5f6499a8..3d3056dd7 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Data-Export-Tool_timecho.md @@ -2,7 +2,10 @@ ## 1. 功能概述 -数据导出工具 `export-data.sh/bat` 位于 `tools` 目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile（开源时间序列文件格式）格式。具体功能如下： +IoTDB 支持两种方式进行数据导出： + +* 数据导出工具 ：`export-data.sh/bat` 位于 `tools `目录下，能够将指定 SQL 的查询结果导出为 CSV、SQL 及 TsFile （开源时间序列文件格式）格式。 +* 基于 PIPE 框架的 TsFileBackup：`tsfile-backup.sh/bat`位于 `tools `目录下，能够使用 PIPE 将指定的数据文件导出为 TsFile 格式。 @@ -20,15 +23,18 @@ - + + + + +
包含自定义 SQL 语句的文件
TsFileTsFile 开源时序数据文件格式
tsfile-backup.sh/bat开源时序数据文件格式，支持 Object 数据类型
-- 不支持 Object 数据类型。 -## 2. 功能详解 +## 2. 数据导出工具 ### 2.1 公共参数 @@ -178,3 +184,71 @@ Parse error: Missing required option: db > /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0 Parse error: Missing required option: db ``` + +## 3. 基于 PIPE 框架的 TsFileBackup + +IoTDB 自 **V2.0.9.2** 版本起支持 `tsfile-backup.sh/bat` 脚本，该脚本能够自动生成并向服务端发送 `CREATE PIPE` SQL 指令，将指定的数据文件导出为 TsFile 格式。 + +**注意：** + +1. ​**使用该脚本需联系天谋团队获取定制安装包 ​`timechodb--extension`**​**。** +2. **该脚本支持 Object 类型数据导出为 TsFile 文件。** + +### 3.1 运行命令 + +```Shell +# Unix/OS X +> tools/tsfile-backup.sh [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table + ] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +# Windows +> tools\windows>tsfile-backup.bat [-sql_dialect ] [-h ] [-p ] + [-u ] [-pw ] [-path ] [-db ] [-table +
] [-s ] [-e ] [-t ] + [-th ] [-tu ] [-tp ] + [--rate_limit] [--plugin_jar] [-help] +``` + +### 3.2 脚本参数 + +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +| -------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------- | -------------- | -------------------- | +| `-sql_dialect` | `--sql_dialect` | 指定数据模型类型，可选值：`tree`(树模型) 或`table`(表模型)。 | 是 | - | +| `-h` | `--host` | 本地主机地址。指当前数据所在的 IoTDB 实例 IP。 | 否 | `127.0.0.1` | +| `-p` | `--port` | 端口号，IoTDB RPC 服务端口。 | 否 | `6667` | +| `-u` | `--user` | 用户名，用于登录 IoTDB 验证。 | 否 | `root` | +| `-pw` | `--password` | 密码，对应用户的IoTDB密码，支持隐藏输入。 | 否 | `root` | +| `-t` | `--target` | 导出目标目录。在 SCP 模式下，此路径指远程服务器上的绝对物理路径。TsFile 和关联的 Object 目录将导出至此。 | 是 | - | +| `-db` | `--database` | 数据库名称 (表模型可选) | 否 | `.*` | +| `-table` | `--table` | 表名 (表模型可选) | 否 | `.*` | +| `-s` | `--start_time` | 起始时间。支持 ISO8601 格式（如 2026-01-01T00:00:00）或毫秒时间戳。仅导出该时间点及之后的数据。 | 否 | - | +| `-e` | `--end_time` | 截止时间。格式同上。仅导出该时间点之前的数据。 | 否 | - | +| `-th` | `--target_host` | 远程目标主机 IP，默认自动识别启动脚本的IP。指定此参数后，脚本将自动配置 Pipe 使用 SCP 模式进行数据传输。 | 否 | - | +| `-tu` | `--target_host_user` | 远程主机用户名。用于 SSH/SCP 登录目标服务器。 | 否 | - | +| `-tpw` | `--target_host_pw` | 远程主机密码。用于远程身份验证，支持隐藏输入。 | 否 | - | +| `-tp` | `--target_host_port` | 远程 SSH 端口。 | 否 | `22` | +| `--rate_limit` | `--rate_limit` | 发送速率限制。单位：字节/秒 (Bytes/s)。防止导出任务占用过多网络带宽。 | 否 | - | +| `--plugin_jar` | `--plugin_jar` | 指定 Pipe 插件的Jar包路径 | 否 | - | +| `-help` | `--help` | 查看帮助 | 否 | - | + +### 3.3 运行示例 + +示例一：SCP 远程导出（将数据发送到另一台服务器） + +```Bash +./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123! +``` + +示例二：带限速的远程 Object 数据导出 + +```Bash +./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880 +``` + +示例三：指定 Pipe jar 目录 + +```Bash +./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar +``` \ No newline at end of file diff --git a/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md b/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md index 6341f0649..d48186fb5 100644 --- a/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md +++ b/src/zh/UserGuide/latest-Table/Tools-System/Data-Import-Tool_timecho.md @@ -38,7 +38,7 @@ IoTDB 支持三种方式进行数据导入：
- **表模型 TsFile 导入暂时只支持本地导入。** -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起，import-data.sh/bat 脚本导入 tsfile 文件时支持 Object 数据类型。 ## 2. 数据导入工具 @@ -189,7 +189,7 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # Unix/OS X > tools/import-data.sh -ft [-sql_dialect] -db -table [-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] # Windows @@ -202,19 +202,20 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste # V2.0.4.x 版本及之后 > tools\windows\import-data.bat -ft [-sql_dialect] -db -table
[-h ] [-p ] [-u ] [-pw ] - -s -os [-sd ] -of [-fd ] + -s [-o ] -os [-sd ] -of [-fd ] [-tn ] [-tz ] [-tp ] ``` #### 2.4.2 私有参数 -| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | -| ---------- | ------------------------ |----------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | -| -os| --on\_succcess| 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || -| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| -| -of| --on\_fail| 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || -| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | -| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| 参数缩写 | 参数全称 | 参数含义 | 是否为必填项 | 默认值 | +|------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------| -------------------- | +| -os | --on\_succcess | 1. none：不删除
2. mv：移动成功的文件到目标文件夹
3. cp：硬连接（拷贝）成功的文件到目标文件夹
4. delete：删除 | √ || +| -sd | --success\_dir | 当`--on_succcess`为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_succcess`为mv或cp时需要填写 | `${EXEC_DIR}/success`| +| -of | --on\_fail | 1. none：跳过
2. mv：移动失败的文件到目标文件夹
3. cp：硬连接（拷贝）失败的文件到目标文件夹
4. delete：删除 | √ || +| -fd | --fail\_dir | 当`--on_fail`指定为 mv 或 cp 时，mv 或 cp 的目标文件夹。文件的文件名变为文件夹打平后拼接原有文件名 | 当`--on_fail`指定为 mv 或 cp 时需要填写 | `${EXEC_DIR}/fail` | +| -tp | --timestamp\_precision | 时间戳精度
tsfile 非远程导入：-tp 指定 tsfile 文件的时间精度 手动校验和服务器的时间戳是否一致 不一致返回报错信息
远程导入：-tp 指定 tsfile 文件的时间精度 pipe 自动校验时间戳精度是否一致 不一致返回 pipe 报错信息 | 否：
1. ms（毫秒）
2. us（微秒）
3. ns（纳秒） | ms| +| -o | --object-file-paths | Object 文件存储路径。
默认模式：若不指定此参数，脚本将自动识别并导入位于 `/` 同名子目录下的 Object 文件。
绝对路径模式：显式指定 Object 文件的外部存储根目录，工具将基于此路径建立数据的关联索引。
注意：该参数自 V2.0.9.2 版本起支持 | 否 | | #### 2.4.3 运行示例 @@ -228,6 +229,53 @@ Fail to insert measurements '[column.name]' caused by [data type is not consiste Parse error: Missing required options: os, of ``` +**Object 类型导入** + +1. 导入格式 + +* 默认 + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +* 指定 Object 目录 + +```Bash +target_dir + ├── tsfile.tsfile +object_dir + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + +2. 命令行示例 + +* 基础导入（自动识别 TsFile 同名目录下的 Object 文件） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/sensor_v1.tsfile -db database1 -os none -of none +``` + +* 批量导入目录（指定并发线程数与成功后的处理动作） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/raw_data/ -tn 16 -os mv -sd /data/archive/ +``` + +* 表模型关联导入（指定外部 Object 存储路径与目标数据库） + +```Bash +./import-data.sh -sql_dialect table -ft tsfile -s /data/import/ -db factory_db -o /mnt/object_storage/ -of mv -fd /data/error_log/ +``` + + ## 3. TsFile 自动加载功能 本功能允许 IoTDB 主动监听指定目录下的新增 TsFile，并将 TsFile 自动加载至 IoTDB 中。通过此功能，IoTDB 能自动检测并加载 TsFile，无需手动执行任何额外的加载操作。 diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index 2968a8301..99d523cdc 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -81,7 +81,7 @@ - 不支持 1.x 系列版本 IoTDB 与 2.x 以及以上系列版本的 IoTDB 之间进行数据同步。 - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 -- 不支持 Object 数据类型。 +- 自 V2.0.9.2 版本起支持 Object 类型数据导出。 ## 2. 使用说明 @@ -223,17 +223,18 @@ SHOW PIPEPLUGINS ```SQL IoTDB> SHOW PIPEPLUGINS -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| PluginName|PluginType| ClassName|PluginJar| -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ -| DO-NOTHING-PROCESSOR| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | -| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.donothing.DoNothingConnector| | -| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.airgap.IoTDBAirGapConnector| | -| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.extractor.iotdb.IoTDBExtractor| | -| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftConnector| | -|IOTDB-THRIFT-SSL-SINK| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.iotdb.thrift.IoTDBThriftSslConnector| | -| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.connector.writeback.WriteBackConnector| | -+---------------------+----------+-------------------------------------------------------------------------------------------------+---------+ ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| PluginName|PluginType| ClassName|PluginJar|ExceptionMessage| ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ +| DO-NOTHING-PROCESSOR| Builtin|org.apache.iotdb.commons.pipe.agent.plugin.builtin.processor.donothing.DoNothingProcessor| | | +| DO-NOTHING-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.donothing.DoNothingSink| | | +| IOTDB-AIR-GAP-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.airgap.IoTDBAirGapSink| | | +| IOTDB-SOURCE| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.source.iotdb.IoTDBSource| | | +| IOTDB-THRIFT-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSink| | | +|IOTDB-THRIFT-SSL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.iotdb.thrift.IoTDBThriftSslSink| | | +| TSFILE-LOCAL-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.tsfile.PipeTsFileLocalSink| | | +| WRITE-BACK-SINK| Builtin| org.apache.iotdb.commons.pipe.agent.plugin.builtin.sink.writeback.WriteBackSink| | | ++---------------------+----------+-----------------------------------------------------------------------------------------+---------+----------------+ ``` 预置插件详细介绍如下(各插件的详细参数可参考本文[参数说明](#参考参数说明))： @@ -260,8 +261,8 @@ IoTDB> SHOW PIPEPLUGINS - - + + @@ -285,6 +286,14 @@ IoTDB> SHOW PIPEPLUGINS + + + + + + + +
默认的 processor 插件，不对传入的数据做任何的处理
sink 插件支持sink 插件支持 do-nothing-sink 不对发送出的数据做任何的处理
opc-ua-sink 用于 IoTDB （V2.0.2 及以上）支持OPC UA协议的数据传输插件，支持Client/Server 和 Pub/Sub 两种通信模式。
tsfile-local-sink用于 IoTDB (V2.0.9.2及以上）支持将 Object 数据导出到 IoTDB 服务器所在的本地文件系统。
tsfile-remote-sink用于 IoTDB (V2.0.9.2及以上）支持通过 SSH/SCP 协议将 Object 数据发送到远程服务器。
@@ -507,6 +516,79 @@ with sink ( ) ``` +### 3.9 Object 类型数据导出 + +IoTDB 自 V2.0.9.2 版本起支持导出 Object 类型数据，通过配置 sink 参数支持如下两种方式： + +* Local 模式（本地导出）：将数据导出到 IoTDB 服务器所在的本地文件系统。 +* SCP 模式（远程传输）：通过 SSH/SCP 协议将数据发送到远程服务器。 + +**示例一：本地导出** + +可直接使用系统内置的 `tsfile-local-sink `插件创建 PIPE 语句导出数据，例如： + +```SQL +CREATE PIPE tsfile_export_local +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile-local-sink', -- 必填，指定 Sink 类型 + 'sink.local.target-path' = '/data/backup/export_2024' -- 导出目标路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**示例二：远程传输** + +1. 联系天谋团队获取 `tsfile-remote-sink` 插件相关的 jar 包，如 `tsfile-remote-sink--jar-with-dependencies.jar`，并放至 IoTDB 可访问的路径（例如所有数据节点主机）。 +2. 使用如下语句注册插件 + +```SQL +CREATE PIPEPLUGIN tsfile_remote_sink +AS 'org.apache.iotdb.pipe.plugin.sink.tsfile.PipeTsFileRemoteSink' +USING URI 'file:///path/to/tsfile-remote-sink-<版本号>-jar-with-dependencies.jar'; +``` + +3. 创建 PIPE 语句 + +```SQL +CREATE PIPE tsfile_export_scp +WITH SOURCE ( + 'source' = 'iotdb-source', + 'table-name' = 'test_table' +) +WITH PROCESSOR ( + 'processor' = 'do-nothing-processor' +) +WITH SINK ( + 'sink' = 'tsfile_remote_sink', + 'sink.file-mode' = 'scp', -- 指定为 SCP 模式 + 'sink.scp.host' = '192.168.1.100', -- 远程主机 IP + 'sink.scp.port' = '22', -- SSH 端口 + 'sink.scp.user' = 'backup_user', -- SSH 用户名 + 'sink.scp.password' = 'ComplexPass123!', -- SSH 密码 + 'sink.scp.remote-path' = '/remote/archive/', -- 远程存放路径 + 'sink.rate-limit-bytes-per-second' = '10485760' -- 限速 10MB/s +); +``` + +**Sink 导出 TSFile 与 Object 格式：** + +```Bash +target_dir + ├── tsfile.tsfile + └── tsfile/ (对应TSFile名字) + ├── regionID/tableName/tag1/tag2/field/timestamp1.bin + ├── regionID/tableName/tag1/tag2/field/timestamp2.bin + └── regionID/tableName1/tag3/tag4/field/timestamp1.bin +``` + + ## 参考：注意事项 可通过修改 IoTDB 配置文件（`iotdb-system.properties`）以调整数据同步的参数，如同步数据存储目录等。完整配置如下：： @@ -683,3 +765,22 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.password | 密码，这里指 OPC UA 的允许密码 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | | sink.opcua.placeholder | 当ID列的值出现null时，用于替代null映射路径的占位字符串 | String | 选填 | "null" | +#### tsfile-local-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|-----------------------------------|-----------------------------------------|---------------------------|------|---------| +| sink | 组件名称 | String: tsfile-local-sink | 是 | - | +| sink.local.target-path | 本地目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 限速阈值。单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 | + +#### tsfile-remote-sink + +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认值** | +|------------------------------------|-----------------------------------|--------------------------|----------|---------| +| sink | 组件名称 | String: tsfile-remote-sink | 是 | - | +| sink.scp.host | 远程主机 IP | String | 是 | - | +| sink.scp.port | 远程 SSH 端口 | Long | 否 | 22 | +| sink.scp.user | 远程 SSH 用户 | String | 是 | - | +| sink.scp.password | 远程 SSH 密码 | String | 是 | - | +| sink.scp.remote-path | 远程目标目录 | String | 是 | - | +| sink.rate-limit-bytes-per-second | 单位：字节/秒。开启限速时生效。rate-limit<=0不限速 | Long | 否 | 0 |