Skip to content

Add object import and export 2092 #1112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HTHou merged 3 commits into from
May 15, 2026
+1,013 −115
Merged

Add object import and export 2092 #1112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
98a7d5f
fix slow query of audit log version
leto-b May 6, 2026
bb72f35
Merge remote-tracking branch 'upstream/main'
leto-b May 11, 2026
0d6b004
support obejct import(import-data) and export(tsfile-backup and pipe …
leto-b May 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 81 additions & 5 deletions src/UserGuide/Master/Table/Tools-System/Data-Export-Tool_timecho.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# Data Export

## 1. Function Overview
The data export tool `export-data.sh/bat` is located in the `tools` directory and can export query results from specified SQL statements into CSV, SQL, or TsFile (open-source time-series file format) formats. Its specific functionalities are as follows:

IoTDB supports two methods for data export:

* Data Export Tool: `export-data.sh/bat` is located in the `tools` directory. It can export the query results of specified SQL statements into CSV, SQL, and TsFile (open-source time-series file format) files.
* PIPE Framework-based TsFileBackup: `tsfile-backup.sh/bat` is located in the `tools` directory. It can export specified data files into TsFile format using the PIPE framework.

<table style="text-align: left;">
<tbody>
Expand All @@ -19,15 +23,18 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an
<td>File containing custom SQL statements.</td>
</tr>
<tr>
<td >TsFile</td>
<td rowspan="2">TsFile</td>
<td>Open-source time-series file format.</td>
</tr>
<tr>
<td>tsfile-backup.sh/bat</td>
<td>An open-source time-series data file format,and this script supports the Object data type.</td>
</tr>
</tbody>
</table>

- Does not support the Object data type.

## 2. Detailed Features
## 2. Data Export Tool
### 2.1 Common Parameters
| Short | Full Parameter | Description | Required | Default |
|----------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ----------------- |-----------------------------------------------|
Expand Down Expand Up @@ -91,7 +98,7 @@ The data export tool `export-data.sh/bat` is located in the `tools` directory an
> export-data.sh -ft csv -sql_dialect table -t /path/export/dir -q "select * from table1"
Parse error: Missing required option: db
```
## 2.3 SQL Format
### 2.3 SQL Format
#### 2.3.1 Command
```Shell
# Unix/OS X
Expand Down Expand Up @@ -168,3 +175,72 @@ Parse error: Missing required option: db
> /tools/export-data.sh -ft tsfile -sql_dialect table -t /path/export/dir -start_time 0
Parse error: Missing required option: db
```


## 3. TsFileBackup Based on PIPE Framework
Since **V2.0.9.2**, IoTDB supports the `tsfile-backup.sh/bat` script. This script can automatically generate and send the `CREATE PIPE` SQL command to the server, exporting specified data files to TsFile format.

**Notes:**
1. **To use this script, contact the Timecho Team to obtain the customized installation package `timechodb-<version>-extension`.**
2. **This script supports exporting Object-type data to TsFile files.**


### 3.1 Execution Commands
```Shell
# Unix/OS X
> tools/tsfile-backup.sh [-sql_dialect <sql_dialect>] [-h <host>] [-p <port>]
[-u <username>] [-pw <password>] [-path <path>] [-db <db>] [-table
<table>] [-s <start_time>] [-e <end_time>] [-t <target_directory>]
[-th <target_host>] [-tu <target_host_user>] [-tp <target_host_port>]
[--rate_limit] [--plugin_jar] [-help]

# Windows
> tools\windows\tsfile-backup.bat [-sql_dialect <sql_dialect>] [-h <host>] [-p <port>]
[-u <username>] [-pw <password>] [-path <path>] [-db <db>] [-table
<table>] [-s <start_time>] [-e <end_time>] [-t <target_directory>]
[-th <target_host>] [-tu <target_host_user>] [-tp <target_host_port>]
[--rate_limit] [--plugin_jar] [-help]
```


### 3.2 Script Parameters
| Abbreviation | Full Name | Description | Required | Default |
| ------------ | ------------------ | ----------------------------------------------------------------------------------------------------------- | -------- | --------------- |
| `-sql_dialect` | `--sql_dialect` | Specifies the data model type. Valid values: `tree` (Tree Model) or `table` (Table Model). | Yes | - |
| `-h` | `--host` | Local host address (IP of the IoTDB instance where the data resides). | No | `127.0.0.1` |
| `-p` | `--port` | Port number for the IoTDB RPC service. | No | `6667` |
| `-u` | `--user` | Username for IoTDB authentication. | No | `root` |
| `-pw` | `--password` | Password for IoTDB authentication (hidden input supported). | No | `root` |
| `-t` | `--target` | Export target directory. In SCP mode, this is an absolute physical path on the remote server. TsFile and associated Object directories will be exported here. | Yes | - |
| `-db` | `--database` | Database name (optional for Table Model). | No | `.*` |
| `-table` | `--table` | Table name (optional for Table Model). | No | `.*` |
| `-s` | `--start_time` | Start time (ISO8601 format e.g. `2026-01-01T00:00:00` or millisecond timestamp). Only data from this time onwards is exported. | No | - |
| `-e` | `--end_time` | End time (same format as above). Only data before this time is exported. | No | - |
| `-th` | `--target_host` | Remote target host IP. If specified, the script automatically configures Pipe to use SCP for data transfer. | No | - |
| `-tu` | `--target_host_user` | Username for SSH/SCP login to the remote server. | No | - |
| `-tpw` | `--target_host_pw` | Password for remote authentication (hidden input supported). | No | - |
| `-tp` | `--target_host_port` | Remote SSH port. | No | `22` |
| `--rate_limit` | `--rate_limit` | Transfer rate limit (unit: Bytes/s) to prevent excessive bandwidth usage. | No | - |
| `--plugin_jar` | `--plugin_jar` | Path to the Pipe plugin JAR file. | No | - |
| `-help` | `--help` | Show help information. | No | - |


### 3.3 Execution Examples

Example 1: SCP Remote Export (Send Data to Another Server)

```Bash
./tsfile-backup.sh -sql_dialect table -db test_db -t /remote/archive/ -th 192.168.1.100 -tu backup_user -tpw ComplexPass123!
```

Example 2: Remote Object Data Export with Rate Limiting

```Bash
./tsfile-backup.sh -sql_dialect table -t /mnt/backup/ -th 10.0.0.5 -tu iot_admin -tpw Admin@2026 --rate_limit 5242880
```

Example 3: Specify Pipe Plugin JAR Directory

```Bash
./tsfile-backup.sh -sql_dialect table -db test -table .* -tu luoluoyuyu -tpw -t /tmp/backup --plugin_jar /local/lib/tsfile-remote-sink-2.0.8-SNAPSHOT-jar-with-dependencies.jar
```
Loading
Loading