Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
12 changes: 6 additions & 6 deletions docs/buiding.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
* gcc/g++ 4.9
* cmake
* 安装gflags:<br />
gflags是google开源的一套命令行参数解析工具,支持从环境变量和配置文件读取参数:
* Install gflags:<br />
gflags is an open-source command-line flag parsing tool from Google that supports reading parameters from environment variables and configuration files:
```shell
git clone https://github.com/gflags/gflags.git
cd gflags
Expand All @@ -12,18 +12,18 @@
sudo ln -s /usr/local/lib/libgflags.so.2.2 /lib64
```
### CentOS
- 执行以下命令安装依赖:
- Execute the following command to install dependencies:
```shell
sudo yum install -y make snappy snappy-devel zlib zlib-devel bzip2 bzip2-devel lz4-devel libasan openssl-devel cmake3 mysql-devel mxml-devel
```
### Ubuntu
- 执行以下命令安装依赖:
- Execute the following command to install dependencies:
```shell
sudo apt-get install make snappy libsnappy-dev zlib1g zlib1g-dev bzip2 liblz4-dev libasan0 openssl libmxml-dev
sudo apt-get install make snappy libsnappy-dev zlib1g zlib1g-dev bzip2 liblz4-dev libasan0 openssl libmxml-dev
```
* `mkdir build`
* `cmake ../`
* 默认编译配置不包含测试用例,如需要编译test文件夹下的测试用例,需要再cmake时添加参数 -DCMAKE_TEST_OPTION=ON
* The default compilation configuration does not include test cases. If you need to compile test cases in the test folder, you need to add the parameter -DCMAKE_TEST_OPTION=ON when running cmake
```shell
cmake -DCMAKE_TEST_OPTION=ON ../
```
Expand Down
78 changes: 39 additions & 39 deletions docs/configure.md
Original file line number Diff line number Diff line change
@@ -1,52 +1,52 @@

## 配置文件
配置文件目录:conf/<br/>
- AGENT服务的配置文件是:
* agent.xml 指定后端dtc server的主、备服务的地址信息和权重;
- DTC服务的配置文件是:
* dtc.yaml dtc模块的配置文件,包括基础配置和表结构信息。
## Configuration Files
Configuration file directory: conf/<br/>
- AGENT service configuration file:
* agent.xml Specifies the address information and weight of primary and backup services of backend DTC servers;
- DTC service configuration file:
* dtc.yaml Configuration file for the DTC module, including basic configuration and table structure information.

下面就各配置文件具体配置选项做介绍:
The following sections introduce specific configuration options for each configuration file:
### agent.xml
agent配置文件位置为当前项目的conf/agent.xml,主要配置以下字段:
- ALL.BUSINESS_MODULE.MODULE.ListenOn 监听IP和端口号
- ALL.BUSINESS_MODULE.MODULE.Preconnect 预连接状态
* True 开启预连接状态,当Agent启动时会自动建立对后端所有DTC Server结点的连接通道。
* False 关闭预连接状态,Agent启动时并不会针对后端Server建立连接,待第一个请求过来时才会创建连接。
- ALL.BUSINESS_MODULE.MODULE.Timeout 单位:毫秒。Agent响应超时时间,超过此时间则直接返回。
- ALL.BUSINESS_MODULE.CACHESHARDING 此节点为Agent的分片配置信息。根据分片服务器的数量,可配置多个CACHESHARDING节点。
- ALL.BUSINESS_MODULE.CACHESHARDING.INSTANCE 分片节点下的具体服务器实例信息:
* Role 实例角色配置:
* master 主服务器角色设置,至少要配置一个。
* replica [可选] 备服务器角色设置,可不配置或配置多个备机。
* Enable 角色开关:True为开启,False为关闭。
* Addr 服务器监听地址和权重。例如 0.0.0.0:20015:1, 分别代表了监听的IP、监听的端口和权重值。权重值在INSTANCE实例下的所有服务器的请求权重,Agent会根据此值设置的大小,来分发不同的流量到后端DTC Server上。默认值为1。
The agent configuration file is located at conf/agent.xml in the current project. The main fields to configure are:
- ALL.BUSINESS_MODULE.MODULE.ListenOn Listen IP address and port number
- ALL.BUSINESS_MODULE.MODULE.Preconnect Pre-connection status
* True Enable pre-connection status. When Agent starts, it will automatically establish connection channels to all backend DTC Server nodes.
* False Disable pre-connection status. Agent will not establish connections to backend servers when starting, and will only create connections when the first request arrives.
- ALL.BUSINESS_MODULE.MODULE.Timeout Unit: milliseconds. Agent response timeout. If this time is exceeded, it will return directly.
- ALL.BUSINESS_MODULE.CACHESHARDING This node contains Agent's sharding configuration information. Multiple CACHESHARDING nodes can be configured based on the number of sharding servers.
- ALL.BUSINESS_MODULE.CACHESHARDING.INSTANCE Specific server instance information under the sharding node:
* Role Instance role configuration:
* master Primary server role setting, at least one must be configured.
* replica [Optional] Backup server role setting, can be not configured or multiple backup servers can be configured.
* Enable Role switch: True to enable, False to disable.
* Addr Server listen address and weight. For example, 0.0.0.0:20015:1 represents the listen IP, listen port, and weight value respectively. The weight value is the request weight among all servers under the INSTANCE. Agent will distribute different traffic to backend DTC Servers based on the size of this value. Default value is 1.


### dtc.yaml
#### primary 主库,提供dtc核心功能
* table dtc的表名
* layered.rule 分层存储的命中规则,匹配到此规则就进入到cache层。
#### primary Primary database, provides DTC core functionality
* table DTC table name
* layered.rule Hit rules for layered storage. Requests matching this rule will enter the cache layer.
* cache/hot/full
dtc根据不同的功能需要配置不同的模块,共分为三层:<br />
cache层:提供缓存功能,只设置cache,不设置hot则表示CACHE ONLY模式,只缓存数据不存储数据到数据库。<br />
hot层:在cache的基础上提供数据存储功能,在datasource模式下需要配置此层来设置具体的数据源信息。在分层存储时,设置此模块能够配置热点数据的数据源<br />
full层:在分层存储的功能时,需要配置此层,全量数据将存储在此数据源中。
* logic/real
该字段分别设置逻辑库表和真实库表。逻辑库表用于在dtc中显示和使用库、表。真实库表信息是真实的数据源信息。
DTC needs to configure different modules based on different functional requirements, divided into three layers:<br />
cache layer: Provides caching functionality. If only cache is set without hot, it represents CACHE ONLY mode, which only caches data without storing data to the database.<br />
hot layer: Provides data storage functionality on top of cache. In datasource mode, this layer needs to be configured to set specific data source information. In layered storage, setting this module allows configuration of data sources for hot data<br />
full layer: When using layered storage functionality, this layer needs to be configured. Full data will be stored in this data source.
* logic/real:
These fields set logical database tables and real database tables respectively. Logical database tables are used for display and usage of databases and tables in DTC. Real database table information is the actual data source information.
* sharding:
该字段用户设置分库分表的信息,key字段用于设置依照此字段进行分片。table字段用于设置分表的信息,例如分表名为opensource_0/opensource_1.....opensouce_9,则只需要设置为{prefix: [*table, _], start: 0, last: 9}
This field is used to set database and table sharding information. The key field is used to set which field to use for sharding. The table field is used to set table sharding information. For example, if table names are opensource_0/opensource_1.....opensource_9, you only need to set it as {prefix: [*table, _], start: 0, last: 9}

#### table
cache配置文件位置为当前项目的conf/table.yaml,主要配置以下字段:
Cache configuration file is located at conf/table.yaml in the current project. The main fields to configure are:
* TABLE_CONF.table_name
* TABLE_CONF.field_count
* TABLE_CONF.key_count 指定
* FIELD*ID* *ID*为当前字段的编号,从1开始使用,可以根据场景需要配置个数。
* FIELD*ID*.field_name 字段名
* FIELD*ID*.field_type 字段类型:1.整数型 2.无符号整数形 3.浮点型 4.字符串(大小写不明感) 5.字符串(大小写敏感)
* FIELD*ID*.field_size 字段长度。定义了该字段的大小。例如int型可配置为4字节,long型可配置为8字节,字符串类型可根据具体使用场景配置长度,但最大长度不得超过64KB。
* FIELD*ID*.unique [可选]默认值0,可配置0或1。当为0时表示该字段的值不唯一,1时表示该字段值唯一。
* TABLE_CONF.key_count Specify
* FIELD*ID* *ID* is the field number, starting from 1. The number can be configured based on scenario requirements.
* FIELD*ID*.field_name Field name
* FIELD*ID*.field_type Field type: 1.Integer 2.Unsigned integer 3.Float 4.String (case insensitive) 5.String (case sensitive)
* FIELD*ID*.field_size Field length. Defines the size of the field. For example, int type can be configured as 4 bytes, long type can be configured as 8 bytes. String type can be configured with length based on specific usage scenarios, but maximum length cannot exceed 64KB.
* FIELD*ID*.unique [Optional] Default value 0, can be configured as 0 or 1. When 0, it means the field value is not unique; when 1, it means the field value is unique.

#### extension 扩展库,提供多租户功能
在dtc的基础缓存和数据代理功能之外,还提供了扩展库,通过配置此模块能够在数据库中进行复杂查询和分库分表功能。
#### extension Extension library, provides multi-tenant functionality
In addition to DTC's basic caching and data proxy functionality, an extension library is also provided. By configuring this module, complex queries and database and table sharding functionality can be performed in the database.
92 changes: 46 additions & 46 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
@@ -1,92 +1,92 @@
## 目录结构
Bin文件目录:/usr/local/dtc<br/>
配置文件目录:/usr/local/dtc/conf/<br/>
日志目录:/usr/local/dtc/log/<br/>
统计数据目录:/usr/local/dtc/stat/<br/>
binlog数据目录:/usr/local/dtc/log/<br/>
## Directory Structure
Binary file directory: /usr/local/dtc<br/>
Configuration file directory: /usr/local/dtc/conf/<br/>
Log directory: /usr/local/dtc/log/<br/>
Statistics data directory: /usr/local/dtc/stat/<br/>
Binlog data directory: /usr/local/dtc/log/<br/>

## DTC模式
## DTC Modes

DTC分为两种数据模式:CACHE ONLY模式和Datasource模式。
- CACHE ONLY模式的DTC当做缓存使用,不连接数据库。
- Datasource模式需要连接数据库,目前支持连接Mysql。此模式下DTC作为数据库的缓存代理,提供分库分表,将热点数据缓存在DTC中。
DTC has two data modes: CACHE ONLY mode and Datasource mode.
- CACHE ONLY mode uses DTC as a cache without connecting to a database.
- Datasource mode requires a database connection and currently supports MySQL. In this mode, DTC acts as a database cache proxy, providing database and table sharding, and caching hot data in DTC.

Demo使用的是CACHE ONLY模式演示。
The demo uses CACHE ONLY mode for demonstration.

## 表结构
表结构文件在conf/dtc.yaml中。<br/>
demo中定义的表名为dtc_opensource, <br/>
结构为:
| 字段名 | 类型 | 长度 |
| ------ | ---------------------- | ------- |
| uid | 整型 | 4 Byte |
| name | 字符串(大小写不敏感 | 50 Byte |
| city | 字符串(大小写不敏感) | 50 Byte |
| sex | 整型 | 4 Byte |
| age | 整型 | 4 Byte |
## Table Structure
The table structure file is in conf/dtc.yaml.<br/>
The table name defined in the demo is dtc_opensource, <br/>
Structure:
| Field Name | Type | Length |
| ---------- | ------------------------ | ------- |
| uid | Integer | 4 Byte |
| name | String (case insensitive)| 50 Byte |
| city | String (case insensitive)| 50 Byte |
| sex | Integer | 4 Byte |
| age | Integer | 4 Byte |

## 启动DTC Server端
为了省去配置环境的麻烦,Demo中提供docker镜像,直接运行即可启动服务端:<br/>
## Starting DTC Server
To save the trouble of environment configuration, the demo provides a docker image that can be run directly to start the server:<br/>
```shell
docker pull dtc8/dtc:latest
docker run --rm --name dtc -p <MY_LISTENER_PORT>:12001 -v <MY_HOST_CONF_DIR>:/usr/local/dtc/conf/ -e DTC_BIN=dtc -e DTC_ARGV=-ayc dtc8/dtc
```

## 运行Client测试示例
当前已经支持mysql 5.X和8.X的客户端访问dtc进行SQL操作。当运行上面docker之后,可以运行以下SQL语句:
* 登录:
## Running Client Test Examples
Currently supports MySQL 5.X and 8.X client access to DTC for SQL operations. After running the above docker, you can run the following SQL statements:
* Login:
```
mysql -h127.0.0.1 -P12001 -uroot -proot
```
* 查看数据库列表
* View database list
```
show databases;
```
* 切换数据库
* Switch database
```
use layer2;
```
* 查看表列表
* View table list
```
show tables;
```
* 插入
* Insert
```
insert into opensource(uid, name) values(1, 'Jack') where uid = 1;
```
* 更新
* Update
```
update opensource set name = 'Lee' where uid = 1;
```
* 查询
* Query
```
select uid, name from opensource where uid = 1;
```
* 删除
* Delete
```
delete from opensource where uid = 1;
```

你也可以根据需要尝试修改示例中的代码或配置,进行更多的体验。配置文件请参考[Configure](./configure.md)
You can also try modifying the code or configuration in the examples as needed for more experimentation. For configuration files, please refer to [Configure](./configure.md).

源码编译请参照[buiding](./building.md)
For source code compilation, please refer to [Building](./building.md).

## 直接部署
* 创建文件夹
## Direct Deployment
* Create directories
```
mkdir -p basepath
mkdir -p /usr/local/dtc/data
mkdir -p /usr/local/dtc/stat
mkdir -p /usr/local/dtc/log
mkdir -p /usr/local/dtc/conf
```
* 将bin文件拷贝到/usr/local/dtc文件夹,并赋执行权限
* Copy binary files to /usr/local/dtc directory and grant execution permissions
```
cp * /usr/local/dtc/
chmod +x *
```
* 运行dtc
执行./dtc -h获取详细信息,可以依据需要分别运行不同的组件。注:需要在root权限下运行core模块。
* Run dtc
Execute ./dtc -h to get detailed information. You can run different components separately as needed. Note: The core module needs to run with root privileges.
```
-h, --help : this help
-v, --version : show version and exit
Expand All @@ -96,20 +96,20 @@ chmod +x *
-s, --sharding : load sharding module
-r, --recovery mode : auto restart when crashed
```
例如:
1.只运行core,不使用agent代理:
Examples:
1. Run only core, without using agent proxy:
```
./dtc -c
```
2.运行agent代理的dtc模式:
2. Run DTC mode with agent proxy:
```
./dtc -ac
```
3.运行分层存储
3. Run tiered storage
```
./dtc -ayc
```
4.运行带有分库分表的分层存储
4. Run tiered storage with database and table sharding
```
./dtc -aycs
```
2 changes: 1 addition & 1 deletion src/agent-watchdog/agent_entry.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/agent_entry.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/base.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/base.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/cold_wipe_entry.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/cold_wipe_entry.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/core_entry.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/core_entry.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
4 changes: 2 additions & 2 deletions src/agent-watchdog/daemons.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -68,7 +68,7 @@ static void sighdlr(int signo)

WatchDog::WatchDog()
{
/* 立马注册进程退出处理函数,解决启动时创建太多进程导致部分进程退出没有收到信号linjinming 2014-06-14*/
/* immediately register process exit handler to solve the issue of some processes not receiving signals when too many processes are created during startup linjinming 2014-06-14*/
notifier = new WatchDogPipe;
//signal(SIGCHLD, sighdlr);

Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/daemons.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
2 changes: 1 addition & 1 deletion src/agent-watchdog/log.cc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down
14 changes: 7 additions & 7 deletions src/agent-watchdog/log.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
* Copyright [2021] JD.com, Inc.
* Copyright JD.com, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -69,12 +69,12 @@ static Logger logger = Logger::getRoot();
#define LOG4CPLUS_CONF_FILE "../conf/log4cplus-wd.conf"

/*********************************************
**TRACE:调试应用的详细步骤
**DEBUG:算法关键部分的相关信息
**INFO: 应用的内部状态信息
**WARN: 可以避免的内部状态信息
**ERROR:发生了错误,且应用程序知道如何处理它
**FATAL:发生了不可逆转的错误,程序无法继续运行
**TRACE: Detailed steps for debugging applications
**DEBUG: Information about key parts of algorithms
**INFO: Internal status information of applications
**WARN: Avoidable internal status information
**ERROR: An error occurred and the application knows how to handle it
**FATAL: An irreversible error occurred, the program cannot continue running
**********************************************/
#define log4cplus_trace(fmt, args...) \
write_log(logger, 1, __FILE__, __FUNCTION__, __LINE__, fmt, ##args)
Expand Down
Loading
Loading