diff --git a/content/operate/rc/databases/rdi/rds-proxy.md b/content/operate/rc/databases/rdi/rds-proxy.md new file mode 100644 index 0000000000..0f286d5138 --- /dev/null +++ b/content/operate/rc/databases/rdi/rds-proxy.md @@ -0,0 +1,112 @@ +--- +Title: RDS Proxy setup for RDI +alwaysopen: false +categories: +- docs +- operate +- rc +description: Set up RDS Proxy for Redis Data Integration (not recommended). +hidden: true +hideListLinks: true +weight: 99 +--- + +{{}} +We do not recommend using RDS Proxy for RDI connections. The [Lambda function approach]({{< relref "/operate/rc/databases/rdi/setup#setup-lambda-function" >}}) provides better failover handling and is the recommended solution for production environments. + +Additionally, RDS Proxy does not work with RDS PostgreSQL and Aurora PostgreSQL because it does not support PostgreSQL logical replication. + +Only use RDS Proxy if you have specific requirements that necessitate it. +{{}} + +## Overview + +RDS Proxy is a fully managed, highly available database proxy for Amazon RDS. While it can be used with RDI, we recommend the Lambda function approach instead for the following reasons: + +- **PostgreSQL incompatibility**: RDS Proxy does not support PostgreSQL logical replication, which is required for CDC (Change Data Capture). +- **Added complexity**: RDS Proxy adds an additional layer between RDI and your database. +- **Lambda provides better failover**: The Lambda function approach handles failover scenarios more efficiently. + +If you still need to use RDS Proxy, follow the instructions below. + +## Prerequisites + +Before setting up RDS Proxy, ensure you have: + +- An RDS or Aurora database (MySQL or SQL Server only) +- AWS Secrets Manager secret containing your database credentials +- AWS KMS encryption key for the secret +- Appropriate IAM permissions + +## Create RDS Proxy + +Follow the AWS documentation to create an RDS Proxy: + +- [Creating an RDS Proxy](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-setup.html) (AWS documentation) +- [How RDS Proxy works](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.howitworks.html) (AWS documentation) +- [RDS Proxy TLS/SSL](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy.howitworks.html#rds-proxy-security.tls) (AWS documentation) + +### IAM permissions + +The Proxy's IAM role must have the following permissions to access the database using the credentials secret and encryption key: + +- `secretsmanager:GetSecretValue` +- `secretsmanager:DescribeSecret` +- `kms:Decrypt` + +Example IAM policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "secretsmanager:GetSecretValue", + "secretsmanager:DescribeSecret" + ], + "Resource": "arn:aws:secretsmanager:region:account-id:secret:secret-name" + }, + { + "Effect": "Allow", + "Action": "kms:Decrypt", + "Resource": "arn:aws:kms:region:account-id:key/key-id" + } + ] +} +``` + +## Get the RDS Proxy IP address + +After creating the RDS Proxy, you need to get its static IP address to use when configuring the Network Load Balancer. + +To get the static IP address of your RDS Proxy, run the following command on an EC2 instance in the same VPC as the Proxy: + +```sh +$ nslookup +``` + +Replace `` with the endpoint of your RDS Proxy. Save this IP address for use in the Network Load Balancer configuration. + +## Configure the Network Load Balancer + +When you [create the Network Load Balancer]({{< relref "/operate/rc/databases/rdi/setup#create-network-load-balancer-rds" >}}), use the RDS Proxy IP address instead of the database IP address: + +1. In **Register targets**, enter the static IP address of your RDS Proxy (obtained in the previous step). +2. Enter the port number where your RDS Proxy is exposed. +3. Select **Include as pending below**. +4. Complete the remaining Network Load Balancer setup as described in the [main setup guide]({{< relref "/operate/rc/databases/rdi/setup#create-network-load-balancer-rds" >}}). + +## Next steps + +After setting up RDS Proxy and the Network Load Balancer: + +1. [Create an endpoint service]({{< relref "/operate/rc/databases/rdi/setup#create-endpoint-service-rds" >}}) through AWS PrivateLink. +2. [Share your source database credentials]({{< relref "/operate/rc/databases/rdi/setup#share-source-database-credentials" >}}) with Redis Cloud. +3. Continue with the [RDI pipeline configuration]({{< relref "/operate/rc/databases/rdi/define" >}}). + +{{}} +When using RDS Proxy, you do not need to set up the Lambda function for failover handling, as the proxy provides a static endpoint. +{{}} + diff --git a/content/operate/rc/databases/rdi/setup.md b/content/operate/rc/databases/rdi/setup.md index 336ebd533a..0f648a949e 100644 --- a/content/operate/rc/databases/rdi/setup.md +++ b/content/operate/rc/databases/rdi/setup.md @@ -130,33 +130,16 @@ For more details on AWS PrivateLink, see [Share your services through AWS Privat To set up PrivateLink for a database hosted on AWS RDS or AWS Aurora: -{{}} -The RDS Proxy does not work with RDS PostgreSQL and Aurora PostgreSQL because it does not support PostgreSQL logical replication. +To connect to your RDS or Aurora database, we recommend using a Lambda function approach. This provides a reliable and secure connection method for all database types. -For PostgreSQL databases, use one of the following alternatives instead: -- **For test environments**: Connect the Network Load Balancer directly to the database IP address (skip the RDS Proxy step). -- **For production environments**: Use the AWS Lambda approach described in [Access Amazon RDS across VPCs using AWS PrivateLink and Network Load Balancer](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/). -{{}} - -1. [Create an RDS Proxy](#create-rds-proxy) that will route requests to your database (MySQL and SQL Server only). -1. [Create a network load balancer](#create-network-load-balancer-rds) that will route incoming requests to the RDS proxy (or directly to the database for PostgreSQL). +1. [Create a network load balancer](#create-network-load-balancer-rds) that will route incoming requests to your database. 1. [Create an endpoint service](#create-endpoint-service-rds) through AWS PrivateLink. - -### Create RDS proxy {#create-rds-proxy} +1. [Set up Lambda function connectivity](#setup-lambda-function) to route requests to your database. {{}} -For RDS PostgreSQL and Aurora PostgreSQL, skip this step and proceed directly to [Create network load balancer](#create-network-load-balancer-rds) and configure it to connect directly to your database IP address. +If you have specific requirements that necessitate using RDS Proxy instead of the recommended Lambda function approach, see the [RDS Proxy setup guide]({{< relref "/operate/rc/databases/rdi/rds-proxy" >}}). Note that RDS Proxy is not recommended and does not work with PostgreSQL. {{}} -In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Database** > **Aurora and RDS**. [Create an RDS proxy](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-creating.html) that can access your database. - -The Proxy's IAM role must have the following permissions to access the database using the credentials secret and encryption key: -- `secretsmanager:GetSecretValue` -- `secretsmanager:DescribeSecret` -- `kms:Decrypt` - -You can set the proxy's IAM role during creation in the **Authentication** section. - ### Create network load balancer {#create-network-load-balancer-rds} In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Compute** > **EC2**. [Create a network load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html#configure-load-balancer) with the following settings: @@ -166,31 +149,25 @@ In the [AWS Management Console](https://console.aws.amazon.com/), use the **Serv - **Load balancer IP address type**: Select **IPv4**. 1. In **Network mapping**, select the VPC and availability zone associated with your source database. 1. In **Security groups**, select the security group associated with your source database, or another security group that allows traffic from PrivateLink and allows traffic to the database. -1. In **Listeners and routing**: +1. In **Listeners and routing**: 1. Select **Create target group** to [create a target group](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-target-group.html) with the following settings: 1. In **Specify group details**: - **Target type**: Select **IP Addresses**. - **Protocol : Port**: Select **TCP**, and then enter the port number where your database is exposed. - The **IP address type** and **VPC** should be selected already and match the VPC you selected earlier. - 1. In **Register targets**, enter the static IP address of your RDS proxy (for MySQL and SQL Server) or your database (for PostgreSQL), enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup. + 1. In **Register targets**, enter the static IP address of your database, enter the port, and select **Include as pending below**. Then, select **Create target group** to create your target group. Return to **Listeners and routing** in the Network Load Balancer setup. - **For MySQL and SQL Server**: To get the static IP address of your RDS Proxy, run the following command on an EC2 instance in the same VPC as the Proxy: - ```sh - $ nslookup - ``` - Replace `` with the endpoint of your RDS proxy. - - **For PostgreSQL**: To get the static IP address of your database, run the following command on an EC2 instance in the same VPC as the database: + To get the static IP address of your database, run the following command on an EC2 instance in the same VPC as the database: ```sh $ nslookup ``` - Replace `` with the endpoint of your RDS or Aurora PostgreSQL database. + Replace `` with the endpoint of your RDS or Aurora database. 1. Set the following **Listener** properties: - **Protocol**: Select **TCP**. - **Port**: Enter your source database's port. - **Default action**: Select the target group you created in the previous step. 1. Review the network load balancer settings, and then select **Create load balancer** to continue. -1. After the network load balancer is active, select **Security**. +1. After the network load balancer is active, select **Security**. If you selected the same security group as your source database, you must not enforce security group rules on PrivateLink traffic. Select **Edit** and then deselect **Enforce inbound rules on PrivateLink traffic**, and then select **Save changes**. @@ -224,6 +201,102 @@ After you create the endpoint service, you need to add Redis Cloud as an Allowed For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). +### Set up Lambda function connectivity {#setup-lambda-function} + +{{}} +Setting up the Lambda function is optional but recommended for production environments. The Lambda function provides automatic failover handling and a more robust connection to your RDS or Aurora database. +{{}} + +The Lambda function monitors RDS failover events and automatically updates the NLB Target Group to point to the new primary instance's IP address. This ensures RDI reconnects automatically after a failover. + +#### Option 1: Use the Redis Terraform module + +Redis provides a ready-to-use Terraform module that automates the Lambda function deployment. This is the recommended approach. + +##### Prerequisites + +- [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) >= 1.5.7 +- [AWS CLI](https://aws.amazon.com/cli/) configured with credentials +- The ARNs from the Network Load Balancer and Endpoint Service you created in the previous steps + +##### Required variables + +Before deploying the Lambda module, gather the following information: + +| Variable | Description | Where to find it | +|----------|-------------|------------------| +| `identifier` | A unique name for the Lambda resources | Choose a descriptive name (e.g., `rdi-failover-handler`) | +| `db_endpoint` | Your RDS cluster or instance endpoint | AWS Console → RDS → Your database → Connectivity | +| `db_port` | Your database port | AWS Console → RDS → Your database → Connectivity (default: `5432` for PostgreSQL, `3306` for MySQL, `1433` for SQL Server) | +| `elb_tg_arn` | The NLB Target Group ARN | AWS Console → EC2 → Target Groups → Your target group | +| `rds_arn` | The RDS cluster or instance ARN | AWS Console → RDS → Your database → Configuration | +| `rds_cluster_identifier` | The RDS cluster identifier | AWS Console → RDS → Your cluster name | + +##### Deploy the Lambda module + +1. Clone the Redis cloud automation repository: + + ```bash + git clone https://github.com/redis/rdi-cloud-automation.git + cd rdi-cloud-automation/modules/aws-rds-lambda + ``` + +1. Create a `terraform.tfvars` file with your configuration: + + ```hcl + identifier = "rdi-failover-handler" + db_endpoint = "your-cluster.cluster-xxxxxxxxx.us-east-1.rds.amazonaws.com" + db_port = 5432 + elb_tg_arn = "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/your-tg/xxxxxxxxx" + rds_arn = "arn:aws:rds:us-east-1:123456789012:cluster:your-cluster" + rds_cluster_identifier = "your-cluster" + ``` + +1. Initialize and apply Terraform: + + ```bash + terraform init + terraform apply + ``` + +##### How the Lambda function works + +The deployed Lambda function: + +1. **Monitors RDS events**: Subscribes to RDS failover events via SNS +1. **Detects failover**: When a failover occurs, RDS triggers an SNS notification +1. **Resolves new IP**: The Lambda function queries DNS to get the new primary's IP address +1. **Updates NLB target**: Automatically updates the NLB Target Group with the new IP + +This process typically completes within 30-60 seconds for Aurora, or 60-120 seconds for standard RDS. + +##### Verify the deployment + +After deployment, verify the Lambda function is configured correctly: + +1. Check the Lambda function in AWS Console → Lambda → Functions +1. Verify the environment variables are set correctly: + - `Cluster_EndPoint`: Your RDS endpoint + - `RDS_Port`: Your database port + - `NLB_TG_ARN`: Your NLB Target Group ARN +1. Check the SNS subscription in AWS Console → SNS → Subscriptions + +#### Option 2: Full infrastructure deployment + +For new deployments, Redis provides a complete Terraform example that deploys the entire infrastructure including the RDS database, NLB, PrivateLink, and Lambda function. + +See the [AWS RDS PrivateLink Failover Example](https://github.com/redis/rdi-cloud-automation/tree/main/examples/aws-rds-privatelink-failover) for: + +- Multi-engine support (PostgreSQL, MySQL, SQL Server) +- Automatic CDC user creation +- Complete VPC and networking setup +- Lambda-based failover handling + +#### Option 3: Manual Lambda setup + +For custom implementations, refer to the AWS documentation: +[Access Amazon RDS across VPCs using AWS PrivateLink and Network Load Balancer](https://aws.amazon.com/blogs/database/access-amazon-rds-across-vpcs-using-aws-privatelink-and-network-load-balancer/) + {{< /multitabs >}} ## Share source database credentials