Redis – Automate Redis Enterprise Cloud Deployment Using Terraform

This blog describes the steps to create and deploy Redis Enterprise Cloud Subscription and Databases as code on GCP.

Install Terraform in a centos VM. If you want to test the connection with the Redis private endpoint in the same VM after provisioning the databases, then use a VM  that is part of the VPC network that you want to peer with Redis Enterprise later.

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

# Verify version
terraform --version

You may also run the terraform scripts in the cloud shell.

Here are the Terraform scripts that you need to upload to your server.

provider.tf

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.21.0"
    }
   rediscloud = {
     source = "RedisLabs/rediscloud"
     version = "0.2.9"
   }
 }
}

provider "rediscloud" {
  # Configuration options
  api_key = "${var.api_key}"
  secret_key = "${var.secret_key}"
}

variables.tf

# You may get the values of api_key and secret_key (Also known as API user keys) by following the steps from this site https://docs.redis.com/latest/rc/api/get-started/manage-api-keys/#:~:text=Sign%20in%20to%20your%20Redis,select%20the%20API%20Keys%20tab
variable "api_key" {
  type = string
  default = "<my_api_key>"
}


variable "secret_key" {
  type = string
  default = "<my_secret_key>"
}


# Project name as Prefix of the Redis instance
# Example: "${projname}-gacc-cache-redis"
variable "projname" {
  type = string
  default = "test-project"
}

variable "region" {
  type = string
  default = "us-west1"
}

variable "preferred_zones" {
  type = string
  default = "us-west1-a"
}

variable "multiple_zones" {
  type = bool
  default = false
}

variable "cidr" {
  type = string
  default = "192.168.1.0/24"
}

main.tf

# Generates a random password for the database
resource "random_password" "passwords" {
 count = 3   # this number should be equal to the number of Redis database to be created
 length = 20
 upper = true
 lower = true
 numeric = true
 special = false
}
 
resource "rediscloud_subscription" "MY-REDISCLOUD-SUBSCRIPTION" {
 name = "My-Redis-Subscription"
 memory_storage = "ram"
 
 cloud_provider {
   # Running in GCP on Redis resources
   provider = "GCP"
   region {
     region = "${var.region}"
     networking_deployment_cidr = "${var.cidr}"
     preferred_availability_zones = ["${var.preferred_zones}"]
     multiple_availability_zones  = "${var.multiple_zones}"
   }
 }
 
  database {
   name = "${var.projname}-redis-database1"
   protocol = "redis"
   memory_limit_in_gb = 6
   replication = true
   data_persistence = "none"
   throughput_measurement_by = "operations-per-second"
   throughput_measurement_value = 10000
   password = random_password.passwords[0].result
   alert {
      name = "dataset-size"
      value = 80
    }
   alert {
      name = "throughput-higher-than"
      value = 10000
    }
 }
  
  database {
   name = "${var.projname}-redis-database2"
   protocol = "redis"
   memory_limit_in_gb = 6
   replication = true
   data_persistence = "none"
   throughput_measurement_by = "operations-per-second"
   throughput_measurement_value = 10000
   password = random_password.passwords[1].result
   alert {
      name = "dataset-size"
      value = 80
    }
   alert {
      name = "throughput-higher-than"
      value = 10000
    }
 }

 database {
   name = "${var.projname}-redis-database3"
   protocol = "redis"
   memory_limit_in_gb = 13
   replication = true
   data_persistence = "aof-every-1-second"
   throughput_measurement_by = "number-of-shards"
   throughput_measurement_value = 4
   password = random_password.passwords[2].result
   alert {
      name = "dataset-size"
      value = 80
    }
   alert {
      name = "throughput-higher-than"
      value = 10000
    }
 }
 
}

outputs.tf

# Terraform output values

output "database_private_endpoints" {
    description = "Output private endpoints"
    sensitive = true
    value = {
    for database in rediscloud_subscription.MY-REDISCLOUD-SUBSCRIPTION.database:
    database.name => database.private_endpoint}
}

output "database_passwords" {
    description = "Output passwords"
    sensitive = true
    value = {
    for database in rediscloud_subscription.MY-REDISCLOUD-SUBSCRIPTION.database:
    database.name => database.password}
}



Few Notes

Update Redis Enterprise Cloud API Keys in variables.tf

You can get the values of api_key and secret_key (Also known as API user keys) by following the steps from this site.

The Account key identifies the account associated with the Redis Enterprise Cloud subscription.

The User key (secret_key) identifies the user and (optionally) the context of a request. Generated by account owners.

In main.tf file, add database blocks with configuration based on application requirements.

Example:

Also in the main.tf, edit the count parameter under random_password resource. The number should be equal to the number  of Redis instance to be created.

Executing Terraform Commands To Create Redis Enterprise Databases

Execute terraform init command to initialize a working directory containing Terraform configuration files.

terraform init

Run terraform plan command to create an execution plan, which lets you preview the changes that Terraform plans to make to your infrastructure.

terraform plan

If the plan is okay, then execute terraform apply.

terraform apply

Outputting sensitive data

The database_private_endpoints and database_passwords are sensitive data. So, the contents will not be instantly outputted after executing terraform apply.

terraform output -json database_private_endpoints   terraform output -json database_passwords  

To output the contents, run the commands below.

terraform output -json database_private_endpoints

terraform output -json database_passwords

Verifying Redis databases from the console.

Bonus Section: VPC Peering

Google Cloud VPC Network Peering allows internal IP address connectivity across two Virtual Private Cloud (VPC) networks regardless of whether they belong to the same project or the same organization.

In your Redis subscription, Go to Connectivty tab, then click + Add peering.

Provide Project ID and Network Name. Then Copy the Google cloud command.

Click Initiate peering.

In GCP console, open cloud a cloud shell, then execute the command you just copied.

Configure your GCP project and region.

gcloud config set core/project 
gcloud config set compute/zone zone

Execute the command to accept VPC Peering. This is the Google cloud command you copied from Redis Enterprise portal.

Example:

gcloud compute networks peerings create rl-c19541-us-west1-1-rlrcp --project  --network --peer-network=c19541-us-west1-1-rlrcp --peer-project=g0c4be81afe238245-tp

Check the peering status in Redis Enterprise portal. It may take several minutes to become active.

Once the peering is successful, the status will become green.

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

Redis – Predixy Proxy Auto Setup

Here is a shell script I wrote to automate the setup of Predixy Proxy in Centos 7.

This script will work if you have a single endpoint (i.e. AWS Elasticache cluster configuration endpoint or a standalone Elasticache endpoint).

#!/bin/bash

# Created By  : Enrique Valencia 
# Script Name : predixy_auto_setup.sh
# Description : Predixy Automatic Setup

################################################################################################################
# Usage	          : ./auto_predixy_setup.sh <configuration_end_point>"
# Example         : ./auto_predixy_setup.sh dba-redis-test01.bbssxp.clustercfg.apse1.cache.amazonaws.com:6379"
# To Check Status : systemctl status predixy 
# To Start	      : systemctl start predixy
# To Stop		  : systemctl stop predixy
# To Restart	  : systemctl restart predixy
################################################################################################################

################################################################################################################
##### Validate Usage
################################################################################################################

if [ $# -ne 1 ]
then
  echo ""
  echo "************************************************************************************************************"
  echo "Usage:   ./predixy_auto_setup.sh <configuration_end_point>"
  echo ""
  echo "Example: ./predixy_auto_setup.sh dba-redis-test01.bbssxp.clustercfg.apse1.cache.amazonaws.com:6379"
  echo "************************************************************************************************************"
  echo ""
  exit 1
fi

################################################################################################################
##### Variable
################################################################################################################

ENDPOINT=$1

################################################################################################################
##### Install Predixy
################################################################################################################

InstallPredixy(){
	mkdir /predixy_data
	cd /predixy_data
	sudo yum install git -y
	sudo yum install libstdc++-static -y
	yum -y install gcc
	yum -y install gcc-c++ 
	sudo git clone https://github.com/joyieldInc/predixy.git
	cd predixy
	sudo make
	mkdir -p /predixy_data/predixy/log
}

################################################################################################################
##### Cluster
################################################################################################################

cluster(){

cat >> /predixy_data/predixy/conf/cluster.conf << EOF
ClusterServerPool {
	MasterReadPriority 100  # 100 
	# Password sjwkk123456 # redis 
	StaticSlaveReadPriority 0  #  redis slave redis 0
	DynamicSlaveReadPriority 0 #  redis sentinel 0
	RefreshInterval 1 # predixy redis sentinel 1 
	ServerTimeout 1 #  predixy/redis predixy redis blpop 0 redis 0
	ServerFailureLimit 10 #  redis 10
	ServerRetryTimeout 1 #  redis 1 
	KeepAlive 120 #predixy redis tcp keepalive 0 0
	Servers {
	##  
		+ $ENDPOINT
	}
}
EOF

	sed -i 's/Auth "#a complex password#" {/Auth "123456" {/' auth.conf
	sed -i 's/WorkerThreads 1/WorkerThreads 4/' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/# Log .\/predixy.log/Log \/predixy_data\/predixy\/log\/predixy.log/' /predixy_data/predixy/conf/predixy.conf
	sed -i '/# LogRotate 1d 2G/d' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/# LogRotate 1d/LogRotate 1d/' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/Include try.conf/# Include try.conf/' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/# Include cluster.conf/Include cluster.conf/' /predixy_data/predixy/conf/predixy.conf
}


################################################################################################################
##### Standalone
################################################################################################################

standalone(){
	
cat >> /predixy_data/predixy/conf/standalone.conf << EOF

StandaloneServerPool {
	RefreshMethod fixed
	Group shard001 {
		+ $ENDPOINT
	}
}
EOF

	sed -i 's/Auth "#a complex password#" {/Auth "123456" {/' auth.conf
	sed -i 's/WorkerThreads 1/WorkerThreads 4/' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/# Log .\/predixy.log/Log \/predixy_data\/predixy\/log\/predixy.log/' /predixy_data/predixy/conf/predixy.conf
	sed -i '/# LogRotate 1d 2G/d' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/# LogRotate 1d/LogRotate 1d/' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/Include try.conf/# Include try.conf/' /predixy_data/predixy/conf/predixy.conf
	sed -i 's/# Include sentinel.conf/Include standalone.conf/' /predixy_data/predixy/conf/predixy.conf
}

################################################################################################################
##### Setup Predixy as a Systemd Service
################################################################################################################

systemd(){
	echo "/predixy_data/predixy/src/predixy /predixy_data/predixy/conf/predixy.conf" > /usr/sbin/predixy_start.sh

	cat > /etc/systemd/system/predixy.service <<-EOF
	[Unit]
	Description=Predixy

	[Service]
	ExecStart=/bin/bash /usr/sbin/predixy_start.sh


	[Install]
	WantedBy=multi-user.target
	EOF

	sudo chmod 640 /etc/systemd/system/predixy.service
	sudo systemctl daemon-reload
	sudo systemctl enable predixy
}


################################################################################################################
##### Check Predixy Status
################################################################################################################

CheckStatus() {
	if [ `ps -ef | grep predixy.conf | grep -v grep | wc -l` -gt 0 ]; then
		echo ""
		echo "************************************************************************"
		echo "Predixy Setup Completed."
		echo "************************************************************************"
		echo ""
	else
		echo ""
		echo "************************************************************************"
		echo "Predixy Proxy is unable to start. Please check."
		echo "************************************************************************"
		echo ""
		exit 1
	fi
}

################################################################################################################
##### MAIN
################################################################################################################

echo -ne "
Menu: 
=====

1) cluster
2) standalone
3) Exit

Please choose: "

read -r choice

case $choice in

1)
	echo ""
	echo "************************************************************************"
	echo "You chose 1 - Cluster. Starting Predixy Setup."
	echo "************************************************************************"
	echo ""
	sleep 3
	InstallPredixy
	cluster
	systemd
	systemctl start predixy
	sleep 3
	CheckStatus
	;;

2)
	echo ""
	echo "************************************************************************"
	echo "You chose 2 - Standalone. Starting Predixy Setup."
	echo "************************************************************************"
	echo ""
	sleep 3
	InstallPredixy
	standalone
	systemd
	systemctl start predixy
	sleep 3
	CheckStatus
	;;
	
3)
	echo ""
	echo "*********"
	echo "再见"
	echo "*********"
	echo ""
	exit 1
	;;
*)
	echo ""
	echo "************************************************************************"
	echo "Invalid Option. Please try again."
	echo "************************************************************************"
	echo ""
	exit 1
esac

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

Redis – Monitor Redis Using Grafana

In this blog post, I will outline the steps to set up Grafana to monitor your Redis database.

Install Grafana

Go to Grafana website download page.

Example: For Red Hat, CentOS, RHEL, and Fedora(64 Bit).

Execute the following commands to make Grafana start automatically when the server is booted up.

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server.service

Install Redis Plugin

The Redis Data Source for Grafana is a plugin that allows users to connect to any Redis database On-Premises and in the Cloud. It provides out-of-the-box predefined dashboards and lets you build customized dashboards to monitor Redis and application data.

grafana-cli plugins install redis-datasource

Start Grafana

Start Grafana.

sudo systemctl start grafana-server.service

Check Grafana Status.

sudo systemctl status grafana-server.service

Access Grafana Monitor Dashboard

Paste the server’s IP, plus port 3000 on your web browser to access Grafana Dashboar<Ipd.

<ServerIPaddress>:3000

The initial username and password is admin/admin.

Configure Data source

Go to Configuration > Data sources.

Add data source.

Search for “Redis“, then select the data source below.

Enter a name and address in the following format.

redis://your-redis-endpoint.ng.0001.sae1.cache.amazonaws.com:6379

Click Save and test and verify that the data source is working as expected.

Import Dashboard

This step will be performed one time only.

Go to Dashboards. Then click Import for both Redis and Redis Streaming.

View Dashboard

Go to Dashboard > Manage.

Choose Redis or Redis Streaming.

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter