Migrate Redis to OCI Cache on Redis

You can migrate your Redis DB to OCI Cache on Redis using the following steps.

You need to use a data dump tools such as pyredis-dump for getting serialised dump from your Redis DB. 

This tool can be used to take a serialised dump from any Redis node running in any cloud environments like AWS ElastiCache. So far, we are testing only single master Redis cluster.

 

[root@jay ~]# git clone https://github.com/tkote/pyredis-dump.git

Cloning into 'pyredis-dump'...

remote: Enumerating objects: 51, done.

remote: Counting objects: 100% (40/40), done.

remote: Compressing objects: 100% (23/23), done.

remote: Total 51 (delta 22), reused 28 (delta 17), pack-reused 11

Receiving objects: 100% (51/51), 15.89 KiB | 15.89 MiB/s, done.

Resolving deltas: 100% (22/22), done.

[root@jay ~]#

 

Using pyredis-dump, you can create a backup of your dataset from OCI Redis as well for safe keeping of your data.

 

You can use the backup you have taken to move the Redis node to a different Redis node in another region/VCN as per your requirement.

 

The pyredis-dump has the following options:

[root@jay pyredis-dump]# python3 pyredis-dump.py -h

Usage: pyredis-dump.py [options] dump|restore|dblist

 

Options:

  -h, --help            show this help message and exit

  -H HOST, --host=HOST  connect to HOST (default localhost)

  -P PORT, --port=PORT  connect to PORT (default 6379)

  -s SOCKET, --socket=SOCKET

                        connect to SOCKET

  -d DB, --db=DB        database

  -w PASSWORD, --password=PASSWORD

                        connect with PASSWORD

  -p PATTERN, --pattern=PATTERN

                        pattern

  -o OUTFILE, --outfile=OUTFILE

                        write to OUTFILE

  -i INFILE, --infile=INFILE

                        read from INFILE

  -t                    use ttl when in restore mode

  -b BULK, --bulk=BULK  dump/restore bulk size

  -S, --ssl             use tls connection

  -k, --watch           watch key value change while dumping

[root@jay pyredis-dump]#

 

Take a serialised backup of the Redis.

Syntax:

# python3 pyredis-dump.py <Options> -H <Hostname/IP> dump -o /path/to/backupfile

 

[root@jay pyredis-dump]# python3 pyredis-dump.py -S -H amaaaaaa2ne4d2aaxhlt47oluwtlrvualroXXocvb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com dump -o /opt/redisdump/rdump1

dumping to '/opt/redisdump/rdump1'

connecting to {'db': 0, 'host': amaaaaaa2ne4d2aaxhlt47oluwtlrvualroXXocvb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com', 'port': 6379, 'ssl': True}

5 keys, bulk_size: 1000, watch_keys: False

dumped 5 records

elapsed time: 0.029 seconds

[root@jay pyredis-dump]#

 

Here we have taken a backup of the OCI Redis cluster. However, you may run the same command to take backup of any Redis clusters in any environments.

 

In the above command, we used the option  ‘-S’ as we are connecting to an OCI Redis cluster and it mandates secure TLS connection. For your Redis environment you might not need to use this option.

 

The pyredis-dump command creates a dump file in the path you specified. The time it takes to create a dump depends on your data size.

 

Once you get the dump file, you can move it to the location or an OCI instance from which you can access your new OCI Redis cluster.

 

 

Restore to OCI Redis Cache

You may use the following command to restore the data to the new cluster.

 

Syntax:

# python3 pyredis-dump.py <Options> -H <Hostname/IP> restore -i /path/of/backup

 

[root@jay pyredis-dump]# python3 pyredis-dump.py -S -H amaaaaaa2ne4d2aaxhlt4XXXrvualrociwb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com restore -i /opt/redisdump/rdump1

restore from '/opt/redisdump/rdump1'

connecting to {'db': 0, 'host': amaaaaaa2ne4d2aaxhlt4XXXrvualrociwb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com', 'port': 6379, 'ssl': True}

restored 5 records

elapsed time: 0.022 seconds

[root@jay pyredis-dump]#

 

 

 

Some catches:

If you get the following error when using pyredis-dump, install redis for python.

 

[root@jay pyredis-dump]# python3 pyredis-dump.py -h

Traceback (most recent call last):

  File "pyredis-dump.py", line 7, in <module>

    from redis import StrictRedis as Redis

ModuleNotFoundError: No module named 'redis'

[root@jay pyredis-dump]#

 

Install redis in python using pip3.

 

[root@jay pyredis-dump]# pip3 install redis

WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.

Collecting redis

  Downloading https://files.pythonhosted.org/packages/d6/f6/19237b28c632935c7359bddf703395ba13bbd134fc5e2eb297c4c120398c/redis-4.3.6-py3-none-any.whl (248kB)

    100% |████████████████████████████████| 256kB 5.7MB/s

Collecting importlib-metadata>=1.0; python_version < "3.8" (from redis)

  Downloading https://files.pythonhosted.org/packages/a0/a1/b153a0a4caf7a7e3f15c2cd56c7702e2cf3d89b1b359d1f1c5e59d68f4ce/importlib_metadata-4.8.3-py3-none-any.whl

Collecting async-timeout>=4.0.2 (from redis)

  Downloading https://files.pythonhosted.org/packages/d6/c1/8991e7c5385b897b8c020cdaad718c5b087a6626d1d11a23e1ea87e325a7/async_timeout-4.0.2-py3-none-any.whl

Collecting packaging>=20.4 (from redis)

  Downloading https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl (40kB)

    100% |████████████████████████████████| 40kB 11.4MB/s

Collecting typing-extensions; python_version < "3.8" (from redis)

  Downloading https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl

Collecting zipp>=0.5 (from importlib-metadata>=1.0; python_version < "3.8"->redis)

  Downloading https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/lib/python3.6/site-packages (from packaging>=20.4->redis)

Installing collected packages: zipp, typing-extensions, importlib-metadata, async-timeout, packaging, redis

Successfully installed async-timeout-4.0.2 importlib-metadata-4.8.3 packaging-21.3 redis-4.3.6 typing-extensions-4.1.1 zipp-3.6.0

[root@jay pyredis-dump]#

 

If you get a ConnectionError exception as follows, its mostly due to OCI Redis cache needing a secure TLS connection.

Error: redis.exceptions.ConnectionError: Error while reading from <Hostname>:6379 (104, 'Connection reset by peer')

[root@jay pyredis-dump]# python3 pyredis-dump.py -H amaaaaaa2ne4d2aaxhlt47xxxktdy77fovocvb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com dump -o rdump1

dumping to 'rdump1'

connecting to {'db': 0, 'host': amaaaaaa2ne4d2aaxhlt47xxxktdy77fovocvb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com', 'port': 6379, 'ssl': None}

[..]

redis.exceptions.ConnectionError: Error while reading from amaaaaaa2ne4d2aaxhlt47xxxktdy77fovocvb3xmg546a-p.redis.eu-frankfurt-1.oci.oraclecloud.com:6379 : (104, 'Connection reset by peer')

[root@jay pyredis-dump]#

 

OCI Redis cache requires secure TLS connection. The pyredis-dump supports TLS connection using switch ‘-S’. Use this option to the connection string.

How to connect to OCI Redis cluster

Redis is a private service and accessible only within the VCN or appropriately peered VCNs.

 

You can connect to the OCI Redis Clusters using the cluster endpoints provided. 

 

To write to the redis cluster, connect using the Primary endpoint.

To read from the redis cluster, connect using the Replica endpoint.

 

You may choose a client library your programming language supports to connect to OCI Cache with Redis.

 

For example, Lettuce or Jedis for Java, redis-py for Python, node-redis for Node.js, go-redis for Go and so on.

 

You can use redis-cli to connect to Redis cluster. The redis-cli is a command-line based tool connect to your redis clusters which can also be used to get valuable insights from the redis cluster.

 

To use redis-cli comes included with the Redis package. You need to establish a secure TLS connection between the client/application and redis server. You should use redis-cli version 6 or above so that it supports the --tls switch to establish as secure TLS connection.

 

By default, redis version 5 is default in the stream on Oracle Linux 8.

See below for Oracle Linux 7/RedHat Linux 7/CentOs 7.

 

[root@jay ~]# yum module list redis

Last metadata expiration check: 0:06:43 ago on Fri 27 Oct 2023 14:34:46 GMT.

Oracle Linux 8 Application Stream (x86_64)

Name                                     Stream                                     Profiles                                     Summary                                                             

redis                                    5 [d][e]                                   common [d]                                   Redis persistent key-value database                                 

redis                                    6                                          common [d]                                   Redis persistent key-value database                                 

 

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

[root@jay ~]#

 

To install Redis Version 6, you need to enable redis 6 module stream.

 

[root@jay ~]# yum module enable redis:6

Last metadata expiration check: 0:10:29 ago on Fri 27 Oct 2023 14:34:46 GMT.

Dependencies resolved.

======================================================================================================================================================================================================

 Package                                         Architecture                                   Version                                         Repository                                       Size

======================================================================================================================================================================================================

Enabling module streams:

 redis                                                                                          6                                                                                                   

 

Transaction Summary

======================================================================================================================================================================================================

 

Is this ok [y/N]: y

Complete!

[root@jay ~]#

 

However, if you end up with the following error: “Error: It is not possible to switch enabled streams of a module unless explicitly enabled via configuration option module_stream_switch.”, you need to uninstall previous redis versions and reset the redis module as follows.

 

[root@jay ~]# yum module enable redis:6

Last metadata expiration check: 0:07:49 ago on Fri 27 Oct 2023 14:34:46 GMT.

Dependencies resolved.

The operation would result in switching of module 'redis' stream '5' to stream '6'

Error: It is not possible to switch enabled streams of a module unless explicitly enabled via configuration option module_stream_switch.

It is recommended to rather remove all installed content from the module, and reset the module using 'yum module reset <module_name>' command. After you reset the module, you can install the other stream.

[root@jay ~]#

 

[root@jay ~]# yum module reset redis

Last metadata expiration check: 0:10:14 ago on Fri 27 Oct 2023 14:34:46 GMT.

Dependencies resolved.

======================================================================================================================================================================================================

 Package                                         Architecture                                   Version                                         Repository                                       Size

======================================================================================================================================================================================================

Resetting modules:

 redis                                                                                                                                                                                              

 

Transaction Summary

======================================================================================================================================================================================================

 

Is this ok [y/N]: y

Complete!

[root@jay ~]#

 

 

Once redis mode is reset and version 6 module is enabled, you may proceed with installing redis version 6 as follows.

 

[root@jay ~]# yum module install redis

Last metadata expiration check: 0:11:16 ago on Fri 27 Oct 2023 14:34:46 GMT.

Dependencies resolved.

======================================================================================================================================================================================================

 Package                              Architecture                          Version                                                                Repository                                    Size

======================================================================================================================================================================================================

Installing group/module packages:

 redis                                x86_64                                6.2.7-1.module+el8.7.0+20764+cc495aeb                                  ol8_appstream                                1.2 M

Installing module profiles:

 redis/common                                                                                                                                                                                       

 

Transaction Summary

======================================================================================================================================================================================================

Install  1 Package

 

Total download size: 1.2 M

Installed size: 4.3 M

Is this ok [y/N]: y

Downloading Packages:

redis-6.2.7-1.module+el8.7.0+20764+cc495aeb.x86_64.rpm                                                                                                                6.9 MB/s | 1.2 MB     00:00   

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Total                                                                                                                                                                 6.8 MB/s | 1.2 MB     00:00    

Running transaction check

Transaction check succeeded.

Running transaction test

Transaction test succeeded.

Running transaction

  Preparing        :                                                                                                                                                                              1/1

  Running scriptlet: redis-6.2.7-1.module+el8.7.0+20764+cc495aeb.x86_64                                                                                                                           1/1

  Installing       : redis-6.2.7-1.module+el8.7.0+20764+cc495aeb.x86_64                                                                                                                           1/1

  Running scriptlet: redis-6.2.7-1.module+el8.7.0+20764+cc495aeb.x86_64                                                                                                                           1/1

  Verifying        : redis-6.2.7-1.module+el8.7.0+20764+cc495aeb.x86_64                                                                                                                           1/1

 

Installed:

  redis-6.2.7-1.module+el8.7.0+20764+cc495aeb.x86_64                                                                                                                                                 

 

Complete!

[root@jay ~]#

 

On Oracle Linux 7/RHEL 7/CentOs7:

 On Oracle Linux 7 / RedHat Linux 7, you should install ‘epel-release-7-14’ and ‘remi’ repositories to get the latest version of Redis. The default repo has only 3.X.X which doesn't support 'tls' in-transit encryption.

# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # yum -y install http://rpms.remirepo.net/enterprise/remi-release-7.rpm # yum --enablerepo=remi install redis

You should get Redis v7.X.X which has support for 'tls' connections.

 

You should now be able to connect to Redis cluster using redis-cli:

 

# redis-cli --tls -h <Redis Endpoint>

 

[root@jay ~]# redis-cli --tls -h amaaaaaa2ne4d2g6eiu32jva-p.redis.eu-frankfurt-1.oci.oraclecloud.com

amaaaaaa2ne4d2g6eiu32jva-p.redis.eu-frankfurt-1.oci.oraclecloud.com:6379>

amaaaaaa2ne4d2g6eiu32jva-p.redis.eu-frankfurt-1.oci.oraclecloud.com:6379> set key value

OK

amaaaaaa2ne4d2g6eiu32jva-p.redis.eu-frankfurt-1.oci.oraclecloud.com:6379> get key

"value"

amaaaaaa2ne4d2g6eiu32jva-p.redis.eu-frankfurt-1.oci.oraclecloud.com:6379> quit

[root@jay ~]#