Saturday, December 12, 2020

How2 change Datadir path under MariaDB 10.3 running on CentOS 8 server

This tutorial will guide with the steps to change datadir in MariaDB and also provide with rollback steps in case it does not work as expected.

Configuration steps are as follows


    a) Plan for the server downtime and shutdown MariaDB.

            # systemctl stop mariadb

        Reconfirm the status of MariaDB is properly shutdown.

            # systemctl status mariadb


    b) Copy existing data to new datadir (in our case we will use /mnt/data), but replace this path as per your new datadir path.

            # rsync -av /var/lib/mysql /mnt/data

        Optional step : Above rsync command will copy all data files keeping ownership intact, but to ensure all directories and datafiles have correct ownership execute following command

           
# chown -R mysql:mysql /mnt/data/


    c) Rename original datadir so as to have a rollback plan in place and also to avoid confusion.

            # mv /var/lib/mysql /var/lib/mysql.old


    d) Update config files with new datadir path

        Update server config file

            
# vi /etc/my.cnf.d/mariadb-server.cnf

        in [mysqld] section, comment out original datadir and socket lines as it can be used to roll back changes

            
#datadir=/var/lib/mysql
            #socket=/var/lib/mysql/mysql.sock

        and add new values

            datadir=/mnt/data/mysql
            socket=/mnt/data/mysql/mysql.sock
 
        and save the file and proceed to updated client config file

            # vi /etc/my.cnf.d/client.cnf

        under client-mariadb section add this line

            [client-mariadb]
            
socket=/mnt/data/mysql/mysql.sock

        and save the file


    e) Start MariDB server

           
# systemctl start mariadb

        and check if mariadb is working fine

            # systemctl status mariadb

        Incase there is any error then you can try to debug the same or roll back steps as mentioned in section c & d and restart MariaDB and you will be back to original state


    f) To check the status, login into MariaDB console

            # mysql -uroot -p

        check if datadir is set correctly
        
           
> select @@datadir;

        and response should be as follows

            +-------------------+
            | @@datadir         |
            
+-------------------+
            | /mnt/data/mysql/  |
           
+-------------------+


    g) To check everything is working fine, create a database

            
> create database test;

        and if created successfully then proceed to clean up the test database. 

            > drop database test;

        and exit out of mariadb.


    h) After ensuring all the data is successfully copied and new Datadir path is working fine, delete the /var/lib/mysql.old folder to save on hdd space.

            # rm -rf /var/lib/mysql.old

With this we are done with moving Datadir path under MariaDB 10.3 and I hope this post help you with the required steps.

Thursday, March 12, 2020

How2 deploy Redis 5 Cluster on CentOS 8

This tutorial will guide you with the steps to deploy Redis 5.0 master nodes cluster across 3 nos of CentOS 8 servers.

Disclaimer :

a) There are multiple ways to deploy Redis cluster
b) This documents attempts to quickly get started and cover basic commands that are required to get cluster running
c) I am also learning and not an expert, so help me improvise :)


Assumptions :

1) We have 3 different servers / VMs with their own IPs to deploy cluster
2) User need to have root access to all servers and user would do su to get root access. And as a policy direct ssh root login should be avoided.

Process :

1] First we need to check if firewall is enabled on the server and atleast running with bare minimum rules

    # systemctl status firewalld

and if not running we start it immediately and set it to start by default when server reboots

    # systemctl enable --now firewalld


2] We need to install EPEL repository on CentOS 8 server and update packages

    # dnf install epel-release && dnf update

optionally we can restart the server and at times it would be kernel update that would require restart

    # reboot


3] Now install few basic tools that we would need for debugging or testing

    # dnf install telnet bind-utils

you should be able to use telnet command to debug open port like

    # telnet localhost 6379

or for inter server connectivity

    # telnet SERVER_1_IP 6379


4] We do some basic OS level tweek for improving Redis performance by editing /etc/sysctl.conf file by adding following line at the end of file

    vm.overcommit_memory = 1

and reboot the server for setting to get effect

or

to enable it without reboot then run

    # sysctl vm.overcommit_memory=1


5] 
Disable transparent huge pages features to improve on memory usage and latency.

    # echo never > /sys/kernel/mm/transparent_hugepage/enabled


6] Install Redis on all 3 servers

    # dnf install @redis

once installed, we start Redis server and also enable Redis services to auto start on reboot

    # systemctl enable --now redis


7] Lets check if Redis is running fine

    # redis-cli ping

    and get PONG in response


8] Now configure redis.conf file for cluster

    # vi /etc/redis.conf

search for following lines and update 

a) Change Redis server listening IP address to servers ip address

    bind 127.0.0.1

to

    bind SERVER_IP_ADDRESS

optionally

    bind SERVER_IP_ADDRESS 127.0.0.1

Please note do not set 127.0.0.1 as first ip otherwise your cluster would fail


b) This is optional step, change port number if required to be running on different port. Please note, any good security tool can identify service running on any port, so don't bother to change port no for security reason

    port 6379

to

    port NEW_PORT_NO


c) Correct the path of the pidfile accordingly

    pidfile /var/run/redis_6379.pid

to

    pidfile /var/run/redis/redis_6379.pid


d) Enable cluster

    # cluster-enabled yes

to

    cluster-enabled yes


e) Enable cluster config file that would be system configured

    # cluster-config-file nodes-6379.conf

to

    cluster-config-file nodes-6379.conf


f) Set node timeout to 5 seconds


    # cluster-node-timeout 15000


to


    cluster-node-timeout 5000



g) Enable Append-only logs of all write operations performed by the server (AOF)

    appendonly no

to

    appendonly yes


h) Enable security by setting authentication password

    # requirepass foobared

with random password string

    requirepass LONG_RANDOM_STRING_WITH_NUMBERS_AND_SPECIALKEYS

and if this is set then user has to authenticate before accessing data

$ redis-cli -c -p 6379
127.0.0.1:6379> keys *
(error) NOAUTH Authentication required.
127.0.0.1:7531> auth LONG_RANDOM_STRING_WITH_NUMBERS_AND_SPECIALKEYS
OK


9] Now configure firewalld on all Redis servers to run cluster by opening 6379 and 16379 ports between all servers for running cluster and opening client IPs

    # firewall-cmd --new-zone=redis --permanent
    # firewall-cmd --zone=redis --add-port=6379/tcp --permanent
    # firewall-cmd --zone=redis --add-port=16379/tcp --permanent

and for following skip your own server ip rule i.e on Redis server 1, you don't need to add open rule for Redis server 1 ip

    # firewall-cmd --zone=redis --add-source=SERVER_1_IP --permanent
    # firewall-cmd --zone=redis --add-source=SERVER_2_IP --permanent
    # firewall-cmd --zone=redis --add-source=SERVER_3_IP --permanent
    # firewall-cmd --reload
    # firewall-cmd --runtime-to-permanent

to check firewalld rules run

    # firewall-cmd --list-all --zone=redis
    # firewall-cmd --list-all-zones


10] Finally lets create Redis Cluster by running following command on one of the server

    # redis-cli --cluster create SERVER_1_IP:6379 SERVER_2_IP:6379 SERVER_3_IP:6379

and you would see something similar. Also you will be asked to confirm the config by typing 'yes'

    Performing hash slots allocation on 3 nodes...
    Master[0] -> Slots 0 - 5460
    Master[1] -> Slots 5461 - 10922
    Master[2] -> Slots 10923 - 16383
    M: SERVERSTRING9dcc5 SERVER_1_IP:6379
     slots:[0-5460] (5461 slots) master
    M: SERVERSTRING42932 SERVER_2_IP:6379
     slots:[5461-10922] (5462 slots) master
    M: SERVERSTRING7174c SERVER_3_IP:6379
     slots:[10923-16383] (5461 slots) master
    Can I set the above configuration? (type 'yes' to accept): yes
    >>> Nodes configuration updated
    >>> Assign a different config epoch to each node
    >>> Sending CLUSTER MEET messages to join the cluster
    Waiting for the cluster to join
    .
    >>> Performing Cluster Check (using node SERVER_1_IP:6379)
    M: SERVERSTRING9dcc5 SERVER_1_IP:6379
     slots:[0-5460] (5461 slots) master
    M: SERVERSTRING7174c SERVER_3_IP:6379
     slots:[10923-16383] (5461 slots) master
    M: SERVERSTRING42932 SERVER_2_IP:6379
     slots:[5461-10922] (5462 slots) master
   [OK] All nodes agree about slots configuration.
   >>> Check for open slots...
   >>> Check slots coverage...
   [OK] All 16384 slots covered.


11] Now we are done with deploying Redis cluster and we can run few basic commands to check the status by running redis-cli with -c parameter

    127.0.0.1:6379> cluster nodes

should see

    SERVERSTRING9dcc5 SERVER_1_IP:6379@16379 master - 0 9999999998108 1 connected 0-5460
    SERVERSTRING7174c SERVER_3_IP:6379@16379 myself,master - 0 9999999997000 3 connected 10923-16383
    SERVERSTRING42932 SERVER_2_IP:6379@16379 master - 0 9999999999000 2 connected 5461-10922


and

    127.0.0.1:6379> cluster info
    cluster_state:ok
    cluster_slots_assigned:16384
    cluster_slots_ok:16384
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:3
    cluster_size:3
    cluster_current_epoch:3
    cluster_my_epoch:3
    cluster_stats_messages_ping_sent:4309
    cluster_stats_messages_pong_sent:4259
    cluster_stats_messages_meet_sent:1
    cluster_stats_messages_sent:8569
    cluster_stats_messages_ping_received:4258
    cluster_stats_messages_pong_received:4309
    cluster_stats_messages_meet_received:1
    cluster_stats_messages_received:8568

Thanks and do share your feedback below.