Elasticsearch – Curator

Curator is an index management tool provided by open source Elasticsearch. This tool allows you to create, delete, and disable indexes.
It also allows you to merge index segments.

This blog postdescribes how to install Curator and how to delete old indices based on time.

Installing Curator

pip3 install elasticsearch-curator

Check curator version

curator --version

Note: If you encounter this error while installing.

ERROR: Cannot uninstall ‘PyYAML’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Execute the command below to fix it.

sudo -H pip3 install --ignore-installed PyYAML

Create a curator.yml file

In this file, indicate the host, port, username, and password.

Reference https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/configfile.html

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 192.168.1.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  username: elastic
  password: Password
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Create a delete_indices_time_base.yml file

Reference: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html

The example configuration below will delete indices with a prefix pattern basketbal-scores- (full index format: basketbal-scores-2022.04.01) older than 14 days.

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 14 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: basketbal-scores-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14
      exclude:

Manually run script

/usr/local/bin/curator /home/scripts/delete_indices_time_base.yml --config /home/scripts/curator.yml

Schedule the script to run daily via cronjob

# Housekeep indices more than 14 days
0 0 * * * /usr/local/bin/curator /home/scripts/delete_indices_time_base.yml --config /home/scripts/curator.yml >> /home/scripts/log/curator_purging_time_base.log 2>&1

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MySQL/MariaDB Memory

In this blog post, I’m gonna talk about MySQL Memory Issues. What causes High memory usage and what is the impact on MySQL database instances.

But first, we can do a quick review on the following topics:

  • How MySQL uses memory? (Log files, buffers, InnoDB buffer pool size)
  • How MySQL allocates memory?
  • Swap memory
  • Troubleshooting
  • Best practices.
Globally shared memory is allocated upon the creation of an instance and is shared by all connections
Example: thread cache, query cache, buffer pool, and log buffer.

Private memory is used to allocate cache upon connection to MySQL server. It is allocated for each thread. For example; sort buffer, join buffer, and temporary table.

InnoDB maintains one or more buffer pools that cache frequently used data and indexes in the main memory.

So, when a read query is executed from the client program, InnoDB checks to see if the required data pages are in the buffer pool. If it is not in the buffer pool, InnoDB requests the data from the tablespace. Then it will put the data pages in the buffer pool. And then, MySQL will return the results to the client.

 
When transactions are performed, data changes are written to the log buffer in memory. then every time a transaction is committed, the transaction logs get flushed to the disk, but this behavior can be changed (InnoDB_flush_log_at_trx_commit). If a crash occurs (while the tables are being modified), the redo log files are used for recovery.
 
On the other hand, Dirty pages in the buffer pool get written to the disk at checkpoints. This is to ensure that the cache has enough free space.

innodb_buffer_pool_size is the most important tuning parameter. It caches table and index data. The buffer pool permits frequently used data to be accessed directly from memory, which speeds up processing.

Setting it too low can degrade the performance. Setting it too high can increase the memory consumption causing the DB to crash.

MySQL allocation is not only from innodb_buffer_pool_size, but also from other buffers in the database such sort_buffer_size, read_buffer_size, read_rnd_buffer, join_buffer_size, and tmp_table_size and this will require additional 5-10% extra memory

Below is a formula to calculate the approximate memory usage for your MySQL:

Maximum MySQL Memory Usage = innodb_buffer_pool_size + key_buffer_size + ((read_buffer_size + read_rnd_buffer_size + sort_buffer_size + join_buffer_size) X max_connections)

Swapping can happen when a system requires more memory than is allocated .A large buffer may lead to swapping in the operating system and make the performance slow.

To avoid your MySQL/MariaDB data being SWAP instead of RAM, you have to play with a kernel parameter called swappiness.

A swappiness value is used to change the balance between swapping out runtimememory and dropping pages from the system page cache. The higher the value, the more the system will swap. The lower the value, the less the system will swap. The maximum value is 100, the minimum is 0, and 60 is the default.

I performed a test in AWS RDS MySQL. I ran a stored procedure that consumed a lot of memory until the server crashed. We will notice that when memory gets low, Swap usage will increase. We will also see a spike in disk IO usage.

When MySQL crashed, we will lose connection to the database. In AWS RDS, we can see the logs and EVENTs to identify the cause of the crash. The log indicates that the database process was killed by the OS due to excessive memory consumption.

I also did the same test in Azure MySQL. The difference between Azure and AWS is that when the DB crash, in Azure, it will failover to the Standby replica, but in AWS, a mysqld process crash would not trigger a failover.

In Azure, under Private DNS Zone, if we see that the IP address changed, then it means that it failed over to the standby replica successfully.

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MySQL/MariaDB – FLUSH TABLES WITH READ LOCK

To block all write access and to mark all tables as ‘properly closed’ on disk, we execute the FLUSH TABLES WITH READ LOCK. Note that the tables can still be used for read operations. Open tables will be closed and all tables will be locked for all databases with a global lock.

FLUSH TABLES and RELOAD privilege are required to execute this operation.

To release the lock, you may use UNLOCK TABLES, which implicitly commits any active transaction only if any tables currently have been locked with the LOCK TABLES. If the UNLOCK TABLES is executed after FLUSH TABLES WITH READ LOCK, then the commit will not occur because FLUSH TABLES WITH READ LOCK does not acquire table locks.

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

Flush Tables With Read Lock

This command closes all open tables and locks all tables for all databases with a global read lock

FLUSH TABLES or RELOAD privilege is required for this operation.

To release the lock, use UNLOCK TABLES. This command implicitly commits any active transaction only if any tables currently have been locked with LOCK TABLES.

Inserting rows into the log tables is not prevented with FLUSH TABLES WITH READ LOCK.

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MySQL/MariaDB – ulimit open files & MySQL open_files_limit

The amount of resources that can be used can be controlled from the operating system perspective. Each user has limits that are set, but for that particular user, the limits are applied individually to each of its processes.

Limits can either be hard or soft. Only the root user can set the Hardlimits. Other users can set Soft limits, but it cannot be more than the hard limit.

The default value of ulimit open files limit is 1024. This is very low for a typical web server environment that hosts many have database-driven sites.

MySQL/MariaDB also uses this setting. The open_files_limit is set by MySQL/MariaDB to the system’s ulimit. Default is 1024.

NOTE: MySQL/MariaDB can only set its open_files_limit lower than what is specified under ulimit ‘open files’. It cannot be set higher than that.

Examine Current Limits

To inspect current limits


ulimit -a

# -a will show all current limits including hard, soft, open files, etc.

To inspect the current hard and soft limits.


# Hard Limits
ulimit -Ha

# Soft Limits
ulimit -Sa

# H for hard limits, or S for soft limits.

To check current open file limits.


ulimit -n

# –n for number of open files limits

Set ‘open files’ Limit Persistently

Open  /etc/security/limits.conf using the text editor of your choice, and add the following lines, then save it.


* soft nofile 102400
* hard nofile 102400
* soft nproc 10240
* hard nproc 10240

Edit the file /etc/security/limits.d/90-nproc.conf using the text editor of your choice, and add the following lines, then save it.


* soft nofile 1024000
* hard nofile 1024000
* soft nproc 10240
* hard nproc 10240
root soft nproc unlimited

Set open_files_limit in my.cnf (MySQL)

Open and edit /etc/my.cnf

Insert the lines below under [mysql], then save it.


[mysqld]
open_files_limit = 102400

Run the command below to see if there are any .conf files being used by MySQL that overrides the values for open limits.


systemctl status mysqld

Below is something you will see after running the command above.


/etc/systemd/system/mariadb.service.d
└─limits.conf

This means that there is a  /etc/systemd/system/mariadb.service.d/limts.conf  file, whicv is loaded with MySQL. Now, edit that file as well.


[Service]
LimitNOFILE=102400

Execute the command below for the changes to take effect.

systemctl daemon-reload && /scripts/restartsrv_mysql

Perform Server Reboot

Run the command below in MySQL to see the value of open_files_limit.


SHOW VARIABLES LIKE 'open_files_limit';

Output:


+------------------+--------+
| Variable_name    | Value  |
+------------------+--------+
| open_files_limit | 102400 |
+------------------+--------+
1 row in set (0.00 sec)

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MySQL/MariaDB – optimizer_switch

To control optimizer behaviour, we can enable/disable specific optimization via optimizer_switch system variable. The optimizer_switch variable can be changed at runtime, and has global and session values.

Execute the command below to see the current set of optimizer flags.`

SELECT @@optimizer_switch \G

To change the value of optimizer_switch, use the following syntax.

SET [GLOBAL|SESSION] optimizer_switch='cmd[,cmd]...';
Syntax Description
defaultReset all optimizations to their default values.
optimization_name=defaultSet the specified optimization to its default value.
optimization_name=onEnable the specified optimization.
optimization_name=offDisable the specified optimization.
There is no need to list all flags – only those that are specified in the command will be affected.

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MariaDB/MySQL – Seconds Behind Master Deep Dive

seconds_behind_master takes the difference between the timestamp on the Replica minus the timestamp of the event that the SQL_THREAD is processing that (timestamp) that was on the master.

DEMO

Execute the script below in Master Server.

create database if not exists sbm_db;

USE `sbm_db`;
DROP procedure IF EXISTS `gendata`;

DELIMITER $$
CREATE PROCEDURE gendata (in loopLimit int)
BEGIN
	declare c int;
    set c = 0;
    
    label: LOOP
		insert into tbl (fld)
        values (FLOOR(1 + (RAND() * 60000)));
        set c = c + 1;
        if c > loopLimit then
			leave label;
        end if;
    end LOOP label;
END$$

DELIMITER ;


drop table if exists tbl;
create table tbl (
tblId int not null primary key auto_increment,
fld varchar(255),
updatedAt timestamp not null default current_timestamp
);

Execute in Replica

STOP SLAVE SQL_THREAD;

Execute in Master

call `gendata`(3000);
mysql> SELECT COUNT(*) FROM tbl;
+----------+
| COUNT(*) |
+----------+
|     3000 |
+----------+

In the Replica, since the SQL_THREAD is not running, these records don’t exists.

mysql> SELECT COUNT(*) FROM tbl;
+----------+
| COUNT(*) |
+----------+
|        0 |
+----------+

Now, run these commands in one shot in the Replica to start the SQL_THREAD, get the unix_timestamp(), then stop the SQL_THREAD again.

START SLAVE SQL_THREAD;

SHOW SLAVE STATUS \G

SELECT unix_timestamp();

STOP SLAVE SQL_THREAD;

SELECT COUNT(*) FROM tbl;

Below are my output.

mysql> START SLAVE SQL_THREAD;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> SHOW SLAVE STATUS \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.33.17
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: db1-bin.000018
          Read_Master_Log_Pos: 3349116
               Relay_Log_File: mysqldb2-relay-bin.000006
                Relay_Log_Pos: 2512613
        Relay_Master_Log_File: db1-bin.000018
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2512410
              Relay_Log_Space: 3349604
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 93
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:


mysql> SELECT unix_timestamp();
+------------------+
| unix_timestamp() |
+------------------+
|       1621660163 |
+------------------+
1 row in set (0.00 sec)


mysql> STOP SLAVE SQL_THREAD;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> SELECT COUNT(*) FROM tbl;
+----------+
| COUNT(*) |
+----------+
|      552 |
+----------+
1 row in set (0.00 sec)


You might be wondering, why seconds behind master is 93 when there were only 5 seconds of worth of data?

How seconds master works is it takes the difference between the timestamp on the replica minus the timestamp of the event that the sql thread is processing (The timestamp that is on the master).

mysqlbinlog --base64-output="decode-rows" --verbose mysqldb2-relay-bin.000006 | less

# at 2512613
#210522  1:07:50 server id 1  end_log_pos 2512485       GTID    last_committed=9012     sequence_number=9013    rbr_only=yes    original_committed_timestamp=1621660070   immediate_commit_timestamp=1621660070915181     transaction_length=279

mysql> select 1621660163 - 1621660070;
+-------------------------+
| 1621660163 - 1621660070 |
+-------------------------+
|                      93 |
+-------------------------+



Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MySQL/MariaDB – slave_compressed_protocol

This parameter enables compression on master/slave protocol. If both master and slave supports it, this parameter will instruct MariaDB/MysQL to use compression.

This parameter is not enabled by default.

It can be enabled in your my.cnf file (ensure to restart the DB service after enabling it to take effect)

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MariaDB – Troubleshoot Out-of-Memory Issues

In this blog post, we are going to discuss MariaDB crash issues due to out-of-memory (OOM), and how to troubleshoot them.

When you get this error, it clearly indicates that the server is running out of memory. In other words, the allocated memory to MariaDB is not enough to handle the process requirements.

If memory is insufficient, then we just have to increase RAM, right? Well, this is certainly one solution, but it is the easiest way out. Simply adding RAM is neither the ideal solution nor a cheaper way. A memory-intensive query or process can eventually eat up the added memory, and can lead to errors in a short span of time.

Common Causes of Out-Of-Memory Issues

  1. Non-optimal tables and queries
  2. MySQL Server Configuration
  3. Memory-intensive processes
  4. Poor hardware resources
    • Insufficient RAM
    • Lack of disk space

Here are the ways to troubleshoot

  1. Check Linux OS and Config
    • Check /var/log/messages and /var/log/syslog. Try to find an entry where it says something like “OOM killer killed MySQL”. Once you find it, the surrounding details will give you clues.
    • Check available RAM.
      • free -g
      • cat /proc/meminfo
    • Issue “top” command to see what applications are consuming RAM.
    • Run vmstat 5 5 to view if the system is swapping and if it is reading/writing via virtual memory.
  2. Check MariaDB. Look for possible MariaDB memory leaks
    • These are the places where MariaDB allocates most of the memory:
      • Table cache
      • InnoDB (execute show engine innodb status, and look for buffer_pool and associated caches in the buffer pool section.
      • Identify if there are temporary tables in RAM (select * from information_schema.tables where engine=’MEMORY’;)
      • Prepared statements, when it is not removed from the set of resources (show global status like ‘Com_prepare_sql’; show global status like ‘Com_dealloc_sql’)

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter

MariaDB – Enabling Semisynchronous Replication

In this blog post, we are going to learn how to enable Semisynchronous Replication in MariaDB.

MariaDB provides semisynchronous replication option besides the standard MariaDB asynchronous replication.

In Asynchronous replication, the Master does not wait for a slave to confirm that an event has been received whenever the slave request events from the Master’s binary log.

Semisynchronous Replication

The procedure to enable Semisynchronous Replication below applies only to MariaDB 10.3.3 and later since it is already built into MariaDB server and is no long provided by a plugin.

It can be set dynamically with SET GLOBAL.

SET GLOBAL rpl_semi_sync_master_enabled=ON;

It can also be set in my.cnf

[mariadb]
...
rpl_semi_sync_master_enabled=ON

To enable Semisynchronous Replication on the Slave dynamically with SET GLOBAL.

SET GLOBAL rpl_semi_sync_slave_enabled=ON;

It can also be set in my.cnf

[mariadb]
...
rpl_semi_sync_slave_enabled=ON

If slave threads are already running when you enable Semisynchronous Replication, you need to restart slave I/O thread. Otherwise, it will continue to use asynchronous replication.

STOP SLAVE IO_THREAD;
START SLAVE IO_THREAD;

Configuring the Master Timeout

In semisynchronous replication, the slave acknowledges the receipt of the transaction’s events only after the events have been written to the relay log and flushed.

A timeout will occur if the slave does not acknowledge the transaction before a certain amount time, and the Master will be switching to Asynchronous replication.

Rpl_semi_sync_master_status status variable will be switched to OFF when this happens.

Semisynchronous replication will be resumed when at least one semisynchronous slave catches up, and at the same time Rpl_semi_sync_master_status status variable will be switched to ON.

The timeout period is configurable. It can be changed by setting the rpl_semi_sync_master_timeout system variable.

It can be set dynamically with SET GLOBAL.

SET GLOBAL rpl_semi_sync_master_timeout=20000;

It can also be set in my.cnf

[mariadb]
...
rpl_semi_sync_master_timeout=20000

Cheers!

Knowledge worth sharing...Share on linkedin
Linkedin
Share on facebook
Facebook
Share on google
Google
Share on twitter
Twitter