Curator is an index management tool provided by open source Elasticsearch. This tool allows you to create, delete, and disable indexes.
It also allows you to merge index segments.
This blog postdescribes how to install Curator and how to delete old indices based on time.
Installing Curator
pip3 install elasticsearch-curator
Check curator version
curator --version
Note: If you encounter this error while installing.
ERROR: Cannot uninstall ‘PyYAML’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
Execute the command below to fix it.
sudo -H pip3 install --ignore-installed PyYAML
Create a curator.yml file
In this file, indicate the host, port, username, and password.
Reference https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/configfile.html
# Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - 192.168.1.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False username: elastic password: Password timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3']
Create a delete_indices_time_base.yml file
Reference: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html
The example configuration below will delete indices with a prefix pattern basketbal-scores- (full index format: basketbal-scores-2022.04.01) older than 14 days.
--- actions: 1: action: delete_indices description: >- Delete indices older than 14 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: basketbal-scores- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 14 exclude:
Manually run script
/usr/local/bin/curator /home/scripts/delete_indices_time_base.yml --config /home/scripts/curator.yml
Schedule the script to run daily via cronjob
# Housekeep indices more than 14 days 0 0 * * * /usr/local/bin/curator /home/scripts/delete_indices_time_base.yml --config /home/scripts/curator.yml >> /home/scripts/log/curator_purging_time_base.log 2>&1