Elasticsearch – Curator

Curator is an index management tool provided by open source Elasticsearch. This tool allows you to create, delete, and disable indexes.
It also allows you to merge index segments.

This blog postdescribes how to install Curator and how to delete old indices based on time.

Installing Curator

pip3 install elasticsearch-curator

Check curator version

curator --version

Note: If you encounter this error while installing.

ERROR: Cannot uninstall ‘PyYAML’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Execute the command below to fix it.

sudo -H pip3 install --ignore-installed PyYAML

Create a curator.yml file

In this file, indicate the host, port, username, and password.

Reference https://www.elastic.co/guide/en/elasticsearch/client/curator/5.0/configfile.html

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 192.168.1.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  username: elastic
  password: Password
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

Create a delete_indices_time_base.yml file

Reference: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_delete_indices.html

The example configuration below will delete indices with a prefix pattern basketbal-scores- (full index format: basketbal-scores-2022.04.01) older than 14 days.

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 14 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: basketbal-scores-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 14
      exclude:

Manually run script

/usr/local/bin/curator /home/scripts/delete_indices_time_base.yml --config /home/scripts/curator.yml

Schedule the script to run daily via cronjob

# Housekeep indices more than 14 days
0 0 * * * /usr/local/bin/curator /home/scripts/delete_indices_time_base.yml --config /home/scripts/curator.yml >> /home/scripts/log/curator_purging_time_base.log 2>&1