Elasticsearch Curator

by Posted on

Elasticsearch Curator helps to curate, or manage, the Elasticsearch indices and snapshots by getting the full list of indices from the Elasticsearch cluster as actionable list and iterate through the list with user defined filters and to perform various actions on the items.

Standard operations  performed on both indices and snapshots include

  • Add or remove indices from an alias
  • Change shard routing allocation
  • Close indices
  • Create index
  • Delete indices
  • Delete snapshots
  • Open closed indices
  • forceMerge indices
  • reindex indices, including from remote clusters
  • Change the number of replicas per shard for indices
  • rollover indices
  • Take a snapshot (backup) of indices
  • Restore snapshots

Curator installation on CentOS 7 using yum

Elastic use the PGP key D88E42B4, Elastic’s Signing Key, with fingerprint 4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4 to sign all our packages. It is available from http://pgp.mit.edu.

Download and install the public signing key:

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Create curator.repo file in /etc/yum.repos.d/

[root@myhost ~]# cd /etc/yum.repos.d/
[root@myhost yum.repos.d]#

[root@myhost yum.repos.d]# vim curator.repo
[curator-5]
name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
baseurl=http://packages.elastic.co/curator/5/centos/7
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Install Curator using the following command

[root@myhost yum.repos.d]# yum install elasticsearch-curator

Modify the Curator config files

[root@myhost yum.repos.d]# cd /usr/share/curator/
[root@myhost curator]# ls
action_file.yml curator.yml

Elasticsearch IP and port are defined in the curator.yml file.

[root@myhost curator]# vim curator.yml
---
client:
hosts:
- 127.0.0.1
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
aws_key:
aws_secret_key:
aws_region:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: False

The action file is configured to delete indices older than 7 days, each index is configured separately.

[root@myhost curator]# vim action_file.yml
actions:
1:
action: delete_indices
description: >-
Delete indices older than 7 days (based on index name), for logstash-
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: syslog-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 7
exclude:
2:
action: delete_indices
description: >-
Delete indices older than 15 days (based on index name), for winlogbeat-
options:
ignore_empty_list: True
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: winlogbeat-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 15
exclude:

You can issue curator –help to get Curator usage with various options available

[root@myhost curator]# curator --help
Usage: curator [OPTIONS] ACTION_FILE

Curator for Elasticsearch indices.

See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
--config PATH Path to configuration file. Default: ~/.curator/curator.yml
--dry-run Do not perform any changes.
--version Show the version and exit.
--help Show this message and exit.

Curator is executed with the following command

[root@myhost curator]# /usr/bin/curator --config /usr/share/curator/curator.yml /usr/share/curator/action_file.yml

You can add a contrab entry of curator for scheduled run

[root@myhost curator]# crontab -e

0 8 * * * /usr/bin/curator --config /usr/share/curator/curator.yml /usr/share/curator/action_file.yml

Curator will be scheduled to run 8.00 AM everyday.

Crontab entry format

* * * * * *
| | | | | |
| | | | | +– Year (range: 1900-3000)
| | | | +—- Day of the Week (range: 1-7, 1 standing for Monday)
| | | +—— Month of the Year (range: 1-12)
| | +——– Day of the Month (range: 1-31)
| +———- Hour (range: 0-23)
+———— Minute (range: 0-59)

Elasticsearch log entry

[root@myhost elasticsearch]# tail -f MY-SIEM.log
[2018-04-23T08:03:15,516][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.02.23/NvGVXFlxReCMg-iVEodMfw] deleting index
[2018-04-23T08:03:15,516][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.02.08/57YPi-xETL2an-HaAQ1qRg] deleting index
[2018-04-23T08:03:15,516][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.04.14/WvSpP8C6TzWvbHVPqhoJBg] deleting index
[2018-04-23T08:03:15,517][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.03.17/Ibd1SNvsScaERqfxjOuj3g] deleting index
[2018-04-23T08:03:15,517][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.02.07/qDOP9Eh9RbmABhhfpPGvWg] deleting index
[2018-04-23T08:03:15,517][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.02.05/0EctcN7nTW6Pl1to8bdG-Q] deleting index
[2018-04-23T08:03:15,517][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.04.02/PAW6MtANQviwZluhCacq8Q] deleting index
[2018-04-23T08:03:15,517][INFO ][o.e.c.m.MetaDataDeleteIndexService] [0Wtoamx] [winlogbeat-2018.01.28/lLEH0ST_QCS1MpqaBLPwwA] deleting index

Disk usage before executing Curator:

[root@myhost elasticsearch]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 209751040 196268592 13482448 94% /

Disk usage after executing Curator:

[root@myhost curator]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 209751040 47807144 161943896 23% /

Published by Sumesh MS