Search Guard Installation for ELK Stack

by Posted on

Search Guard Installation and Concepts

Search Guard is an Open Source Elasticsearch plugin that offers encryption, authentication, and authorization and can be used to secure your Elasticsearch cluster by working with different industry standard authentication techniques such as Active Directory, LDAP, Kerberos, JSON web tokens and many more, and includes fine grained role-based access control to clusters, indices, documents and fields. It enables multi tenancy in Kibana, and stays compliant with GDPR, HIPAA, PCI, SOX and ISO by using audit logging.

With Community Edition Search Guard offers all basic security features for free. The Community Edition of Search Guard can be used for all projects, including commercial projects, at absolutely no cost. The Community Edition includes:

  •     Full data in transit encryption
  •     Node-to-node encryption
  •     Index level access control
  •     Document type based access control
  •     User-, role- and permission management
  •     HTTP basic authentication
  •     User Impersonation
  •     Proxy support

If you just want to use the free Community Edition, install Search Guard Enterprise and then disable all commercial features.

Basic workflow of Searchguard mechanism

  • A user wants to access an Elasticsearch cluster, for example by issuing a simple query.
  • Search Guard retrieves the user’s credentials from the request (they can be extracted from HTTP Basic Authentication headers, from a JSON web token or from a Kerberos ticket)
  • Search Guard authenticates the credentials against the configured authentication backend(s).
  • Search Guard authorizes the user by retrieving a list of the user’s roles from the configured authorization backend
  • Search Guard maps the user and backend roles to Search Guard roles.
  • Search Guard determines the permissions associated with the Search Guard role and decides whether the action the user wants to perform is allowed or not iewhich role,  can perform which action, against which cluster or index.
  • If Document- and Field-Level-Security is required you can also apply more fine grained permissions based on documents and individual fields.

Search Guard ships with pre-defined groups of permissions like READ, WRITE, SEARCH etc called action groups. An action group is an alias for a set of permissions. Action groups can be nested. For example, the following snippet shows two action groups, where the SUGGEST action group is referenced by the SEARCH action group:

SEARCH:

  - "indices:data/read/search*"

  - "indices:data/read/msearch*"

  - SUGGEST

SUGGEST:

  - "indices:data/read/suggest*"

Action groups can be used in the role configuration instead of or in combination with fine-grained permissions like indices:data/read/search*.

A first time installation of Search Guard on a cluster always requires a full cluster restart. TLS encryption is mandatory on the transport layer of Elasticsearch, and thus all nodes must have Search Guard installed in order to be able to talk to each other.

All configuration settings for Search Guard, such as users, roles and permissions, are stored as documents in a special Search Guard index. This index is secured so that only an admin user with a special SSL certificate may write or read this index. You can define one or more of these certificates, called admin certificates, in elasticsearch.yml.

The Search Guard configuration consists of the following files. These are shipped with Search Guard as templates.

sg_config.yml - configure authenticators and authorisation backends.
sg_roles.yml - define roles and the associated permissions.
sg_roles_mapping.yml - map backend roles, hosts and users to roles.
sg_internal_users.yml - stores users, roles and hashed passwords (hash with hash.sh) in the internal user database.
sg_action_groups.yml - define named permission groups.

Configuration settings are applied by pushing the content of one or more configuration files to the Search Guard secured cluster by using the sgadmin tool. For details, refer to the chapter sgadmin.

The installation Steps

  • Stop Elasticsearch
  • Install the Search Guard plugin
  • Generate or obtain TLS certificates
  • Add at least the TLS configuration to elasticsearch.yml
  • Restart Elasticsearch and check that the nodes come up
  • Configure authentication/authorization, users, roles and permissions by uploading the Search Guard configuration with sgadmin

Search Guard demo installer is an installation script that ships with Search Guard with certificates that you can use to run a PoC. This example uses the internal database for authentication.

Download Search Guard Elasticsearch plugin

[root@myhost bin]# wget https://repo1.maven.org/maven2/com/floragunn/search-guard-6/6.2.4-compliance-rc1/search-guard-6-6.2.4-compliance-rc1.zip
--2018-06-04 08:34:37-- https://repo1.maven.org/maven2/com/floragunn/search-guard-6/6.2.4-compliance-rc1/search-guard-6-6.2.4-compliance-rc1.zip
Resolving repo1.maven.org (repo1.maven.org)... 151.101.40.209
Connecting to repo1.maven.org (repo1.maven.org)|151.101.40.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19553206 (19M) [application/zip]
Saving to: ‘search-guard-6-6.2.4-compliance-rc1.zip’

100%[===========================================================================================================================================================>] 19,553,206 226KB/s in 87s

2018-06-04 08:36:06 (219 KB/s) - ‘search-guard-6-6.2.4-compliance-rc1.zip’ saved [19553206/19553206]

Install the plugin

[root@myhost bin]# ./elasticsearch-plugin install file:///usr/share/elasticsearch/bin/search-guard-6-6.2.4-compliance-rc1.zip
-> Downloading file:///usr/share/elasticsearch/bin/search-guard-6-6.2.4-compliance-rc1.zip
[=================================================] 100%  
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.io.FilePermission /proc/sys/net/core/somaxconn read
* java.lang.RuntimePermission accessClassInPackage.com.sun.jndi.ldap
* java.lang.RuntimePermission accessClassInPackage.sun.misc
* java.lang.RuntimePermission accessClassInPackage.sun.nio.ch
* java.lang.RuntimePermission accessClassInPackage.sun.security.x509
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission accessUserInformation
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission shutdownHooks
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission getNetworkInformation
* java.net.NetPermission getProxySelector
* java.net.SocketPermission * connect,accept,resolve
* java.security.SecurityPermission getProperty.ssl.KeyManagerFactory.algorithm
* java.security.SecurityPermission insertProvider.BC
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setProperty.ocsp.enable
* java.util.PropertyPermission * read,write
* javax.security.auth.AuthPermission doAs
* javax.security.auth.AuthPermission modifyPrivateCredentials
* javax.security.auth.kerberos.ServicePermission * accept
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed search-guard-6
[root@myhost bin]#

Installing the demo configuration script

[root@myhost tools]# chmod a+x install_demo_configuration.sh
[root@myhost tools]# ./install_demo_configuration.sh
Search Guard 6 Demo Installer
** Warning: Do not use on production or public reachable systems **
Install demo certificates? [y/N] N
[root@bnghqsiem tools]# ./install_demo_configuration.sh
Search Guard 6 Demo Installer
** Warning: Do not use on production or public reachable systems **
Install demo certificates? [y/N] y
Initialize Search Guard? [y/N] y
Cluster mode requires maybe additional setup of:
- Virtual memory (vm.max_map_count)
See https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

Enable cluster mode? [y/N] N
Basedir: /usr/share/elasticsearch
This script maybe require your root password for 'sudo' privileges
Elasticsearch install type: rpm/deb on CentOS Linux release 7.2.1511 (Core)
Elasticsearch config dir: /etc/elasticsearch
Elasticsearch config file: /etc/elasticsearch/elasticsearch.yml
Elasticsearch bin dir: /usr/share/elasticsearch/bin
Elasticsearch plugins dir: /usr/share/elasticsearch/plugins
Elasticsearch lib dir: /usr/share/elasticsearch/lib
Detected Elasticsearch Version: launchers-6.2.4
Detected Search Guard Version: 6.2.4-compliance-rc1

### Success
### Execute this script now on all your nodes and then start all nodes
### Search Guard will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../sgconfig and execute:
sudo /usr/share/elasticsearch/plugins/search-guard-6/tools/sgadmin.sh -cd /usr/share/elasticsearch/plugins/search-guard-6/sgconfig -icl -key /etc/elasticsearch/kirk-key.pem -cert /etc/elasticsearch/kirk.pem -cacert /etc/elasticsearch/root-ca.pem -nhnv
### or run ./sgadmin_demo.sh
### To use the Search Guard Configuration GUI see http://docs.search-guard.com/v6/configuration-gui
### To access your Search Guard secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
[root@myhost tools]# cd /etc/elasticsearch/
[root@myhost elasticsearch]# ls

Run the sgadmin_demo.sh script for installing the demo configuration files

[root@myhost tools]# ls
hash.bat hash.sh install_demo_configuration.sh sgadmin.bat sgadmin_demo.sh sgadmin.sh
[root@bnghqsiem tools]# ./sgadmin_demo.sh
WARNING: JAVA_HOME not set, will use /bin/java
Search Guard Admin v6
Will connect to localhost:9300 ... done

*** This is a RC version and NOT suitable for production. ***
Elasticsearch Version: 6.2.4
Search Guard Version: 6.2.4-compliance-rc1
Connected as CN=kirk,OU=client,O=client,L=test,C=de
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: MY-SIEM
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/
Will update 'sg/config' with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_config.yml
SUCC: Configuration for 'config' created or updated
Will update 'sg/roles' with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles.yml
SUCC: Configuration for 'roles' created or updated
Will update 'sg/rolesmapping' with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles_mapping.yml
SUCC: Configuration for 'rolesmapping' created or updated
Will update 'sg/internalusers' with /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml
SUCC: Configuration for 'internalusers' created or updated
Will update 'sg/actiongroups' with /usr/share/elasticsearch/plugins/search-guard
SUCC: Configuration for 'actiongroups' created or updated
Done with success
[root@myhost tools]#

Download and install the Search Guard Kibana plugin

[root@myhost bin]# ./kibana-plugin install file:///usr/share/kibana/bin/search-guard-kibana-plugin-6.2.4-13.zip
Attempting to transfer from file:///usr/share/kibana/bin/search-guard-kibana-plugin-6.2.4-13.zip
Transferring 2958310 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete
[root@myhost bin]#

elasticsearch.yml settings

######## Start Search Guard Demo Configuration ########
# WARNING: revise all the lines below before you go into production
searchguard.ssl.transport.pemcert_filepath: esnode.pem
searchguard.ssl.transport.pemkey_filepath: esnode-key.pem
searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: esnode.pem
searchguard.ssl.http.pemkey_filepath: esnode-key.pem
searchguard.ssl.http.pemtrustedcas_filepath: root-ca.pem
searchguard.allow_unsafe_democertificates: true
searchguard.allow_default_init_sgindex: true
searchguard.authcz.admin_dn:
- CN=kirk,OU=client,O=client,L=test, C=de

searchguard.audit.type: internal_elasticsearch
searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: ["sg_all_access"]
cluster.routing.allocation.disk.threshold_enabled: false
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 3
searchguard.enterprise_modules_enabled: false
######## End Search Guard Demo Configuration ########

kibana.yml settings

# Use HTTPS instead of HTTP
elasticsearch.url: "https://localhost:9200"
#
# # Configure the Kibana internal server user
elasticsearch.username: "kibanaserver"
elasticsearch.password: "kibanaserver"
#
# # Disable SSL verification because we use self-signed demo certificates
elasticsearch.ssl.verificationMode: none
#
# # Whitelist the Search Guard Multi Tenancy Header
elasticsearch.requestHeadersWhitelist: [ "Authorization", "sgtenant" ]

Internal user database installed by the demo installer script. All the passwords are hashed. You can use hash.sh to hash passwords.

[root@myhost tools]# vim /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_internal_users.yml
# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh

#password is: admin
admin:
readonly: true
hash: $2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG
roles:
- admin
attributes:
#no dots allowed in attribute names
attribute1: value1
attribute2: value2
attribute3: value3

#password is: logstash
logstash:
hash: $2a$12$u1ShR4l4uBS3Uv59Pa2y5.1uQuZBrZtmNfqB3iM/.jL0XoV9sghS2
roles:
- logstash

#password is: kibanaserver
kibanaserver:
readonly: true
hash: $2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H.

#password is: kibanaro
kibanaro:
hash: $2a$12$JJSXNfTowz7Uu5ttXfeYpeYE0arACvcwlPBStB1F.MI7f0U9Z4DGC
roles:
- kibanauser
- readall

#password is: readall
readall:
hash: $2a$12$ae4ycwzwvLtZxwZ82RmiEunBbIPiAmGZduBAjKN0TXdwQFtCwARz2
#password is: readall
roles:
- readall

#password is: snapshotrestore
snapshotrestore:
hash: $2y$12$DpwmetHKwgYnorbgdvORCenv4NAK8cPUg8AI6pxLCuWf/ALc0.v7W
roles:
- snapshotrestore

Roles configuratin

[root@myhost tools]# vim /usr/share/elasticsearch/plugins/search-guard-6/sgconfig/sg_roles.yml

# For logstash and beats
sg_logstash:
cluster:
- CLUSTER_MONITOR
- CLUSTER_COMPOSITE_OPS
- indices:admin/template/get
- indices:admin/template/put
indices:
'logstash-*':
'*':
- CRUD
- CREATE_INDEX
'*beat*':
'*':
- CRUD
- CREATE_INDEX
'exchange*':
'*':
- CRUD
- CREATE_INDEX
'iis*':
'*':
- CRUD
- CREATE_INDEX
'mcafee*':
'*':
- CRUD
- CREATE_INDEX
'san*':
'*':
- CRUD
- CREATE_INDEX
'cisco*':
'*':
- CRUD
- CREATE_INDEX
'usb*':
'*':
- CRUD
- CREATE_INDEX
'bakup*':
'*':
- CRUD
- CREATE_INDEX
'wazuh*':
'*':

Logstash output configuration

Make sure that the url is modified with https and credentials are givem

output {
if [indexType]== "WIN-LOGS"
{
elasticsearch {
user => logstash
password => logstash
ssl => true
ssl_certificate_verification => false
hosts => "https://localhost:9200"
manage_template => false
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => doc
}
}
else if [indexType]== "SYS-LOGS"
{
elasticsearch {
user => logstash
password => logstash
ssl => true
ssl_certificate_verification => false
hosts => "https://localhost:9200"
manage_template => false
index => "syslog-%{+YYYY.MM.dd}"
document_type => doc
}
}

Seach Guard Community License shows “No license required because enterprise modules not enabled.” message

https://127.0.0.1:9200/_searchguard/license

{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"MY-SIEM","sg_license":{"msgs":["No license required because enterprise modules not enabled."],"license_required":false},"modules":{"INTERNAL_USERS_AUTHENTICATION_BACKEND":

Kibana Login page

 

 

Published by Sumesh MS