Zero-Cost Threat Hunting with Elastic Stack

Setting up a Zero Cost Threat Hunting Platform with Elastic Stack and Alienvault Reputation List

Elastic Stack is an awesome suit of products used for several analysis activities since its inception utilizing its amazing searching and visualization capabilities. Here we are trying to leverage the Elastic Stack with few other components to a threat hunting platform to build a reliable blacklist block containing malicious IPs obtained from OSINT and analyze the network traffic in real-time against this for any malicious traffic to any of these IPs.  To achieve this we are using FIREHOL, a service provided by OSINT. It analyses security IP Feeds, mainly related to online attacks, on-line service abuse, malware, botnets and other cybercrime activities. It has several lists. In our case we are going to choose the alienvault_reputation list.

Components Used


Elasticsearch will act as our log repository. It’s incredibly powerful and versitile, and when coupled with Logstash for log ingestion and Kibana for visualization, provides a robust platform for all types of data.


Logstash is mainly made up of three parts Input, filter and output. The input section is where we define the source of the logging data. The filter section could be used for parsing, normalizing, transforming or multiple other methods to prepare the data for sending out to ElasticSearch or any other analytics engines. The output section defines where the data processed by Logstash is stored. This can be ElasticSearch, Kafka or any other database options. Please refer the output filter documentation for supported database options.


Kibana is an open source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data


ElastAlert is an open source project started by the engineers at Yelp to provide an alerting mechanism for Elasticsearch. It’s an independent project that doesn’t need to run on the same server. It simply queries Elasticsearch through the REST API and has numerous outputs to alert on a match. One of those outputs will feed the information into Slack.


Slack is a cloud-based proprietary instant messaging platform developed by Slack Technologies


Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

Memcached is simple yet powerful. Its simple design promotes quick deployment, ease of development, and solves many problems facing large data caches. Its API is available for most popular languages.


The software utility cron is a time-based job scheduler in Unix-like computer operating systems. Users that set up and maintain software environments use cron to schedule jobs to run periodically at fixed times, dates, or intervals


A comprehensive, fast, pure-Python memcached client library.

Log collection

The Logstash input filter is configured to receive the logs from Firewall which is configured in syslog format

Like most of the NextGEN Firewalls, in our scenario the Firewall is producing the logs in Key=Pair format.

We use kv filter and CIDR filter to process to RAW logs to a searchable form

Logstash – KV Filter

This filter helps automatically parse messages (or specific event fields) which are of the foo=bar variety.

For example, if you have a log message which contains ip= error=REFUSED, you can parse those automatically by configuring:

The above will result in a message of ip= error=REFUSED having the fields:

  • ip:
  • error: REFUSED


Logstash CIDR Filter

The CIDR filter is for checking IP addresses in events against a list of network blocks that might contain it. Multiple addresses can be checked against multiple networks, any match succeeds. Upon success additional tags and/or fields can be added to the event.

Here the CIDR filter is used to add a tag to source and destination IPs to distinguish whether is it internal or external IP


Getting the blacklist block

Since the IP list is dynamic we are fetching the list on a daily basis. Once the updated list is downloaded, it will be updated on the memcached cache. A Bash script is created and updated in the cron taks to fetch the IP list and update the memcached

Memached – Preparation

Memcached installation and configuration is simple.  It is also supported by ElasticSearch/Logstash which makes it perfect for our requirement. It also comes with the huge additional benefit of storing the data in memory, so lookups from Logstash to the data will be blazing fast.

The Memcached application is a very simple key-value store running in memory, you can telnet into the application running by default on port 11211.

The application is made up of only a few commands. The ones we are in need of here, are the “get” and “set” commands. Both of which are quite self explanatory….


Once it is installed start the service using

Also systemctl enable memcached enables the service in the startup

Once the service is started test the connection using telnet

The set command will be used by our Python script, to set the data into the store.

The get command will be used by the Logstash filter plugin, to query the store for a specific IP and return the result back to Logstash.

 Python – Memcache integration

There are many modules available for python to interact with memcached. Here we use pymemcache which is simple yet powerful library for get / set operation of memcached.


File –  /root/

Shell script to fetch and update the memcached with IP blocks

File – /root/

Cron job entry to run the script daily at 7.00 am

Logstash – Memcached filter

The Memcached filter provides integration with external data in Memcached.

It currently provides the following facilities: – get: get values for one or more memcached keys and inject them into the event at the provided paths – set: set values from the event to the corresponding memcached keys


Filter configuration

What it does here, once the IP is identified as an external IP, it will be searched in the memcached database. If the search is successful the Value of the Key=Value pair will be returned and stored in a separate field that is “ioc_ip”

Full code

Now search the firewall log index for the additional field created for traffic to external IP. We require to refresh the field list to make the new field searchable

Indication of traffic to Blocked IP lists

Elastalert Rule Setup

Please follow : for Elasticsearch installation

Custom Rule to send Slack Alerts

Slack Configuration

We are using Incoming-Web hooks to send data to the Slack channels

Incoming Webhooks are a simple way to post messages from apps into Slack. Creating an Incoming Webhook gives you a unique URL to which you send a JSON payload with the message text and some options.

The web hooks URL generated is used in the Elastalert rule to send alerts to Slack channel when a traffic is originated from the internal network to any of the blacklisted IPs.

Please follow and like us: