Architecting Splunk Deployment

Architecting Splunk Deployment

  • Scale Splunk Enterprise functionality to handle the data needs for enterprises of any size and complexity.
  • Access diverse or dispersed data sources.
  • Achieve high availability and ensure disaster recovery with data replication and multisite deployment.

Splunk Enterprise performs three key functions as it processes data:

  1. It ingests data from files, the network, or other sources.
  2. It parses and indexes the data.
  3. It runs searches on the indexed data.

Types of processing components

There are three main types of processing components:

  • Forwarders
  • Indexers
  • Search heads
  • Data input.Data enters the system through forwarders, which consume external data, perform a small amount of preprocessing on it, and then forward the data to the indexers.
  • Two or three indexers receive, index, and store incoming data from the forwarders. The indexers also search that data, in response to requests from the search head. The indexers reside on dedicated machines.
  • Search management.A single search head performs the search management function. It handles search requests from users and distributes the requests across the set of indexers, which perform the actual searches on their local data. The search head then consolidates the results from all the indexers and serves them to the users.

An Indexer cluster is a group of Splunk Enterprise indexers that are configured to replicate each other’s’ data, so that the system keeps multiple copies of all data.

  • They include a capability to coordinate configuration updates easily across all indexers in the cluster.
  • They include a built-in distributed search capability.
  • They feature indexer discovery, which enables the set of forwarders to automatically load-balance across all indexers in the cluster.

The data pipeline has these segments:

  • Input
  • Paring
  • Indexing
  • Search

The correspondence between the three typical processing tiers and the four data pipeline segments is this:

  • The data input tier handles the input segment.
  • The indexing tier handles the parsing and indexing segments.
  • The search management tier handles the search segment.

Types of management components

A deployment usually includes one or more of these management components:

  • The license master  handles Splunk Enterprise licensing.
  • The monitoring console performs centralized monitoring of the entire deployment.
  • The deployment server  updates configurations and distributes apps to processing components, primarily forwarders.
  • The “cluster master”, coordinates the activities of an indexer cluster. It also handles updates for indexer clusters.
  • The search head cluster deployer handles updates for search head cluster.

These are some of the main types of deployments, based on size:

  • A single instance that combines indexing and search management functions.
  • Small enterprise ->   One search head with two or three indexers.
  • Medium enterprise -> A small search head cluster, with several indexers.
  • Large enterprise -> A large search head cluster, with large numbers of indexers.
Departmental Small enterprise deployment:

Single search head with multiple indexers

 

Medium to large enterprise deployment:

Search head cluster with multiple indexers

 

·        One combined indexer/search head

·        Multiple forwarders

 

 

·        one search head

·        Multiple indexers

·        Multiple forwarders

·        One search head cluster, containing multiple search heads

·        Multiple indexers

·        Multiple forwarders

Use case

Characteristics of this type of deployment include:

·        Indexing volume of under 20GB/day.

·        A few users, typically less than 10.

·        A relatively small number of forwarders sending data to the instance, typically less than 10 and rarely exceeding 100.

 

 

 

 

Use case

This distributed search scenario provides the first level of horizontal scaling. It allows users to run searches across a set of indexers. As your needs increase further, you can add more indexers.

Characteristics of this type of deployment include:

·        Indexing volume between 20 and 100GB/day.

·        Between 10 and 100 users.

·        Up to several hundred forwarders feeding data to the indexers. The forwarders typically make use of load balancing to distribute the data across the set of indexers.

Medium enterprise deployment use case

A medium enterprise deployment provides greater horizontal scaling than a small enterprise deployment. It services larger numbers of users and searches. As your needs continue to increase, you can continue to add indexers and search heads.

Characteristics of this type of deployment include:

·        Indexing volume between 100-300GB/day.

·        Users numbering possibly a hundred or more.

·        Up to a few thousand forwarders feeding load-balanced data to the indexers.

Large enterprise deployment use case

A large enterprise deployment provides even greater horizontal scaling.

Characteristics of this type of deployment include:

·        Indexing volume ranging from 300GB to many TBs per day.

·        A large number of users, potentially numbering in the several hundred.

·        Many thousands of forwarders.

 

http://docs.splunk.com/Documentation/Splunk/7.0.3/Deploy/Deploymentcharacteristics

 

Design considerations also change as the deployment scales.
This table summarizes some of the issues you need to consider when designing your deployment.
Departmental Small enterprise Medium enterprise Large enterprise
Forwarder issues Management, monitoring Load balancing, management, monitoring Load balancing, management, monitoring, intermediate forwarders Load balancing, management, monitoring, intermediate forwarders
Search issues User counts, alerts, apps Search head/indexer knowledge management, user counts Search head/indexer knowledge management, user counts, search head clustering, job servers Search head/indexer knowledge management, user counts, search head clustering, job servers
Scheduled search workload Alerts, app/dashboard dependent, summary searches Alerts, app/dashboard dependent, summary searches Alerts, app/dashboard dependent, summary searches, job server Alerts, app/dashboard dependent, summary searches, job server, API/SDK
Input types Network, scripted Network, scripted, batch, integrations Network, scripted, batch, integrations Network, scripted, batch, integrations
Availability Platform-dependent (RAID, power supplies) Data fabric (forwarder load balancing, storage, index replication) User interface (search head clustering, load balancers); data fabric (forwarder load balancing, storage, index replication) User interface (search head clustering, load balancers); data fabric (forwarder load balancing, storage, index replication)
Recoverability Backup, retention Backup, index replication, bucket/index restoration Backup, index replication, bucket/index restoration Backup, index replication, bucket/index restoration
Accessibility Local vs. enterprise authentication Authentication method Authentication method Authentication method
Staffing Admin: 0.5-1 person; search/dashboard/appdev/ knowledge manager: 0.25-1 person Admin: 0.5-1 person; search/dashboard/appdev/ knowledge manager: 0.5-1.5 persons Admin/architect: 1-2 persons; knowledge manager: 0.5-2 persons; search/dashboard/appdev: 1-3 persons; program/project manager: 1 person Admin: 2-4+ persons; architect: 1+ persons; knowledge manager: 2-5+ persons; search/dashboard/appdev: 2-6+ persons; program manager: 1 person; project manager: 0.5-2 persons

 

Departmental Small enterprise Medium enterprise Large enterprise
Indexing volume (daily) 0-20GB 20-100GB 100-300GB 300GB-1TB+
# of forwarders Median < 10; maximum 100 Median in the 10’s; maximum in the 100’s Median in the 10’s; maximum in the low 1000’s Median in the 10’s; maximum in the 1000’s
# of users Median < 10 Median in the 10’s Median in the 10’s; maximum in the low 100’s Median in the 10’s; maximum 500+
# of apps (pre-packaged and customer-developed, combined) 1-10 1 – 10 1 – 20+ 10 –  50
Indexing tier 1 indexer 2-3 indexers, possibly in a cluster 4-9 indexers, possibly in a cluster 10+ indexers, possibly in a cluster
Search management tier Combined with indexer 1 standalone search head 3 search heads in a cluster 3+ search heads in a cluster
Configuration management function Manual configuration or deployment server Manual configuration or deployment server Deployment server or 3rd party tool for forwarders and indexers. Deployer for search head cluster. Deployment server or 3rd party tool for forwarders and indexers. Deployer for search head cluster.
Please follow and like us: