Deploying Splunk on AWS

 When deploying a non-clustered environment, either single-server or distributed, we recommend utilizing EBS volumes and EBS-optimized instance types.

  • An EBS volume is persistent, even in the case of instance termination or crash. Each of these options come with their own feature set and price points. Exact storage selection will depend on a customer’s desired requirements and price sensitivity.
  • When using Splunk premium solutions, such as Splunk Enterprise Security (ES) or Splunk® IT Service Intelligence (ITSI), we recommend indexer instance types with a larger memory footprint. The following assumptions also assume EBS gp2 volumes attached.
  • Splunk provides an add-on that will automate ingest data from several AWS services, including Config and Config Rules, CloudTrail, CloudWatch, Inspector, CloudFront and more. Installing and configuring the add-on will allow data collection from any of the services a user selects directly to the user’s Splunk Enterprise deployment.

The assumptions below assume EBS gp2 volumes attached to the c4 instance types. The d2 instance types come with instance storage, so EBS is not required. In all situations, deploying on dedicated hosts to avoid potentially noisy neighbor situations is recommended.

 Indexers
Instance Type Daily Indexing Volumes (GB) 
(c4 or d2).2xlarge < 100
(c4 or d2).4xlarge 100 – 200
(c4 or d2).8xlarge 200 – 300 +
 Search Heads
Instance Type Concurrent Users Performance
(r4.4xlarge Up to 8 Good
(r4.8xlarge) Up to 16 Better
Deployment Server, License or Cluster Master
(c3.2xlarge ) Good
(c3.4xlarge) Better

 

 

Small-Scale Deployment

 

Medium-Scale Deployment

 

Large-Scale Deployment

 

Indexing up to 100 GB/day

Six concurrent searches

 

Indexing up to 500 GB/day

Search and load up to 8 to 16 users

Indexing up to 1TB/day

With concurrent search load for 16 users

 

1 – c4.4xlarge with EBS-backed storage • 3 – c4.8xlarge with EBS-backed storage (Indexers)

 

• 5 – c4.8xlarge with EBS-backed storage (Indexers)
1 – c4.8xlarge with EBS-backed storage (Search Head) • 1 – c4.8xlarge with EBS-backed storage (Search Head)

 

• 1 – c3.xlarge (License Master) • 1 – c3.xlarge (License Master)

 

 

BEST PRACTICE

Custom AMI creation

  • Create your own AMI using Linux based or Splunk provided
  • Leverage current configuration tooling with AMI (don’t have to use deployment server, but can be very helpful)

Search Head Clustering

  • Deploy to the same AWS Region
  • Replication and searches across Regions can be a challenge
  • Use a Virtual Private Cloud (VPC)

 

Please follow and like us: