Building a Multi-node Vagrant Splunk Environment, Part 1

By 13 September 2016Blog, Tech Tips

Multi-node Vagrant Splunk Environment

By Michael O’Leary, Senior Consultant

Overview

As a part of our daily consulting practice, Avocado engineers work with many clients at different stages of DevOps maturity. For clients in the early stages of their DevOps journey, we recommend creating several environments prior to getting to production. These environments need to be repeatable and should mimic environments up the chain to production. A common set of environments for releasing application and automation code consists of:

  1. Local Development Environment.
  2. Shared Development Environment.
  3. Testing Environment.
  4. Stage Environment.
  5. Production Environment.

For the purposes of this article we are going to focus on creating a multi-node development environment for a Splunk architecture. This is useful for DevOps/Splunk engineers developing on their local workstation or laptop. For this we are going to create a standard Splunk infrastructure (Cluster Master, two Search Head and two Indexers). In addition, we are going to use an automation technology to provision the servers and ensure the infrastructure is immutable. For this reason I have also created a management server called “mgmt”, this will allow Ansible playbooks to be run within the environment for users on Windows operating systems.

Note: Feel free to follow along step by step (however I haven’t documented everything here) but all this code will be released publicly on GitHub. The architecture (which will be a plain vanilla install of the Splunk with some clustering configured) will look like below.  This meets the minimum requirements of a Splunk Cluster with data replication.

cluster

For this article we are going to the used the following technologies:

  1. Virtualbox – An open source virtualisation software for creating virtual machines.
  2. Vagrant – A wrap tool for orchestrating the creation and configuration machine of virtual machines.
  3. Ansible – A powerful, self-documenting automation tool.

Note: While I’m using the above tools, I don’t cover the installation of these tools. The installation of these tools is sufficiently covered in the documentation for the relevant toolset. See the links above to find the documentation.

Getting Started

Let’s start by creating a blank project directory and a Vagrantfile to house the configuration for our vagrant lab environment.

Starting in my Workspace directory, issue the following commands:

mkdir multinode-lab
cd multinode-lab
vagrant init base

If those commands worked correctly for you, you should now see the Vagrant file if you do a directory listing.

vagrantfile

Doing a cat of the file should reveal all the default configuration and comments that come along with a default generation of the Vagrant file.  

vagrantfile_with_comments

That looks a little bit busy so let’s tide it up by removing all the comments we don’t need.

✔ ~/Workspace/multinode-lab [master|✚ 1]
10:18 $ sed -i '' '/^.*#.*$/d' Vagrantfile
✔ ~/Workspace/multinode-lab [master|✚ 2]
12:38 $

Your Vagrant file should look something like this now.

vagrantfile_without_comments

There are multiple different vagrant box types that can be utilised, including Windows, Linux and Unix so there are plenty of options out there to meet your development needs. Vagrant boxes that have been created by the community are located at the following URL: https://atlas.hashicorp.com/boxes/search 

Instead of the base box, I’ve decided to use Centos 7 as the default box.  To do this I’ve just replaced the value of the config.vm.box to be centos/7 instead of base. I’ve also set the config.ssh.insert_key value to false.  The reason for this is marginally quick boot times and I don’t need to replace the default vagrant key as this is my local environment. 

Vagrant.configure("2") do |config|
 config.vm.box = "centos/7"
 config.ssh.insert_key = false
end

Ensure your Vagrantfile looks like the code above.

vagrantfile_with_centos_7

Now that I have my basic configuration lets test that we can load up our default box and ssh to it.

✔ ~/Workspace/multinode-lab [master|✚ 1]
09:10 $ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'centos/7'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'centos/7' is up to date...
==> default: Setting the name of the VM: multinode-lab_default_1473635489166_21269
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: No guest additions were detected on the base box for this VM! Guest
    default: additions are required for forwarded ports, shared folders, host only
    default: networking, and more. If SSH fails on this machine, please install
    default: the guest additions and repackage the box to continue.
    default:
    default: This is not an error message; everything may continue to work properly,
    default: in which case you may ignore this message.
==> default: Rsyncing folder: /Users/moleary/Workspace/multinode-lab/ => /vagrant
✔ ~/Workspace/multinode-lab [master|✚ 1]

 

vagrant_up_default

Everything looks good from the above screenshot apart from the fact that we don’t have guest editions installed.  That’s not a huge problem however it does limit us from mounting a shared volume between my host and guest machine. We can tackle that later as that is functionality I find useful.  In the meantime lets destroy the vagrant machine we have created.

✔ ~/Workspace/multinode-lab [master|✚ 1]
09:19 $ vagrant destroy
==> default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
✔ ~/Workspace/multinode-lab [master|✚ 1]

Install the Supporting Vagrant Plugins

Vagrant comes with a long list of plugins that extend it’s functionality. For this tutorial I need three plugins. One of these plugin is Landrush. Landrush provides local DNS resolution support within the vagrant environment. The others are the Ansible and the Shell provisioner, Ansible and Shell are the provisioners for our vagrant environment.  Ansible can be used to directly provision the vagrant boxes if you are using a Unix or Linux operating system.  If you are using a Windows operating systems then we can create an additional box to be used as a provisioner.

✔ ~/Workspace/multinode-lab [master|✚ 2]
11:49 $
vagrant plugin list
shell (0.0.1)
vagrant-aws (0.7.0)
vagrant-hostmanager (1.8.1)
vagrant-share (1.1.5, system)
✔ ~/Workspace/multinode-lab [master|✚ 2]

Let’s install the plugins using the following command

✔ ~/Workspace/multinode-lab [master|✚ 2]
11:49 $ vagrant plugin install landrush ansible
Installing the 'landrush' plugin. This can take a few minutes...
Installed the plugin 'landrush (1.1.2)'!
Installing the 'ansible' plugin. This can take a few minutes...
Installed the plugin 'ansible (0.2.2)'!
✔ ~/Workspace/multinode-lab [master|✚ 2]
12:00 $

For a list of vagrant plugin see the Github home page https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins.

Let’s go Multi-Node

Most architectures don’t consist of one box on a workstation. We are going to replicate the architecture mentioned above. I’ve added a sub directory “ansible/vars/cluster/guests/guests.yml” under our lab directory. In this directory I have added guests.yml to define some nodes. I’ve also added some ram values, so I can override the default ram value of 256MB.  In addition I’ve updated the Vagrant file to reflect the changes for the multi-node environment.

default_ram = '256'
default_domain = 'avocado.lab'
default_env = "cluster"
environment = ENV['ANSIBLE_ENV'] || default_env
environment_domain = "#{environment}.#{default_domain}"
require 'yaml'
guests = YAML.load(File.open(File.join(File.dirname(__FILE__), "ansible/vars/#{environment}/guests/guests.yml"), File::RDONLY).read)



Vagrant.configure("2") do |config|
 config.landrush.enabled = true
 config.landrush.tld = default_domain
 config.vm.box = "centos/7"
 config.ssh.insert_key = false



 guests.each do |name,options|
   config.vm.define name do |guest_config|
     hostname = [name, environment_domain].join('.')
     guest_config.vm.host_name = hostname
     memory = options['ram'] || default_ram
     guest_config.vm.provider :virtualbox do |v|
       v.name = hostname
       v.cpus = 1
       v.memory = memory.to_s
     end
   end
 end
end

In addition to configuring the nodes, I’ve used the Landrush plugin to provide me with DNS names, each node will have their name followed by the domain name.  Thus the FQDN of the nodes will be:

  • clm01.cluster.avocado.lab
  • shd01.cluster.avocado.lab
  • shd02.cluster.avocado.lab
  • idx01.cluster.avocado.lab
  • idx02.cluster.avocado.lab
  • ufw01.cluster.avocodo.lab
  • mgmt.cluster.avocado.lab

The contents of the guests.yml file is as follows:

---
clm01:
 ram: '512'
shd01:
 ram: '512'
shd02:
 ram: '512'
idx01:
 ram: '512'
idx02:
 ram: '512'
mgmt:
 ram: '512'

Running vagrant status will show the hosts define in the guests yaml file.

✔ ~/Workspace/multinode-lab [master|✚ 2]
14:18 $ vagrant status
Current machine states:
clm01 not created (virtualbox)
shd01 not created (virtualbox)
shd02 not created (virtualbox)
idx01 not created (virtualbox)
idx02 not created (virtualbox)
ufw01 not created (virutalbox)
mgmt not created (virtualbox)



This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
✔ ~/Workspace/multinode-lab [master|✚ 2]

Running vagrant up command will start all three nodes

✔ ~/Workspace/multinode-lab [master|✚ 2…41]
08:56 $ vagrant up
Bringing machine 'clm01' up with 'virtualbox' provider...
Bringing machine 'shd01' up with 'virtualbox' provider...
Bringing machine 'shd02' up with 'virtualbox' provider...
Bringing machine 'idx01' up with 'virtualbox' provider...
Bringing machine 'idx02' up with 'virtualbox' provider...
Bringing machine 'ufw01' up with 'virtualbox' provider...
Bringing machine 'mgmt' up with 'virtualbox' provider...
==> clm01: Importing base box 'centos-7.2-64-base'...
==> clm01: Matching MAC address for NAT networking...
==> clm01: Setting the name of the VM: clm01.cluster.avocado.lab
==> clm01: Clearing any previously set network interfaces...
==> clm01: [landrush] virtualbox requires an additional private network; adding it
==> clm01: Preparing network interfaces based on configuration...
    clm01: Adapter 1: nat
    clm01: Adapter 2: hostonly
==> clm01: Forwarding ports...
    clm01: 22 (guest) => 2400 (host) (adapter 1)
==> clm01: Running 'pre-boot' VM customizations...
==> clm01: Booting VM...
==> clm01: Waiting for machine to boot. This may take a few minutes...
    clm01: SSH address: 127.0.0.1:2400
    clm01: SSH username: vagrant
    clm01: SSH auth method: private key
    clm01: Warning: Remote connection disconnect. Retrying...
==> clm01: Machine booted and ready!
==> clm01: Checking for guest additions in VM...
==> clm01: Setting hostname...
==> clm01: Configuring and enabling network interfaces...
==> clm01: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
           [landrush] Using enp0s8 (172.28.128.3)
==> clm01: [landrush] adding machine entry: clm01.cluster.avocado.lab => 172.28.128.3
           [landrush] Using enp0s8 (172.28.128.3)
           [landrush] Host DNS resolver config looks good.
==> clm01: Mounting shared folders...
    clm01: /vagrant => /Users/moleary/Workspace/multinode-lab
==> shd01: Importing base box 'centos-7.2-64-base'...
==> shd01: Matching MAC address for NAT networking...
==> shd01: Setting the name of the VM: shd01.cluster.avocado.lab
==> shd01: Clearing any previously set network interfaces...
==> shd01: [landrush] virtualbox requires an additional private network; adding it
==> shd01: Preparing network interfaces based on configuration...
    shd01: Adapter 1: nat
    shd01: Adapter 2: hostonly
==> shd01: Forwarding ports...
    shd01: 22 (guest) => 2410 (host) (adapter 1)
==> shd01: Running 'pre-boot' VM customizations...
==> shd01: Booting VM...
==> shd01: Waiting for machine to boot. This may take a few minutes...
    shd01: SSH address: 127.0.0.1:2410
    shd01: SSH username: vagrant
    shd01: SSH auth method: private key
    shd01: Warning: Remote connection disconnect. Retrying...
==> shd01: Machine booted and ready!
==> shd01: Checking for guest additions in VM...
==> shd01: Setting hostname...
==> shd01: Configuring and enabling network interfaces...
==> shd01: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
           [landrush] Using enp0s8 (172.28.128.4)
==> shd01: [landrush] adding machine entry: shd01.cluster.avocado.lab => 172.28.128.4
           [landrush] Using enp0s8 (172.28.128.4)
           [landrush] Host DNS resolver config looks good.
==> shd01: Mounting shared folders...
    shd01: /vagrant => /Users/moleary/Workspace/multinode-lab
==> shd02: Importing base box 'centos-7.2-64-base'...
==> shd02: Matching MAC address for NAT networking...
==> shd02: Setting the name of the VM: shd02.cluster.avocado.lab
==> shd02: Clearing any previously set network interfaces...
==> shd02: [landrush] virtualbox requires an additional private network; adding it
==> shd02: Preparing network interfaces based on configuration...
    shd02: Adapter 1: nat
    shd02: Adapter 2: hostonly
==> shd02: Forwarding ports...
    shd02: 22 (guest) => 2411 (host) (adapter 1)
==> shd02: Running 'pre-boot' VM customizations...
==> shd02: Booting VM...
==> shd02: Waiting for machine to boot. This may take a few minutes...
    shd02: SSH address: 127.0.0.1:2411
    shd02: SSH username: vagrant
    shd02: SSH auth method: private key
    shd02: Warning: Remote connection disconnect. Retrying...
==> shd02: Machine booted and ready!
==> shd02: Checking for guest additions in VM...
==> shd02: Setting hostname...
==> shd02: Configuring and enabling network interfaces...
==> shd02: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
           [landrush] Using enp0s8 (172.28.128.5)
==> shd02: [landrush] adding machine entry: shd02.cluster.avocado.lab => 172.28.128.5
           [landrush] Using enp0s8 (172.28.128.5)
           [landrush] Host DNS resolver config looks good.
==> shd02: Mounting shared folders...
    shd02: /vagrant => /Users/moleary/Workspace/multinode-lab
==> idx01: Importing base box 'centos-7.2-64-base'...
==> idx01: Matching MAC address for NAT networking...
==> idx01: Setting the name of the VM: idx01.cluster.avocado.lab
==> idx01: Clearing any previously set network interfaces...
==> idx01: [landrush] virtualbox requires an additional private network; adding it
==> idx01: Preparing network interfaces based on configuration...
    idx01: Adapter 1: nat
    idx01: Adapter 2: hostonly
==> idx01: Forwarding ports...
    idx01: 22 (guest) => 2420 (host) (adapter 1)
==> idx01: Running 'pre-boot' VM customizations...
==> idx01: Booting VM...
==> idx01: Waiting for machine to boot. This may take a few minutes...
    idx01: SSH address: 127.0.0.1:2420
    idx01: SSH username: vagrant
    idx01: SSH auth method: private key
    idx01: Warning: Remote connection disconnect. Retrying...
==> idx01: Machine booted and ready!
==> idx01: Checking for guest additions in VM...
==> idx01: Setting hostname...
==> idx01: Configuring and enabling network interfaces...
==> idx01: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
           [landrush] Using enp0s8 (172.28.128.6)
==> idx01: [landrush] adding machine entry: idx01.cluster.avocado.lab => 172.28.128.6
           [landrush] Using enp0s8 (172.28.128.6)
           [landrush] Host DNS resolver config looks good.
==> idx01: Mounting shared folders...
    idx01: /vagrant => /Users/moleary/Workspace/multinode-lab
==> idx02: Importing base box 'centos-7.2-64-base'...
==> idx02: Matching MAC address for NAT networking...
==> idx02: Setting the name of the VM: idx02.cluster.avocado.lab
==> idx02: Clearing any previously set network interfaces...
==> idx02: [landrush] virtualbox requires an additional private network; adding it
==> idx02: Preparing network interfaces based on configuration...
    idx02: Adapter 1: nat
    idx02: Adapter 2: hostonly
==> idx02: Forwarding ports...
    idx02: 22 (guest) => 2421 (host) (adapter 1)
==> idx02: Running 'pre-boot' VM customizations...
==> idx02: Booting VM...
==> idx02: Waiting for machine to boot. This may take a few minutes...
    idx02: SSH address: 127.0.0.1:2421
    idx02: SSH username: vagrant
    idx02: SSH auth method: private key
    idx02: Warning: Remote connection disconnect. Retrying...
==> idx02: Machine booted and ready!
==> idx02: Checking for guest additions in VM...
==> idx02: Setting hostname...
==> idx02: Configuring and enabling network interfaces...
==> idx02: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
           [landrush] Using enp0s8 (172.28.128.7)
==> idx02: [landrush] adding machine entry: idx02.cluster.avocado.lab => 172.28.128.7
           [landrush] Using enp0s8 (172.28.128.7)
           [landrush] Host DNS resolver config looks good.
==> idx02: Mounting shared folders...
    idx02: /vagrant => /Users/moleary/Workspace/multinode-lab
==> ufw01: Importing base box 'centos-7.2-64-base'...
==> ufw01: Matching MAC address for NAT networking...
==> ufw01: Setting the name of the VM: ufw01.cluster.avocado.lab
==> ufw01: Clearing any previously set network interfaces...
==> ufw01: [landrush] virtualbox requires an additional private network; adding it
==> ufw01: Preparing network interfaces based on configuration...
    ufw01: Adapter 1: nat
    ufw01: Adapter 2: hostonly
==> ufw01: Forwarding ports...
    ufw01: 22 (guest) => 2450 (host) (adapter 1)
==> ufw01: Running 'pre-boot' VM customizations...
==> ufw01: Booting VM...
==> ufw01: Waiting for machine to boot. This may take a few minutes...
    ufw01: SSH address: 127.0.0.1:2450
    ufw01: SSH username: vagrant
    ufw01: SSH auth method: private key
    ufw01: Warning: Remote connection disconnect. Retrying...
==> ufw01: Machine booted and ready!
==> ufw01: Checking for guest additions in VM...
==> ufw01: Setting hostname...
==> ufw01: Configuring and enabling network interfaces...
==> ufw01: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
           [landrush] Using enp0s8 (172.28.128.8)
==> ufw01: [landrush] adding machine entry: ufw01.cluster.avocado.lab => 172.28.128.8
           [landrush] Using enp0s8 (172.28.128.8)
           [landrush] Host DNS resolver config looks good.
==> ufw01: Mounting shared folders...
    ufw01: /vagrant => /Users/moleary/Workspace/multinode-lab
==> mgmt: Importing base box 'centos-7.2-64-base'...
==> mgmt: Matching MAC address for NAT networking...
==> mgmt: Setting the name of the VM: mgmt.dev.avocado.lab
==> mgmt: Clearing any previously set network interfaces...
==> mgmt: [landrush] virtualbox requires an additional private network; adding it
==> mgmt: Preparing network interfaces based on configuration...
    mgmt: Adapter 1: nat
    mgmt: Adapter 2: hostonly
==> mgmt: Forwarding ports...
    mgmt: 22 (guest) => 2222 (host) (adapter 1)
==> mgmt: Running 'pre-boot' VM customizations...
==> mgmt: Booting VM...
==> mgmt: Waiting for machine to boot. This may take a few minutes...
    mgmt: SSH address: 127.0.0.1:2222
    mgmt: SSH username: vagrant
    mgmt: SSH auth method: private key
    mgmt: Warning: Remote connection disconnect. Retrying...
==> mgmt: Machine booted and ready!
==> mgmt: Checking for guest additions in VM...
==> mgmt: Setting hostname...
==> mgmt: Configuring and enabling network interfaces...
==> mgmt: Landrush IP not installed in guest yet (or it's an outdated version). Installing now.
          [landrush] Using enp0s8 (172.28.128.7)
==> mgmt: [landrush] adding machine entry: mgmt.dev.avocado.lab => 172.28.128.7
          [landrush] Using enp0s8 (172.28.128.7)
          [landrush] Host DNS resolver config looks good.
==> mgmt: Mounting shared folders...
    mgmt: /vagrant => /Users/moleary/Workspace/multinode-lab
✔ ~/Workspace/multinode-lab [master|✚ 2]
14:36 $

You should now have servers virtual machines that can be pinged using their corresponding DNS names.

virtualbox

In Part 2 we will explore the integration of the Ansible playbooks to assist with on the fly provisioning of the virtual machine.

Related Posts