SPLK-2002 Splunk Enterprise Certified Architect – Indexer Clustering Part 4

7. Configuration Bundle – Part 02

Hey everyone and welcome back. Now in the earlier video we were discussing the basics about what the configuration bundle was all about and how you can push custom configuration like for example indexes cones to the peer nodes from the master indexer. So continue my journey with the configuration bundle. In today’s video we’ll look into some more features of configuration bundle. So last time we had pushed indexes conf, this time we will be pushing certain custom add ons to the peer indexes and look into how exactly it would work. So in order to do that, let’s go to find more apps and we’ll type Linux over here so we’ll go ahead and install the Splunk add on for Unix and Linux perfect, so it is installed and I’ll quickly restart so until the time it gets restarted let’s go to the CLI.

So I’m in my splunk IDX one. This is from the earlier demo. We’ll move it to the midx one server because this is where our configuration bundle was pushed from. So typically if you go to etc opt Splunk etc apps, and if you do an LS, you would see that our Unix add on you have your Splunk tanks is installed. Now what we want to do is we want to push this specific add on to all the peer nodes. Now we already know the directory where we need to place which is the master Hyphen apps directory so let’s do a copy r for splunk tanks and we’ll put it to opt splunk etc master app.

So now that we have it, let’s quickly do a LS Hyphenl opt plunk etc master apps so now we have our splunk underscore Tanscore nix add on which is present under the master apps. So once we have this let’s do a login to our indexer master node and this time we’ll go to settings indexer clustering we’ll go to edit configuration bundle actions and now let’s do a validate and check restart and now it basically got to validate it successfully. You see, it gives an error saying that this is not critical but no spec file for opt splunk etc master apps plunk tanging defaultevengen Conner was found so this is a little warning, kind of a message we’ll ignore that for now and let’s go ahead and we’ll be pushing the changes. Perfect. So the push has been successful, so let’s quickly validate it. So let me log into the peer node one and now you see you have Splunk add on for Unix and Linux which is pushed now in case many times it happens that whatever you push it does not really work out and things break.

So you also have functionality for rollback. So typically every bundle that you push over here, it has a bundle ID and you can also see that there is a previous bundle ID so you can always roll back in case the changes does not work out well. So let’s look into the rollback functionality. I’ll click on rollback and let’s say rollback changes. Perfect. So the rollback is now completed and you see the current active bundle ID is four three, a four and AC 33 was our previous bundle ID. Now, after rollback, let’s log into the peer node once again. And now you see whatever add on that we had pushed, it has been rolled back and it has been removed from the indexer node.

So this is what basically the rollback functionality is all about. Again, in production it would be required, depending upon the add on and the architecture you have, that you would have to typically push certain addons to the indexer peer nodes. And this is how you can do that. And in case if things do not go well, you can even do a rollback. So that’s about it about the second video of the configuration bundle Actions. I hope this has been informative for you and look forward to seeing the next video.

8. Forwarding Logs to Indexer Cluster

Hey everyone and welcome back. In today’s video, we will look into how we can forward logs from the universal forwarder to the indexer cluster based architecture setup. Now, once we have the indexer cluster built, universal forwarder would need to start sending logs to the peer indexer nodes. Now, we already know the difference between master indexer and the peer indexer. So basically, currently we already have the indexer clusterbased setup, which is already done. So we would like a universal forwarder to start sending logs to the indexer cluster.

Now, in order to do that, there are two ways. One is with the help of the index discovery feature, and second is by directly specifying the list of the peer indexer nodes within the configuration file. So in today’s video, we’ll look into the second aspects where we can connect the forwarders directly to the peer nodes, and then we’ll look into the indexer discovery feature. So currently, if you look into my master indexes indexer clustering console, you will see that everything seems good. I have two peer nodes. One is Splunk IDX zero one and Splunk IDX two. The only thing that we do not have right now is the forwarders, which can start to send locks to the indexer cluster. So what we’ll do, I’ll go to my terminal and if we do a docker PS, I have the three containers up and running.

One is MITx and two other indexer PS. The only thing that we are missing right now is the forwarders. So let’s do a docker images and let’s launch one more container based on the center six repo. So I say docker run hyphen hyphen name. I gave it forwarder three. The host name that we’ll assign is also the same forwarder three. And we’ll be launching this from the cento west container repo. Perfect. So this is the cento S image that we have currently. And if you do a docker TS, you should see that there is one container which is up and running. So let’s quickly log in there. Perfect. So currently we have logged in and I’ll just install few package. Perfect. So now that we have this installed, we need to install few more packages. wgat is one and Mano is something that is optional for you, but it is mandatory for me because this is one of my preferred text editor.

So once you have this installed, what we can do is you can follow the universal forwarder installation. So this is something that we had looked in the previous video. So what we’ll do is we’ll be installing the Splunk universal forwarder within the container that we have created. Great. So once this is downloaded, you can just go ahead and install the Splunk forwarder. All right. So now if you go to opt Splunk bin just towards Splunk status, it will basically ask you for the licensing agreement. You can just press so in case you want to go down, you can press space so by pressing space the license agreement will go down and then you can press Y to accept the license, also give the username and password. Great and that’s about it. So let’s go ahead and start as plunk instance and once you have done that, we’ll follow a few more things like we’ll be adding this plunk add monitor dialog.

So in case you do not know this, I would really recommend you to go through the earlier videos where we were discussing about universal forwarders. So let’s go ahead and run this where we’ll say splunk add monitor wire log and you’ll have to give the username and password that you have just set. So you have wire log added and then you need to add the forward server.

Now, this forward server in our case needs to be the two peer nodes which are part. So basically our cluster consists of two peer nodes and it is recommended to add both the peer nodes because if one peer note goes down, the universal forwarder will still be able to send data to the second peer node so that the data will always be available for searching. So in order to do that, let’s do one thing. I’ll come out of the docker container by pressing control D and let’s do a docker inspect. I’ll say Splunk IDX one and the container IP address is 170 2170 three.

Let’s look into one more 170 2170 four. All right, so now we can go back to our forwarder three container. We’ll go to opt plunk forwarder bin and we’ll basically go ahead and add our forwarder server which is Splunk. At forwarder server you have to specify the IP address. So let’s do that. Splunk at forwarder server 170 2170 four. So this is important, you need to make sure that you also given the receiving port. So once you do this, you can press Enter. So this is one server, you also need to specify the second because we have two peer indexes.

You need to specify the IP address of both the peer indexes. Once you have done that, you can go ahead and you can do a Splunk restart. All right, so Splunk has been restarted. So before we continue further, let’s look into the configuration file so as the etc system local and there would be a file called outputs conve as well as inputs conve. So if you look into the inputs conve, it basically will have host is equal to forward a three and outputs convey will basically contain the IP address of the peer indexes that we had configured which is 170 217 0497 and 170 2170 three seven. So these are the IP addresses of both the peer indexes. So now you once you have done that, let’s go into opt Splunk forwarder where log splunk and we are more interested in a file called Splunkdot log.

So I’ll quickly do a tail on Splunkdot log and basically now you see you should typically get a TCP out proc saying that it has connected. So if you quickly want to verify, let’s go to Splunk Enterprise. I’ll go to search and reporting app. Let’s do a skip. And within the data summary you should see that there is the host call as forward as three. And this host is basically sending all the logs to your Splunk instance. So before we conclude, let’s do certain test cases, right? Because it might happen that currently you have two indexes which are available. It might happen that one of the peer indexers can go down at any moment of time. So let’s bring down one of the peer indexes and see whether Splunk forwarder will still be able to forward logs or not. So currently it was connected to 170 2170 four within the log file.

And here you see it is also connected to 170 to all right, so basically what we’ll do is we’ll bring down one index pier. So let’s do a docker PS I’ll log into one of the index, appear, let’s say IDX two and I’ll do a bash and let’s do opt bin blank stop. Perfect. So now one of the indexer peer has stopped working and what we’ll do is we’ll login back to our forwarder server, I’ll say forwarder zero three bash and we’ll we’ll try and generate certain locks to verify whether they are coming in Splunk or not. So I’ll do Yum by install, let’s say Apple release oops, great.

So now we have one package downloaded. So basically this would typically come. So if you look into wire log and if you do a tail on Yum log, let’s say you will see that you have the FL release package which is available over here. Now along with that, if you go to Splunk forwarder, let’s go to wire log and do a Splunk Splunkd log. Here it says that it tried to connect to 170 2170 four. The connection has been refused, so it is not able to connect to the one peer indexer. So let’s quickly verify whether the log which was part of PM log is still visible within the Splunk instance. So in order to do that, if you see over here currently it says that Splunk IDX two is stopped.

It has the status of stop. Now all the buckets are available from Splunk IDX one instance. So let’s go to the Splunk Enterprise. We’ll typically go to the search and reporting app and within the data summary we’ll go to the forwarder three. And typically now you see, it says that an Apple release package has been installed. So that basically means that even though one of the peers were down, the Splunk Universal forwarder were still able to send the data to one more peer which was up and running. So this is the basics about sending logs from universal forwarder to the indexer cluster. I hope this video has been informative for you and I look forward to seeing you in the next video.

9. Indexer Discovery

Hey everyone and welcome back. In today’s video we will be discussing about the indexer discovery method. Now, with the indexer discovery method, what happens is that the universal forwarder, basically they query for the master node for the list of peer nodes which are within the cluster. So we do not hardcore the list of peer indexer node. We query the master indexer, we get the list and then we send the data to the list which the master indexer will provide us. It then uses the load balancing to forward the data across the set of peer nodes. So this can be explained with a simple animation that I have created, where on the left hand side, let’s consider this as a laptop which has a universal forwarder. This can be server as well. And on the right hand side we have an indexer cluster architecture, where you have a master indexer here and you have three set of VR nodes.

So with the indexer discovery method, what happens is first the universal forwarder will send a query to the master indexer which states can you share the list of indexer nodes within the cluster? So now the master indexer nodes knows that there are three indexer peers which are present. So it will go ahead and it will send a list of all the three peers saying this is splunk IDX one, splunk IDX two, splunk IDX three. So these are the list of three peers. So now, once the universal forwarder receives that these are the peer nodes which are present within the index or cluster, it will go ahead and send the data to all of the peer nodes. Now, one of the benefit of this approach is that after one week it might happen that there are seven or there are ten sets of peer nodes.

So in such cases, the static configuration within the universal forwarder need not be updated. It can get the dynamic list of the amount of peers which are present and then it can start to send the data to all the peers. This is one of the recommended way in which the data should be sent. So now, coming back to the CLI, if you do a docker PS, you would see that I have launched one more universal forwarder based docker container. So this is forwarder four. Now, in the videos where we were doing static listing of the peer indexes, we had created forwarder three, in the same way I have created forwarder four and I have installed the basic package, including in its scripts WKIT and Nano. So let’s go to the forwarder four. And now what we can do is we can go ahead and we can perform the basic universal forwarder installation steps.

So now that we have it downloaded, I’ll go ahead and I’ll start a universal forwarder installation. Perfect. So once it is installed, we can go to opt Splunk forwarder, we can go to Bin, just do a Splunk status it will ask you for the license agreement, press space and press Y. You’ll need to put the username and password for the forwarder and perfect.

So now the license agreement has been initiated. So now you can go ahead and do a Splunk start. Great. So now the universal forwarder installation has completed. So basically for indexer discovery to work, there are two configuration steps that you will need to take. First is you need to inform master indexer about the configuration or I would say enablement of the indexer discovery. So we have to enable the feature of indexer discovery in the master indexer node. Once that is done, we need to also put the configuration in the universal forwarder configuration file.

Now I have a simple document for them. So in order to enable the indexer discovery feature, basically you need to set a passport same key. So you will have to put this stanza in the indexer master node and once you have done this, you’ll have to specify the tanza which is mentioned here in the universal forwarder node.

So let’s look into how that would work. So before we do that, let’s log into the master indexer node which is Splunk MITx one bash. Now within opt Splunk etc system local. Now we’ll have to put it under the server connect configuration. All right, nano seems not to be there. So let’s go a bit down and I’ll paste it here. So basically what we are doing is we have a stands off indexer discovery and basically how things would work is when the universal forwarder sends this request to the master indexer, it needs to authenticate against it and hence you have field call as password simply. So here we’ll specify a simple password called as password and we can go ahead and we can save it. Once we have saved this, let’s quickly restart splug. Perfect. So now that it has been restarted and if you quickly do a cat on server connex you will see whatever password that we had written in plain text. It is no longer in plain text and it is in an encrypted format. So this is one important part to remember here. Perfect. So now we have added the configuration within the master indexer.

We need to also tell the universal forwarder the details about the master indexer node as well as the authentication related password SIM key field values. So in order to do that, we’ll quickly log in to the forward as zero node four and this tank will go to opt Splunk. Let’s go to etc system local and we’ll create a file called connex and within the outputs conve we’ll put a simple configuration. I’ll be posting this document, you can try it out yourself. So this is a very simple configuration that we have here. So if you will see you have indexer underscore discovery you are specifying master. Then you have the passport synthe. You are specifying what is the password that will be used for authenticating the request. Then the master Uri you specify the master uri of the master indexer. Whatever is the IP address of the master indexer you can specify over here and you have a TCP out group one where you have an indexer discovery mode of master.

So this is the master mode. So this is the simple configuration that is needed. We can go ahead and we can do an opt splunk forwarder bin splunk restart. Now before we do that, we need to ensure that we have a proper inputs configuration. So if you have anything to monitor like wire log or any other configuration log files then you can specify it over here. So in my case, let’s quickly do it. So we have already seen on what exactly the stanza means in the previous videos. So we’ll quickly do a splunk or let’s go to opt splunk bing and let’s run the same command where we are doing a splunk ad monitor on warlock directory. Now, once you have done this, we do not need to add the forwarder server over here because this time we are doing an index discovery method and we have manually added the configuration within the outputs conve.

So now once you have done that you can go ahead and do a splunk restart. So our splunk has been restarted. So to just generate some sample requests, what I’ll do is I’ll download some package like Epel release. So now that we have our sample package installed, we know that this will go to the Yum log file if we now log into the Splunk enterprise instance. So this is the master indexer and if you go to search and reporting within the data summary you should typically see the forwarder four coming and it has the log related to Apple release. So now we have the logs which are coming in to the indexer peer nodes through the indexer discovery method. So that’s about it, about how indexer discovery method works. I hope this video has been useful for you and I look forward to seeing the next video.