SPLK-1002 Splunk Core Certified Power User – Splunk Indexer And Search Head Clustering Part 4

16. Handson Multisite Indexer Clustering : Part 03

As you can see, this screen is similar to the step that we have done as part of a single site cluster where you saw, as soon as we configured the master node, it reported itself as a search engine. So now we need to add peers to this master. In order to add peers, there are two methods. One, you can use splunk cli, but it’s better to understand splunk configuration files so that you will know what configuration are changed and how they are placed. We’ll go to System Local server. com same file as we did for single site clustering.

As you can see, we have disabled the single site clustering. Now we will add the configuration that is necessary as part of our multi site clustering. That is, these are the configuration that are required for our multi site clustering. You need to change this IP as per our indexer cluster master. As you can see, we have added these lines here. We are mentioning this as site one. That is indexer one and cluster master. Both are in the same site and the replication port. As we all know, this port is required for copying data in between the cluster members.

We are defining the same port as we defined for our single site. That is 9080 and the master uri the mode will be slave here the multi site configuration and the single site configuration. The only difference is we are defining the site name.

I’ll copy the same configuration and change the site name to my other indexer so that it simulates that it is in a different site. In order to verify our indexer cluster, I’ll go to my second instance system localserver. com. You can edit this configuration using any editing tools. That is, text editors, GE editor, nano, vi, notepad, plus plus anything you feel comfortable. You can edit and copy these files and restart us. Blank instance should totally work fine. As you can see, I just changed side to side to rest everything. It is same. Our first indexer is about to start up. Okay, that successfully started. Now let me go ahead and restart my second instance.

So once these two are restarted, you should be able to see as part of your cluster indexing clustering master information where peers will be two and the site information will be displayed right next to them. That is our site one and site two. Our indexer two is also up. Now, as you can see, we have two sites which we have simulated splunk to read two instances over a period of time. Once we have replication factor and the indexes which needs to be replicated are enabled, we’ll be able to see all these factors will be met. Replication factor, search factor and data. That is searchable.

17. Handson Search Head Clustering : Part 01

In this video, we’ll be seeing how we can configure search ed clustering in our Amazon aws instance. For the purpose of this tutorial, I have created two searched instances that are searched one and Search at two. Let us quickly log in and verify first, plunks services are running. It is running on the one searcher. That is our first searcher. Both the searches are up and running. Now let us log into their web console and have a look at it how they are visible before enabling search ed clustering. This is our search head set up web component search at Two.

Let us see our Search at one. Search at one seems to be up. What about search at two? This looks pretty normal. This is searched two. As you can see. Now this is a plain splunk in sensors. There are nothing installed. As of now, this is something that we can ignore because we configured this instance in the previous tutorials as a V fourer. So we can ignore this. So this instance also looks normal, including this one. As you can just keep a track on this settings menu because once we enable searched clustering, this whole menu will change. So as of now, this both looks fine. Now let us go back to our console. In order to configure search at clustering, we know the benefits and the added advantages of having the multi site clustering and single site clustering.

In this demonstration, we will be dealing with single site clustering of search ed in the later stage, where we’ll be building our complete splunk environment on Amazon aws, which includes multisite searched clustering, multisite indexer clustering with high availability on Amazon aws for configuring searched clustering, you need to first determine which will be your search captain. The search captain is nothing but the searcher which is responsible for taking care of all the configuration files, dashboards user files in sync with other search ads, that is the search at cluster members. If there are three search ads in a cluster, one of them will be your captain, whose responsibility is to make sure all the searches gets the copies of each data. In search at clustering.

The data that is replicated among the cluster members are dashboard reports and the user files and also application files. These are the common files which are replicated among your searched cluster members. Now let us say we’ll make our searched one as our captain.

You can make any search ed as captain. Not only search at one or any other preferable searches, any member can be acting as a search at captain. Before enabling captain, make sure all the members are initialized for search at clustering. Here we have two members in our cluster. We’ll make sure one by one they are initialized for search at clustering. In order to do that, we’ll be doing this enter searched clustering via cli, that is, plunged cli, which is much more efficient. And also we’ll go through how to edit configuration files in order to enable searched clustering.

18. Handson Search Head Clustering : Part 02

Go to splunk home. Bin splunk. This is the utility that we are using for the splunk cli. As you all know by now, this is the only utility we will be using throughout your splunk implementation or operations. For such a clustering we need to initialize this member. It is with init parameter that is searched cluster which stands for ssh cluster iphone config. In this part of the command we are seeing this search ed to initialize the configuration that are necessary for a search ed cluster. But we are still not staying which is the management uri.

The management uri will be your own splunk instance that is acting as searched. So this is search ed one. This is the url and the port number. We all know by now management port by default is 80 89. So this is the syntax for specifying management uri. Once you have set this, we have couple of other parameters which are necessary. As you know, in indexer cluster we have seen one of the most important factor is replication factor. The replication factor here we will be defining as just one copy should be more than enough for the replication so that each searcher holds one copy of the data. And for replication to occur, we need a replication port. In indexer we give 9090.

I believe we’ll give 90 91. This port number, it’s completely random. You can choose whichever the port you like. And as an optional parameter you can also mention a cluster label. This is just to identify your search ed cluster which is good to have. We’ll give it searched cluster number one. So once this command just copy this command, it will be useful in running other searched cluster members.

Hit enter. It will ask for your username and password if you have not already entered. Since my credentials are already entered for splunk user, it automatically executed the command. It says searched clustering has been initialized on this node. We must restart in order to see the effects of these changes. Let us go ahead and restart the instance. This is our search head one, which is our search at this one down. As you can see, our application port has been successfully initiated. 90 91. That means our searcher clustering configuration has been reflected. Let us verify it via gui.

Click on settings. As you can see here, lot many options have been disappeared. As you can see there are lot many options in the search ed two where we have not initialized for search ed cluster. But on the search head one, everything else has been disabled. Because these configurations are related to your indexers and every forwarder and other splunk component.

For search edge you need only these two. We’ll see how we’ll proceed and we’ll configure searched initialization requirements for our search. At number two I’ll copy the same command. This is our command. So here I’ll just change the IP address of the search ad. That is a search at two IP address. So here. As you can see, my splunk credentials are invalid as of now. So I’ll reenter my credentials. It was successful. Let us go ahead and restart our splunk instance.

19. Handson Search Head Clustering : Part 03

So this is our search ed two. Now we have made sure our cluster members that are searched cluster members are ready for searched cluster by initiating searched cluster configuration. Once this searched is up, we should be ready for final configuration of our search ad. So this process continues for any number of searches in an organization. Let’s say there are ten different searches. You need to make sure this command gets executed on ten different searches in Shearly or you can use your deployment server. We’ll come to that in a later part where we will see how we can use deployment server to configure these kind of settings across ten searches in a matter of minutes. Now we have initialized both of them. Let us log into our second searcher.

This is our second searcher. This also seems to be fine. Let us validate. As you can see, we now got a new menu search at clustering and a lot of the other menus have been removed. When you click on Search ed clustering you’ll be able to notice the search ed clustering is not yet ready because we have initialized all the cluster members of the search ads but we have not configured the captain yet. In order to configure the captain go to our search head which we would like to make it as captain, we need to run one more specific command for making our search ed as a captain of the cluster. In order to make this instance to act as captain, you need to run bootstrap command bootstrap with searched cluster should we underscore captain?

So this specifies this instance to make us captain and also we need to specify all the servers that are part of our searched cluster that is searched one. We need to completely specify the management uri of our search edge 80, 89 comma the second search. So before configuring the captain, this is how the search at clustering looks like. It says the search at clustering service is not ready and it is waiting for our captain. So let us make sure our search at captain is configured. So it says there is a syntax error. Let us quickly figure that one out. We have servers underscore list that was correct. Searched cluster underscore Captain no, it is iphone Captain.

So it is searched iphone Captain. So it says successfully bootstrapped this node as a captain with the given server. So all the servers that is your cluster members will be reporting to our cluster captain for configuration. As you can see in rgui, the splunk setup searched clustering is complete. We have a captain that is our search ed one. We have another instance that is our cluster member. We can transfer this role of captain at any moment of time. So presently our search ed one is acting as the captain. Similarly, you can have any number of search at cluster members but there will always be one captain managing the configuration replication as you can see, both searches reflect same configuration. Now, how do we validate this configuration?

20. Search Head Clustering Validation

Now how do we validate this configuration? So this is our search ed two. I’ll go ahead and create a user. Let us see what happens. So in our search ed one, if you go to access control under access control check for the users. As you can see as of now there is only admin and in search at two, there is only admin on the search at two which is my cluster member.

It is not even my captain. I’ll go ahead and create a new user. I’ll name this user as cluster user. This is not the cluster user, this is the cluster test user. I need to know whether this configuration will be replicated to my other members in the cluster. So I’ve just created a splunk cluster test user. I’ll go ahead and save this with some dummy password.

So it says it was successfully created and this mentions that it is a trial license and the user will be disabled. That should not be an issue. So this test user was successfully created. Now let us go to our other search ad and refresh this page. As you can see, the user has been automatically copied into other searches. So this is how your search ed clustering makes sure all the configuration that are created or modified in one searcher will be replicated among your cluster.

Even though if this search ed instance goes down, the user should be able to log in using this searched. One more way of verifying your cluster is let us go to our search search ed one. Yes, this is our search ed one. Let us try to create is dashboard.

So we have demo copy xml drill down in built visualization. We have lot many which we have gone through all this dashboards in our previous understanding about splunk. Now let us create one more dashboard. Before that, let us validate whether the same dashboards are present in our cluster member. Click on Dashboards and Dashboards you are able to see all the similar dashboard. I’ll go ahead and copy this. I’ll clone it and name it as demo clone for cluster test.

I’ll go ahead and save it not view this. As you can see there is demo clone for cluster test on a search ed one. If you go ahead and refresh this, you should be able to see your dashboard on the other search head. As you can see there is a demo clone for cluster test on your search ed. So this way your cluster makes sure all the configurations are replicated and maintained for a hip available and sharing of the load purpose. So this confirms our clustering was successful.