SPLK-2002 Splunk Enterprise Certified Architect – Indexer Clustering Part 3

5. Testing Replication and Failover capabilities

Hey everyone and welcome back. So in the earlier video, we had configured the entire architecture where we connected both of the peer nodes with the master indexer. Now in today’s video we will actually see whether the replication is actually working or not. Because we just configured this architecture, we have not tested it yet. So before we do that, typically if you you go into the indexer clustering, like this is the 8001 which is the peer node one. Now if you typically go inside the indexer clustering, you will see a page similar to this where it says that this is the name which is flung IDX one, the replication port is 8080, the master location is 170 2170. Two, that basically means that this node is acting as a peer node. You can even see it from here. And the master node is the one which is specified under the master location key.

Now along with that, I would quickly like to show you, let’s quickly log in to the splunk IDX one, which is the peer node one. And this time if we go to opt splunk ATC system, we’ll go to local and if you do a cat on server connect, you will see that there are two stanzas. One is the replication port of 80 80. This is something that we had configured and second stanza is related to the clustering aspect where you have the master Uri present over here. This time the mode is safe and this is the pass for SIM key which is basically used for authenticating the request in the communication that happens within the cluster.

So now let’s go ahead and test whether the things are actually working or not. So what we’ll do is we’ll go back to our splunk homepage and what we’ll be doing here is that I will be adding some data to the peer node one and we’ll see whether the data is also replicated to the pier node. Two, this is something that we’ll be looking in today. So now let’s go ahead and we’ll add a data. So we’ll do upload you select the file and basically I’ll be uploading the access log that we have within our tutorial data directory. I’ll do a next so it automatically detected the source type.

We’ll do a next and we’ll put it to the default index. We already know what the default index is and we’ll do a subject perfect. So now the data should be searchable. So now if you do a start searching now you typically see that the data is present. So what you basically did was you added data with a source type of access combined w cookie to the default index which is index of main. Now the same thing you will typically see within the master indexer. So you are within the indexer clustering. And now if you go to the indexes, you have index calls made and the searchable copy has greener and the replicated copy also has greener. So now let’s try it out whether we are able to see the data within the indexer node two or not. So I’m in my index and node two.

It ends with port 8002. And if you go to search and reporting, you see that there are 13,451 events. If you quickly look into the source, the source is access dot log. That means the data has been replicated successfully. You can again verify that host this time is plug hyphen IDX One, typically because we had uploaded it from the IDX One server. So this is the overview about the testing that I wanted to show you whether the replication is actually working or not.

Now, before we conclude this video, we’ll do one more testing. What we’ll do is we’ll bring down the indexer node one and we’ll verify whether the data is still searchable or not. So in order to do that, what we’ll do, I am already logging to the IDX One server. What I’ll do, I’ll log out or let me log in again and we’ll do opt Splunk Bin. We have to do a CD. And I’ll say splunk stop. So now that I have stopped my Splunk, if you go into the indexer clustering, you will see that it says that the search factor is not met, the replication factor is not met. And now instead of two, because we had two identical data copies, only one is green. Now, if you will see, only one is green and the second is not green. Now and again, let’s check whether you are able to see the data or not. Let’s open up the data. I’ll do it at all times. And currently the data is still accessible as usual.

The only problem here is that we only have one more indexer node where all the data is still present and you will be able to still search the data as the usual manner. Now let’s do one thing. Let’s start the indexer node once again. Or before we start it, let’s do one thing. This time we’ll add some data within the indexer node two. So let’s do that. So I’ll do add data. So we’ll try and cover all the test cases that can be done within the short demo that we have. And this time what we’ll do, we upload a file called as MySQL Underscore Error. I’ll do a quick Next, so event types. I’ll just do it as every line. Or you can just assign a right source type here, say MySQL Underscore Error.

All right, so you have assigned the right source type. You will do a quick next index. We’ll leave it as default. We’ll go ahead and will upload the data over here. So now what we have here is that we have two source types. One is the Apache and second is the MySQL. So we can quickly verify it on the data. Summary one is from the host IDX 1 second is from the host IDX two. You have two sources access log and MySQL log and source type access combined and MySQL D error. Perfect. So now what we’ll do is we’ll bring up the indexer node zero one and we’ll verify whether the data which was added in the indexer node two gets replicated to the indexer node one. All right, so this is one aspect that we’ll be checking. The number of buckets here is currently one and it only is not replicated. So you see it’s green and on the searchable and it’s one green on the replicated.

So with this, let’s go ahead and start our splunk instance on IDX one. So once the splunk instance is started, if you go back here, you will see that for main index you have two out of T two. However, for internal index it is still replicating the buckets. So the replication is still on for the internal. And basically what happens is that once the failed node comes up again, the replication of data will keep on happening. And once if everything gets replicated, you will have green on both the sides. That is something that needs to be expected.

So it will take little amount of time. So currently the main index is synced between both the nodes and audit and internal index will also get synced eventually. So the amount of time it might take for everything to be green depends upon the network connectivity, depends upon the network bandwidth, and it also depends upon the amount of data that you have that needs to be synced. For example, let’s say that the peer node one went down and you uploaded 200 GB of data in node two.

Now, once the node one comes up, then what will happen is all the 200 GB of data needs to be synced back to node one. So that syncing will take little amount of time. Again, that time depends upon various factors including networking bandwidth, also the amount of data that you might have. So now if you see everything is green, it took a little amount of time, but we have everything is green now. So this is the basics about the test cases that we performed on Index Clustering. I hope this video has been informative for you and I look forward to seeing you in the next video.

6. Configuration Bundle

Hey everyone, and welcome back. In today’s video we will be discussing about the indexer configuration bundle. Now, typically, when you have an architecture, something similar to this, where you have a master indexer and multiple peer nodes, one important aspect to remember is that you should not configure anything locally in the peer node because those aspects might not be replicated across of the other peer nodes which are part of this cluster. So let’s do one thing, let’s understand this with a practical aspect so that we know that what the configuration bundle is all about. So what I’ll do is currently you will see, I have Splunk Enterprise running and this is the first peer node. It’s starting with 8001. And what I’ll do is I’ll add certain data locally. So in order to do that, what I’ll do, I’ll create one index locally within the peer node one. So I’ll say the index name as Pier One and I’ll save this. All right? So now I have this. Index, pier One. Once I have this, I’ll go ahead and I’ll add the data in the new index that we have created.

I’ll do a select file and this time let’s add the Warlock secure file. I’ll do a next and let’s just say the source type as test for the timing and within the index. This time we’ll send it to the index call as peer one that we have created manually within one of the peers. And I’ll go ahead and I’ll do a submit. And if you quickly do start searching within the peer node one, you see I have all the data which is coming in. Now, one of the problem that has happened is that what we did was we created an index manually within the peer node one and within that index we started to add the data. Now, if you quickly wanted to verify, let me actually open the master indexer node and within this, if we go to indexer clustering within the indexes, you see that the peer one index has not been populated over here. Let me just quickly do a refresh and let’s see whether it gets populated and it has not. Now to double verify, what we’ll do is that I am in my Splunk Enterprise.

So this is the second peer node and let’s do index is equal to peer one. And I’ll set all time and I’ll do a search and you see here I am not seeing any data which is part of the peer one index. So typically what has happened was since we configured certain aspects locally, so we configured the indexes locally, it did not get replicated among the peer node which are part of the cluster. And hence it is recommended that whatever you do like if you want to push add ons, you want to push custom transforms, conve or indexes, it needs to be pushed from the master indexer to the peer nodes. This is the idle way of doing things. And this pushing of configuration files or add ons is called as the configuration bundle push. Now, understanding the configuration bundle, it is basically a set of configuration files as well as app which are common to all the peers.

Now, this set of configuration bundle is managed by the master indexer which is distributed to the peers via the bundle push operation. So anything that we do it needs to be done at the master indexer level. So let’s look into how we will handle the scenario where a custom index needs to be pushed. Let’s say you want to create a custom index. Now, instead of creating that index in the peer nodes manually, you can create it from the master indexer and you can push it to the peer nodes. And this is what we’ll be looking into in today’s video. So what we’ll do is this time we’ll go to the MITx one and if you go to opt Splunk we’ll go to etc. And this time within etc. We’ll be looking into a directory call as master hyphen apps. So let’s go to master hyphen apps and if you do LS over here there is a directory call as cluster. We’ll go to cluster and now you have a default and local. Now we know that any changes that we’ll be making would typically go inside the local directory. However, let’s look into what default contains. It contains a file call as indexes connect which has certain indexes which is configured. So let me just copy, I’ll copy this specific index and we’ll create one more file in local and I’ll say indexes conve and basically we’ll push the copied standard that we had taken.

Now, let’s say that our index name is KP Labs. I’ll specify the DB path. We’ll also specify the cold DB path and the third DB path once then you can go ahead and you can save it. So any configuration that you might want to push, typically it would go under the master apps directory. One important part to remember now, once you have created a file call as indexes con. So what we have done is we have created a file in the master apps directory of the master index. So now we want to push those files or push those configurations to the peer nodes and that is referred as the bundle push operation. Now, this is something that you can do from the GUI. So currently if you do an edit you have an option called as configuration bundle action. So just click here and within this there are two options. In fact there are three. One is not highlighted, one is validate and check restart. Second is push and third is rollback. So if you click on validate and check restart. So what this basically will do is it will validate the configuration bundle and it will verify if a restart would typically be required.

So now you have done that you see, it says last validate and check restart. It was successful. Restart. It is saying restart is not required. Perfect. So now we know that restart is not required, we can go ahead and we can push the changes to the peer indexes. Great. So now you see the changes have been pushed. So in order to verify whether the changes are being pushed, you can go to settings, you can go to indexes and typically you should see that there is a file called SKP Labs which is present. So this index configuration has come from the master indexer. In fact, what you can do is you can even check it from the configuration file directly. Let’s explore that as well. So if you go to Opt, Plunk, etc. This time instead of master apps, master apps, we already know it basically contains the configuration for the master app. So master indexer would typically use that. We are more interested in the slave apps because this acts as a peer node. So if you do a slave apps here and you do LS, let’s go inside cluster, we’ll go to local, we’ll open the indexes cones and KP Labs indexes is something that you see over here. So now let’s go ahead and add the data to the kplabs index from one of the peer nodes. So I’ll copy the same file, which is the authentication logs. Let me just say linux underscore Secure.

So this is one of the default source type which comes with splunk index will this time select kplab one and I’ll do a review and I’ll click on submit. Perfect. So once you have clicked on Submit, typically this specific index of KP lab should appear within the master indexer console. So if you go back to the master node and you go to the indexes, now you see you have a new index called as KP lapse. So now that we have started to add the data to the KP lapse index, it will go ahead and replicate it across the index.

So currently we have two indexer nodes which are working as peer. So we uploaded the data here. So now the data will get replicated across the node two. And then you see, now you get the green status for both the buckets for searchable as well as the replicated copies. So this is the reason why any configuration that you would like to push, it’s very important that you do it via the bundle push operations. Otherwise it will not it replicated across the other peer nodes.