SPLK-1002 Splunk Core Certified Power User – Building Splunk Enterprise Architecture on Amason AWS Under 60 Minutes

1. Introduction to building Enterprise Architecture on Amazon AWS

We have seen this one as one of the most advanced architecture in Splunk which has multisite replication, deployment server to manage the configuration, separate license manager and also avoider out of this architecture. We’ll be implementing the similar architecture but without av forwarder and license master because we don’t have that much traffic. In order to have heavy forwarder, we are not doing any filtering or routing of these events.

So we will be directly sending our logs from our laptop or Windows machines into directly into our indexer cluster. When we come to indexer cluster we’ll be having two servers which will be clustered in site two and two servers as indexes in site one. To finalize we’ll be having one deployment server, two searches, two indexer and the universal forwarder from which we’ll be sending log.

So that is a total of two plus two, four in site two and two plus two, four in site one and one deployment server which we can configure it either as a site one or a site two. We’ll see that during our configuration and we will use one universal forwarder and see how we can replicate the data and search it. Clustering is working indexed clustering is working as part of our tutorial. These are the instances that we will be using for today’s implementation. That is we have two indexers which belongs to site one, I’ve labeled them as site one so that it will be easy for us to reference and two searches which belonging to site two.

Similarly we have site two, two searchers, site two, two indexers, so total of eight machines and one machine which is used as deployment server, license manager and also as a cluster master. We’ll see how we can implement this architecture at presently via just the service I just started this service with. A couple of minutes back we’ll see how quickly you can set up a Splunk Enterprise level multi site clustering probably in a matter of minutes to ours. So we have a stopwatch in order to calculate how much time does it take to implement Splunk Enterprise. That is with multisite clustering enabled.

I’ve just started up these instances and I have made sure the prerequisites are met. That is your Firewall rules splunk prerequisites like disabling php, disabling your C, linux and also port communication between the Splunk components that is 8080, 89, 780, 80 for replication such as probably one more port. All these ports have taken care so that these prerequisites are met. Once these prerequisites are met, I’ve also configured in my terminal that is based on site so that I’ll be having clear idea on which instance I am working on during my installations of Splunk Enterprise on the Amazon cloud. Now, in order to start our installation, we need two things, that is one prerequisite server mat.

I made sure all the prerequisites on all these instances have been set. That is we have created our own application user account that is Splunk and also firewall rules, thp disabling, se, linux disabling and internal clustering communications. All these are clearly met. In order to proceed, we need one more thing. That is our Splunk installation package. From my previous downloads, I have already shown you how to download directly your Splunk installation package based on your flavor of linux that you are installing from the Splunk portal. Now we’ll go directly into downloading these packages onto our server using the wjet. So before I start, I’ll make sure to start the timer so that we will estimate how soon we can set up our Splunk enterprise.

This is not a tutorial where you will be taught of syntax that are used for configuration and stuff. This is just similar to race against time to see how quickly we can spin up a Splunk environment that can be implemented in most of the enterprise organizations. To be clear, we will be implementing one of the most flexible Splunk architecture that is searched clustering indexer clustering which supports multisite with data application. So before I begin downloading, I’ll make sure to start the timer so that we record our time accordingly.

2. Building Splunk Enterprise Architecture on Amason AWS Under 60 Minutes

In this video we will be using the latest version of Splunk. That is six six two as of today at the time of this video. And also we will be installing Splunk on the linux platform. We have seen the benefits of having Splunk on linux platforms and what are the disadvantages of having in Windows environment in a regular video course. All these instances will be on our Red at linux Seven. You can also call it as cento s seven. Before installing, I’ve already downloaded the package required. Let me verify. So this is the installation package that we are going to use for this video tutorial. And this is the OS version that I’m going to use. That is Enterprise. linux seven.

As of now, I will just name these instances in order for my understanding. So that before installation of Splunk I need to understand which servers as the storage for the indexer and which as optimum configuration for the searches. Similarly, which instance belong to which site. This is a general idea since all are on Amazon aws. This is just for our understanding in order to know which instance should be configured as which Splunk component. And also I have made sure all the prerequisites are met like application ports which all the necessary ports that are required to communicate between your Splunk instance and also from the universal forwarder to the indexes and also replication ports are made sure they are enabled. thp has been disabled.

AC linux has been disabled which all these prerequisites we have dealt through in our beginning of Splunk learning. Once you are made sure all the prerequisites are met, you can begin with installation. Before beginning, I’ll start my timer so that we can keep track of how much time does it take to bring up a production Splunk installation with Multisite High Availability Clustering. So this is the command which is used for installation of Splunk in linux. We know this by now. I’ll just go through them. This is just a tutorial in order to understand how much time it actually takes for implementing a Splunk. As you can see, probably it will not take more than a couple of hours, which is like we’ll be installing one of the most efficient Splunk environment.

So now we have our installation done. I’ll log into application account, that is Plank, which is as part of prerequisites I’ve created already and I made sure it has all the privileges required for opt directory. Now I start my Splunk instance by accepting license. Accept License. So that was successful. Yes. Splunk deployment server or the Red ad instance which will act as deployment server is up. I’ll make sure all other Splunk instances are up before proceeding further. As you can see now we have all our Splunk instances up. That is just under two minutes or over two minutes where we have completed our Splunk installation. Let us quickly go to our Splunk web of Deployment server and the first step would be to enable ssl by default. Splunk comes with during installation just Http. This is the Splunk default username and password. I’ll change the password for something simple. Now skip. So this is the Splunk login screen. I’ll just go forward and enable ssl server settings. We have seen this how to enable ssl probably in three ways.

That is editingwebcom Splunk, Cli and Web. For simplicity, I’ll just be using web here save we need to restart. Go ahead and restart. This shouldn’t take much time. I’m just looking for Splunk log weather to check whether it started up. Seems like okay, it is up. You are successfully logged in. So before proceeding further, we do the best method for modifying configuration in any Splunk instance is to make the other Splunk instance report to our deployment server. Let me quickly grab our deployment server IP address opt bins plank set deploy fun pool Https followed by complete management uri enter we didn’t change the password yet.

We’ll come to this later where you can change this configuration at once. I’ll just copy the same configuration to all other clients so that they start reporting to our deployment, enter the password wrong again and change me. You can also pass this username and passwords as part of your Splunk cli. But of course it is not a good practice because in your bash history you will leave the passwords of your Splunk exposed at any time. There is a risk of any other person who is accessing the same cli as Splunk as you will be able to under the same user account. Of course they’ll be able to see the passwords of Splunk. So I do not recommend entering your username and credentials as part of your Splunk cli command. Whenever it is necessary, it will pop up for the username and password. That time you can enter the credentials. Basically what I’m doing is I’m making sure all the clients that is our Splunk components are reported to deployment server, which makes it much easier to modify any configurations on this deployment server.

As you can see, our deployment clients have already started reporting to our deployment server. From the logs, let us go back to our ui for the management. As you can see, there is site one, indexer one, which is that instance. Our first configured index has already reported successfully. Probably within five or ten minutes we’ll be able to see all our eight clients which are as part of our Amazon aws instances will be reported to our deployment server. This is basically because each instance of Splunk contacts deployment server by default. 30 seconds. So it takes 30 seconds time in order to report back to deployment server. This is called Phoning home. We have understood more about Phoning Home and how to increase it and what is the best method to configure it. In our forwarder management tutorial we have five clients, three more, two more, one more. Okay, now we have all our eight clients reporting to our deployment server. Let’s quickly go ahead and create a server class. I’ll name this as search it Save. I’ll add some clients.

That is our four searches sh Star Preview which all of them as you can see, I specified the wild characters. We have gone through this forwarder management thoroughly in our forwarder management tutorial so I’ll click on Save that is our search ads back to forwarder management. All Splunk instances we require this in order to modify any of the Splunk configuration or base config. I’ll just enter Star and also you can use Filter match type in order to let me use Filter match type so that all our Splunk instances are linux machines so that there won’t be any misconfiguration or pushing configuration into Windows machine. So that is our all Splunk instances. I’ll create one group for indexes and also we have seen in our videos where we have created this forwarder groups or server class groups using Splunk Cli and Splunk web and also directly through editing configuration. Why? We have only three clients in our index. What happened to the other one? I named it as dix instead of Idx mention with an R condition and test it out. Or should I rename comma should work. Yes, as you can see although we have renamed our indexes pretty badly, but I was able to fit in using a comma. So we have four indexes, four searches depending on different site. Now we need one more in order in order to segregate the based on site.

I’ll create site one server class. This will contain everything with site one. Okay, that is four. Perfect. We’ll create one for site two. We didn’t add the clients. This is site two. Go ahead and save it. We know that it will be four instances. Perfect. So now we have defined these many. You can also define site one indexers. Site two indexers. Not necessary. We’ll come to this in case if you have to create multiple groups again. First let us create an app which we can deploy for all our Splunk instances. We know how to do that by now because we have seen in our previous discussions how we have created them. I’ll go to etc deployment apps. I’ll copy the sample app, so I’ll rename it baseconfig. You can create your own base config any time. We have seen how to create our own configuration apps. I’ll get rid of some of the unnecessary components because it is just configuration files we’ll be dealing with here. You don’t need logs too. What do we have now?

Default will not require inputs, indexes, props we don’t require all this. We’ll add them whenever they are necessary. We need app. com. It’s not good to edit under default and we’ll see what this inside metadata. These are basic information. So we’ll create one directory called Local. Under local I’ll copy the app. com underap. com I’ll change this one to Enabled. Enabled is visible. Now I’ll change the label to Base config. This is just nothing but a configuration app that I’m creating for the baseconfig. In this I’ll just get a syntax the start web server from the system default. This is a good place to refer the configuration but not to edit the configuration.

So this is the one. We don’t need Start web server we need enable ssl so it will be web conf. We know the configuration hierarchy, how it works and how the deployment apps work. All this we have discussed as part of our video tutorials earlier. So we’re directly modifying the changes in order to see how quickly we can get this environment up and running. Although we have made some good progress under 18 minutes now we have our deployment server up and running which is the core setup for Big implementation. Once you have deployment server and your instance is running it’s a matter of minutes where you start to push all this configuration.

As you can see now we have our newly configured base app. I laid it this so this option we know what it does. I’ll add server class to all Splunk instances where it needs to enable ssl configuration. I’ll click on Save automatically. This configuration will be deployed to our Splunk instances not yet. So it has one app it deployed. Let us see. Is it deployed? As you can see it automatically copied from our deployment server into our index server one. Similarly it will be under all other locations of our Splunk instances. Perfect. So all our Splunk instances now have enabled Https configuration almost with no effort of logging into individual servers. So now we have our deployment server setup. Let me quickly create one more base config. I’ll copy the base config itself because that has the minimal files so we can edit it quickly. I’ll say site one instances.

This is for Site One to localvia app. com. This state is enabled. I need to change this right privilege for this file. The label will be site configuration. In order to configure site configuration we don’t need web. I’ll delete this configuration or we can even remove this write server. com and let me quickly grab our Site One configuration. This is our Site one configuration. Perfect. cpif and R I’ll create one more app for our Site Two instances go to Site Two under local server. com changes to Site Two. Now we have configured Site two also let us go ahead and deploy this apps. Let us see we have completed the total downloads that is it. That shows there was no errors during deployment. Now this app should go to our Site One instances.

That is this one site One searcher indexes perfect. I didn’t click on restarts plunket. Damn it. Go to site one. Edit restart So automatically when this file is or this configuration is downloaded it will restart your Splunk D process in order to reflect the recent changes. Okay, add in. Add this one. Site two. This is site two. Yes. Perfect. Let us validate. This is our site one index. It should have site one app now sooner or later. It’s always a good practice to give it five to ten minutes. As I said, the phoning home interval determines when this app is being downloaded. Once the clients communicate to your Splunk deployment server, it automatically downloads the app and restarts your Splunk instance in order to reflect. As you can see, we have our site on app deployed. That’s fantastic. We are making some good progress. We are just under 25 minutes.

We have created already three apps. We have enabled ssl on all Splunk instances. We have configured the instances based on their specific sites. Now it’s a matter of configuring searches, indexes and clustering. This is all we are left with. Now we have 15 downloads. Four, four, eight plus 816. We should have one more eight. But here it says 15 downloads. Rest the one more. It should be done anytime. Now let’s go ahead and configure our main configuration. That is our index clustering. So this will be our cluster manager, which is our deployment server.

As you can see, this will be our deployment server. Also act as cluster manager, managing indexer clustering on both sides. Let us go ahead and enable indexer clustering. This is our indexer clustering? Yes. Master node replication three, search factor two. It should be fine. We need to restart. Restart it no problem. This is one of the scenarios where it is recommended to have another Splunk instance to act as your deployment server so that whenever you’re restarting your indexer cluster master, it doesn’t impact your download of application or configuration changes. Successfully restarted. Let us log in. Clustering has been enabled by default. It will report itself as a searcher. It is waiting for our cluster peers to join. Let us enable our cluster peers.

We didn’t give any security key or password into

our cluster because this is our demo setup. So this is our clustering master configuration. We know that and we need to replace this IP address with our cluster master IP address. All this configuration we have seen how we got this configuration during our indexer clustering phase where we have gone through how to enable clustering in Splunk and we have been going through indexer clustering for quite some time and we have understood how the configuration replication takes place and all the complexities involved with clustering. I’ll again copy a base config to start with. I’ll call this index cluster slave app. That’s a long name conf and also I’ve explained how to edit the configuration files. If you are not familiar with linux, you can use any of the text editor to edit this configuration. It’s always good to give a nice label so that whenever you see what this app is will come to know. I don’t need web all betting yes server. com replication port is 90 80 and this is our slave 49 in the app as oh yeah idx cluster slave let us go to deployment server.

This is our deployment server we’ll deploy this indexer clustering app that is slave configuration to all our index slaves that is idx one idx two idx three sorry site two idx one and site two idx two so this is the indexes and this is our index our clusters layer up you go ahead and deploy this. Before this we need to edit one more configuration that is replication factor or auto replication. We need to enable in order to see the actual clustering taking place. We need some indexer to replicate so we’ll set that to default. Always don’t remember the configuration because it’s always available on the internet and also splunk default is a good place to start with. So this is the configuration we need to set it to one that is true in Splunk. By now you should know that zero one true false represent the zero one and true false represent same meaning. Idxlave app local via index replication factor. Set it to auto. Save this file and we’ll deploy this application.

Now where we have in our index clustering server confined indexes confirm 35 minutes. We are already into configuring or finishing our splunk index cluster configuration. This is our indexer save. I didn’t again set my Restart Splunk instance to true. So whenever the ur endixer checks with deployment server for any updates it will find the new update or the configuration update. Like restart splunk D it’s a new configuration update. So what it does is it redownloads the app and if the checksum is varied, it redownloads the app and restarts your Splunk demon process which reflects your configuration. After that we should be able to see our newly reporting clients here.

It should probably take around five to ten minutes or probably faster depends on the Splunk restarting. Let us see. Yes we have deployed our app successfully. Let us see if we had any issues bringing up Splunk instance replication factor search factor okay that is perfect. We didn’t set the multi site option true for our Splunk master or the clustering master. We need to set that. This was pretty bad mistake because all our Splunk indexes are now in downstate. There is my multi site configuration it’s better to refer system default. You can see by default it is set to off. I’ll make it to true it’s under clustering stands. Now since we deployed the wrong configuration of Splunk we need to restart our indexes manually.

So let me make sure our Splunk deployment server as a multi site configuration I think we didn’t mention the site to which this cluster master belongs so we need to read it the configuration again and give it a restart. I’ll place our deployment server in the site one itself. You can place it in site One or site two. It shouldn’t be a matter for index or clustering for search ed. Listing searched clustering as of six. Six two doesn’t support site awareness. That means your search ed is not aware of to which site it belongs, which is not a big deal. Localserver. com here under 37 minutes. After this probably we shouldn’t have any issues bringing up our Splunk instances that are our indexes, which was a horrible mistake to miss the site multisite clustering attribute. Let us see now, will our Splunk start? Perfect. We have our Splunk up and running. We have an issue again. Before starting, we’ll delete some of the apps so that it can download them once more.

Since we have this configuration already on our cluster master, I’m directly deleting these apps. This is our one more index. This is the other one. Let us try to bring them back. So if we have edited our configuration properly, this shouldn’t be the issue. As you can see, our Splunk instance is now up without any issues. So basically we missed a couple of factors. Hence we got into this trouble of logging into indexers and restarting them one by one. Let us see now how our index are clustering is looking. Do we have any clients reported? Gives us a warning that we didn’t set any. We have lost a good amount of time here, around 15 to 20 minutes in order to troubleshoot hq because the introspection db by default it is not replicated. Let us see if our client is reported on our clustering. Not yet. So it has downloaded the app, now it’s installing restarting.

That was completed. Perfect. We have one indexer reported. Let us restart our other indexes. This is our second site two indexer. Two indexer one start. By now we should be able to see all our indexes reporting once they are up. Of course, once they are up. Downloaded the package that is Idx clustering app. Perfect. Now we should be able to see all our instances reporting. We have two instances belonging to site One and site Two. We have not made any indexer as of now. On the indexing configuration you will make couple of indexes like Windows linux and default index. Of course, it is already present, but it doesn’t contain any data.

We are yet to see our site to indexes. Have they restarted? Seems to be yes. We have three indexes reporting to our deployment server now, or the cluster master in this case. There are two different ways of deploying apps to the searched cluster or indexer cluster that is using underscore cluster app which is already part of your Splunk, or using Sh cluster app where you will have separate sub directories. As you can see now we have our underscore audit and underscore internal which are replicated. Now. Once all the clients have been in place, you’ll be able to see the data starts to replicate in between those two. Now we have our indexer clustering set up over a period of time. You’ll be able to see data will be searchable, such factor will be met and replication factor will be met.

Now it’s time to point your searcher to your deployment server which we have already done as part of our initial integration. Let us log into one of the search ads now in order to see the indexer configuration, let us log into site one search at. We didn’t edit any configuration including default password but we know that we have deployed the configuration using deployment server in order to change this to Https change me.

This is the default password at any time. You can log into Splunk using Cli or Web Console and change the passwords which are a default before going live with your production environment. You’ll skip this, you’ll go to Distributed Search and if you click on Search here, we should be able to see our indexers. In this case we are not because we have not pointed our search edge to our indexer cluster. This is similar as Search Ed node and you need to present the master Uri. But this way we are going to configure it using our deployment server. We will copy the Idx cluster slave app in order to deploy to our searcher name. It search idx cluster. Remove most of the configuration file. So this is the search head configuration. We need server. com. We need search at.

The mode will be searcher. We don’t need replication web. I’m just checking whether it should be Search underscore at or it is plain Searcher. Yeah, that is mode is equal to search at. It should be perfect. We’ll deploy it to our search ads. Now there is our indexer clustering. This is our indexer clustering and this is our deployment server. I’ll go to Deployment Server and deploy this search Ed configuration to point to my indexer cluster. So these are my search ads. App. This is my newly created configuration for searching in indexer cluster, click on Save.

It will automatically download this configuration into your search arts. We’ll wait for the search at to restart so that it downloads the configuration and it also downloads your indexes so that you don’t need to manually log into each searcher in order to point to the right indexes. Downloaded and restarted. Let us log in and validate whether we have our updated indexer ips on our searcher. It will be under Distributed Search K. We made the same mistake as we did earlier. We didn’t update the configuration of this one. Let us make it multi site equals true. Since we have demonstrated during the course the single site cluster and how to configure single site cluster, I add the same configuration. So now I’ve edited a configuration. I can go ahead and restart my Splunk deployment server. Instead I’ll just reload the configuration we have seen.

What is the difference of reloading the configuration and restarting the Splunk server or the deployment server in our forwarder management tutorial. I’m just deploying the new changes. Let us see our indexer clustering. Is it complete? No, it’s still copying. That’s perfect. If we didn’t add the firewall rule for the replication to begin, I’m just updating my rules. These are all typical scenarios which you face in case of production implementation. The missing firewall rules, missing configuration files. Once you have implemented a successful implementation, you can copy the same base config which will reduce almost half of your implementation time.

So we have our search and picked up new configuration which we reloaded as part of deployment server. What does it say now? Perfect. It says replicated index may not be searchable fully searchable. That means our indexers configurations has been successfully deployed on our search act. Go to search here. You should be able to see our indexers now. Perfect. So, as you can see now, we didn’t edit any configurations on the search head, but it was still able to pick up all the configurations from the deployment server. In this way you’ll be able to manage high availability of your Splunk environment. So now we have completed our indexer clustering searched pointing to our indexer cluster. And also we have seen how to configure indexer clustering which is independent of your site awareness in Splunk.

In our clustering videos, this is how a typical production environment will look. Now you can successfully say you have implemented one production environment of Splunk which is under your belt. And you can proudly say you know Splunk and how you can implement Splunk. Once the connection issues have sorted out, you’ll be able to see these buckets will be successfully replicated among these instances. And this warning symbol will be gone. And it will be a green check symbol. This is basically a period of ten to 15 minutes. You should be able to see all these buckets replicated. I didn’t mention the rule name or the port number previously. Let me go ahead and save. I saved it. Now we should be able to receive traffic. Our replication should have started now. Perfect. As you can see, it all started filling it out. Now we have three copies of replicated data as per our replication configuration here, wherever the data has been originated, we have two copies.

And the data that has been copied into other site will have total of three copies. That is two at the origin site we can consider site one will have two copies and site two will have one copy. Similarly search factor it will be site one will have two searchable copies, site three will have one sorry. Site two will have one searchable copies. In a similar fashion, you can edit the configuration of your forwarder to send the logs to your indexers. So that automatically the newly created indexer under your indexes con which you will be deploying using deployment server underscore cluster. Let me quickly show you the configuration file so it will be under Etc master Apps underscore cluster. So in this you’ll be deploying all the configuration related to your indexer.

It says this configuration settings here are applicable only if the clustering is enabled and set up. We have already enabled and we have successfully set up our clustering configuration. Here you can deploy your indexes, confront the size of each indexes retention policies. All this information related to your indexes can be deployed under this director. We’ll wait for this in order to final confirmation of our indexer configuration. All this configuration including server confexers conf searched configuration all this configuration which we implemented as part of today’s video that is complete multi set clustering configuration of your aws splunk which we have implemented just now will be as part of this tutorials configuration files where you will be able to download and review the configuration and set as per your needs. You can use the same configuration in order to deploy your base configuration which will work seamlessly during your implementation.

We are almost done. We have two searchable copies. That means one will be inside one, one will be inside two, which should be fair enough. You can also have three copies. We have seen all this. What is Searchable Factor and what is Replication Factor? Site Replication Factor site Search Factor in our index or clustering videos we have our underscore internal index completely successfully and replicated. As you can see, all our factors are met. One is Data is Searchable, search factor is met, replication Factor is Met and our indexes is also able to deploy the configuration without any intervention of any configuration updates.

It automatically downloaded these configuration from your deployment servers rather than manually going and editing the configuration in order to add a search search peer. So this concludes the Splunk implementation of enterprise level high availability with multisite clustering in our Amazon. aws.