SPLK-1002 Splunk Core Certified Power User – Installation and Configuration of Splunk Components

11. Installation of Splunk Indexers

And when it is starting, there will always be a statement or a phrase. Splunk is specified by As we can see, it says another one where we’ll be starting in the index. This phrase was chosen at random, and there are a few others, such as “big data superhero” and “picking a needle in a haystack.” These are the key phrases that appear every time Splunk starts, and it checks for prerequisites. That is our port’s availability. 8000 for web, 80 89 for management, 80 65 for our app server, and C-store for 8191. It created the web directory, which we knew before installation. The term “directory” will not be used until after installation or during the first boot, when directories will be created. Now we have installed our Splunk searcher, and we have successfully started it up. Let us try to access this. So, what is this IP address? Since you searched for the complete package or the Splunk Enterprise full package, the results are 52 and 40. We will also have the Splunk web component by default.

When you log in for the first time, this is your login screen, where the username is admin and the password is “change me” as soon as you hit “enter” by logging in. Enter the new password. Let me enter my new password. That should be it. Yes. So you change the password and login. This will be our welcome screen for the first time. From our previous tutorial, we now understand what this screen is and what each menu item means. This shows our Splunk search engine has been installed, but it is not configured to function as a searcher. but our instance is up. Let us follow the same procedure to bring up our indexes. This is our index, which we installed using a privilege account. Let me log into my application account. That is splunk. I’ll be using the complete path. This is our Splunk home bin directory. Splunk start. We know there will be a licence that will be shown.

If we don’t want that to be shown to us, we’ll just add one more parameter, which is “accept license.” If you type this, it will automatically accept the licence and start our Splunk instance. As you can see, it didn’t display the licence key, but it completed the rest of the steps. Now we have our index set up and running. Change the password for the same. We all know by now that 8000port is used for Splunk Web. Is it taking so long? Is it up? This IP seems to be correct. Let me check quickly whether I have any firewall rules enabled on this. I have applied some new firewall rules. Let me try to reload. Yes, I think there was a firewall that was configured to block this connection. This is our index, and we are logging in for the first time with “change my password,” and once logged in, I’m changing my password. Our indexer is also updated.

12. Installation of Splunk Heavy Forwarders and Deployment Servers

For this instance, we’ll be mode. g GUI mode. For instance, we’ll bunce we’ll be using commode. I’ve selected Messages; click here to restart to make my new server or instance run on HTTP; please allow me to restart Splunk.  my Splunk.247.  is my 247.That should be my intima. t any time. If you want to see the status of your Splunrent status. As an argument, use bin splunk and status. n argument. This is home. plunk home. In is wheexecutables executable are stored and Splunk is the name of the utility that is used to check for status. Hit Enter. It will show that Splunk is running. Let’s see how things go. Yes, it is up. We can proceed with our IP because our self-signed certificate is secure. We know this is hosted by us. and this is our Splunk instance. And this is a self-signed certificate. Let us log in with our new password. This is done now. We have changed it successfully to https, although it is our own sign.

13. Enable SSL on Splunk Enterprise Instance

In our previous tutorial, we have seen how to instal Splunk on Linux and Windows using a universal forwarder, and we have also seen how to enable HTTPS for a Splunk search app. This is our Splunk indexer. We will go through a couple of other methods of enabling SSR. This is our Splunk indexer machine. Let us verify. So these are the 57 and 57, which are currently running HTTP.

So we will see other methods of enabling HTTP on the Splunk instance. For that, we need to start editing our configuration. So we’ll go to Splunk home first, followed by the Splunk directory structure, which we’ve gone through a couple of courses back. We know all the configurations are stashed under, etc. We know there is a system directory that is highly critical and holds all the Splunk configuration. And we know there is a default directory where all this default Splunk configuration required to start a Splunk instance is present. So we’ll not modify that directory. Let’s see if we can find a local. Yes, we have a local directory and a couple of configuration files that are generated during installation. We’ll probably go over all of the configuration in the later part of our course.

However, for the time being, we must examine how to enable our Splunk SSL through configuration file editing. The SSL configuration is in a file called Webcam, which as of now is not present. We’re going to organize a web conference. I’m using VI as a text editor throughout this course. You can also use nougat, or probably some other text editor, to edit the file and upload it to this location on Splunk. Make sure you are in the right location. Splunketc and the system local VI Web are located here. This is the file name, and we’ll edit the Settings configuration to enable Splunk web SSL, which is set to one. One needs nothing but truth to explain this configuration. This is our configuration throughout Splunk. The configuration syntax is almost the same for all the configuration files. It starts with square braces saying “configuration file” or “configuration type.” You can invoke them by providing a parameter and its value. This can be a boolean that is true or false, a constant, or a regular expression.

There are multiple options for the value, but the parameter will always be one of the Splunk configurations that we are going to change. So this is the common configuration for how the Splunk configurations are edited. This is collectively known as stanza in a configuration file. Each configuration file is a set of multiple stances. That means it contains multiple different configurations. Let us go through an example of WebConf, which is by default present in Splunk. So we’ve edited our own local configuration, but we’ll check the web that is available by default as part of our installation package. So this command I’m using it just to see the contents of our default web configuration so that we can go through all the configurations that are probably there quickly. Here plunk clearly says not to edit these files. That is the reason we are just seeing the contents of those files. If we edit any of this configuration or make any changes that are not compatible with Splunk, we might be unable to bring up the Splunk instance until we fix those problems. It says it has a default configuration, Stanza.

All of these standards, or the definition of standards, are the defined set of configurations. So there is a setting in Stanza that says to start the web server, and it is set to true by default. One stands for true, and zero stands for false. You can either mention the text true or false, which is mentioned as “year.” See here. By default, it says “enable Splunk Web SSL” is false. That is the reason that as soon as we start Splunk, we can access it over HTTP rather than HTTPS. It is by default set to false, which we have changed it to. There is a local file, and this is our local file. We have changed it to true. One is simply the truth. We’ve made it available on the web as well. While we were enabling HTTPS for our Splunk web, we saw a “restart” message under the messages. So we need to restart Splunk any time you edit a configuration file, but there are a couple of other configurations that just require a reload of configurations instead of a complete restart of Splunk.

That’s what we’ll be looking at in the advanced section of this course: how to reload the configuration without restarting Splunk. But few of the configurations, like enabling SSH and changing ports, No matter what, Splunk must be restarted. Now we are using this command to restart Splunk. It’s Splunk’s home bin directory and the Splunk utility with the argument restart once more. We should be able to see Splunk in HTTPS by the time it restarts. But keep in mind that we did this partially through CLI and not entirely through CLI. It is the Linux CLI, not the Splunk CLI. We have edited the configuration on the system level. The configuration is called web conf, and we’ve set Splunk web SSL to true or one. Let us see whether our configuration is reflected or not. So it has been disconnected from the server during the restart. We know there is no HTTP anymore. As a result, it is HTTP. Yes, as you can see, this is a self-signed certificate, and as a browser precaution, it wants us not to proceed. But we know this is a site hosted by us, and it is a self-signed certificate. We are clicking “Proceed.” Let me log in with my new password now that our index is also up and running on HTTP.

14. Enabling SSL from CLI

In our previous tutorial, we saw how to instal all the components of Splunk and enable HTTPS on them. We have seen two methods to enable HTTP. One method is to use Web and go to Settings server settings, then check the box to enable SSL. It will be under “general settings.” Enable SSL. That concludes the first section. The second part we saw was by editing the Web conf configuration file under System local and setting enable Splunk web SSL to drop. So those were the two methods we saw. If you found some of them too difficult or too simple, there is another option to enable Splunk SSL that is the Splunk CLI. To invoke Splunk CLI, you need to call the Splunk utility. That is the same utility that we called for starting, stopping, or restarting Splunk service.

This is our AV forwarder, which, as you can see, still runs on HTTP. So it is still running on HTTP. This is our third method for enabling SSL. There is a simple command that says “enable web iPhone SSL.” This useful entry can be typed. It will ask for the username and password of the admin, not the OAS. So make sure whenever you enter the Splunk utility, it is asking for or looking for the Splunk user password. With admin privilege, I’ll enter my Splunk admin user and password. It says you need to restart the Splunk server for your changes to take effect. Of course, we had to restart it using either configuration editing or the Splunk UI for it to take effect. Now we used the Splunk CLI command to enable Webs’ configuration, and it popped up with a prompt saying you needed to restart your Splunk service. Let’s go ahead and restart the Splunk service. Once this has been restarted, we will be able to access our AV forwarder instance via HTTP.

So there is no longer HTTP, only HTTPS. See? Since this is the first time, I’ll proceed with accepting the certificate. Now we have seen three methods of enabling HTTP. One through Splunk Web. Two via editing configuration Three. The simplest. Splunk.CLI. Now, during this course or any further course that will be going on from now on, you’ll be able to see me showing you all these three steps for doing each configuration. If you are configuring a searcher, we will see how we can configure and edit configuration in Web CLI. Similarly, if we are configuring a V folder, we will see how to edit the configuration, how to edit using CLI, and how to configure using the Web console. We’ll be going through all three of these methods in order to understand and plan, and you can choose to learn whichever is more convenient or easier to start with.

15. Index, Indexes and Indexers

In our previous lectures, we have seen how to install Splunk, how to install a Splunk universal forwarder on Windows, and how to enable SSL on Spunk’s other components using three methods. Now, let us proceed with the indexer configuration. Since the indexer is the core component of Splunk, let us start with the indexer. We know from previous modules that an indexer passes the data received from the AV forwarder or the unicorn forwarder and is the one who stores the data after processing. The indexer contains multiple logical storage units known as “indexes.” The index is like small blocks in a big train. For example, each block will hold specific data and be a specific size. Example one of an index can be named “Windows,” which consists of all the Windows data. Similarly, another index can be named Linux, which holds all the Linux-related data.

These Windows and Linux custom defined indexes that hold specific data can be of any size, such as 100 GB. As a result, each Windows index is less than 1 GB in size. Linux can hold another 100 GB, whereas internal indexes like underscore audit underscore internal can be of a lesser size, but they can also exist on the same storage. To make it simpler, I’ve created a small diagram that is a visual file for a better understanding of what the different components are, like how the indexes are stored on an indexer. Let’s say this is total indexer storage; that is the outermost container, and there is a Windows index and there is a Linux index. As you can see, they are both about the same size. Considering our example of 100 GB, these two are sitting on the same storage, but they are logically separated by Splunk size or even location. This will be in a different folder. Both the folders can hold up to 100 GB of data. Similarly, this is our indexer storage, which by default will be under the Splunk 100 score home.

That is, we already know that we will create a splunk directory, and it will hold all the databases that are total index storage, all the databases or indexes that are created under Splunk, under VAR lib Splunk here. So this will be our complete index location. Similarly, we have under this location Windows, Linux, underscore audit, and internal databases, which are sharing the same storage but can be of different sizes. This is typically how the underlying storage of these indexes works. One of the most confusing parts to understand is “index,” “indexes,” and “indexing,” which all seem to be kind of similar. Let me help you clear out these terms. To define an index, one needs a logical separation for storing data based on technology, teams, or even organisations level. If you are building Splunk for a shared environment, you can probably define each index based on individual company names. What are indexes? Then we understood from our diagram that these individual blocks within the same storage can be called an index. Now, what is an index? It typically represents a group of indexes present on the same storage we call “windows” as an index. Collections of these are known as indexes. The next term is “indexer.” As previously defined, an indexer is a Splunk component that performs the indexing process.

16. Configuring Indexer: Enable Reciever

Then what is indexing? Indexing is the process of dividing events into smaller chunks, also known as parsing and storing data. So indexing is a process where the component of Splunk processes the data and stores the data. Now let us see how we can configure the indexer in a couple of ways. The first step in configuring our indexer is to ensure that I don’t mix things up by closing all other instances of Splunk and leaving only my indexer screen or the session open. This is my index.

Let me go to Spunk’s home. That is Splunk Opt. Now I am at home. This is the first step in configuring an index series to receive logs. This can be from either AV forwarders or our universal forwarders enterprise instance, which is not doing anything. The first step in configuring or making this component an indexer is to enable log receiving, as with any other Splunk configuration. We have three methods to achieve this. The first method will be the Splunk GUI. Let us log into our Splunk GUI, or we can also call it a Splunk web instance, which is running under HTTP. Because I’ve already logged in, it’s taking me to the homepage. Once we are logged in, click on Settings.

You can see forwarding and receiving. Click on “forwarding and receiving.” Once the Forwarding and Receiving page loads up, we need to configure the risk instance installed to act as your receiver. On the configure receiving part, click on “Add new.” It takes you to this page, where you just need to specify a port number on which you would like to receive the logs by default splunklike by default representation of receiving of logs.It is triple nine seven on the index. Let us enter the same port and save it. It says “successfully save triple line seven.” Splunk is now listening on port 387, as we can see. If triple line seven is already in use by another application, ensure that you can change or add a new port to receive on, say, triple nine eight. You can add any number of ports to make Splunk listen on the specific port we’ll be using throughout our tutorial. That is triple-line seven. This is one of the ways in which you can add receiving ports to your index.

17. Enabling Reciever from CLI and Configuration File Edit

In our previous tutorial, we understood that the first step in configuring any index is to enable receiving logs from AV Forwarder or Universal Forwarder. We have confirmed that we can enable receiving from the Splunk console by using the Splunk GUI. Let’s look at how we can enable log receiving or indexer reception using Splunk CLI. We’ll go over all three except Web CLI and editing configuration. You can start using it with either the Web CLI or by editing a configuration file, whichever you prefer. To enable reception of logs on Splunk, we need to invoke the same utility as Splunk, which will be widely used for adding, changing, or modifying any configuration in Splunk, including removing. We’ll use the Splunk utility to enable listening on port number seven.

So it says Splunk started listening for data on triple nine seven, and it also says it is particularly listening for Splunk data. It doesn’t say to listen to all the data. We will see how to enable listening to all the data in the later part of the tutorial. For now, make sure it is listening only for Splunk data on the triple-nine-seven port. To verify this, let us refresh our forwarding and receiving. This is before configuration; nothing is displayed here. Once it reloads, we should be able to see our newly created connection. Yes, this is our triple 97, which was added through the CLI rather than the Web Console. Now, let me go ahead and delete this. We’ll see how we can add this configuration directly by editing the configuration. To confirm, let me refresh this. We do not have any receiving data on any of the ports. Allow me to return to one of the editors and select Splunk, etc. or the configuration directory.

System Local During the course of this module of the tutorial, we’ll be editing all the configuration in the local. But it is highly recommended not to enter any configuration in the System Local because, as we know from the file hierarchy, System Local is at the top, so anything you write here will override any configuration present anywhere on Splunk. For this tutorial, we will be using System Local to change the Splunk configuration, but it is highly recommended not to edit any configuration or place the configuration under System Local. We’ll be seeing how to edit the configuration at the enterprise or organisations level when we move to building our own infrastructure at the enterprise level with multi-site clustering and high availability in our Amazon cloud. As I explained, this is part of an explanation for beginners to understand that you can change configuration by editing files. Hence, we’ll be placing the files in System Local. Moving on to System Local, you’ll find a file called Inputs Conf, which is created by default to accept the host name.

We will enter a new stanza. If you don’t know what the stanza is, there is a quick way to find out before going to any documentation. Just go to default. Not here; just go to default. Directory inputs conf: this is basically your documentation, which contains all the default configuration. So I’ll search for Splunk TCP. This is our Splunk TCP syntax. Let me see if there are any others. So that’s it. We understood. What is the stanza for Splunk listeners? I’ll be adding the same. This is the configuration for receiving Splunk logs, followed by “cooler double” and “triple nine seven.” This is the configuration for receiving Splunk data on the triple line seven port. So let us restart to see whether we have picked up the configuration.

No, since we have directly edited the configuration, give it a restart. Once we have restarted, we should be able to see our new inputs that were added by editing the configuration file inputs. Splunk is almost up. Yes, it is up. Now our session has expired because of the restart. Please use my new password to log in. We now have our triple-line-seven port, which we can configure by editing configuration files. There are three ways to edit configuration now that you know there are three ways to add receivers to the indexer or take the first step in configuring a Splunk instance as an indexer. Feel comfortable using all three options because you’ll be able to understand better which file is reflected and where it has been placed when you change a configuration.

18. Default Index

Now we have completed our first step in configuring the indexer. The first step is to understand what different components or different terminologies are used as part of indexer configuration. Once we have configured the first step, the second would be to create indexes. What is an index? Again, it is a group of indexes; we have a Splunk indexer setup and ready to accept data, but once the data is received, it will be stored in the default index, which is Splunk. As part of its installation, it will have a default index. All the data will go to a default index called Main. So by default, if you go to settings, indexes, you’ll be able to see an index named Main.

This will be our default index. Anything that starts with “underscore” is a representation that these indexes are used for internal Splunk application purposes. This is used for storing your searches. These are some more internal indexes that are used by Splunk itself. The main point here to understand is that the default index in Splunk is named Main. We will see how it works once we start sending the data from our Windows machine, which is our local PC where we are accessing the Splunk console, to the indexer. So from our local PCs, we have collected a lot of Windows configuration during the installation, which we will be sending to our indexes on indexes.The second step is creating the indexes.

When we come to indexes, we understand that we can create our own index based on technology like Windows, Linux, Mac, et cetera. Throughout this tutorial, we will see different kinds of indexes being created, but the creation of any number of indexes is completely at the will of the Splunk administrator or to create any number of indexes. For example, you can even create an index based on the applications that are running or the logs that are being fetched. From that machine, we can create the index named Apache, IAS, or even Database ETCA. Even so, you’ll be able to create indexes named after teams in your organization, such as Team One, Team Two, or Team Three.

But always create reasonable names because indexes are the best place for doing data segregation in Splunk. For its users to access the reports, dashboards, or any visualization that you create, let’s say you have two teams. Windows and Linux team, you can create two indexes named Windows and Linux. You can give access to the windows team just to the index windows, where they can search throughout the splunk. But they will get results only from the Windows index. That means their access is restricted to only this data. Similarly, if it is a Linux team, give them access only to the Linux index, so that even though they try to search throughout this plan, they only get results that are necessary for them. Since we know indexer refers to holding multiple indexes, let us create some of the indexes we discussed earlier, like Windows, Linux, Team One, Team Two, Apache, or even a database similar to any other configuration. We can create indexes using three methods. We will see them one by one.

19. Index Creation From Splunk Web and Splunk CLI

The first one This is our indexer machine. The first one is using Splunk Web. Go to settings. Click on the indexes. On the top right, you can see a new index. Click on the new index. Give the file the name windows. So now, as you can see, there are a lot of optional components that you’ll be able to understand further when we come to indexing and how the clustering works and how the data is being stored in splunk to understand.

What are “homepath,” “cold path,” and “third path”? For the sake of simplicity in understanding the indexer configuration in this part of the tutorial, we’ll be skipping this and coming back to it at a later stage. For creating a new index, just give it a name. If you want this to be of huge size, make sure you select whatever the option required, let’s say 5000 GB, so that my index can store up to 5000 GB. Not necessary, I’ll keep it at 100 GB. 100 GB of index for Windows should be more than enough for this tutorial’s purposes, but depending on your organization, you can increase it to any limit. It’s totally customizable; the rest of the options we’ll come back to as part of a later process. To create an index, simply enter the index’s name and size, then click Save. Here we will be able to see our newly created index windows.

Now that we’ve built our index windows with Splunk, let’s build our Linux with the Splunk CLI. This is our Splunk indexer. Let me check: where am I? I am in Splunk home, but I have practiced entering the complete command, so it is better if you can run this full command wherever you are. Opt splunk pin is the Splunk utility, and again, the command would be splunk add index; the name of the index is Linux. Let me enter; it’s asking for it again. username and password This is your Splunk privilege user account, so once your password is successfully authenticated, you will receive a message that your index has been successfully added. Let us see here. Once you refresh, you’ll be able to see the Linux index, which has been created with default parameters. Similarly, if you want to mention the size, you can mention Splunk add index Linux followed by its argument that the size is equal to 200 GB, which should create an index with a size of 200 GB. For now, we understand how to add an index from the CLI. Now consider the third option: editing the configuration file. All of the configuration for creating an index, including its path, size, and location, is stored in a file called Indexes.

20. Index creation from Splunk Edit configuration file

In the previous configuration of indexes, we did a couple of things. One, we designed receiving using three distinct methods. We have started creating indexes using two different methods: CLI and Web. The third important method is web configuration editing. Whenever you try to edit a data configuration file, the best practise would be to go to the Splunk documentation. Splunk Enterprise administrator and admin manual I always keep these manuals on hand because I’ve mentioned them in previous tutorials. So because I can get all these configuration references from here, The index configuration is always stored under indexes.com. all the configuration related to adding an index, deleting an index, changing the size of the index, and changing the location of your index file. All this configuration is present in indexes conf.

To add a new index using configuration files, we need to add a new file. If there is no index.conf file, it’s already present under the local directory. We will add a new file under system local just for the example of understanding the indexer, but in the future we will see how we will be replacing or creating these indexes using our deployment. So to move on, I’ll click on indexes.com, for example, where there is one database that has been created. I’ll copy the hatch one; I will not modify anything. We can change the name if you want, but since we’re just demonstrating, I’ll just copy everything, including the name localindexes.com, from the configuration file reference.

We have not done anything. We have just copied a sample index that was created using this, and if we want, we can rename the names of the indexes, but we are not going to deal much with this indexer, so let us keep it as it is. These locations, HomePath Cold Path and Tower Path, will come to this later, probably with a deeper understanding of Splunk and indexers and the clustering part and the retention policies where this plays a key role. As of now, for simplicity in configuring index, we’ll stick to creating index for now; this is the regular syntax. We created an index by editing the indexes conf configuration. Let us restart our Splunk instance; it shouldn’t take much time. Yes, it is starting. This should be our index. Yes, this is our indexer login. So we are under indices, correct?

So where is our new hatchback, DB? Yes, this is Hatch DB, which we created with configuration edit; we created Windows with Web linux and Hatch with configuration edit. But we all know by now that the DB location of Splunk is opt-splunk, where lib-splunk This is your default database location on Splunk, where all the files that have been received, processed, and stored here You’ll see a lot of directories under this, including your newly created indexes. Let us go over our window index, of which we mentioned hot DB, cold DB, and tau DB. When we move on to retention rolling of buckets, we will understand why data model summary is present, what is cold DB and what is tau DB, what is a data model, and how data model summary works. For simplicity in configuring an indexer, this will be your index location, and all the files will be stored here at any time. If you want to check the size of the indexer, you can check the size of the folder, or you can just go to your indexer GUI and check for those sizes.

21. Configure Search head From Splunk Web

We can configure them as indexes. We’ll see at a later stage how we can rename the Splunk instance name. For now, to validate, go to your search and reporting app to see that we are successfully set up, and when you set it up, you get a couple of screens.

So one is our Uri, and the second one is the host name. This gives us a quick picture of whether our configuration was successful. Replication should be successful. In a matter of time, it will be successful. Similarly, if this belongs to a cluster label, what is its current health status? It seems healthy. And have there been any recent L-check failures? None. And the status of this indexer is enabled. If you want, we can disable it. Similarly, once we see status up, which indicates success, we can delete or make current in the actions. To verify, go to the search and reporting app.

The best option would be to search for internal logs. Let me skip the two. So this is our search bar. We know a couple of basic searches that we have performed in our early part of the tutorial, and we have gone through all these menus and functionalities. So I’ll directly jump into writing a query. It will be a simple query. index is equal to internal. I’ll check for the last 15 minutes because within 15 minutes we will have configured our indexer. We’ve got like 5,000 events. As you can see, there are two hosts in our internal index. The first is our search engine, and the second is our indexer. To better understand these values, we can check the host name or replace the host name on our index.