SPLK-1002 Splunk Core Certified Power User – Splunk Post Installation Activities : Knowledge Objects Part 3

22. Searching in Splunk

In this module, we will learn how to search in Splunk. We’ll be seeing some of the basic operations as part of everyday searching in Splunk. Now we have our on boarded data, our Splunk installation set up, and we have created some basic knowledge objects. Now we will see how to search efficiently using Splunk. This is our searcher. We know what indexes are indexes. So I’ll go to index. Main and just hit enter for the last 30 days. We have relatively little data, so it should load up pretty quick. We have had this much information as part of our Splunk database for the last 30 days.

There is lots of information in our fields. I would highly recommend going to our first chapter, where we have discussed the complete UI of Splunk and described each and every field that is available. As per this part of the tutorial, we’ll be completely following up on that, and we’ll be strictly focusing on how to search in Splunk. Now I have narrowed down the index. If I want to search for Windows, I’ll go for “index is equal to Windows.” This is determined by the lock sources you are directed to in that index; you will be able to search the same. We now have an index that equals the mean. We are sending that as the default index for all the locks. So we’ll be sticking with “index is equal to mean.” Even though you can also search using the wild card, which is the star, You can search the star, and you’ll end up getting the same number of results. because it will be searching our default index itself.

Let us go. And this is the best way to narrow down the results. Instead of a plain old wildcard, I’ll first narrow down My search for index is equal to main. In index, one is equal to one in main. I have two hosts. I have something called my universal forwarder laptop and Arun Kumar’s PC. So I’ll click on this host, which is of interest to me, and I’ll see what information it has collected. Let’s say I need to look for the error. What do I type? I simply typed the freeform error keyword. So what happens when I type an error?

This search query is looking under the main index for a host named this, with a string containing errors. All the logs with this error and this error keyword are matched and displayed in front of you. What is happening here is that we have not mentioned anything, but Splunk takes it as implicit, so if you don’t mention anything, that means you are searching for both conditions. Splunk interprets it this way: if there is nothing between two phrases, it assumes you are looking for both of these values in your logs. That is how our search query works or is interpreted by Splunk. So there is another function, or with this logical function, I’ll search for an error or warning message in this log. This is a condition within which it is looking for an error or warning scenario and will display all the events that match the specific criteria.

There are also other operations like not, which show only error and no warning. Let’s say this should make more sense. I’ll search all the logs in my previous criteria but not warning messages or information, which again narrows it down to only my error blocks. As you can see here, there won’t be any values that contain warnings or information. We’ve learned that it doesn’t matter whether you do a free-form search in caps or small letters. However, whether you use a wild card or not, it only matters what phrase you are typing. But if it is a field name, it should always be whatever is displayed here; the field is case-sensitive, but the value is not case-sensitive. Let us now examine how other fields operate, whether we understand them or not.

23. Search Modes in Splunk

Let me go back to my main search, and this time I’ll go for my access logs. So this is our access log, which is on our web server and has a large number of fields. We have two more conditions for which we can verify how to specify greater than or equal to values. Assume I have status fields in my logs; let me select them so they appear upright in our screen for viewing the status field. It has 200, 504, and many other different values. Let’s say I want status equal to 200. It displays only 200 values; there’s nothing special about that.

But if I choose status greater than 200, it ignores all the 200 values and displaces the rest of the values. As can be seen, 50348 and 500 are all status values greater than 500. And let’s say I need to filter out my five and five, three and 500. I’ll use the implicit, and because we didn’t mention anything here, it will treat it as such, so I’ll mention that status should be less than 500, or you can specifically mention and condition so that Splunk interprets as we want by default. I’ll leave it as it is. It treats it as a and condition. And we have five values now, which is completely 400 values. We have learned about greater than and less than. Let’s see how we can specify that status is equal to or not equal to 200. This is how we say “not equal to,” then an exclamation point, “equals,” and the value that should not be matched. Let’s hit enter now.

Go back to the status field, and we can check that there is no 200 value. Let us eliminate 500 with some wild cards. I’ll use a status exclamation equal to a phi star. As a result, anything beginning with “phi” in the status field will be removed. We’ll return to our status field based on the outcome. As you can see, we only have our 400 errors. This way, you can narrow down your results much further. The fastest way would be to click on some field, click on the value, and it automatically updates your search. If you want to not match the condition, you can click on it or update your search to match it as “not.” This is also one method for including a note condition. You can also use the boolean “not” function to say that don’t match wherever action equals “add to cart.” These are some of the methods by which you can fine-tune your search and populate the results. We’ve also learned about fast mode, smart mode, and verbus mode. Just remember that fast mode is the fastest of all. Let’s modify the same search and see what all the changes will reflect. So quick mode I’m selecting fast mode here, and you’ll be able to see hardly ten fields.

It has completed the search, but we have very limited information in our fields. Menu. Even if you extend, you have only a few fields, but there is a lot more information in the locks. The fast mode is more focused on getting you results faster than parsing them completely and giving you the complete information in the fields. The smart mode, on the other hand, provides you with all of the requested information. So in this search, we have requested only events. We have not specified any visualizations. Let’s go ahead and mention some top action values visualizations. I’ll mention that top is synonymous with action. Top is a command. Action is a field name. The view is said to be the most valuable component of the action. So now we have a visualisation from our smart mode, which automatically produces a visualization.We need to make sure the recommended charts are set to see the value. Now we have a Smart Mode, which automatically provides the visualisation and statistics requested by our talk command. But it will not give you any events. But Smart Mode gives you whatever has been requested.

So by using the top command, we have requested statistics and visualization, which are populated by default. So if we want events to be seen, it clearly says you need to search in verbose mode. Verbose mode, like any other process utilization, gives you everything. It provides you with all of the extracted fields, as well as statistics and visualization, in addition to events. So this is just a quick overview, but if you’d like to know more about this, I would highly recommend you go back to our first module of the lecture, where we have gone through all these search modes. Now that we know about search modes, let us see how we can validate whether the data that has been uploaded to Splunk is being parsed properly or not. So the best way to check for parsing of the data is to say that these are our logs. Make sure you are running in smart mode. This is our uploaded log. We know this is a complete line as a log. Once you have uploaded, you can see from Notepad or any other programme that you’ll be able to notice the difference between the lines. This is our actual log, and if you expand this arrow, you’ll be able to see all the fields extracted by this log. In case any of the fields have been misnamed or wrongly named, you’ll be able to notice, and we can consider them as not being parsed properly. Also, if you see any values that haven’t been extracted, let me know. For example, let’s say I see the value 159 here but can’t find it anywhere else. That means the logs have not been passed properly or completely. We can consider this a response time, and we can create a new field so that our logs are passed completely. This is a quick way to identify whether logs have been parsed completely or not.

24. Creating Alerts in Splunk

In this module of our discussion, we will see how to create alerts in Splunk. In our previous modules, we have learned how to do basic searching in Splunk, how to use a couple of boolean operations, and whether or not to filter out the necessary events that we are looking for. Some of the topics we are going to study about alerts are: first, creating a search query, which is an important part of your alert creation for what condition you need.

This alert will be defined as part of a search query, along with when it should be run, whether it should be real-time or scheduled, what action this alert should take, whether we should enable trolling, who has visibility of these alerts, and who can modify these alerts. If the action is emailing, should we email the link, just the results, or as an attachment? We will see all these operations in our exercise. This is our search e-mail. We can start with searching. So let’s consider an example. Whenever my server throws either 400 or 500 errors as per our HTTP code, we know that something is wrong with our server or that some page has not been made available for the user. Let us see. I’ll search for my index main where my data is present and the source type of my web server logs are accessible. I didn’t mention complete access combined with cookies.

That is our source type, and status should start with either 500 or 400 something. This typically means there is something wrong with my server or the client requests have not been successfully processed. Let us validate. Here is our status field, where it shows 503, which is our internal server error. Similarly, 408 and other server errors are defined as per our HTTP status code. In fact, in our previous lecture, we also learned how to use our lookup field to enhance the data. Let us use the same type of field so that we’ll be able to make more sense out of the data output. description of the status field: this is not input. This will be your lookup command. Yes, there is a description; here it is.

Let me select it here. As you can see, these are the error descriptions as per our status. Let us see how we can create an alert so that whenever this event occurs, we should be alerted almost in real time. I’ll write a query to display just the statistics based on the count of those statistics, status, and values, so stats command I’m using it to just display this should be counted by what it basically does, which is sort everything based on your status and how many events you’ve received in the period that you’ve chosen. Now we have set up our query. This is our condition. We should be alerted whenever these status quotes are received on our web server. Click on Save Alert, and here you will see a short title where it displays a brief description of what this alert does. And here you can probably write “web server generating 405 hundred errors,” and “permission,” which is either private or shared in the app. If it is private, the alert will be visible only to you. And if it is shared in the app, anybody using this app will be able to access it.

25. Splunk Alert Condition and Sharing

it supports Go for HTML and plain text for a better formatted output of the alert. So here I’ll just add my email so that if any alert triggers, I’ll be getting an email notification, and I’ll keep everything as default. I’ll save this alert, so once it is set, you will get a permissions tab. If this is a search schedule search, it says that since we are running an enterprise trial license, it will not be run after the license express. So if it is not, then if we have an actual Splunk license, this warning will not pop up. The next is the permissions step, in which you will be able to give privileges to the people who can edit or read this alert. As you can see, Admin and Power have edit privileges, and all other people using Splunk have read permission. I’ll just make it global; I’ll make sure this alert is visible through all the apps.

26. Editing Splunk alert and Alerts Actions

So once we have created them, if you want to identify where your alerts are located, go to Alerts at any moment of time. You can edit these alerts by going to the Alerts tab. As you can see, this is the only alert that has been created. Click on Status; that is the alert name, and you can edit this alert at any time to specify what actions should be taken, and you can see the search query. By opening in Search, you’ll be able to see the actual search query for which the alert condition has been triggered.

Similarly, if you don’t want this alert to continue, you can go ahead and disable it. As you can see, the present status is enabled. If you don’t want this alert, you can disable this alert.And if you want to change this from real-time to scheduled, make sure you run it frequently. You can run it every 1 hour because it’s an error alert or a periodic reporting type of thing. So that you have a view of what is happening and it is a less-priority event, you can schedule it to run every hour past 15 minutes. That race will run every hour. At the 15th minute, it’s like 115, 215, 315, and so on. If you choose 30 minutes, it will be 132, 30, and so on. So you have the option of going that route or using the Cron schedule.

If you are familiar with Cron, let’s say you need to run this alert every 15 minutes. Then I’ll choose “star,” followed by “15 stars,” “star,” and “five stars.” Make sure you have five values with a space between them, and each one represents a minute, day, month, and year. Let me know if you need more details about the crown. I’ll be able to help you in our discussion section. For now, I’ll cancel this and keep it as a real-time alert. Now, to summarize, we have learned how to create an alert, how to share it, and how to add actions. And each alert can have even more than one action. Let’s say I have one action for this alert, like sending an email. I can add one more action to add it to the triggered alert. While adding the new action, we can specify whether it should be high or low based on our current situation. Similarly, you can add another action to run a script and specify a dummy script name. Because as of now, we don’t have any scripts that have been created, and these scripts should be present in your search bin. That is your opt-splunk ATC Search app under SearchBin Directory Scripts, and you are going to place your script.sh or script.py if it is Python. And every time this alert is triggered, that script will be invoked to take the next action.

27. Creating Splunk Reports

From our previous discussion, we know by now how to create an alert in Splunk and what all the different actions can be enabled as part of alerting. And also, just to add a note on alerting, there are other add-ons for Splunk where you can get alerts based on your SMS or a post notification. We will be seeing this when we discuss more about the Splunk mobile app. To continue with our discussion from where we left off in our previous post, which is creating alerts, Now we’ll be seeing how to create a report in Splunk. For creating a report, make sure your specific criteria have been defined or a use case has been defined for reporting every day, every week, or some specific period of time. Before creating a report, determine what action should be taken and whether it should be sending an email, generating a lookup file, or executing a script after it has been generated. You must be aware of such actions, such as what to do after the report has been generated.

The next step is to report acceleration. We’ll be coming back to this while we are creating our report to explain more about how the acceleration works. The next part is whether, if we want to embed our report in any other third-party applications, we can see how we can do that and get permission for this report’s visibility as any other object embed. We’ll see how we can set the permissions for this report, and we can also see how we can email this as a PDF or export it as a PDF. Now let us see in our lab how we can set up our report. So, are we on the right track with our search? It appears to be up. Have we logged in? Let me refresh. Yes, it seems like our Splunk is up, and we are logged in successfully. Let us create a scenario where we need a report. Let’s say since we have our tutorial data, we have lots of visitors for our site. I need the IP address from where my site has been accessed in the last month. To do that, let’s create a query. We know from our previous discussions that all the access logs are in index, which is equal to main, and the source type for my access log is access combined with cookie.

This is the name, and there is a field in our log. Allow me to demonstrate; it will be within the last 30 days. There is a field called “Client IP,” which represents the IP of the end customer or visitor who is accessing my site. I’ll just write a simple query to get their information. That is top client IP. These are the reports, let us say, or this is the report that was requested to be sent or generated every month as part of this use case. So here we are generating the top client IP for every month—that is, in the last 30 days, this was the most-visited user or the IP address for our site. If you want more information, we can enrich this by adding some geographical location. “Get the geolocation for this client IP using IPLocation Command and add some more information about the country, city, and region of this so that we have more information in our report,” I’ll say. And it’s also better understood that this IP, which is from the UK, is the most visited in the last 30 days, followed by China, the United States, and so on. We can consider this the most active user on our side. So I need a report for every month at the beginning of the month. Let’s see how we can create that.

28. Splunk Report Scheduling and Accelerating Reports

So we have finalised our query. go to “Save as Report” and here you can select a title, let’s say “Active Users in a Month,” and a description for the last 30 days. We’ll select a time range because when we schedule this, this will become useful. If you want to change our time period, I’ve saved the report successfully. Your report has been created, and you can change many parameters or the features of this report. Before that, we’ll click View so that all our reports will be under the Reports tab. Now this is our newly created report. Let’s explore some of the options. At present, we have a statistical view of the report. If you want, we can change it to another view, but for simplicity, let’s leave it as it is and see other features. This is the report that we have created. We’ll see if we can open in search; we already know the query, and if you want to change the description, you can change the Edit permission. We’ll make it global and editable only by admins and power users. And we’ll schedule this to run on the first day of each month, probably at 09:00.

This will make sure that every month it runs on the first day of the new month, which is around 9:30 in the morning. And if you want to change the time range to the previous month, we can leave it as is or change it to the last 30 days. Let us make this our highest priority, and if you want to schedule a window, let’s say I have a window of five to ten minutes for this execution; it doesn’t want to execute exactly at 09:30. It can start at 9:05, 5:05, or 8:55. You can choose this five-minute window so that this will be executed around that time. We’ll click on “next,” and then you’ll get an option that is your email action. If you click on email, This is similar to what we have seen in our alertingto, followed by the subject message description, and if you want to attach it as a PDF or inline results, let us attach it as a PDF, and you can give your email ID so that these reports will be sent to your email. And you can also run a script so that the necessary actions will be taken. Since it is a report and mostly information, we can write these results to a CSV file. As of now, this is not required, but in some cases, it may be necessary to write it to CSV files so that it can be used later. I’ll click on “Save.” Once this has been saved, you will not be able to see these results because it says this scheduled report runs only on a monthly basis on this particular date. Its time spans the last 30 days. It gives you a brief description of what it does. So until it has been run, you will not be able to see the results. That is our scheduling part; the next part is acceleration.

What is acceleration? Acceleration is best defined as a pre-run report or dashboard. In the future, we’ll be seeing dashboard acceleration,  also known as “acceleration,” or something like “pre-run,” so that before loading this dashboard on the background, Splunk will be consistently running the searches on new data, and it will be accelerating your searches so that whenever you open this reporter dashboard, you will get the latest information. And it says, “When we enable acceleration of a report, it says, what should be the range we’ll give it for three months?” So these three months of data will be stored on your search ad so that whenever you can summarise your data for the last three months or you want to fetch this report for the last three months of data, we’ll get it in a faster manner than searching for the whole index for the last three months. That is the case with acceleration.

29. Embeding Reports in External Applications

The next option is “disable.” If we don’t want this alert or report anymore, we can disable them. If you want to clone and edit a new report, we can clone the report, and the final option is the Embed option. If we want to embed, the first condition is that the search should be scheduled. Since it is a monthly report for testing our embed option, let us make it more frequent. I’ll schedule it to run based on the cron schedule to run every two minutes so that we can schedule or test our embedding features for our report. So I’ll just make sure everything is specified. I don’t need an email; I just need this report to be run. Save. Okay, I need to disable my embedded feature. It has already been enabled. I’ll disable this. It has been disabled. I’ll edit the schedule now. Chrome-based applications run every two minutes. As I said, if you are new to Cron, just let me know in the discussion. I’ll be able to help you with the Cron syntax more specifically. This is nothing more than to run this report every two minutes, which makes no sense, but it will be helpful in demonstrating our embedded features in Splunk. Now we are scheduled to run every two minutes. Allow us to enable our Embedding feature and copy this URL or the iframe, depending on which is displayed in this window. Make sure you are copied; go to reports, and once it has run, we’ll be able to test this.

So the next schedule will be ready in about two minutes. So do we have our URL? Yes, we do. This is the URL I used for testing iframes, where you can basically copy and paste this iframe that we got from our Splunk embedding. And if you click on “Run,” you’ll be able to see the display that is our Splunk output. As a result, it is likely that the scheduled report has not yet run. So we’ll wait for this, probably another minute, and we’ll be able to see how we’ll be able to get results from Splunk and the visualisation of Splunk. In any third-party application, this represents an embedding feature. Let’s say you have your own monitoring application or intelligence application where you need to fetch the visualisation part, which you can demonstrate or present in Splunk itself. This report can be used for visualisation as well as retrieving search results. And you can also use the API on your Splunk server to fetch the results and get the values so that you can make use of them as part of your application. It’s not yet completed.

Once the search has been run, the execution time, as you can see, changes from eleven six to eleven eight. Now let us rerun our iframe so that we should be able to see the results. Now, as you can see, we have our top results for the visitors. We got the same results regardless of what we defined as “active users in a month” in our report. This way, you can embed it in any of the applications by providing just this frametag as part of your application. This is the same result that has been displayed over here.

30. Creating Dashboards in Splunk

In our previous modules, we have learned more about report creation and alert creation. Now we’ll move on to one of the most important and widely used knowledge objects, which is our dashboards, which is the most common thing in our Splunk that is used independently of our industry. That Splunk has been implemented because the final output is always a dashboard, a report, or an alert.

In comparison to reports and alerts, a dashboard will provide you with a concise picture as well as excellent visualization. So in this example, we’ll go ahead and see how we can create a dashboard and add multiple panels to the dashboard, the different ways in which we can add panels to our dashboard, and also how we can view or edit our dashboard sources. That is the source code for your dashboard and for scheduling a dashboard similar to that of scheduling a report. Accelerating Dashboard. It is also similar to how you accelerate a report and how to export a dashboard as a PDF and how to share this dashboard with other users.

Now let’s get into our lab and identify how we can achieve all this. This is our searcher. Let me confirm 52 to 36. Yes. So this is our searcher. By now, we all know I’m searching for index, equal domain, and the access log, which is our web server logs of our tutorial data. Now let us create a dashboard for demonstration purposes and add a couple of panels so that the visualisation is effective. The first scenario, or the first case, where I need to see the complete user details or number of visitors for my dashboard and how they are visiting, which countries they are from, We will grab onto a couple of use cases as we proceed with creating some of the dashboards. from the use case that we created. We’ll add our top Clayton-type piece based on the country,  region, and city that we have created in our report. I’ll use the same panel, but this time we’ll be adding it to our dashboard. We have our first panel. I’ll add it as part of my dashboard. To add it as part of the dashboard Click Save, as that is the dashboard panel. We will be creating a new dashboard. I will give it the name “demo” so that this is just part of our learning.

This will be the most frequently visited iPad address in the last 30 days. I’ll share this with the app so that anybody using it will be able to see it. The panels or dashboards that we are going to create This will be included in the panel title. This will be our demonstration dashboard. This is a dashboard description. We’ll keep it as a statistics table. If you want a chart, make sure to say “dashboard” whenever you say it. We’ll go over how to do that in our next panel. We’ll save this as a chart. We’ll see how we can do that. But now we will be adding a statistics table. as you can save. We’ll use this quick menu to view the dashboard. I’ll open this in a new tab and we’ll return to it later. So now we have this. I need a graphical representation. We have too many fields here, so it might not look good on a pie chart. We’ll add it. Only the IP address should be shared so that our pie chart looks much better. I’ll add a pie chart to our already-existing dashboard by clicking on “existing.” This is the newly created dashboard demo and example.

31. Adding Panels to Dashboards And adding Panel from Report

I’ll add the same data with one more visualization. That will be our bar chart or column chart. I’ll go ahead and save this as an existing dashboard panel. Choose the dashboard you are editing, and I’ll add a column chart example, and as you can see, let me add one more, that is, a bar chart. “I’m just changing this format field and selecting a different visualization, and I’m using the same link save as the existing Dashboard panel, and this will be my bar chart example,” click.

There is an option. Panel content will be either statistics or a bar chart. We have already added statistics, so we’ll add a bar chart. So now let’s go and view our dashboard. In order to view the dashboard, you can click on Dashboards, and you’ll find your dashboard here. If you have a lot of money on your dashboard, you can simply type its name and it will appear. Choose the dashboard that you have created. Once the dashboard loads, you should be able to see all the different panels that we have added. As you can see here, we have added a pie chart, a column chart, and a bar chart.

Let me try to rearrange this panel. To rearrange any panel in a dashboard, go to the top-right corner edit, and as you can see, the edit changes your UI for how the dashboard was looking and gives you an option to move these panels as you can rearrange them at your own will. So I want the top three and three charts, and I’ll move my statistics below those. I’ll click on “Save.” So as you can see now, all my charts have been represented on top, and I have rearranged my statistics below to give me a complete picture. So this is one of the ways you can add panels. Let’s say I need to add a panel to this dashboard. I’ll copy the same query in order to know more about dashboards. We’ll see how we can create other use cases, probably as part of another lecture. But here we’ll see what other options we have for adding a panel. I’ll click on “edit.” Again, there is an option here called Add Panel. As you can see, I’ll click that, and you’ll get a new menu where you can choose any report that we have already created. As you can see from our previous tutorials, we have created a report for “Active User Cinema.” We can do that, but first we’ll make a new panel, which will contain statistics. Table, the panel title will be Panel added from Dashboard Edit.

This is an edit function where you can add a panel, choosing which type of visualisation you need. And if you already have a report or a dashboard, you can copy it from those dashboards or add it to your existing reports. If you want new ones, you can click on the specific visualisation and add those created by our dashboard edit function. I’ll choose a timeframe of the last 30 days. I’ll click on “add to dashboard.” As soon as you click, the dashboard starts loading up.So as you can see, this is our panel created by the dashboard edit function. After you click Save, you should be able to see our newly added dashboard. Sorry, newly added panel to our dashboard; there are two options from this edit function. One is via the search bar, where you can save it as a dashboard panel. The second is using your edit function. You can also use this edit function to add a report to an existing dashboard, clone it, and add existing scheduled reports as panels. These are some of the scheduled reports that you can schedule to run at a specific time and load onto your dashboard.