Designing Splunk Architecture

5. Estimation of Storage for Indexers

Finally, there is one last option, "User," for determining your storage. Till now, we have learned about licences, licence size, number of indexes, number of searches, heavy forwarder deployment, servers, and licence managers, including why we need to have clustering and high availability options. We will now go forward with calculating storage. The storage is the important part of your indexes, and the storage would have always been greater than 200 I'm up for getting a rough estimate on storage for the indexer.

The link mentioned in the document should take you straight into calculating the index storage, which has been used by most of the Splunk consultants, and this is not officially connected to Splunk or authorised by Splunk authorises it.You can see it's not supported by Splunk, and it is completely independent, probably built by one of the Splunkers to help other fellow Splunkkers. This is a very good side for getting a rough estimate of storage for the indicator. When I say rough estimates, it's accurate, like 80 or 90%. That's not like complete rough. Okay, so we are getting like 80% to 90% of accurate storage as of now. In my whole experience of four to five years, I've been using this site, and I have not faced any miscalculations or misjudgments while assigning a storage.

We can consider that a good example. Most of the time, when I go to a customer and ask them, "What is your log size, and what do you need it for?" They can't understand or give a straight answer based on log size. Few of them give us size in events per second because most of the traditional SIM solutions were like Q and Lapse, our site, and the logarithm. They were using EPS, though they were familiar with it. They will come back and say to you that I have a $5,000 or even a $2,000 event. So what do you do? You visit this site and check on this box size per event. It will give us a rough estimate of how much licence you need. Probably you can go up to 60 gigabytes in the case of two k, and if you choose five k, you'll get around 115 gigabytes. You can add a 10% buffer, and you can go up to 130 gigabytes. This is how you calculate, as I argued and defined, that this is what my licencing I need.The next concept we come to is data retention. This depends on your complete policy. Let's say I need six months of data.

I'll choose three months hot and three months cold. We'll be going through what is this "hot" cold archive, "frozen retention, how to calculate them, how to configure them, and the later part when we dive deep into indexes and see how it operates and stuff. For now, consider this the time during which you need to keep your data in Splunk. Let's say I need three plus three plus six months of data in Splunk. We are not going to use any premium maps. If you use, you can choose one of them and considerably the architecture. Or it recommends, as you can see, Splunk Enterprise Security, which is one of the premium apps. It will recommend 100 GB per indexer. This should be automatically updated. Yet here you can see that it says it requires 5.1 TB of storage per indexer. So totally, you need like 100 TB. So here it is saying we need only one indexer because if you choose Planck Enterprise, it says maximum volume per indexer, and you can go up to 100 GB. Will you choose without anything?

So it says it can handle up to 300 GP. But we know how it works—the storage configuration. If you have separate volumes mounted for it, you can mention them. But as of now, the only important number is 10.2 TB. You have calculated your storage requirement based on EPS. Let's say I'll come back, I'll have 100 GB of licences, and I'll need six months of retention. I'll come back here and check the storage I require for indexes, which is 8.8 TB. So this is how you calculate the storage requirement for your indexes. You.

6. Small Enterprise Architecture review

Now, since we have all the required factors, let's proceed to have a look at what different architecture looks like for demonstration purposes. These are purely designed based on my experience on Splunk after working for close to five years. In this tutorial, I have made three scenarios of architecture: small, medium, and large enterprises, and the crazy one, which I call the large enterprise with high availability and clustering mentioned, Let's go through them one by one. The first one, as we see in our screen, is the small architecture, which can be compared to an organisation with a licence limit that is less than 100 GB per day.

As you can see in this picture, we have one indexer, one searcher, and a couple of people using it. And we have lots of data forwarded from many different sources. This is a typical small enterprise architecture. It looks like here even the searcher can be optional. If you have just one or two users, all you can do is have one indexer be able to handle the one or two users' load. You can deploy everything as a single standalone instance. But always keep in mind that in a large organisation, even if the licence is 100 GB per day, it's good to start deploying them in distributed mode. Let's say that Splunk is in "distributed mode," which means that all its components are assigned dedicated roles. That means search is separated, indexers are separated, and forwarders are separated, especially heavy forwarders. So this is how each dedicated rule is configured.

So we call it a distributed mode of deployment. I say this because the organization, let's say a big corporation, has purchased a ten-gigabyte licence for just monitoring their locks. My experience is that once the organisation realises the value of Splunk and the variety of data it can process, it can scale from ten GB to several hundred GB in a matter of months. So as an architect, you should be aware of what level of data you have in your organisation and how big it is your organization.So that you should always think one step ahead so that you cover all these scenarios. And you should be aware that if I expand from 10 to 100 gigs, I will need to add more indexes and more searches. So it's better to deploy without going into distributed Distributed Mode.And as we have discussed, it can be scaled up from a single instance setup to a distributed setup, where each component will be responsible for specific tasks at any point in the Splunk installation or operations phase. In previous videos, we have gone through saying that you can add your search at any time and that you can add your index at any time.

There is no impact, no data loss, and no operational disruption because that is how Splunk has been designed. It is easy to scale horizontally and vertically. You can add resources to one machine, and you can add additional components to one architecture, but you still have all the data. You can search; you can report; you can do everything. But it's always a good practise to start with a good foundation. Let me open up the designs that I have created for architecture. So here is the architecture that we have created for small enterprises with less than 100 GP of data. We can see we have only one indexer, one searcher, and many universal forwards, which are covered as a block because it will look ugly if I draw lines from all these endpoints to point to one index. So I made it a container and sent it as a single. The final motor is simple All the logs that are collected by agents or syslog devices will be sent to the indexer.

This is what a typical small Splunk architecture looks like. The data sources can be syslog, firewalls, universal forwarders, Apple Solaris, Linux, Windows, or even scripted inputs that will be sending logs to indexers and the search. If you come to the data flow, the logs have been collected from the universal forwarder and sent to the indexer where they are parsed, broken down into pieces, and then stored in our storage. The indexer holds 100% of the data here The search engine is the one which queries the indexer based on the searches we write or the alerts or reports, and this query is run on the storage and the indexer fetches the results, giving them to search for visualisation or alerting purposes.

7. Medium Enterprise Architecture review

Now we have a good idea of what a small enterprise looks like. Let's proceed further to medium, enterprise architecture, or Splunk? From the previous discussions and considerations, we have come to the conclusion that we should use more than the GB per day licencing limit to have multiple indexes. In this scenario, we have a medium enterprise architecture, which can consume or ingest 100 to 200 GB per day into a Splunk instance. By now, we know that we have multiple indexes in a medium enterprise. As you can see, there is one group of indexes.

So it's better to have more than one index. If you can afford three, you probably can have three. Also, by now, we know that we are having multiple indexes in medium enterprises, and when it comes to searching, it's better to have more than one searcher if you have more than eight users, and if you're using any of the premium apps offered by Splunk, it's good practise to dedicate one complete searcher to the premium apps. Since premium apps like Enterprise Security, VMwareApp, ITSI, or PCI have very high resource utilisation, They will be running constant searches, alerts, dashboard acceleration, data model acceleration, and a lot of other stuff inside the premium apps. So it's better to dedicate one complete searcher to your premium apps. The premium apps come with preset collection offers, ports, dashboard alerts, and data sets. These are the things that are constantly running in the background.

And along with this, if you add a document such as one run by normal users and the alerts or reports that you have created, it adds additional load to the search ads, and search response starts to deteriorate. In our diagram we can see we have two searchapps and two indexes, or probably multiple indexes, which is pretty common for a medium enterprise plank architecture. Here, the data flow will be like the universal forwarders. Let me open up the bigger architecture. This is a medium-sized enterprise. Here the data flow will be like the universal forwarders are fetching the data and feeding it to a group of indexes and syslog devices. Also sending logs to our indexes directly; there is no intermediate or any forwarding. If the syslog devices are noisy and causing trouble for the indexer, we can probably install a forwarder in between. Now the data flow: we collected the logs using universal folders, sent them to indexes, and the same indexer passes them and stores them. The searcher runs the query indexer, takes the query and searches it on the storage, fetches the results, and gives it back to search. The searcher uses this data for visualisation or alerting purposes. Now, in this architecture, everything seems fine for a medium enterprise, but as a Splunk, your job is not yet done.

Before finalising the design, evaluate the need for deployment, server analyze, and predict how many agents or universal forwarding clients will be in your deployment if it is greater than 25 to 50 clients. Of course, it's good practise to have a deployment server so that your management of these clients will be pretty good. You can put in a deployment server somewhere here because it will not be in your operational architecture, so it will be somewhere standing out. But if you are having just ten to twenty clients, which are quite noisy and generate hundreds of gigabytes of data, it should be good to trade off just to postpone your deployment server in case of your next plan of scaling up your architecture. Now that we have evaluated deployment, let's see about Aviv's forwarder. As we discussed earlier, if our data sources in our deployment or in our organisation are very chatty or sending lots of junk data to Splunk, then it would be best to have heavy forwarders in between your universal forwarders and indexers.

That will be somewhere here, where in between universal forwarders and indexes you can have your heavy forwarder, the Syslog service. Then the data flow will change to heavy forwarding, which will receive the logs from Syslog servers it passes. It then sends the logs to your indexer, which will be so efficient that the logs will also be processed before reaching your Splunk indexes so that the passing load on the indexes is also reduced. instead of indexing on multiple ports for Syslog or multiple IPS or receiving data from multiple points. It can efficiently use its precious inputs and outputs. Operations per second (IOPS) for concentrating and receiving the logs from heavy forwarding Because index will have only one source for the reception of logs, that will be our only forwarder. so that most of its I O can be used for storing and fetching results for our queries. Based on this evaluation criteria for having a folder, as an architect, you will decide whether to have one or not. We have gone through the medium Splunk enterprise thoroughly as of now. You might have a good idea of how Splunk's architecture grows from a single instance.

8. Large Enterprise Architecture review : Part 1

Since you understand now how Splunk artefacts can grow from a single standalone machine to a large enterprise plank with the same flow, let's go through a simple enterprise scenario. According to our previous architecture designs, like Small and Medium Enterprise reviews, we have come to the understanding that it's good to have forevery one hundred gigabytes of logs or every hundred gigabytes of licence and additional indexes. Since in our previous example we considered anything between 100 and 200 GB as our medium enterprise for a large enterprise, in this example we will consider any licence of 250 GB or greater as a large enterprise.

We have come to the conclusion that we will have more than two indexes that can increase linearly with your licence limit. We have a number of indexes figured out. Let's see how many search ads we need. As we have discussed earlier in other architecture reviews, if we have more than eight users who are constantly active on Splunk and are running multiple searches, we can have additional searches, and it is totally fine to begin with one searcher and scale up the searches as the number of users are increasing.To begin with our architecture here, which represents a typical large environment, we can see there are groups of searches. Probably one will be with premium apps, and the other will be, let's say, premium apps, Enterprise Security VMware, or PCI, and a group of servers for normal users who are accessing and running hard searches.

Investigating these kinds of use cases, it is reasonable to assume that there will be multiple teams accessing Splunk because they are indexing more than 200 gigabytes of data. Of course, they will have more than eight or 15 people accessing their Splunk environment, as well as any premium purchases. As we discussed Enterprise Security, PCI, VMware, and ITSI—all these premium apps—they might be using one or two for that app.So for those apps, you need dedicated searches. We can allocate them to individual searches. Hence, in this architecture, we have multiple searchers carrying the user load of the Splunk environment. In our architecture, we see that we have groups of multiple indexes. Based on our understanding, we know that by now we will have more than two indexes for this environment, and these indexes store all the data that has been processed. In this architecture, the storage for these indexes will be holding 100% of your data.

Whatever data has been processed in Splunk will be shared among these three or four indexes in standalone mode. By default logic, each index out of these three receives the data from the forwarder over a certain amount of time, and then the forwarder forwards the data to the next available index to send the next chunk of data. And let's say we have three indexes; the data is now like 33/33% available per index. This is a typical round robin algorithm based on time. In this mode, if one indexer goes down, your searches will return just approximately 67% of the results from the two available indexes. Let's say this went down. You will have around 66% to 67% of results from only two indexes. To get around this, we will see how clustering can help. The clustering is planned as two different types: a single side cluster and a multi site cluster. We'll see about this in the later parts. And we'll also go through the deep technical aspects of Splunk clustering when we're implementing our own enterprise-level, high availability multisite clustering setup in Amazon Web Services as part of this tutorial.

9. Large Enterprise Architecture review : Part 2

For simplicity's sake, this module. In this slide, we'll be considering side indexing or clustering. In single-sided indexing or clustering, each index exchanges the data that it has received from the forwarder so that, at any point, if one indexer fails, the data can be retrieved from the other two instances. To understand how this actually works, we need to know more about replication and search factors, which we'll be covering at a later stage because of the complexity involved in choosing them and the depth of the concept of application and search factors. Moving on. Now we understand that single set clustering in Splunk exchanges data between indexes in order to overcome any data loss or disruption into our search results during an event of any indexer going down. Now we got that clear.

Let's move on further down and compare this with our medium architecture. If you see our medium architecture, this is our medium architecture. You can see that there are some components like the deployment server, heavy forwarder, and licence Forwarder and License Manager. There are three blocks that are new in Large Enterprise. As you can see, these three components are present in the large enterprise of Splunk's architecture. We all know by now that to expand the deployment of Splunk, we must have a deployment server for managing Splunk, Universal Forwards, and other Splunk components. By having a single deployment server, we will help reduce the complexity of managing Splunk infrastructure. Consider the example of having 200 clients and logging them individually to just change the IP address or the host name of that instance.

Instead of that, you can push this configuration from the deployment server probably in 5 minutes rather than logging into 200 different machines one at a time. The deployment service acts like a boss in an architecture, where it talks to all the Splunk components across the architecture and tells each component how to operate and what configurations are deployed on each component. And now we can also monitor the health of the components throughout the infrastructure using Deployment Server. Now we're clear about the deployment server. Let's move on to our License Manager, where we see from the architecture that it keeps tabs on our licence utilisation and can also alert based on a licence violation upon reaching our thresholds as per our alert configurations for this licence usage.

This functionality of License Manager is almost identical to that of a non-cluster or clustered environment, where its only function is to collect the licence usage from all the components and keep track of the violations in the daily usage statistics of our licences and report them accordingly. And we also see in this architecture that there are heavy forwarders between Universal Forwarders or our data sources and indexes.

This heavy forwarder is receiving the data from forwarders and passing it along. It filters some of the events and sends it to the indexer, where the indexes will be receiving the data from. These heavy forwards, which are passed and filtered by the heavy forwarders, are being received on the indexes and stored on the indexes themselves. This is a good option, and it is best practise to have an AV forwarder in your architecture for large deployments. It will add significant value to the Splunk architecture. By now we are clear on the three architectures of small, medium, and large enterprises. Now let us see one of the best architectures of Splunk, considering a scaled up version of a large deployment.

Prepared by Top Experts, the top IT Trainers ensure that when it comes to your IT exam prep and you can count on ExamSnap Splunk Enterprise Certified Admin certification video training course that goes in line with the corresponding Splunk SPLK-1003 exam dumps, study guide, and practice test questions & answers.