Designing Splunk Architecture

4. Evaluation : Heavy Forwarder, License Manager and Deployment Server

Let's evaluate them one by one. Consider a V forwarder. There are three brilliant cases in which you can use a forwarder in your architecture. Number one is to filter out the locks. Let's say my firewall is just killing my licence and it is sending 200 GB of data per day. I'll filter out all the denied connections I can filter out denied connections at the heavy forwarder level. And also, I can remove some event codes from the Windows event log to reduce the noise on my licence or the locks on my indexes. The first one is to filter out the logs. The second one is to mask your sensitive information from the logs. Let's say I need to anonymise some of the data which I'm sending to logs. I have some credit card information from my database which needs to be retrieved on Splunk to analyze,but the credit card numbers should be marked. You can do this using a heavy forwarder. Getting to the third point, having a heavyforwarder can add a major performance boost for your indexer. Assume your indexer is receiving 200 sysloginputs, iOS, or previous IOPSinput output operations per second. It's like gold for your indices. The more it has, the more efficient it is. When you are receiving 200 different log inputs from 200 different IPS on the indexer,your indexer is reading from 200 different sources. Let's say it is trying to read. For this example, we'll consider it committing 200 read operations for just receiving those logs, which is highly acceptable because I have only one left for processing, for storing, for fetchingthe results and giving them to my server. So what do I do? I place a V forwarder, intercept all this 200,reduce the noise initially, the logs break down into pieces, then I'll feed it to my indexer. This will add a good performance boost and release some of your iOS from your indexor, which is done by your average time to go through all three cases where you need AV forwarders. One to filter out the noise in the logs, the second to mask sensitive information in the logs. Third, to boost your index or performance. These are the three cases where you can justify having or evaluating successfully having a heavy forward. The next one is evaluating the need for a deployment server and a licence master. Remember, the deployment server is a must in large scale deployment where you will be having hundreds of forwarders to manage along with your Splunk instance. The deployment server will be your friend which can help in changing the configuration of a large number of clients. When I say clients, it can be your universal forwarder,it can be your indexer, it can be your searcher,it can be your average, it can manage all the clients and change the configuration in a matter of minutes. But if you have a small deployment of 10 to 20 clients to manage, there is absolutely no real reason for having a separate deployment server. If you're scaling up in the future, make sure you add a deployment server to manage your clients. And also, one point to remember is that our deployment server plays a vital role in managing the entire Splunk infrastructure and its clients. Now coming to the licence manager, this component, as we already know, keeps track of licence utilisation by communicating to all our indexes. In most of these cases, this is an optional step because the searcher, indexer, or your deploymentserver can perform this function. This licence master functionality is very minimal when compared to other components in most environments or organizations, as you can see. It is usually clubbed along with your searcher or your indexer. The next and last option before moving onto storage calculation is clustering and High Availability.We will be discussing more about clustering in a separate module. As of now, we'll proceed with why we need clustering and Splunk. There are two main reasons for considering clustering and High Availability.Number one is availability of your data so that if any single instance of your indexer goes down,there should not be any impact or search results should be retrieved from one indexer. So that if I have two indexes in my environment, let's say I don't have clusteringand high availability enabled, the indexer has like 50% of the data at any point of time. If one indexer goes down, I get only 50% of the results, which is not accurate. That is one scenario to go for a cluster. The second option is the integrity of data, so that the file system gets corrupted. You're not able to restore the data on one of the indexes. That shouldn't give you a chance to lose 50% of your data, but if you are clustering enabled, it should be available on other indexes. These are some of the major factors to know before designing an architectural project.

5. Estimation of Storage for Indexers

Finally, there is one last option that is used for determining your storage. Till now we have learned about licences, licence size,number of indexes, number of searches, heavy forwarderdeployment, server and licence manager, including why we need to have clustering gun high availability options. We will now go forward to calculating storage. The storage is the most important aspect of your index, and it should always be greater than 200. I'm up for getting a rough estimate on storage for the indexer. The link mentioned in the document should take you straight into calculating the index storage which has been used by most of the Splunkconsultants and this is not officially any way connected to Splunk or Splunk authorises it. You can see it's not supported by Splunk and is completely independent, probably built by one of the Splunkers to help other fellow splunkers. This is a very good way of getting a rough estimate of storage for the indicator. When I say rough estimates, it's accurate like 80, 90%. That's not like a complete rough Okay, so we are getting like 80% to 90% of accurate storage as of now. I've been using this site for four to five years now, and I've not faced any miscalculations or misjudgments while assigning a storage. We can consider that a good example. Most of the time, when I go to a customer and ask them what size log they need, They can't understand or they can't give a straight answer from a log size. A few of them give us size by events persecond because most of the traditional SIM solutions were Q and Lapse, our site and logarithm. They were using EPS, though they were familiar with EPS. They will return and tell you that you have a five-kilometer, four-kilometer, or even two-kilometer event. So what do you do? When you visit this site, check the box size per event. It will give us a rough estimate of how much licence you need. You can probably go up to 60 gigs in the case of two thousand, and if you choose five thousand, you'll get around 115 gigs. You can add a 10% buffer and you can go up to 130 gigs. This is how you calculate, as I argued, and this is what my licencing I need.The next concept we come to is data retention. This depends on completing your policy. Let's say I need six months of data. I'll choose three months hot and three months cold. We'll be going through what is this hot coldarchive, frozen retention, how to calculate them, how to configure them, and the later part when we dive deep into indexes and see how it operates and stuff. For now, consider this the time that you need to keep your data in Splunk. Let's say I need three plus three months of data in Splunk. We are not going to use any premium maps. If you use them, you can choose one of them and, considerably, the architecture. Or it recommends like, as you can see, SplunkEnterprise Security. It's one of the premium apps. It will recommend 100 GB per indexer. This should be automatically updated. Yet here you can see that it says it requires 5.1 TB of storage per indexer. So totally, you need like 100 TB. So here it is saying we need only one indexer because if you choose Planck Enterprise, it says maximum volume per indexer, you can go up to 100 GB. Will you choose without anything? So it says it can handle up to 300 GP. But we know how it works, the storage configuration. If you have separate volumes for it, you can mention them. But as of now, the only important number is 10.2 TB. You have calculated your storage requirements based on EPS. I'll come back. I have 100GB of license. I need six-month retention. I'll come back here and check the storage required for indexes is 8.8 TB. So this is how you calculate the storage requirement for your indexes. You.

6. Small Enterprise Architecture review

Now, since we have all the required factors,let's proceed to have a look at what different architecture looks like for demonstration purposes. These are purely designed based on my experience with Splunk after working for close to five years. In this tutorial, I have made three scenarios of architecture that are small, medium, and large enterprises, and the crazy one, I call it, as it's the enterprise with high availability and clustering mentioned. Let's go through them one by one. The first one, as we see on our screen, is the small architecture, which can be compared to an organisation with an alicense limit of less than 100 GB per day. As you can see in this picture, we have one indexer, one searcher, and a couple of people using it. And we have lots of data forwarded from many different sources. This is a typical small enterprise architecture. It looks like here, even the searcher can be optional. If you have just one or two users, all you can do is have one indexer that should be able to handle the one or two users' load. You can deploy everything as a single standalone instance. But always keep in mind that in a big organization,even if the licence is 100 GB per day, it's good to start deploying them in Distributed Mode. Let's say Distributed Mode. It means that all the Splunk components are in dedicated roles. That means search is separated, indexers are separated, forwarders are separated, heavy forwarders. So this is how each dedicated rule is configured. So we call it a "Distributed Mode of deployment." I say this because the organization, let's say a big corporation, has purchased a tenGB licence for just monitoring their locks. My experience showed that once the organisation realised the value of Splunk and the variety of data it could process, it could scale from ten GB to several hundred GB in a matter of months. So, as an architect, you should be aware of what level of data you have in your organisation and how big your organisation is. So you should always think one step ahead so that you cover all these scenarios. And you should be aware that if I expand from ten to 100 gigabytes, I will need to add more indexes and searches. So it's better to deploy without going into a Distributed Mode.And as we have discussed, it can be scaled up from a single instance set up to a distributed setup where each component will be responsible for specific tasks at any point of Splunk installation or operations phase. In previous videos we have gone through saying that you can add your search at any time, you can add your index at any time. There is no impact, no data loss, or no operational disruption because that is how Splunk has been designed. It is easy to scale horizontally and vertically. You can add resources to one machine. You can add additional components to one architecture. But you still have all the data. You can search, report, and do everything, but it's always a good idea to start with a good foundation by going to architecture. Let me show you some of the designs I've created for architecture. So here this is the architecture which we have created for a small enterprise which has less than 100 GP of data. You can see we have only one indexer, one searcher and many universal forwarders, which are covered as ablock because it will look ugly if I drawlines from all these endpoints to point to one index. So I made it a container and sent it as a single. The final motor is simple All the logs which are collected by agents or syslog devices will be sent to the indexer. This is how a typical small splunk architecture looks like. The data sources can be syslog. Firewalls Universal forwarders Apple Solaris Linux. Windows or even scripted inputs will be sending logs to indexersand the search if you come to the data flow. The logs have been collected from the universal forwarder and have been sent to the indexer where they are parsed. broken down into pieces and then stored in our storage. The indexer holds 100% of the data here The search engine is the one which queries the indexer based on the searches we write or the alerts, or the reports, and this query is run on thestorage and the indexer fetches the results, giving them to search for visualisation or alerting purposes.

7. Medium Enterprise Architecture review

Now we have a good idea of what a small enterprise looks like. Let's proceed further to medium-enterprise architecture or Splunk? From the previous discussions and considerations, we have come to the conclusion that we can use more than the GB per day licencing limit to have multiple indexes. In this scenario, we have a Medium Enterprisearchitecture which can consume or ingest 100–200 GB per day into a Splunk instance. By now we know that we have multiple indices in a medium-sized enterprise. As you can see, there is one index group of indexes. So it's better to have more than one index. If you can afford three, you can probably have three. Also, we know that we have multiple indexes in a medium enterprise, and when it comes to search, it's better to have more than one searcher if you have more than eight users, and it's also good practise to dedicate one complete searcher to the premium apps if you're using any of the premium apps offered by Splunk. These are apps that consume a lot of resources. They will be running constant searches, alerts,dashboard acceleration, data model acceleration, and there is a lot of stuff that will be going on inside the premium apps. So it's better to dedicate one complete searcher to your premium apps. The premium apps come with preset collectionoffer, ports, dashboard alerts and data sets. These are the things that are constantly running in the background. And along with this, if you add a doc such as run by normal users and the adopt alerts or reports that you have created, it adds additional load to the search ads and the search response starts to deteriorate. In our diagram we can see we have two searchapps and two indexes, or probably multiple indexes, which is pretty common for a medium enterprise plank architecture. Here the data flow will be like the universal forwarders. Let me open up the bigger picture. This is a medium enterprise.Here the data flow will be like the universalforwarders are fetching the data and feeding it to the group of indexes and syslog devices. We are also sending logs directly to our indexes; there is no intermediate or forwarding. If the syslog devices are noisy and causing trouble for the indexer, we can probably install a forwarder in between. Now the data flow, we collected the logs using universal folders, sent them to indexes, the same indexer passes them and stores them. The searcher runs the query indexer, takes the query and searches it on the storage, fetches the results, and gives it back to search. The searcher uses this data for visualisation or alerting purposes. Now, in this architecture, everything seems to be fine for a medium enterprise, but as aSplunk, your job is not yet done. Before finalising the design, evaluate the need for deployment; server analyse and predict how many agents or universalforward will be in your deployment if it is greater than 25 to 50 clients. Of course, it's good practise to have a deployment server so that your management of these clients will be pretty good. You can put in a deployment server somewhere here because it will not be in your operational architecture,so it will be somewhere standing out. However, if you only have ten to twenty clients that are quite noisy and generate hundreds of gigabytes of data, it should be sufficient to trade off simply postponing your deployment server in case your next plan to scale up your architecture. Now we have evaluated deployment, let's see about Aviv forwarder. As we discussed earlier, if our data sources in our deployment or in our organisation are very muchchatty or sending lots of junk data to Splunk,then it would be best to have heavy forwarders in between your universal forwarders and indexers. That will be somewhere here, where in between universal forwarders and indexes you can have your heavy forwarder, the Syslog service. The data flow will then change to heavy forwarder, which will receive logs from Syslog servers as they pass. It is then sent to your indexer, which will be much more efficient so that the logs will also be processed before reaching your Splunk indexes, so that our passing load on the indexes is also reduced. Instead of indexing on multiple ports for Syslog, multiple IPS, or reception of data from multiple points, It can efficiently use its precious inputs and outputs. operations per second, or IOPS, for concentrating and receiving the logs from heavy forward us.Because index will now have only one source for log reception, it will be our every forwarder. So that most of its I/O can be used for storing and fetching results for our queries. Based on this evaluating criteria for having a folder, as an architect, you will decide whether to have one or not. We have gone through the Splunk enterprise thoroughly as of now. You might have a good idea of how Splunk's architecture grows from one single instance.

8. Large Enterprise Architecture review : Part 1

Since you understand now how Splunk artefacts can grow from one single standalone machine to an enterprise plank with the same flow, let us go through our simple enterprise scenario. According to our previous architecture designs like Smalland Medium Enterprise reviews, we have come to understand that it's good to have forever a hundred gigabytes of logs or every hundred gigabytes of licence and additional indexes. Since in our previous example we considered anything between 100 and 200 GB as our medium enterprise for large enterprise, in this example we will consider any licence of 250 GB or greater than a large enterprise. We have come to the conclusion that we will have more than two indexes which can linearly increase with your licence limit. We have a number of indexes figured out. Let's see how many search ads we need. As we have discussed earlier, in other architecture reviews,if we have more than eight users who are constantly active on Splunk running multiple searches, we can have additional searches and it is totally fine to start with one searcher and scale up the searches as the number of users are increasing.To begin with our architecture here, whichrepresents a typical large environment, we can see there are a group of searches. One will probably be with premium apps, and the other will probably be, let's say, premium apps, Enterprise SecurityVMware or PCI and a group of servers for normal users who are accessing and running hard searches. Investigating these kinds of use cases, it is reasonable that there will be multiple teams accessing Splunk because they are indexing more than 200 gigabytes of data. Of course, they will have more than 8 or 15 people accessing their Splunk environment and also any premium purchases. As we discussed Enterprise Security, PCI, VMware, ITSI,all these premium apps, they might be using one or two for that app. So for those apps, you need dedicated searches. We can allocate them to individual searches. Hence, in this architecture, we have multiple searcherscarrying the user load of the Splunk environment. In our architecture, we see that we have a group of multiple indexes. Based on our understanding, we know that by now we will have more than two indexes for this environment. These indexes store all the data that has been processed in this architecture, so that storage for these indexes will be holding 100% of your data. Whatever the data that has been processed in Splunk will be shared among these three or four indexes in standalone mode. By default logic, each index out of these three receives the data from the forwarder for a certain amount of time, and then the forwarder forwards the data to the next available index to send the next chunk of data. And let's say we have three indexes. The data is now like 33/33% available per index. This is a typical round-robin algorithm based on time. In this mode, if one indexer goes down, your searches will return just approximately 66% of the results from the available two indexes. Let's say this went down. You will have 33 to 67% of results from only two indexes. To get around this, we will see how clustering can help. The clustering is planned as two different types: a single-side cluster and a multi-site cluster. We'll see about this in the later parts. And we'll also go through the deep technical aspects of Splunk clustering when we're implementing our own enterprise-level, high-availability multisite clustering set up in AmazonAWS as part of this tutorial.

ExamSnap's Splunk SPLK-1002 Practice Test Questions and Exam Dumps, study guide, and video training course are complicated in premium bundle. The Exam Updated are monitored by Industry Leading IT Trainers with over 15 years of experience, Splunk SPLK-1002 Exam Dumps and Practice Test Questions cover all the Exam Objectives to make sure you pass your exam easily.