CompTIA CYSA+ CS0-002 – Analyzing Application-related IOCs Part 1

1. Analyzing Application-related IOCs (Introduction)

In this section of the course, we’re going to discuss how we can detect and analyze applicationrelated indicators of compromise. Now, in this section, we’re going to continue in domain four with a focus on Objective 4. 3 and Objective 4. 4. Objective 4. 3 states that given an incident, you must analyze potential indicators of compromise. In this section, we are going to focus on those application related IOCs. Now, Objective to 4. 4 states that given a scenario you must utilize basic digital forensic techniques specifically in terms of virtualization and mobile devices, for this section of the course. Now, as we move through this section, we’re going to start with a look at exactly what an application related IOC is and how they’re used to detect malicious activity within our networks. Then we’re going to move into some common IOC types and how to identify them.

This includes identifying anomalous activity, service interruptions, analyzing application logs, and how to detect new accounts that may have been created. Also, we’re going to cover forensics for virtualized systems or applications and mobile devices inside this section of the course. This is because we’re going to be more focused on application related IOCs when we’re doing forensics on these type of systems. Finally, I’m going to perform a demonstration where we’re going to analyze some application related IOC together inside a lab environment so you can better understand what various attacks look like when you’re working as a cybersecurity analyst. All right, let’s go ahead and get started analyzing our workstations and servers to see if we can find any of these application related indicators of compromise.

2. Application-related IOCs (OBJ 4.3)

Applicationrelated indicators of compromise. In this lesson, we’re going to talk about the importance of looking at applicationrelated indicators of compromise. Now, this is important because by observing application behavior, we can start to identify and reveal signs of an intrusion. Now, a lot of this is going to overlap with some of the things we looked at in hostbased analysis for those IOCs as well. But this is more than just looking at the operating system itself. It’s more focused on the applications that run on top of that operating system. For example, while it’s common that most of our applications that are attached to our operating systems or run on top of those operating systems could be subject to malware these days, so can web applications.

By observing their behavior inside these applications and analyzing their code or analyzing the changes they make to a system or procedure call, we’re going to be able to identify indicators of compromise that we can use to identify other infected systems from those same applications. Now, when it comes to identifying these different types of indicators of compromise, where can we find them? Well, the most commonplace is our application logs. These application logs can provide indicators of compromise for us and we’re going to go through some of those logs in this section of the course. Now, the final area that we have to consider is virtualization because a lot of our applications these days are run in containers using things like docker and other containerization mechanisms. This allows applications to run inside a virtualized environment. And so we are going to spend some time looking at virtualization as well inside this section.

3. Anomalous Activity (OBJ 4.3)

Anomalous activity. In this lesson we’re going to look for indicators of compromise that are made up of anomalous activity. Now, when I talk about anomalous activity, this means we need to look in different areas such as web applications, databases, DNS services, and remote access servers. By looking at these, we can identify anything that looks unusual to us. If it’s something that’s unusual that will be deemed anomalous and based on that anomalous activity, this can then be categorized as benign, meaning harmless, or something that is malicious that we have to identify as a potential indicator of compromise. Now, as we start looking at this anomalous activity, there are lots of different symptoms of it. This can include things like strange log entries, excessive per process ports and resource consumption, and unusual user accounts.

We’ve mentioned some of these when we looked at hostbased IOCs earlier, but now we’re going to dig a little bit deeper. Now, as we start looking at these, we might have things like unexpected outbound communication, unexpected output, or service defacement. And we’re going to talk about each of these three in this lesson. First, unexpected outbound communication. In this area, we are looking through all of our network connections and trying to identify them and make sure we understand them and that any outbound connections are approved. If I’m looking through my firewall logs or my router logs and I start seeing a lot of connections going out and I don’t know where they’re going and I haven’t approved them, that could be an unexpected outbound communication.

That could be a sign that there’s a C two channel or beaconing or something like that that’s going on and it’s something that I want to look into further. All of that is stuff that could be categorized as anomalous activity. The second area is unexpected output. Now, unexpected output can occur when you have unusual request patterns or responses and these can be indicative of an ongoing or past attack. For example, if you have a web application and when you go and run a query against it, you get some sort of unexpected output from that query that could indicate that that server has been under attack either now or in the past. And so it is something you’d want to look into further. Now, in addition to looking for unexpected output, sometimes that output can come in the form of code.

For example, if you’re trying to detect a code injection, you can do this by monitoring the number of database reads or examining the Http response packet sizes. If you’re using a web application and it makes a call to a database and gets back a very large amount of information, it could be a symptom that you had a code injection like an SQL injection that is now returning all of the database instead of just a single query. Additionally, if you find an application that’s displaying unformatted error messages or strange strings, this could be an indication of an application that’s been tampered with. And again, this is another form of unexpected output. Now, the third category we have is service defacement.

When we talk about service defacement, this occurs when an attacker gains control of a web server and alters the website’s presentation. This can be done for lots of different reasons. For example, some defacement attacks are like this, where they’re just trying to show somebody, hey, I was able to break into your system and you should really fix your system because it’s vulnerable to attack. And they’re doing it more just as a way to get credibility as a hacker in the community. But it can also be more subtle than this. There are some cases where companies have been attacked and somebody has gone in and modified their website just slightly to put out different branding and messaging.

This can actually have a negative effect on the company. For example, maybe an attacker has taken over a website for a health food store and they started posting a lot of products that have chemicals in them and things like that that can actually be negative for that store and much more subtle, less likely to be detected for a longer period of time. This is again, a type of website defacement. If you find that your organization’s website has been defaced, you want to go in and restore to a known good backup. And then you want to go through and figure out how they were able to get into your website and change it so you can block them from doing it again.

4. Service Interruptions (OBJ 4.3)

Service interruptions. Service interruptions are another area that you should consider when developing your IOCs. Now, just because there is a service interruption though, it doesn’t mean that this is an indicator that you’ve been attacked. Sometimes application services may fail to start or stop unexpectedly for any number of different reasons. Some may be a maintenance issue, some may be normal wear and tear, and some may be that you’re under attack. Now, if you have failed application services, this is any application interruption that’s caused by a service either failing to start or halting abruptly. If you want to identify any of these failed application services, the best thing to do is look through your logs. In your logs, you should be able to have a full history of every service that was started or abruptly stopped.

Now, as you look through those failed application services and those logs, you’re going to have to determine if this was something that was malicious or if it was something that’s just benign and happened as an accident. As you do this, you need to consider a lot of different things from your cybersecurity perspective as an analyst. First, you want to consider if security services are prevented from running. This can be an indication that there is something malicious on the system. Many times Malware will install itself and then prevent other security software such as antivirus, from loading in the background. Second, you want to consider the process that’s running the service and whether or not it’s compromised. If you find something malicious, you can then look at that process and see which process launched that malicious process.

If it was something that was a system process, for example, that may indicate that the entire system is compromised and something you need to look into. Third, the service may be disabled by a DDoS or a Dos. In this case, you may have a denial of service going on. And if this is happening, this would be a clear indicator of compromise that you’d want to make note of. And fourth, you want to check for excessive bandwidth usage because if there’s excessive bandwidth usage, this could disrupt your service as well. Now, just because there’s a large amount of bandwidth being used doesn’t mean you’re actually under attack though. This could actually be real traffic from your users. That’s all just happening at the same time. For example, if there’s a big world event and all your users tried to go and look at the news right at the same time, that could cause a denial of service for your service due to excessive bandwidth. It’s not malicious, it’s just something that happened that everybody was interested in all at the same time and that puts an excessive load on your systems.

Now, as you’re trying to check into these different service interruptions, you’re going to use different tools to do that. If you’re using a Windows system, there are certain service analysis tools that you can use these tools can help you identify suspicious service activity even when anti malware scanners fail to identify it. Now, one of the ways you can do this is by looking in Task Manager. By going into your Task Manager, you can view all the running services. Another way you could check for running services is by looking at Services MSC, which is a snap in plugin inside of Windows that lets you see all the running services. And from there you could see a description of what each one does and you could start or stop those services as appropriate.

If you prefer working in the command line, you can use the Net Start command. NetStart will display all running services on a computer from the command line. When you run that command, you’ll get a list outputted to your screen of everything that is currently running and then you can go ahead and start or stop those using additional commands. Finally, within Windows you can also use PowerShell and use the commandlet Get Service to be able to see all those different services that are running. And again, this will show you the status whether they’re stopped, started or running, the name of it, and a short display name that gives you a little bit more information about it. Now, if you’re using Linux, there are other tools available for service analysis as well.

When you’re dealing with Linux, one of the big ones you’re going to use is Cron. Cron is a task scheduler in Linux that can configure processes to run as daemons or background processes or services during the machine’s startup. We’ve talked about Cron before when we compared it to the task scheduler inside of Windows. Now, when you look at Cron, you can look at the different Cron tab jobs that are scheduled. As you can see here on the screen, there are several different Cron jobs that we have scheduled daily, weekly or monthly showing you what will be done at a certain time based on the scheduled tasks that we have. Another command within Linux is system CTL.

This can list and monitor the startup processes using the appropriate control for the initiative, which is that startup process that happens every time you start the Linux system. Other commands you can use from the command line would be things like PS and Top. The PS and Top commands are used to monitor running processes. PS will show you all the processes on a system and Top will show you all those running processes as well as the memory being consumed and it will allow you to sort that into a particular order. By using all of these different tools inside Windows or Linux, you’re going to be able to identify what services are being run and which ones are being interrupted and why.

5. Application Logs (OBJ 4.3)

Application logs. In this lesson, we’re going to talk about finding indicators of compromise inside your application logs. Now remember, the simple idea of unexpected growth of a log could itself be an indicator of compromise. But in addition to that, there is contents inside these logs that we can use as signatures as we’re searching for indicators of compromise in a system. Now, most applications can be configured configured to log different events. As we go through this lesson, we’re going to start out by covering four different log types. These are DNS event logs, Http access logs, FTP access logs, and SQL event logs. Let’s start with DNS event logs first. What is a DNS event log? Well, this is a log that contains a log of all the different events for each time.

The DNS server handles a request to convert between a domain name and an IP address. So if I pull that up on a Windows system, it looks something like this. Notice this is an information level event and it’s showing it is a DNS client event. As I look at the details, I can see that a DNS query was called for this particular domain name, the type of query it was, the options it was given, and all the other information about this request. I also know what user did this and which computer did this. This is helpful information as we’re trying to figure out where malicious information is on our different systems within our network. The next type of log we have is an Http access log. And this is a log containing Http traffic that encountered an error or traffic that matches some predefined rule set.

So as we start looking through that information, we can find relevant information in that log. And this information is recorded in the common log format CLF or W three C extended log file format. There is no standard of which one of these it’s going to use. That is going to depend on the Http server that you’re running, but it will usually be one of these two formats. Now, as you look at it, you’re going to get all these different status codes in there. And we’ve talked before about the Http status codes. These status codes of responses will indicate if there was an error and whether it was caused by the client or the server. If it’s a client based error code, it’s going to be something in the 400 range. If it’s a server based error code, it’s going to be something in the 500 range.

Now, some Web server software logs are going to have the Http header information for both the requests and the responses. And this is really helpful for us because we can then get even more information that we can look through it. Now, as you’re looking through these logs, one of the fields you’re going to see is a user agent field. This identifies the type of application that’s making the request such as the web browser version or the client’s operating system. Now, a quick word of warning here for you. The user agent field is very helpful, but it is not a reliable indicator of the client’s environment. It can be easily spoofed or simply misreported because a lot of browsers use the same underlying code. So let’s take a look at a couple of examples.

Here the first example of an Http access log looks like this. Here you can see three entries on the screen and we’re going to work through them together so we can understand what’s going on. The first entry has ten dot 10 102. In this first entry we see an unknown user. Notice the second dash after the IP address indicates this is an unknown user. And then we can see this unknown user is trying to receive a file from the downloads directory, which is why the command get was being sent. And then we get a response code of 401, which indicates that the user is being challenged to provide a username and password by the server. Then we’re going to get some information about the client browser and the operating system.

In this case, it’s identified as a Windows 64 bit operating system running Windows Nt 10. 0, which is just some version of Windows Ten, like Windows Ten Home, Windows Ten Pro, Windows Ten Education or Windows Ten Enterprise. Now, from this line alone, we’re not sure which one, but we do know it’s some version of Windows Ten. Now, once we see that, we can also see the browser has been identified here as either Chrome, Safari or Edge because they all share a common code base. And it makes it really hard to identify positively which of the three it really is. The next entry we have, we can see that the user was identified as Jason. This is because the user has provided their username and password in response to the status code 401.

From the first entry here we see a status code of 200, which indicates that the resource was successfully accessed. In this case, that means the file requested in the first entry was now downloaded. Then we get to our third entry. This third entry we can see another user trying and again, this is an unknown user by showing that second dash there after the IP address. Again, this user is trying to download that same file from the server, but from a different computer, this time from the ten dot one, dot zero dot 103 IP. Notice we don’t see another entry with the proper credentials being accepted with a status code 200. So we can assume that this unknown user has provided the incorrect credentials and received an error message instead and they weren’t able to download that file.

Now again, this assumes that these were the only three entries we had. But if we went further in the logs, we may see that status code 200 and a successful attempt later on. All right, so let’s go ahead and take a look at a second example. Here we’re going to go into another Http access log example. And here I have a single entry for you. In this log entry, we can see something suspicious is going on. Notice this time it’s from an unknown user and we have a get command being issued. But inside that query we can see that there is that question mark and then variable equals this is trying to send that variable that’s listed here on the screen. Now, what is this variable? Well, it’s really just a snippet of JavaScript. And what we’re seeing here is that we can see this snippet being sent by the user agent of Nikto, which is a web application vulnerability scanner.

This entry comes from a vulnerability scan against our web server. And this scan is actually running the test number zero zero 8116, which is attempting to see if this web server is vulnerable to a code injection attack through a query string. Also, we see status code 302 was returned inside this entry. This is used to indicate that the server found the requested object, but it redirected the browser to a login page. If it wants you to be able to access that for the exam, you need to be able to read and understand these types of logs and then identify if something is suspicious or not. Notice in the second example, this is suspicious because we see someone trying to perform a code injection against your web application. Now, if this wasn’t your security team, how would you be able to mitigate this type of attack? Well, you can install a web application firewall or a WAF, because this will help you analyze the inbound requests and strip out things like code injection.

To be able to prevent this type of attack for the exam, this is the level of detail you need to be able to see they were trying to inject something, and you can stop that by doing what thing? What is your mitigation? If you can know those two things, you’re going to do well on the exam. Let’s move on to our FTP access logs. Now for FTP access logs, these are logs containing FTP traffic in a W three C extended log format. This might look something like this. Here you can see a couple of different codes that we have. You see three three one and five 30 and two 30. For example. Now five 30 indicates that a user needs to log in before a command is processed. So in this case, we have a username and a password being submitted.

Then you see further down we have code 230. Code 230 indicates that a user has successfully logged in with their credentials and has access to perform the command. In this case, they’re probably trying to download a file because this is the file transfer protocol. Now, if we look in this example, you can see five requests for credentials like a username and password and only one of those was accepted at the end. So what does this mean? Are we under attack? Well, maybe. Maybe we’re under attack from somebody attempting a brute force attempt to crack our password. Or maybe somebody just forgot their password and is trying their top five most common passwords until they found the one that was right.

We really can’t be sure here, but we do know that they are trying to login four times unsuccessfully before being successful on that fifth attempt. So if you were worried about a brute force password attack, how could you mitigate it? Well, you could set up a global policy that added a 15 minutes or 30 minutes lockout for any account after three failed login attempts. This is a common mitigation against this type of attack because it extends the amount of time needed to successfully crack a password by an attacker. And this is all you’re trying to do here is delay the time out, make it take longer for the attacker because if you can do that, hopefully they’ll give up and they’ll go to an easier target.

Our next type of log is an SSH access log and this is an unstandardized type of log that can provide basic client server session information. Now, when you’re dealing with SSH, this is secure shell. Here on the screen is an example of an SSH log. This is showing somebody who successfully is logging in as root. They went in and they gave their password and it was accepted and so a session was open. How do we know it was root? Notice the UID UID equals zero. The root of a Linux system is always going to have user ID of zero. So keep that in mind. Now, let’s say we had a bunch of failed logins. What will we do to be able to identify all the failed root logins on a server as part of our threat hunting or analysis? Well, we could do something like this.

We can go through here and use EGRIP, which is a grip command to search the authentication log at the VAR log auth. And when we do that, we can display any instances of the word failed or failure. Here, for example, you can see 15 instances on December 5 from 939 to 940 at night when somebody tried to log in and they failed to do that using the SSH service. Now again, for the exam you need to be able to read logs like these and identify what type of attack might be going on. In this case, it’s probably some kind of password guessing since we have repeated login attempts that are all failing. Also notice in this log you can see exactly what IP was attempting these connection requests. This way you could use this as part of your analysis and see if that client machine may have been compromised.

That device may be something that was compromised by an attacker and now they’re using it and trying to pivot onto another machine using the root credentials. This is a way to identify what actions have been taken and what possible root cause there is. The fourth type of log we need to talk about is an SQL event log. This is an event or error log that records events with fields like date, time and the action taken, such as server startup, individual database startup, database cache clearing, and databases not starting or shutting down unexpectedly. Now, when we’re dealing with this, we are dealing with SQL Servers, and SQL Servers can log individual query strings that are being sent to the database so we can actually look at everything that’s been sent to the database by a particular user. Now, as you go through and start looking at your SQL Server logs, you’re going to find a lot of information there other than the date, time and the user who sent the query.

For example, you might see things like the query operation that was performed, the schema associated with that operation, and the object of the query. By retrieving all this information, you’re going to be able to get information from the database on what has occurred and identify what data may have been breached. If you’re investigating a data breach, the particular format of those logs and how you access them is going to vary based on the type of SQL database you’re running. For example, if you’re using MySQL, you can go into the MySQL workbench and use your audit inspector to go through those audit logs. Notice here you’ll see things like the record ID, the timestamp, the type, which is a query or a ping, the connection ID, the user, the host IP, the status, the command class that was being used, and any information about that particular command.