CompTIA Linux+ XK0-005 – Unit 15 – Troubleshooting and Maintenance part 1

1. Troubleshooting and Maintenance

Well, this unit is going to be about troubleshooting and maintenance of your Linux system and even of your network connectivity. So one of the first goals for us is to talk about Linux in the entire process of it being turned off, turning it on and dealing with any errors you might have in the boot process. We’re going to look at the different portions of the boot process so you understand what’s happening behind the scenes and how you can choose some different run levels to help you with some special diagnostics or troubleshooting. Then, assuming that the Linux system boots up properly and you’re still having issues of connectivity, we’re going to talk about some of the basics of what can you do to troubleshoot the network connections that you have, what are some of the tools like

peeing and trace route. And then finally we’re going to talk a bit about writing and testing shell scripts. Now remember, shell scripts or scripting was the ability for you to take a routine or mundane task, put some automation to it, and if you wanted to eventually even schedule it, to run automatically. But the purpose here is that you might come up with a series of commands, a series of things you want to run and rather than remembering them or having to remember them, or running down a little checklist, you just launch the script and it does the things that you have pre programmed it to do. So that’s our goal is to talk about what we can do in troubleshooting this process, from turn it on to making it to the Internet or some other location through our network, and automating some of those tasks.

2. Topic A: Booting Linux

All right, so we’re going to begin troubleshooting by talking about what happens during the boot process of Linux. Now remember, this is what’s even going to occur after installation of Linux. I mean, installing it is one thing, but actually getting it to come up online without errors, that’s the booting process that you’ll have to go through every single time. So let’s take a look at what it means to go through the boot process.

3. Booting Linux

One of the first things you’ll see, and this is true on almost any system you deal with. At least everyone that I’ve seen routers switches, servers, different operating systems is what they call the post or the power on selftest. Now, the goal of the power on self test might be different on different machines. In fact, these are instructions that are permanently written, usually into a firmware or readonly memory portion of the system board. But what it does is it’s supposed to go out there and not only detect what hardware is connected to it, but also detest that hardware, including things like memory and this processor and anything else that’s going on. We need to basically have the idea that there are no errors or no problems before we go any further.

Now, one of the nice things about post today is that it can not only detect the hardware, it can detect changes in hardware so that when the operating system loads up, it can actually report that change. So the operating system can look smart by saying, oh, we found something new. Now, after the power on self test, memory is working well, system board is working well, processor, all of that comes the MBR load. Now, remember, the MBR is the master boot record. And basically what happens is that the boot disk, once it’s been located on your computer, it’s going to read the MBR from that disk disk so that it knows, hey, this is where the bootable partition is, this is where the bootable files are so that it knows how to launch Linux.

4. Booting

Now after the MBR has been investigated and it’s located at a location of where the operating system is we’ll go into the boot loader. Now the boot loader will contain operating system specific instructions for how to start up Linux. And once that starts up, remember that was considered the first of the daemons was the actual kernel itself. Then it’s going to begin the daemon loading and of course the first Damon it loads is the INITV and from there all as the other child processes will start.

5. Boot Loaders

Now, when we talk about bootloaders, there’s a couple of different types that you’ll see. One of the first ones was Lilo, which was a traditional Linux boot loader. Typically, it was stored entirely within the master boot record. And through your configuration information, you could change the way in which it went through the boot loading. In other words, you could take that file, that Lilo config file and actually change information about it. So it would change the way in which the boot up would happen. Today, the more preferred option is Grub. Grub. It’s the preferred boot loader for your modern distributions, and only part of that Grub file is stored within the master boot record. That portion basically will contain the information needed to be able to locate the rest of the Grub information on your system disk.

6. GRUB Boot Loader Screen

Now, when you see a typical bootloader screen, or you see the grub boot loader, what you’ll be looking at is a series of different systems of processes starting up, being found, being located. It’s almost like you’re watching a little checklist of the different components, the different systems, processes being loaded. Hopefully you’ll even see whether there were successes or failures.But it allows you to be able to watch during that process of loading the operating system to see if there were any errors. Now, sometimes that screen can go very quickly, so you might not see it all, but you can use your up arrows and down arrows to try to watch or navigate through it, or after the system boots up. You can always read these things in your startup logs.

7. GRUB Configuration

Now, as we said, we can change these bootloader files. We can also change the configuration for Grub. Now, depending on the systems, you’ll probably be able to find Grub in the boot Grub Grub configuration file. You might also find it in the Etsy folder as a Grub configuration file. Or that could actually be one of those symbolic links going back to the original one. Other systems, depending again on the distribution, might find it as a menu LST option that you have for making those configuration changes.

Or you can go to the command line, you can use commands like Grub install to install Grub onto the MBR. You can use the update Grub to be able to make changes or choose options to make changes to the Grub configuration file. For that matter, you could open up a Grub shell and make the changes directly from there as well. What are you hearing from me? Have a lot of choices, a lot of flexibility of how you can change the booting up of your Linux system.

8. Demo – Investigating GRUB

All right, let’s take a look at some of the information we use for booting up. In this case we’re going to look at Grub and we’ll find it in its listings here under the LS Oops, a little space in there, boot and Grub. And underneath there you’ll see this file called the menu list file. So that’s the one that we’re going to use and we’ll use the less command to go to the bootgrub menu list. And there we can see some of the configuration files that you would find as you’re booting up the system. So this boot up list, most of everything you can see is remed out. Right now we have a timeout of five for defaulting to the booting to the default entry. If you don’t interact. We have this, what they call pretty colors.

Somebody from Canada put that in or England because they got the extra U in there and chose the colors of the background and the font style. I hit the page down and you have the examples of the titles but nothing of interest until we get almost to the end here and when we get to the end of this here we go here’s the default options. One is where they’re both to boot up to the Debian Ganu Linux to the same kernel. One is going to be in single user mode which we can tell by the path to the root, whereas this other one is going in quiet mode. So it just kind of does its job without a lot of interaction.

So those are the two options that you have that when you boot up you could use either one of those. Obviously the single user would be great when we’re doing password recovery or some other troubleshooting. And the default is the one that’s going to be this first one. And by being the default, like I said, you have that five second time out to make your choice as we saw on our earlier settings. So let me quit out of there so we can get into that out of the mode of looking at the file and let’s actually try, let me do a pseudo Grub and I have to get my password in there. And now that I’ve gotten into Grub, I’m going to enter Help and from here the Help will show me at the Grub prompt will show me all the commands that I can run.

One of the commands I could run is display mem. I think you see that there in your list on the right column and that will show me the type of memory that we’re using, usable Ram, reserved and all that information about the amount of memory that the Grub is going to be using, what’s present on my system and hopefully how much it’s going to use. Anyway, you can take a look at some of those other help files and kind of get an idea of how things are booting up. I just hit quit to get out of the Grub shell. Hopefully that made some sense, how you can make some changes to the boot up records and that you can look at how Grub is going to run.

9. Runlevels

One of the things you might say is well what might I want to change? What do I really care about? Well maybe you want to deal with things like run levels. Now run levels are really good when you’re trying to make either updates serious changes to Linux or just trying to troubleshoot the run level says how this thing is supposed to actually boot up. Now one of the things you’d see is that a normal boot up is generally going to have a run level of two or three depending on what you want to do. Three I would say is your most normal because that is going to be your networked multi user normal boot up for things like Debian or Red Hat.

Now if I have a run level of one suddenly I have just said I’m going to have a single user only that means that when you boot it up it lets you log in and no one else can get in a great time for you to make configurations without worrying about other users affecting your configs. You can create a graphical or you can put a run level that says reboot or restarting. You have some options because it allows you to have control including whether you do or don’t want networking as again an example single user says no networking connectivity level two is a single user with networking.

Why do we care? Well like I said if you have to make changes if you have configurations something’s wrong with the setup you certainly don’t need other people in there or if it’s an issue with the network itself causing some problems with your booting up process start without it, fix whatever you need to fix. Do a backup, do a restore I mean there’s a lot of things you can do because you’re the only user and then when it’s all said and done you can change that run level back.

10. Runlevel Initialization

All right, now the run level initialization, you have to remember is something that’s going to be looked at after the power on selftest, but prior to our actually booting up the Linux operating system. Because this information is going to tell the operating system to what run level it initiates. I know I kind of almost sound like I’m talking circular. Now you’ll find this information in a file under the Etsy drive called the Init tab. Now this file is red at, as I said, after the initd system starts and it tells Linux the run level to which it should boot. Now there are also some options that you can have, some startup scripts that you can list in the Etsy directory that tells the system basically how it should alsorun or what it should do during this startup process.

Now these files are usually under the format of RC and some numerical reference which is actually a symbolic link to a file in it the directory itself. Now all of these files, you’ll have the names, you’ll have the reference numbers that you’ll set up often with the letter S or a K preceding it. But the idea is that you have these links to be able to say here’s my startup scripts. Now you can use the command line to change your run levels at any time. The init command allows you to follow up that run level number that we’ve talked about or you use the tell init command, which its only difference is, is that you can actually tell it to wait so many seconds before it initiates that run level.

11. Demo – Changing Runlevels

All right. I’m going to start off by switching myself over to the root user just because I like to be the root, like having that power. And it’s going to make life a little bit easier when we’re dealing with run levels. So we’re going to do is we’re going to take a look at the listings that we have in the Etsy and folder for the RC Asterisks. That’s going to show us the different run levels that we’re we have. Now, since I’ve come in here as the route, I kind of lost my alias command. So let’s see if I can do a little bit better with setting this up. Let’s not worry. Let’s do LD in capital F. Now there so much nicer I wanted to see. Just the directories, of course, and then the long process of displaying them for us.

Anyway, I have RC zero through RC six and those are my different run levels. And if I now run the command run level, it tells me that I am run level two. All right, so just like that, I know what run level I’m at, which is normal. I’m going to clear the screen and we’re going to take a look by listing again. LF what is at run level two. In other words, we want to see exactly what we can expect to have started up. So it’s the RC two. So that number two stands for the run level that I’m at. And look at that, all of the processes. In fact, maybe this one might have been nicer to not use the long format and just have done the LS to see what’s running at that particular run level.

So those are all the daemons that would be started up and running for us. All right, well, now that I’ve done that, I’m about to kill off everything that I own here visually, all the graphical stuff. So I’m going to clear the screen and I’m going to run the command tell in it to change my run level to one. Now, when I do this, I’m going to be asked to put in my password, if I’m not already the root, which I am. And it’s going to close down the GUI and put me right into text mode. Boom. Just like that. Now that debian logging is actually going to go away here in a second, I do believe. And when it does, I’m going to log in as the root. All right, there we go.

There’s the kill signal. And now it’s asking for the root password. So I’ll put the root password in and look at that, I’m in as the root. If I type PWD, it says that I’m the root. LS gives me whatever is there file system. Nothing’s really changed as far as command line goes. I’ve just lost all of my other options. If I wanted to, I could do a tell in it and go back to level two and log back in, but that will take a little bit of time for you to watch the whole screen thing boot up. But that would be the way you get back to your old run level. But it was very straightforward and hopefully it’ll be a process that you’ll only deal with when you’re troubleshooting.

12. Password Recovery

All right? One of the things that’s interesting about any operating system or any device is what happens when you forget the password. Okay? It does happen and sometimes it’s not an issue of you forgot, but other people have forgotten. I’ll give you a little quick story. I was working with this company in Portland, Oregon. This is many years ago and they downsize. They have offices in Irvine, California and Portland, Oregon and they downsized the Portland office from 110 people down to ten. I mean, that’s a serious downsize. Eventually they went out of business, but of those people they let go, most of them were their network It people. And the folks that were remaining who were trying to get the network back online couldn’t get in because suddenly all the passwords were different on these systems and on these routers.

And conveniently, I suppose, the people who had been laid off couldn’t remember the passwords. So they gave me an opportunity as a contractor to come in and say, hey, can you get us the passwords into these things so we can retrieve our data? Now of course, because they seem to be going out of business. I did tell them they had to pay upfront. But anyway, it is nice to know that you can break into these systems. Now you have to remember that these corporations, including Linux and the different distros of Linux, all publish how to recover a password. It’s not a great secret, it’s not a new hacking trick at all. All it is is an easy way to be able to go in and change a password if you forget it.

What it emphasizes though, is that to make these changes you have to have what we call physical access. I have to be able to touch the machine and interact with a keyboard, video monitor or the KVM to get in there. I’m sorry, not monitor, but mouse KVM. I have to have that connection. If I don’t, the odds of my getting in are going to be a lot hampered. It’s hard to reboot a system remotely. It’s hard to be able to break in when you can’t watch the post process, which you’re only going to see if you’re directly connected. All right? So we tell that to people all the time. If I touch it, I own it. Basically that’s what happens.

13. Root Password Recovery

So if you are booting up when you get to the grub menu, you’re going to press the arrow at the grub menu and press E to edit. From there, you’re going to select the kernel line, press E again, add the word single, and then or in debian, you might have to add the phrase in. It equals bin bash. Press Enter. Now you boot your entry with the letter B. You remount the root file system as RW and then use the command password to change the password file and restart. What was all that about? Well, you were creating new directory systems that were not under the same password protection.

You were changing the way in which it booted up so that you could issue commands from that boot up process. And not as a route, because you don’t know the route. It was just a method of being able to get past the actual point where the operating system would have initialized enough. That said, now you need to be the route to get logged in. That’s all you were trying to do. Many other devices have similar types of instances where you can go into a specific boot mode that doesn’t require a user account, but because you have that ability to touch the machine and get to that boot mode, you’re able to make changes to those files and then to reboot and have that new password.