LPI 101-500 – 103.4: Using streams, pipes and diversions

1. STDIN, STDOUT, STDERR, <, <<, <<<, >, >>, |

We ended up in a new section, section 103 four using flows, pipes and diversions. Let us first discuss three important terms. These are standard input or STD in for short, standard output or STD out for short, and standard arrow STD error for short. What do we mean by that? Let’s start with the standard output. The default output channel is mostly the screen. When I, for example, enter LS in the terminal to list the directory content, then the command is executed and the result is output in the standard output channel. So in Stout, it’s written in this case in this form STD out, which in this case is the screen. You can also redirect the result of LS, for example, into a file, for example, with the following command LS greater than directory contents. For example, the greater than character is a redirect sign, which means that the result of LS is written to a file, in this case to the file directory contents. We do not get the result displayed on the screen, but it is saved in the directory content file, which means that the file is now STD out and not the screen.

Remember that STD out has the number one. Please keep that in mind. STD out has the number one. What the numbers mean will become clearer shortly. If you only know what STD out is, you can already guess what STD in is. So the standard input or standard input channel this is mostly our keyboard, with which we enter the commands in the console. So I enter again LS. In this case, LS is STD in, so the standard input channels. But the keyboard does not always have to be STD in, because a file, for example, can be STD in too. For example, we can use the NL command. We remember NL counts the lines of a text file. So, for example let me go to the tar directory. For example, NL file 22. We have a result here, and we can use NL smaller than or lower than 522 and returns the same result once the STD in is the keyboard. So it is used with the first command, and the second command uses STD in as a file. Because we are not using the keyboard here. In this case we are using the keyboard and we say we want to analyze 22, and in this case we say, no, we don’t use the keyboard.

We give the NL command the file 22. I hope you understand it’s very difficult to explain. So we have a file here that we assign to the NL command. The standard input channel is number zero. Keep that in mind. Standard input channel is number zero. The standard error channel, or Stder, has the number two. Please keep that in mind too. So if a command causes an error and this error is output, it happens on the STD error channel, which is either the output on the screen or is saved as output in a file. To redirect results from STD out, we usually use the greater than sign, the double greater than sign or the pipe. So LSLA prints the outcome to STD out. In this case it is our screen. And if you want to redirect this output to a file which is also STD out, but in this case it is a file and not the screen I use the greater than corrector or the double greater than corrector.

So for example flag this one then greater than list TXT, for example. So let’s so we are saying that the content of root should be copied to list. TXT and we see the outcome of this command is not available on our screen, so only in list. TXT, we will take a look at the file cat list TXT so that looks like this one, I think. Sorry, let me do that again and then let’s la I’ve forgotten the route and it’s the same. The only difference is that with a second method the directories are represented as directories or in colors, and in this case not. Let’s call the command again, but this time not with root, but the directory in which we are currently located and output it again in list. TXT. So for example LS la so not root at this time, and then to list TXT cat list. TXT and what happened? The file has been overwritten, so everything that was previously in it has been deleted and overwritten again. Incidentally, this is also how you empty log files. There are programs where log files can grow up to a size of several gigabytes. And you can sometimes not simply delete a lock file, because then the program may not longer work at all, but you can empty them accordingly. In this case with the following command echo so nothing in it here and then list TXT and you see it is empty. Now we carry out the order from now again LSLA list TXT so that something is in the file. Accordingly, in order not to overwrite a file, but to add the result of the data that already exists, one would then select double the larger character.

For example for example LSLA bin and then we are using two times greater than and then lift TXT the double greater than means write the result. So the result of lslasla bin to the file list. TXT. But don’t delete the old things, just attach it to the bottom. We check that. Okay. And here we have our first command, and this is our second command here. Maybe it was not a good example because bin is now only a link to user bin. Let me do this again with LSLA userbin and then greater than list TXT cat list TXT and now you can see the difference a little bit better than before. Another way to redirect STD out is the pipe symbol, which we have also used a number of times.

For example, LSLA user bin pipe WCL so the result of the LS query. So the contents of the user bin directory is forwarded to another Linux command here WC using the pipe symbol, which in turn takes over the result and does something with it, depending on the command. In this case we passed the result to redcount and used the l option. So the number of lines in the bin folder, the user bin folder should be the output, and that’s correct 1451. So let’s get back to the numbers I just mentioned. As a reminder, STD in has the number zero, STD out always has the number one, and STD error always has number two. So if you want to redirect STD out, as we have already done in various examples, we would have to tell the terminal that we are now redirecting STD out.

We do this with the corresponding number, in this case with the one. So example LSLA one and then the greater than and list two TXT. For example, so we put a one in front of the larger than sign, because one is redirected for the standard out channel. The result here is exactly the same as if we omit the one only like this. Because if we do not use a number, then the terminal automatically chooses the one by default. If we now want to redirect the standard error, we would proceed as follows LS la two because standard error is two, so we redirect channel two, so the standard error channel, so all errors that can occur. Let’s take a look at the Maybe. We are using a better name, maybe error TXT and let’s take a look at the error text file and we see it is empty. This is because there is no error here and also no error in this command. This command produces no error. We only told that the standard error channel so the channel where the errors occur should be forwarded to the file error TXT, but in this case we had no error, so nothing was forwarded to error TXT.

We can now consciously use the wrong command. For example, so LS with the option e, which I think does not exist, and then do the same error TXT and then cat error TXT and we see here invalid option invalid option e. This option does not exist with the command LS. So this is the error message we would normally have received if we had not redirected the channel two to the error text. If we used the command without redirecting anything, then the standard error channel is the screen, and so we see the error message on our screen. We can also redirect both channels to two files at the same time. It works like this LSLA greater than and maybe list TXT and the standard error channel should be forwarded to error TXT. That means the output LSLA should be forwarded to here is the imaginary one. So the standard output channel should go to list TXT and the standard error channel with the two here. As you remember, the two is always the error channel should go or should be forwarded to the file error TXT. We look at the list TXT and here we have the output we expected. We looked at the arrows TXT file and this is empty because this command did not return an error.

How do we proceed if we want both STD out and STD error in one file? This command gets a little complicated, admittedly. So, for example, LSLA greater than list three TXT. So that’s the command. And please keep that in mind for the exam. It’s very important. The result of the query LSLA is forwarded to the file list three TXT via the standard out channel. So channel one. Normally you can use one here, but you don’t need to. Then it is forwarded to the standard error channel. So channel two, and in that case it is forwarded to the end symbol and the one. This means something like take the same file that you have already used for STD out. So the list three TXT. In this case, of course, there are no errors here. Again, I hope this is clear. Please keep in mind the two is the standard error channel and this here greater than the and sign.

And the one means Please use for the standard error channel the same file or the same file as for the standard out channel. Standard out is here LSLA standard out is list three TXT. So we only tell the shell Please use the list TXT also for the standard error channel. Let’s take another look at an example of redirection to STD in. For example so in this case we no longer use the keyboard at sftdn, but the file itself. And this is so the file here. Laura Ripson the text file. And this is transferred with a less than sign to this command root count with the option L. In principle, it is the same command as the following the result is nearly the same. But in the first case we used STD in as a file and in the second case as text input. Theoretically, we would have to insert a zero for STD in.

Then the command would look something like this zero and then smaller than Laura m ipsum. But here too, since the lower than character is used, the shell automatically interprets this as a zero. So here we have the zero. Here there is no zero. Therefore, it does not have to be written. With standard output, we could redirect both with a greater than sign and with a twice greater than sign. For the standard input, we use the smaller than sign. The question is can we also use two smaller than signs here you can do that, but we are already programming, and that’s not the point here. Nevertheless, I show how that could look for example smaller than or lower than 1234. So we can write several lines here. Such a document is also known as here doc or Here Document. The name Here doc that I have used here is just an example for any other word can be used.

You close the text entry by entering the first word again and then the command word count with the option l tells us that we have four lines 1234 in the STD in area. You can even use three smaller characters as characters. This is necessary if you want to output the content of a variable. So first we create a variable with the name variable and this has the Content Tests. So variable is the name and the content is test. Now we want to output the content of the variable in capital letters. The tr command is suitable for this. So tr A to that and then three times lower than dollar variable. This means that the content of the variable with the name variable the Content is test is passed to the tr command, which should then turn the lowercase letters into uppercase letters and you see it was successful.

Please also keep that in mind for the exam you can use one times greater than, two times greater than and one times smaller than two times smaller than and three times smaller than three times greater than is not possible. So keep that in mind for the exam. We have now discussed all possible redirection variables. Only one is still missing. How do we manage that? The result of a command, for example LSLA is both output to the screen and saved to a file. This doesn’t work if we just use some assignment characters or pipes. For such a case we use the t command. So let’s have a take a quick look at the man page. Man T. Read from standard input and write to standard output and files.

That means Read from the Standard input so from our keyboard in this case and write to Standard Output so the screen and a file or more files. Let’s use our simple example again LSLA pipe t list four TXT. With the pipe we pass the outcome of LSLA to the t command and we select list four dot TXT here so we can see at first glance that the content has been displayed on the screen.

Let’s look at the file list four TXT four TXT and this has the same content that is t doubles the output to output it once and save it once. If we execute another command and select the same file, the content would be overwritten. To avoid this and only append the content, we use the A option of t. So with LSLA root now pipe t with the option A and we’re using again list four TXT and we want that. The command shows us every file and directory under root. Pass this to t use the option A and select the list again so that it is only appended and that the old content of the list for TXT file is not deleted. So cat list four TXT and we see that here is what was originally written. And there is the appendix output of our second command.

2. xargs

The last command in this chapter is x Arcs. What does X Arcs do? Suppose we use the Find command to search for files with a certain extension in order to delete them. An example could be find. Bak. So we are looking for files with the extension bak, and as a result, we will see all files with the extension bak in this folder. But since we still have, as you can see here, text files or CSV files in this folder, we cannot simply delete the entire content of the folder. Of course, you could delete the file very easily with the file blobbing. For example, with RM the option F. And then this could be the command. So remove all files in the directory we are currently in with the extension bak.

That would be very easy, but that would certainly be the much easier variant at the variance I would prefer, of course. But the point here is to explain the command. And therefore we want to delete these files with the help of x Arcs. So delete this. And if we use the Find command, as we have just done, the files are one below the other in the result. You can see it here. With the help of XRC. However, we can also output files side by side. To do this, we pass the result to x Arcs. So we are using Find bak again and forward the result to xorx.

And now the files are all next to each other. Since we can delete several files at the same time with the Remove command by writing them one after the other, as you can see it here. Theoretically, I would just have to put an RM in front of this line here, and all files would be deleted. But you can also do it like this find. Bak and then forward it to orcs and use the RM command. So you add RM after the x Arc and the corresponding BAQ files are deleted. And you can see it here. No more Bak files are here anymore. As a rule, x Arc is not called by itself. It always contains the result of a previous call, in this case from Find. Instead of using the standard input, x Arcs can also read from files.

I have prepared a file here just to show you x Arcs file, just with five lines. Line one, two, line five. And here we would like to output the content side by side. And there are now two possibilities. Either we use cat on the x Arcs file and give the result to x Arcs. So cat x Arcs file pipe xarchs. That would be one possibility. Possibility. And the other one can be x Arcs with the option A x Arcs file. And here we get the same result, option A here. In this case, option A says that a file should be read and not via standard input. Let’s look at another example.

Let’s say we have a list of employee abrogation’s. So I’ve prepared a file here to employees TXT. And we only have two five names. M. Jones, R. Jackson, M. Cooper, A. Bundy, J. Darcy. And we have written all abrogation’s here in capital letters. And for some reason we need this list to be written off one after the other. But in lowercase letters, using x arcs in conjunction with tr, we can create such a file. So, for example, x arcs with the option A, because we want to read from a file. Now the file is employees TXT pipe tr to that and small letters greater than AM Place two TXT. So x arcs with the option A ensures that the names are next to each other. So the names here are next to each other and no longer below each other. The result is passed to the command tr, and thus we ensure that the uppercase letters are converted into lowercase letters. In addition, the whole thing should be saved in the file Employees Two TXT. Now let’s look at the file Employees Two, and we see that the employee abbreviations previously written capital letters are now written next to each other. And in lower case letters there are also many options with x arcs that you can use. And here too, I recommend studying them a little bit.