VMware VCAP6-NV 3V0-643 – Command Line

1. Use esxcli

Before we begin diving into advanced networking topics, I want to introduce some of the command-line tools that I’ll be using, starting with Esxcli. Esxcli was introduced in 50, and its functionality has grown over time so that it now supersedes many of the other command-line utilities, such as the ESX CFG commands. Esxcli has a very long command list, and I’ll go through some of the networking commands now and discuss others throughout this course.

I’m currently logged into a virtualized ESXi host. I’ve enabled SSH on it so that I have command-line access. You can also access this remotely through the VMware Management Assistant, or VMA. ESX CLI is built on a namespace structure, so if I just type ESX CLI and enter, it’s going to show me all of the available namespaces that I have. For each one of these, I can go a little bit deeper, and then it’s going to show me again the available namespaces. The first one that I’ll look at for networking is the Device Driver List. This will list all of the device drivers that I currently have loaded as well as the devices that they support.

This will show both my network interfaces as well as my storage interfaces. ESX CLI System Module List is going to list out all of the modules that I have loaded. However, as you can see, this is a very long list. If I scroll up to the top here, we’ll see what the second and third columns mean. The second one indicates whether the driver is loaded, and the third one indicates whether it’s enabled. If I’m looking for a specific driver, I can use the pipe symbol in a command line, which will take the output of one command and use it as the input for the next command. So I’m going to use Grip, which is a search utility, and I’m going to look for VMX net three.

So now I can see just that specific driver, and then it is loaded and enabled. If I want to see the parameters that are used to load that driver, I can use ESX CLI System Module Parameters List and then specify the module. This gives me a list of parameters. The first column indicates the name. The second one is the type of parameter that it is, and the third one is the value. Notice that currently all of the values are blank. It’s because I haven’t actually set any of these for this particular module. It’s using the defaults. The fourth column under “Description” will often show the default as well as give a little bit of a description of what that particular parameter does. For each network card that is loaded in your system, the parameter options will be different, so it’s best to consult the vendor documentation to see which ones you can modify and what exactly they do.

If I want to see a list of all of the Nix that I have on this systemsxcli network nic list, spelling is important, and this shows the four VM Nix that I have installed. I can also take a look at the ARP table from the command line. The ARP table shows which Mac addresses the system has learned about and their corresponding IP addresses. Here’s the IP address, the Mac address that it matches, and then which VMnick has learned that particular address. I can also look at all the information for my V switches. From the command line, I want to go to standard. There’s not a whole lot you can do with a distributed switch. You can mess around a little bit with the link aggregation groups if you’re using LACP, and that’s about it. Everything else on the distributed switch is managed through the Vcenter server. So let’s take a look at our standard switches, and we’ll see from here that we can do things like list outgoing port groups and add or remove uplinks. We can list out the available standard switches, and then we can make configuration changes on those switches as well. So I’m logged on to my physical Six host now, and the last commit I want to show for now is the ESX CLI network VM list. This will display the number of Nicks that my virtual machines have configured, as well as what those Nicks are connected to. This is the port group that those Nix are connected to. We’ll take a look at some other ESXCLI commands as we go through the course. Next up, let’s take a look at the ESX.

2. Use vsish

By running VSI sh from the command line, I get access to the VSI shell. The VSI shell gives me visibility into the inner workings of the VM kernel, similar to the Proc file system in Linux. Here I want to provide a warning that playing around in the VSI shell can be dangerous. Looking at information is safe. However, using the set command to modify settings can make your system unstable or even cause it to crash. As a result, you should only do so under the supervision of VMware support. From here, I can type “help” to get more information on the commands that are available. I’ll mostly be using CD to change the directory I’m in LS to show the nodes that are available in that directory and then getting information from a specific node. So we do, LS. This is going to show all of the nodes that are available. I’m going to First into VMK modules and then CDP from the root.

From here, I can go to one of my physical NYCs, VM Nix Zero, and then I can do an LS and I can do a get on stat, which is going to show statistical information indicating that I am receiving LLDP packets. on this particular interface. I can do an LLDP summary. However, since I happen to know that this Vmnickzero is bound to a standard virtual switch, I’m not going to get anything from this output. But if I were connected to a distributed switch and it were configured for LDP, I could see all of the LLDP information from the command line. Next, let’s go back to the beginning and explore several different options from here. The first one I’m going to go to is PNIX, which is my physical network interface. If I do an LS in here, I can see all of my physical interfaces. I have all of one on this system.

Now there are lots of different nodes that I can access on this physical interface. First, let’s take a look at properties. This is going to give me a lot of information about that particular nick, including its driver version and the actual driver. It’s going to give me information on the type of interface that it is. And then down here is the PCI information, including the PCI vendor and device ID, which we’ll talk about when we get to the hardware compatibility list. I can also see capabilities that are supported for this card as well as capabilities that are activated, such as Veneto Cap. It is High-DMA capable, which means it can access High-DMA direct memory access.

Here’s information on TSO, as well as all of the other capabilities of the interface. Here I can see network hints. This is all of the VLANs that are visible for that particular nick. I’m only seeing traffic on VLAN 0, and it’s on the subnet 10 3 6 6. I can also do a stats get. And from here, I can see statistical information on this nick for all of the packet counters for the receive and transmit queues. This information is going to look different for every type of Nic, as they report differently. Let’s return to the Internet. TCP IP. This will be information about the VM kernel as of 5:50, where we have multiple instances of the TCP/IP stack. So the first thing we have to do is go to the correct one. Vsish supports tab completion in instances. So if you type part of a command and then hit tab, it’s going to show you additional options. Or if there’s only one option, it’ll complete the command for you.

So I want the default TCP/IP stack, then I can do an LS in here. The statistics are visible to me. If I do an LS here, we’ll see that I can get statistical information for all of the different protocols that are available in this stack. If I go to connections, it’s going to be a list of all of the connections that are currently running on this stack, similar to the output of the Netstat command in the Linux system. So this displays all of the available TCP connections. You can look into those to see if it’s listening, how much traffic is being transmitted, and other such details. If I select V-4, this will be IP version 4, information routing, table information, and any routes I’ve established, as well as the default gateway. Next, let’s go back to netport sets. Port sets are all of our virtual switches on this host. I only have V-switch zero, so I can go in there. Now I can see some properties and statistics information for the entire V switch, and I can also see information for individual ports by going to CDPports. All of the port sets listed here are the port IDs from ESX top. So if I go into one of these, I can do a status check to get more information about it, and I can see what it’s connected to. This is connected to one of my virtual machines. It’s using the VMX Net 3 driver. And then I can also get statistical information. This is from the point of view of the switchport itself, or I can get client statistics information. And this is from the point of view of the VMXnet-3 adapter itself. So this is what you would see from inside the operating system if you looked at the statistics.

There is a separate node for the VMX net three as well as the E 1000. If I go in there, I can see additional information for those specific driver types. I can also get team uplink, which is going to show which uplink this particular port is bound to. And finally, I want to take a look at one other node inside of VSI sh, and that’s under system heaps. Heaps are pools of memory that are set aside to perform a particular function. In specific, I want to look at the network packet counts. Take note that those are now capitalized. VSI sh is case-sensitive, so be sure that you enter that incorrectly. And there are two in particular that I’m interested in: net packet heap low and net packet heap e. The reason that there are two is that historically, network interface cards that had DMA or direct memory access could only access low memory, which is below the four-gig boundary. So we had to create two separate heaps. One for DMAs that could only access low memory, and then one for DMAs that could access high memory. Most modern interfaces can access large amounts of memory, so you’ll see both of them being utilized.

So we enter the net packet heap low, perform an elastic search, and examine the nodes here. I want to look at statistics. This is going to show how my heap is currently being utilized. And I can see that on this system, which isn’t doing much in terms of free of max size, it’s at 99%, and the lowest it’s ever been is also 99%. So I have no issues with this system. However, if either of these numbers drops to about 30%, we can potentially start dropping packets on the ESXi host without them ever showing up in any of the counters. You will sometimes see a message show up in your VM kernel log file that says it could not allocate bytes for a dynamic heap, and then it’ll specify which heap to use. This is what it’s talking about. These heaping packets can be adjusted. In particular, they might need to be adjusted if you’re using multiple receive queues or jumbo frames. If you see these numbers below 30%, I don’t recommend adjusting the queues yourself. Instead, contact VMware support. More information about the network packet heap can be found in VMware Knowledgebase articles 204-2587 and 204-2874. Now let’s take a look at collecting packet captures from the command line.

3. Use esxtop

By running VSI sh from the command line, I get access to the VSI shell. The VSI shell gives me visibility into the inner workings of the VM kernel, similar to the Pros file system in Linux. Here I want to provide a warning that playing around in the VSI shell can be dangerous. Looking at information is safe. However, using the set command to modify settings can make your system unstable or even cause it to crash. As a result, you should only do so under the supervision of VMware support. From here, I can type “help” to get more information on the commands that are available.

I’ll mostly be using CD to change the directory I’m in LS to show the nodes that are available in that directory and then getting information from a specific node. So we do, LS. This is going to show all of the nodes that are available. I’m going to First into VMK modules and then CDP from the root. From here, I can go to one of my physical NYCs, VM Nix Zero, and then I can do an LS and I can do a get on stat, which is going to show statistical information indicating that I am receiving LLDP packets. on this particular interface.

I can do an LLDP summary. However, since I happen to know that this Vmnickzero is bound to a standard virtual switch, I’m not going to get anything from this output. But if I were connected to a distributed switch and it were configured for LDP, I could see all of the LLDP information from the command line. Next, let’s go back to the beginning and explore several different options from here. The first one I’m going to go to is PNIX, which is my physical network interface. If I do an LS in here, I can see all of my physical interfaces. I have all of one on this system. Now there are lots of different nodes that I can access on this physical interface. First, let’s take a look at properties. This is going to give me a lot of information about that particular nick, including its driver version and the actual driver. It’s going to give me information on the type of interface that it is. And then down here is the PCI information, including the PCI vendor and device ID, which we’ll talk about when we get to the hardware compatibility list. I can also see capabilities that are supported for this card as well as capabilities that are activated, such as Vmnet Cap. It is High-DMA capable, which means it can access High-DMA direct memory access. Here’s information on TSO, as well as all of the other capabilities of the interface. Here I can see network hints. This is all of the VLANs that are visible for that particular nick. Right now, I’m only seeing traffic on VLAN 0, which is on the subnet 10 3 6 6. I can also do a stats get.

And from here, I can see statistical information on this nick for all of the packet counters for the receive and transmit queues. This information is going to look different for every type of Nic, as they report differently. Let’s return to the Internet. TCP IP. This will be information about the VM kernel as of 5:50, where we have multiple instances of the TCP/IP stack. So the first thing we have to do is go to the correct one. Vsish supports tab completion in instances. So if you type part of a command and then hit tab, it’s going to show you additional options. Or if there’s only one option, it’ll complete the command for you. So I want the default TCP/IP stack, then I can do an LS in here. The statistics are visible to me. If I do an LS here, we’ll see that I can get statistical information for all of the different protocols that are available in this stack.

If I go to connections, it’s going to be a list of all of the connections that are currently running on this stack, similar to the output of the Netstat command in the Linux system. So this displays all of the available TCP connections. You can look into those to see if it’s listening, how much traffic is being transmitted, and other such details. If I select V-4, this will be IP version 4, information routing, table information, and any routes I’ve established, as well as the default gateway. Next, let’s go back to netport sets. Port sets are all of our virtual switches on this host. I only have V-switch zero, so I can go in there. Now I can see some properties and statistics information for the entire V switch, and I can also see information for individual ports by going to Deports. All of the port sets listed here are the port IDs from ESX top. So if I go into one of these, I can do a status check to get more information about it, and I can see what it’s connected to. This is connected to one of my virtual machines. It’s using the VMX Net 3 driver. And then I can also get statistical information. This is from the point of view of the switchport itself, or I can get client statistics information. And this is from the point of view of the VMXnet-3 adapter itself.

So this is what you would see from inside the operating system if you looked at the statistics. There is a separate node for the VMX net three as well as the E 1000. If I go in there, I can see additional information for those specific driver types. I can also get team uplink, which is going to show which uplink this particular port is bound to. And finally, I want to take a look at one other node inside of VSI sh, and that’s under system heaps. Heaps are pools of memory that are set aside to perform a particular function. In specific, I want to look at the network packet counts. Take note that those are now capitalized. VSI sh is case-sensitive, so be sure that you enter that incorrectly. And there are two in particular that I’m interested in: net packet heap low and net packet heap e. The reason that there are two is that historically, network interface cards that had DMA or direct memory access could only access low memory, which is below the four-gig boundary. So we had to create two separate heaps. One for DMAs that could only access low memory, and then one for DMAs that could access high memory. Most modern interfaces can access large amounts of memory, so you’ll see both of them being utilized. So we enter net packet heap low, perform an elastic search, and examine the nodes here.

I want to look at statistics. This is going to show how my heap is currently being utilized. And I can see that it’s at 99% on this system, which isn’t really doing much free of max size, which is also the lowest it’s ever been. So I have no issues with this system. However, if either of these numbers drops to about 30%, we can potentially start dropping packets on the ESXi host without them ever showing up in any of the counters. You will sometimes see a message show up in your VM kernel log file that says it could not allocate bytes for a dynamic heap, and then it’ll specify which heap to use. This is what it’s talking about. These heaping packets can be adjusted. In particular, they might need to be adjusted if you’re using multiple receive queues or jumbo frames. If you see these numbers below 30%, I don’t recommend adjusting the queues yourself. Instead, contact VMware support. More information about the network packet heap can be found in VMware Knowledgebase articles 204-2587 and 204-2874. Now let’s take a look at collecting packet captures from the command line.

4. Perform packet captures

Packet captures allow us to see detailed network traffic as it passes through an ESXi host. Historically, the only method we had for doing packet captures on an ESXi host was TCP dump UW. The UW stands for User World, indicating that it’s not a part of the VM kernel. The TCP dump command was able to do packet captures only on a VM kernel interface. So in this example here on VMK 0, that meant that if I wanted to capture packets that were closer to the network card, I would have to put a VM kernel interface on the same VLAN and port group as that virtual machine’s Vnick and set it to promiscuous mode and try to capture packets there.

That meant there wasn’t a whole lot we could see about the packet as it flowed through the ESXi network. We now have a new option called Pktcap UW, and unlike TCP dump, it has a lot more options for where we can collect data. We can still collect data on a VM kernel interface, but we can also collect packet captures at the VNC level, at the virtual switch port groups at the physical Nic level, as well as at something called a DVfilter that I’ll talk about in an upcoming video. Let’s go to the command line and look at doing a packet capture. First, I’ll use TCP Dump. UW. I need to specify the interface that I want to capture the packets on. And then there are a couple of settings that I use pretty consistently. The first one is NN, indicating that I want to see numbers instead of names for both the hostname and the port group. S zero sets the snap length to zero, meaning we’re collecting the entire packet. By default, it only collects the first 68 bytes, which is just the header. And then I can specify the type of traffic that I want to collect. In this case, I’m going to collect traffic on port 902.

Now we’re seeing some traffic flowing across there. Those would be heartbeats from our VCenter server. I can also generate traffic using netcat, and this just opens up a connection to port 902. And whatever I type here is going to be sent over to the other side, and ultimately it’ll close the connection because I’m not properly communicating with it over here. On the other side, we can see that we’ve collected all of this data. Now, just reading it from the screen is not very helpful. So we can output it to a file using W, and this time I’m going to collect ping traffic. So ICMP specifies the file name. The extension is PCAP, indicating that it’s a packet capture. And now I want to do a ping on Control C to initiate the capture, and we can see that we’ve collected six packets. Now if I want to replay that file, I can use R to read it. Then I don’t have to specify an interface because I’m not doing a live packet capture. And there are the six packets that I just collected. If I want to see the Ethernet output, I can add it, but not before the dash.

Now it’s showing me additional information here, including the source and destination Mac addresses. Now let’s compare that to the output of the packet capture utility. For this, I’m going to switch over to my physical ESXi host because I have more stuff running on that at the moment. First of all, in order for me to use the packet capture utility, unless I want to just capture input on a VM kernel interface, I need to determine exactly where I want to capture it. I’m going to try to capture traffic on a switch port, so I need to see what my switch ports are. I can do that from ESX top, like I showed you earlier, or I can also use Net Stats Space 1. This is going to list out all of my currently used switch ports, and I want to capture traffic on one of my virtual ESXi hosts. Now, you can’t assume that these are the first, second, third, and fourth Vnixes that I’ve created on that virtual machine. The best thing to do is match up the Mac address here with what you see inside the centre server. Because I know this is my first VNC, I’d like to doPkt Cap UW switchport, specifying the number 3355-4451. Then I want to specify the protocol that I want to capture. The syntax traffic is a little different here. The protocol will specify the IP protocol, and ICMP is the protocol one, and it needs to be in hex. Now, if we go over here and do a ping, we’ll see output over here. Notice it’s in a significantly different format. It’s not showing me that nice, pretty output; it’s telling you what I’m seeing. It’s just doing a hex dump of the data.

So that’s not super useful, but I can still write it out to a file using O, and then I can read that file using TCP dump. Now, notice that there are only three packets in here. The PKT Cap utility only captures traffic in one direction. I can specify whether that’s incoming or outgoing, but unfortunately, there’s not an option to do both. So, if you want to record both sides of a conversation, you’ll have to run Pkt Cap twice. Specifying the direction For additional information on doing packet captures, you can go to the VMware Knowledge Base article 103-1186, which has information about the Pkt Cap utility. There’s also a nice TCP dump document that shows all of the options for how you can specify ports and protocols and all of that capture information for TCP dump. In addition, on my site, www.travelingtechy.com/gopacket-captures, I have all of my notes on collecting packet captures. For additional information on command-line utilities, see my VMware vSphere.Network troubleshooting course.