Lately I troubleshoot some NIC driver problems in VMware ESXi 5.x. During the troubleshooting I needed to Identify the NIC driver, software version used and the latest driver version supported by VMware. Here are the steps outlined. Identify the NIC driver and software version:. Make a SSH connection to the ESXi host.
Identify all the NIC’s by using the following command: esxcli network nic list. In this example you can see that Intel (igb) and Broadcom (tg3) NICs are used. To identify the driver version of the Broadcom NIC (vmnic0) use the following command: ethtool -i vmnic0 driver: tg3 version: 3.135b.v50.1 firmware-version: FFV7.6.14 bc 5720-v1.31 bus-info: 0000:01:00.0. The output list the driver and firmware version. Run the “ethtool” command for all different NICs drivers to obtain the driver version. With the introduction of the new native mode driver, ethtool is no longer supported.
Using ethtool with a native drivers results in the following warning “Can not get control fd: No such file or directory”. The following esxcli command list the driver and firmware version: esxcli network nic get -n vmnic0 Identify the latest NIC driver supported by VMware:. We need to find the hardware ID’s of the NIC by using the following command: vmkchdev -L grep “vmnic” The hardware ID contains the following parts: VID = Vendor Id (Red part) DID = Device Id (Yellow part) SVID = Sub-Vendor Id (Green part) SSID = Sub-Device Id (Blue part). Open the VMware Compatibility Guide (HCL) webpage and choose IO Devices,.
On the right you see the “Additional Criteria” Colum. Enter the VID, DID, SVID and SSID generated by the “vmkchdev” command. Click the “Update and View Results” button. The NIC is displayed, click on the Model. Expand the version of ESXi (release) used. Click on the the Download link (if exist).
VMXNet NIC drivers. The vmxnet and vmxnet3 networking drivers improve network performance. Which driver is used depends on how you configure device settings for the virtual machine. Search the VMware Knowledge Base for information on which guest operating systems support these drivers.
Download the driver and use VMware Update Manager (VUM) or use “ esxcli software vib update” command to install the driver.
Earlier this month I wrote an demonstrating a functional USB ethernet adapter for ESXi 5.1. This was made possible by using a custom built driver for ESXi that was created over three years ago by a user named. After having re-discovered the thread several years later, I had tried reaching out to the user but concluded that he/she has probably moved on given the lack of forum activity in the recent years.
Over the last few weeks I have been investigating to see if it was possible to compile a new version of the driver that would function with newer versions of ESXi such as our 5.5 and 6.0 release. UPDATE (01/22/17)- For details on using a USB-C / Thunderbolt 3 Ethernet Adapter, please see this post.
Updating NIC Drivers in ESXi from the CLI A video walk-through on updating your NIC drivers from the command line for maximum control. There are a number of reasons you may want to update your NIC drivers and firmware. Install the NIC Driver on ESXi For the best performance, use SR-IOV with Intel 10GB network interfaces which requires the ixgbe 4.4.1 driver to support multiple queues for each interface. Obtain a list of network interfaces on the ESXi host.
UPDATE (11/17/16) - New driver has been updated for ESXi 6.5, please find the details. After reaching out to a few folks internally, I was introduced to Songtao Zheng, a VMware Engineer who works on some of our USB code base. Songtao was kind enough to provide some of assistance in his spare time to help with this non-sanction effort that I was embarking on. Today, I am please to announce that we now have a functional USB ethernet adapter driver based on the that works for both ESXi 5.5 and 6.0. This effort could not have been possible without Songtao and I just want to say thank you very much for all of your help and contributions. I think it is safe to say that the overall VMware community also thanks you for your efforts. This new capability will definitely enable new use cases for vSphere home labs that were never possible before when using platforms such as the Intel NUC or Apple Mac Mini for example. Thank you Songtao! I would also like to extend an additional thank you to Jose Gomes, one of my readers, who has also been extremely helpful with his feedback as well as assistance on testing the new drivers.
Now, Before jumping into the goods, I do want to mention there are a few caveats to be aware of and that I think it is important to understand them before making any purchasing decisions. First and foremost, this is NOT officially supported by VMware, use at your own risk.
Secondly, we have observed there is a substantial difference in transfer speeds between Transmit (Egress) and Receive (Ingress) traffic which may or may not be acceptable depending on your workload. On Receive, the USB network adapter is performing close to a native gigabit interface. However, on Transmit, the bandwidth mysteriously drops by 50% which includes very inconsistent transfer speeds.
We are not exactly sure why this is the case, but given ESXi does not officially support USB based ethernet adapters, it is possible that the underlying infrastructure was never optimized for such devices. YMMV. Lastly, for the USB ethernet adapter to properly function, you will need a system that supports USB 3.0 which kind of makes sense for this type of a solution to be beneficial in the home lab. If you have a system with USB 2.0, the device will probably not work at least from testing that we have done. Note: For those interested in the required source code changes to build the AX88179 driver, I have published all of the details on my Github repo. Disclaimer: In case you some how missed it, this is not officially supported by VMware. Use at your own risk.
Without further ado, here are the USB 3.0 gigabit ethernet adapters that are supported with the two drivers:. (verified by reader 03/29/16). (verified by VMware employee 05/19/16). (verified by twitter user 07/29/16) Note: There may be other USB ethernet adapters that uses the same chipset which could also leverage this driver but these are the only two that have been verified. Here are the ESXi driver VIB downloads:. or.
or. or Note: Although the drivers were compiled against a specific version of ESXi, they should also work on the same major version of ESXi, but I have not done that level of testing and YMMV. Verify USB 3.0 Support As mentioned earlier, you will need a system that is USB 3.0 capable to be able to use the USB ethernet adapter. If you are unsure, you can plug in a USB 3.0 device and run the following command to check: lsusb What you will be looking for is an entry stating ' Linux Foundation 3.0 root hub' which shows that ESXi was able to detect a USB 3.0 port on your system. Secondly, look for the USB device you just plugged in and ensure the 'Bus' ID matches that of the USB 3.0 bus.
This will tell you if your device is being claimed as a USB 3.0 device. If not, you may need to update your BIOS as some systems may have USB 2.0 enabled by default like earlier versions of Intel NUC as desribed.
Esxi 6.7 Nic Teaming
You may also be running pre-ESXi 5.5 which did not support USB 3.0 as mentioned, so you may need to upgrade your ESXi host to at least 5.5 or greater. Install Driver You can either install the VIB directly onto your ESXi host or by creating a custom ESXi ISO that includes the driver using a popular tool like by Andreas Peetz. To install the VIB, upload the VIB to your ESXi host and then run the following ESXCLI command specifying the full path to the VIB: esxcli software vib install -v /vghetto-ax88179-esxi60u2.vib -f To verify that the USB network adapter has been successfully claimed, run either of the following commands to list your physical NICs: esxcli network nic list esxcfg-nics -l To add the USB uplink, you will need to either use the vSphere Web Client or ESXCLI to add the uplink to either a Virtual or Distributed Virtual Switch. To do so using ESXCLI, run the following command and specify the name of your vSwitch: esxcli network vswitch standard uplink add -u vusb0 -v vSwitch0 Uninstall Driver To uninstall the VIB, first make sure to completely unplug the USB network adapter from the ESXi first. Next, run the following ESXCLI command which will automatically unload the driver and remove the VIB from your ESXi host: esxcli software vib remove -n vghetto-ax88179-esxi60u2 Note: If you try to remove the VIB while the USB network adapter is still plugged in, you may hang the system or cause a PSOD. Simply reboot the system if you accidentally get into this situation. Troubleshooting If you are not receiving link on the USB ethernet adapter, it is most likely that your system does not support USB 3.0. If you find the a similar message like the one below in /var/log/vmkernel.log then you are probably running USB 1.0 or 2.0.
2016-03-21T23:30:49.195Z cpu6:33307)WARNING: LinDMA: LinuxDMACheckConstraints:138: Cannot map machine address = 0x10f5b6b44, length = 2 for device 0000:00:1d.7; reason = address exceeds dmamask (0xffffffff)) Summary For platforms that have limited built-in networking capabilities such as the Intel NUC and Apple Mac Mini, customers now have the ability to add additional network interfaces to these systems. This will now open up a whole new class of use cases for vSphere based home labs that were never possible before, especially with solutions such as VSAN and NSX. I look forward to seeing what our customers can now do with these new networking capabilities.
Additional Info Here are some additional screenshots testing the dual USB 3.0 ethernet adapter as well as a basic iPerf benchmark for the single USB ethernet adapter. I was not really impressed with the speeds for the dual ethernet adapter which I had shared some more info. Unless you are limited on number of USB 3.0 ports, I would probably recommend just sticking with the single port ethernet adapter.
IPerf benchmark for Ingress traffic (single port USB ethernet adapter): iPerf benchmark for Egress traffic (single port USB ethernet adapter). Thx for this great post. I’m quite interested with USB ethernet adapter on ESXi indeed. I just wondering if there’s a reason you have added that ‘jjkjkj’ odd string at line 1721 of ax88179178.c: ——————— #ifndef RXSKBCOPY jjkjkj axskb = skbclone(skb, GFPATOMIC); #else ——————— OK it has no impact since you have removed the comment of #defined RXSKBCOPY in ax88179178.h!
Was it to be sure skbclone will not be used? From Jose Gomes comment, this makes the server crash after 5 min!?
William, can you say if more work will be done (by whomever) to try and fix the egress traffic issues when using these USB adapters? If that were fixed and the driver made more stable, that would really be the opening of the gate for so many home labs with or without the Intel NUC. Many running (or that could run) a home lab with this hardware really do need at a minimum of two solid 1 GbE adapters. With just 1.5 at the moment, it’s still a bit of a plunge to take.
Awesome work and I’m watching your blog carefully to see if any new developments happen over the next several months! Hi, Just got around to running iPerf.
On my i3 BRIX 6.0u2 I. Get the following throughput::/usr/lib/vmware/vsan/bin./iperf -m -i t300 -c 172.x.x.x -fm WARNING: interval too small, increasing from 0.00 to 0.5 seconds.
Hi, Great post!!! Hi William, I am thinking about updating my ESXi-based lab by replacing two 2012 Mac Minis (quad-core i7 2.6 GHz CPUs) with two NUC Skull Canyons(i7 quad-core 6th generation NUC). It appears that standard images of ESXi 5.5 U3 and ESXi 6.0 U1 may not require any tweaks to run on the NUC Skull Canyon (6th generation NUC) based on this article: Are you planning to get a NUC Skull Canyon (6th generation NUC) to try in your lab.
Do you believe that based on the article I linked to above, the 6th generation NUC Skull Canyon will be able to run ESXi 5.5 U3 and/or ESXi 6.0 U1 without any additional tweaks? If 6th generation NUC Skull Canyon can run ESXi 5.5 U3 or ESXi 6.0 U1, the only downside to replacing Mac Minis with 6th generation NUC Skull Canyon is that the latter has only one built-in Gigabit Ethernet adapter, and there are currently (as of this writing) no Thunderbolt3-to-GigabitEthernet adapters (that I can find) on the market. Your work on the driver for USB3-based Ethernet adapters is commendable, but I really do not want to take a hit on the egress throughput from ESXi via a USB3-based Ethernet adapter judging by the iPerf stats that you provided. Therefore, I’ve been looking at alternatives. What do you think about using this (soon-to-be-released-as-of-this-writing) Thunderbolt3-to-Thunderbolt adapter from StarTech to be used together with the Apple’s Thunderbolt-to-GigabitEthernet adapter so that the NUC Skull Canyon (6th generation NUC) can have two GigabitEthernet NICs (just like the 2012 Mac Minis have in my lab)? —– Thank you. I was wondering if anyone else is running into this issue while trying to install.
I’m using a clean ESXi 6U2 install on gen5 i7 NUC. Error states the VIB does not contain a signature. I changed to CommunitySupported, and the issue still persists.
Any help is very appreciated. Unsure where to locate the log it’s referencing – checked a few in /var/log, but no clues. : lsusb Bus 002 Device 006: ID 05ac:024f Apple, Inc. Bus 003 Device 002: ID 0b95:1790 ASIX Electronics Corp. AX88179 Gigabit Ethernet Bus 002 Device 005: ID 8087:0a2a Intel Corp. Bus 002 Device 004: ID 15d9:0a4c Trust International B.V. USB+PS/2 Optical Mouse Bus 002 Device 002: ID 05ac:1006 Apple, Inc.
Hub in Aluminum Keyboard Bus 001 Device 002: ID 8087:8001 Intel Corp. Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub : esxcli software vib install -v /vmfs/volumes/qnapNFS/ISO/VMware/ESXi/vghetto-ax88179-esxi60u2.vib InstallationError (‘virtuallyGhettobootbankvghetto-ax88179-esxi60u26.0.0-1.0.0’, ‘The VIB virtuallyGhettobootbankvghetto-ax88179-esxi60u26.0.0-1.0.0 does not contain a signature.’) vibs = virtuallyGhettobootbankvghetto-ax88179-esxi60u26.0.0-1.0.0 Please refer to the log file for more details. : esxcli software acceptance get CommunitySupported. Does anybody have a problem when trying to use the VMXNET3 virtual network adapter together with your Startech USB 3.0 to Gigabit Ethernet dongles?
I’m running pfSense 2.3 and an Intel NUC D54250WYK and have the following setup: Startech No. 1 (WAN) to my ethernet jack in my wall which gives me internet Startech No. 2 (LAN) to my smart switch Intel Intel i218v (Intel NUCs integrated NIC) is dedicated to ESXi management to my smart switch In ESXi I have one vswitch for “WAN” and one vswitch for “LAN” and a separate one for VMware Management. Both “WAN” and “LAN” is assigned pfSense and when I choose “E1000” as the adapter type it works but if I change to “VMXNET3” then I don’t get an IP from my ISP on the “WAN”-side and I am unable to ping pfSense “LAN”-side or connect to it through the webinterface. My BIOS is 0041 and I’ve changed the USB-settings to “Enabled” from “Auto-something” in the BIOS as well. I integerated the drivers above – “ESXi 6.0 Update 2 USB Ethernet Adapter Driver VIB” with “ESXi-Customizer-PS”.
The VM was created with USB 3.0 support if that could impact anything. I have installed pfSenses own package of “VMware Tools”. Would be grateful to hear if any of you guys have been experiencing the same problems. More detailed post: Best Regards, Karl. Playing around with a USB to Ethernet adapter (Orico UTR-U3) with the installed vib on ESXi 6.0u2 the limit seemed to be 4088 MTU. It seems to have the same chipset as the others listed.
Esxi 6.7 Realtek Drivers Install
: lsusb Bus 001 Device 004: ID 8087:0a2b Intel Corp. Bus 001 Device 003: ID 413c:2003 Dell Computer Corp. Keyboard Bus 001 Device 002: ID 18a5:0302 Verbatim, Ltd Flash Drive Bus 002 Device 002: ID 0b95:1790 ASIX Electronics Corp. One other thing I noticed the -v/–viburl parameter used during the vib install needs to have an absolute path, relative paths fail. :/vmfs/volumes/57685ad8-d3ef4fd8-9a6e-001fc69b9d34 esxcli software vib install -v./vghetto-ax88179-esxi60u2.vib -f VibDownloadError (‘./vghetto-ax88179-esxi60u2.vib’, ”, “Errno 4 IOError: “) url =./vghetto-ax88179-esxi60u2.vib Please refer to the log file for more details. :/vmfs/volumes/57685ad8-d3ef4fd8-9a6e-001fc69b9d34 ls -la total 792576 drwxr-xr-t 1 root root 1400 Jun 20 21:08.
Drwxr-xr-x 1 root root 512 Jun 20 21:17. Hi William, I’ve got myself the 1port and 2port adapter, the once others have sucessfully installed. I am so far that i can see the device but link negotiation can’t be changed and whatever i do, it’s not detecting a link. I emailed the guys some time back at GoRite about the Skull Canyon Lids with GiG E. I just recently emailed them again about offering alternate Chipsets.
I suspect in order for it to be plug/plug we need to be using some chipset that supports the e1000e adapter can someone confirm that? I was hoping they were early enough in dev to change. But this is what they had said the chipset would be DL-3950/5910. If early enough and there is enough demand, we might be able to show interest for a chipset that would be esxi compatible. Hi Mike, The DisplayLink chipset offers more than just Ethernet (HDMI/DVI/DP), but in anyway, they are USB 3.0 based. I don’t think it would make sense for GoRite to provide Ethernet only capability, as it would be too niche.
Also, except for the NUC5i5MYHE, no other NUC exposes PCIe based interfaces than trough mSATA/M.2, so having a second ESXi compatible Ethernet PHY is almost impossible. For USB based Ethernet we already have drivers for AXIS and Realtek based adapters anyway. G2 Digital seems to be developing an M.2 based Ethernet adapter, but there is no indication that the project is actually progressing.
I don’t know which chipset they are planning on using either. William, first of all thank you for the detailed stepseverything worked fine for me. I tested the procedure on a Gigabyte Brix and TrendNet U3 adaptersworked just fine.
Now the caveat on my case was the addition of 3 adapters which I need for a specific use case. Although they were successfully detected and mapped to vswitches, after a reboot the last adapter needs to be manually mapped again – and this keeps happening after every reboot. This is an acceptable workaround for me but I’m wondering if you’ve ever come across such issue and why the issue only applies to the 3rd adapter (vusb2 in my case). Any hints will be appreciated! Cheers, Peixe.
I recently bought an 7th Gen Intel NUC (Kaby Lake) – NUC7i5BNH and a StarTech USB 3.0 to Gigabit Ethernet NIC Adapter. I installed driver as instructed above on ESXi 6.0 U3. I attached the vusb0 and the onboard vmnic0 both to my distributed virtual switch. In itself it all works fine, but every couple minutes I get this sequence of events: USB configuration has changed. Lost uplink redundancy on DVPorts:. Physical NIC vusb0 is down. Uplink redundancy restored on DVPorts:.
Physical NIC vusb0 is up. Alarm ‘Network uplink redundancy lost’ on srvchanged from Green to Red Alarm ‘Network uplink redundancy lost’ on srvchanged from Red to Green Any suggestions?
I purchased the Startech 31000S USB-NICs mentioned in this blog article, but I used it to lab up a Hyper-V (2016) cluster. Turns out there’s an issue with these NICs when it comes to trunking VLANs. I found that, despite assigning “Management”, “Cluster” and “Live Migration” traffic to separate VLANs, all traffic went out on the same VLAN (the Management VLAN).
Only discovered that after reviewing the upstream switch’s MAC address table So, I used the NUC’s onboard NIC instead and it worked right away, using the same PoSH script that I tried with the Startech NIC, to configure the converged network stack. It may be a flaw with the driver, as the Hyper-V hosts were 2016 and perhaps the Startech NICs aren’t fully compatible yet.
Hope this comment saves someone else from spending hours trying to get Hyper-V converging networking working with these NICs.