
                               README Notes
                     Broadcom bnxtnd.sys Windows Driver

                 Copyright (c) 2015 - 2024 Broadcom Incorporated
                           All rights reserved


Table of Contents
=================

  Introduction
  Device Manager Errors
  Port Speeds
  SR-IOV and Jumbo Frames
  NPAR and SR-IOV
  SR-IOV and Get-NetAdapterSriov
  SR-IOV and Promiscuous mode
  Adjusting Number of MSIX vectors
  PAM4 Support



Introduction
================================
This file describes the bnxtnd.sys Windows driver for the Broadcom NetXtreme-C and NetXtreme-E BCM573xx, BCM574xx. BCM575xx, BCM576xx 10/25/40/50/100/200/400 Gbps Ethernet Network Controllers.



Device Manager Error Codes
================================
This device cannot find enough free resources that it can use. (Code 12)
Most likely this is due to the OS running out of MSIX messages. See "Adjusting Number of MSIX vectors" below.



Port Speeds
================================
On some dual-port devices, the port speed of each port must be compatible with the port speed of the other port. 10Gbps and 25Gbps are not compatible speeds.
For example, if one ports Speed & Duplex advanced property setting is 10Gbps and link is up, the other port cannot be set to 25Gbps.
The firmware will allow incompatible speeds to be set on the two ports if link is not up yet.
  - Subsequent link up on one port will render the incompatible speed on the other port to become unsupported.
  - This will lead to a system error log like this may appear when this scenario happens:

    - Incompatible speed selection between Port 1 and Port 2. Link configuration became illegal.

If Speed & Duplex settings for the two ports are incompatible, you will also see this system error log message:

   Incompatible speed selection between Port 1 and Port 2. Reported link speeds are correct and might not match Speed and Duplex setting.

On some NPAR (NIC partitioning) devices where one port is shared by multiple
PCI functions, the port speed is pre-configured by the CCM BIOS utility (invoked by pressing ^S at the Broadcom banner during Legacy BIOS boot), or, if using a UEFI system,
the HII interface for the NIC. This is why the Speed & Duplex advanced property is not available for NPAR miniports.



SRIOV and Jumbo Frames
================================
Windows SRIOV includes a failover feature such that if the VF path fails to work for some reason, or if the VM is migrated to a host without SR-IOV enabled, the non-VF
synthetic NIC will continue to be connected. This is why a VM using SR-IOV will show 2 devices in Device Manager per one network connection. The synthetic path is
established before, and torn down after, the VF path.

If jumbo frames are needed to the VF, make sure that the VF and the corresponding synthetic NIC have identical Jumbo Frame setting and the failover can work properly.

With or without SR-IOV, the jumbo frame settings for the host OS PF driver must be the same or greater than any Jumbo Frame settings for VM NICs using that host PF. 



NPAR and SR-IOV
================================
The number of VFs available when NPAR is enabled will be lower than when NPAR is not used.
To maximize the number of VFs available reduce the MSI-X vectors to 4 or less in the UEFI Device configuration menu.

Guest Mode RDMA (RDMA on a VF) is not supported when NPAR is enabled.



SR-IOV and Get-NetAdapterSriov
================================
Get-NetAdapterSriov will show supported or not supported strictly based on the presence of the *SRIOV keyword in the device registry. This means that if
SR-IOV is disabled on the device via NVRAM, it may still show as supported via the Get-NetAdapterSriov powershell command because the *SRIOV keyword is present
by default in the registry. If you are not able to create virtual functions, check your NVRAM configuration to make sure SR-IOV is enabled.



SR-IOV and Promiscuous mode
================================
Promiscuous mode is not supported on VFs.



Guest Mode RDMA (RDMA on a VF)
================================
Windows Server has a known issue when using Guest Mode RDMA that may cause the MS synthetic NIC to not start and show a yellow bang. To resolve this,
disable and re-enable the MS syhtetic NIC in the VM device manager.



Get-NetAdapterStatistics on Windows Server 2022
================================
On Windows Server 2022, the PowerShell command (Get-NetAdapterStatistics).RdmaStatistics will show junk values when RDMA is disabled on the given adapter. When
RDMA is enabled correct statistics are shown. This is a known issue in Windows Server 2022.



Adjusting Number of MSIX vectors
================================
The number of MSIX vectors is set soon after the NIC is powered on, and before the PCIe bus enumerates the NIC.
Thus the values can only be changed by the CCM BIOS utility (invoked by pressing ^S at the Broadcom banner during Legacy BIOS boot), or, if using a UEFI system, the HII interface for the NIC.
There is a separate parameter for the "Max number of PF MSIX vectors" for a PF (Physical Function) in the host operating system, and for the "MSI-X Vectors per VF" (Virtual Function) when using SRIOV in a guest virtual machine.

The default values were chosen to cover the majority of configurations, but there are definitely cases where a system administrator might want to increase or decrease the number of MSIX vectors.

If you are running many VM's using VMQ (not SRIOV) and saw that some vmNIC's were not getting a VMQ, you would *increase* the Max number of PF MSIX vectors.
If you get a code 12 "yellow-bang" on a PF, you would *decrease* Max number of PF MSIX vectors. If you get it on a VF, you would decrease MSI-X Vectors per VF

Device manager code 12 yellow-bangs are more likely if you have many miniports (due to either enabling NPAR or by having many NIC's installed) and if you have only a few processor cores (1, 2, or 4).
Many miniports increases the number of MSIX message resources needed in the OS. Few processors decreases the number of available MSIX message resources.



PAM4 Support
================================
Windows does not contain a PAM4 signalling configuration parameter in adapter device advanced properties
For speeds supported by both NRZ and PAM4 Windows will default to NRZ without the following work around.

Examine the NXE device in Windows Device Manager, Network Adapters, Broadcom xxx. 
Right mouse click and select properties, select Details tab, select Driver Key from the Property drop down, and record the last 4 digits after the backslash.
Enter the registry editor (regedit.exe) and traverse to 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\xxx where xxx is the number recored above
Add a new string type value named PreferredSignalingMode, and set the value to 1. Save and exit the registry editor.

