A build guide for running Cisco Nexus Dashboard 3.2.2 as a single-node virtual appliance on Proxmox Virtual Environment 9.2, and onboarding a Nexus 9000v as a manageable device.
Proxmox is not on Cisco's supported hypervisor list for Nexus Dashboard. The platform runs Kernel-based Virtual Machine underneath, so Cisco's documented Linux Kernel-based Virtual Machine procedure mostly applies. Three differences will stop the deployment, and each fails without a useful error message. They are described below.
This guide is for lab and evaluation use.
- Three failure modes to know first
- Topology
- Addressing scheme
- Hypervisor host preparation
- Build the Nexus Dashboard virtual machine
- Workstation supporting services
- First boot
- Bootstrap wizard
- Post-bootstrap settings
- Prepare the switch
- Discovery
- Troubleshooting
- Resource requirements
- References
Peripheral Component Interconnect device addressing. Nexus Dashboard builds its management bond from interfaces named mgmt0 and mgmt1. Proxmox places virtual network devices at bus addresses that produce ens18 and ens19 instead. The bond forms with no members, the appliance never gets a working management interface, and nothing in the console explains why. The fix is to define both network devices in raw QEMU arguments at bus addresses 0x3 and 0x4.
Three management persistent addresses. Fabric Controller requires three addresses in the management external service pool. Two is not enough. With two, the LAN discovery worker pod never reaches a ready state and discovery emits no packets. The platform health command reports all components healthy and the service badge shows Healthy. The only record of the problem is a single line in the Fabric Manager log file.
Management network route. If the switches are on a different subnet from the management interface, Nexus Dashboard needs an explicit route entry in its cluster configuration. The service pods use that list to decide where they can send. The node's own routing table is separate, so a successful ping from the appliance shell proves nothing.
flowchart TB
subgraph HOST1["Hypervisor host — Proxmox Virtual Environment"]
ND["Nexus Dashboard node<br/>mgmt0 into bond1: 10.0.0.10/16<br/>fabric0: 10.1.0.2/24"]
VMBR0["vmbr0<br/>10.0.0.0/16<br/>host address 10.0.0.2"]
VMBR1["vmbr1<br/>10.1.0.0/24<br/>internal, no physical port"]
ND --- VMBR0
ND --- VMBR1
end
subgraph HOST2["Workstation — Ubuntu"]
ENO1["eno1: 10.0.0.5/16<br/>router, Domain Name System, Network Time Protocol"]
VIRBR0["virbr0<br/>192.168.122.1/24"]
EMU["Network emulator host<br/>192.168.122.9"]
SW["Nexus 9000v<br/>mgmt0: 192.168.122.60"]
ENO1 --- VIRBR0
VIRBR0 --- EMU
VIRBR0 --- SW
end
VMBR0 ---|direct cable| ENO1
The workstation is the only device joining the two segments. It holds an address on each and forwards between them.
A discovery request follows this path:
- Nexus Dashboard sources from 10.0.0.10 and sends to its gateway, 10.0.0.5.
- The workstation forwards the packet from
eno1ontovirbr0. - The switch receives it on 192.168.122.60 and replies to its own gateway, 192.168.122.1.
- The workstation forwards the reply back across the cable.
Both directions cross the workstation twice, once per interface. Three conditions must hold: Internet Protocol forwarding enabled, an accept rule permitting inbound connections toward the emulator bridge, and a masquerade exemption so the switch sees the real source address. All three are covered in Part 3.
The data network on vmbr1 has no physical port. Nothing outside the hypervisor can reach it. That is acceptable for a single-node lab because Fabric Controller reaches switches over the management network. Add a physical port to vmbr1 if you later need the fabric network to carry real traffic.
Substitute your own values. Everything here is private address space.
| Address | Assigned to | Notes |
|---|---|---|
| 10.0.0.2 | Hypervisor host, vmbr0 |
Web interface on port 8006 |
| 10.0.0.5 | Workstation, eno1 |
Gateway, Domain Name System, Network Time Protocol |
| 10.0.0.10 | Nexus Dashboard node, bond1 |
Browser interface on port 443 |
| 10.0.1.10 | Management external service pool | Claimed by the Power On Auto Provisioning service |
| 10.0.1.11 | Management external service pool | Claimed by the syslog and trap service |
| 10.0.1.12 | Management external service pool | Third address, required for validation to pass |
| Address | Assigned to | Notes |
|---|---|---|
| 10.1.0.1 | Nominal gateway | Does not exist; the wizard requires a value |
| 10.1.0.2 | Nexus Dashboard node, fabric0 |
Host address, not the network address |
| 10.1.0.10 | Data external service pool | |
| 10.1.0.11 | Data external service pool |
| Address | Assigned to | Notes |
|---|---|---|
| 192.168.122.1 | Workstation, virbr0 |
Gateway for the switches |
| 192.168.122.9 | Network emulator host | |
| 192.168.122.60 | Nexus 9000v, mgmt0 |
Discovery seed address |
Two constraints are worth planning around. The management persistent addresses must sit in the same subnet as the management interface. The data interface address must be a host address rather than a network address, and must not collide with the data persistent pool.
- Processor settings: virtualization technology enabled, logical processor enabled
- Integrated devices: virtualization technology for directed input/output enabled
- Boot mode: Unified Extensible Firmware Interface, secure boot disabled
- Storage controller: host bus adapter mode if using the Zettabyte File System, otherwise leave in Redundant Array of Independent Disks mode and use Logical Volume Manager thin
- Out-of-band management port: set to dedicated, with a static address
On a Linux workstation:
wget https://enterprise.proxmox.com/iso/proxmox-ve_9.2-1.iso
sha256sum proxmox-ve_9.2-1.iso
lsblk -o NAME,SIZE,MODEL,MOUNTPOINTS
sudo umount /dev/sdX*
sudo dd if=proxmox-ve_9.2-1.iso of=/dev/sdX bs=4M status=progress oflag=direct conv=fsync
syncCompare the checksum against the value published on the download page. Write to the whole device, not a partition.
If umount reports the target is busy, run lsof <mountpoint> to find the holder. It is usually the desktop file manager.
Boot the stick and select the Unified Extensible Firmware Interface entry from the one-time boot menu, then choose the graphical installer.
On a physical console, Tab moves focus and Space activates buttons. Enter does not always work. The Previous button on the license screen is greyed out because there is no earlier screen.
At the network configuration screen, enter the address the host will keep. The installer requires a gateway and Domain Name System server even on an isolated network.
Browse to https://<hypervisor-address>:8006.
Repositories. Node, then Updates, then Repositories. Disable the enterprise repository, add the no-subscription repository, then refresh and upgrade. Skip this if the host has no internet path.
Second disk as storage. Node, then Disks, then LVM-Thin, then Create: Thinpool. Select the disk, name it, and leave Add Storage ticked. If that box was cleared, the pool exists but is not registered as Proxmox storage. Register it from the shell:
vgs
lvs
pvesm add lvmthin vmdata --vgname vmdata --thinpool data --content images,rootdir
pvesm statusData bridge. Nexus Dashboard needs two interfaces. Create an internal bridge with no physical port:
cat >> /etc/network/interfaces <<'BRIDGE'
auto vmbr1
iface vmbr1 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
BRIDGE
ifreload -a
ip -br link show vmbr1If you create the bridge in the web interface instead, click Apply Configuration. Without it the bridge is staged but not active.
Download the Linux KVM image from Cisco's software download site. The file is named nd-dk9.<version>.qcow2 and is roughly 15 gigabytes. A Cisco account with entitlement is required.
The ISO and OVA images do not work on Proxmox. Use the KVM image.
Copy it to the hypervisor:
scp nd-dk9.<version>.qcow2 root@10.0.0.2:/var/lib/vz/template/iso/Use Create VM in the web interface:
| Tab | Setting |
|---|---|
| General | Virtual machine identifier 100, name of your choice |
| Operating system | Do not use any media; Linux, 6.x - 2.6 |
| Disks | Delete the default disk |
| Central processing unit | 1 socket, 16 cores, type host |
| Memory | 65536 megabytes, ballooning unticked |
| Network | Tick No network device |
| Confirm | Do not start after creation |
The processor type must be host. The appliance requires the Advanced Vector Extensions instruction set, which the default emulated type does not expose.
Networking is defined manually in section 2.4, which is why no network device is added here.
qm set 100 --machine pc --numa 0 --balloon 0
qm set 100 --serial0 socket
qm disk import 100 /var/lib/vz/template/iso/nd-dk9.<version>.qcow2 local-lvm --format raw
qm config 100
qm set 100 --virtio0 local-lvm:vm-100-disk-0
qm set 100 --virtio1 vmdata:500
qm set 100 --boot order=virtio0Read the imported volume name from qm config 100 before attaching it, since it may differ.
Both disks use the virtio bus. The machine type is pc, which is i440fx, not q35. Ballooning is disabled because it conflicts with the bus addressing used in the next section.
If the 500 gibibyte volume exceeds your thin pool, thin provisioning will still allow it. Monitor the pool with lvs and keep the data percentage well below 100.
This approach comes from Joseph Rinehart of Cisco, posted in the Cisco Community thread listed under References. Calin C. documented an alternative in the same thread that renames the interfaces from inside the appliance using systemd link files. That method works but modifies the image and has to be repeated after every redeployment.
Create the bridge attachment scripts on the hypervisor:
cat > /etc/nd-mgmt <<'SCRIPT'
#!/bin/sh
ip link set $1 master vmbr0
ip link set $1 up
SCRIPT
cat > /etc/nd-fabric <<'SCRIPT'
#!/bin/sh
ip link set $1 master vmbr1
ip link set $1 up
SCRIPT
chmod +x /etc/nd-mgmt /etc/nd-fabricEdit /etc/pve/qemu-server/100.conf and add the following as a single line at the top of the file:
args: -netdev type=tap,id=net0,ifname=tap100i0,script=/etc/nd-mgmt,downscript=no -device virtio-net-pci,mac=<MANAGEMENT-MAC>,netdev=net0,bus=pci.0,addr=0x3,id=net0,rx_queue_size=1024,tx_queue_size=256,host_mtu=1500 -netdev type=tap,id=net1,ifname=tap100i1,script=/etc/nd-fabric,downscript=no -device virtio-net-pci,mac=<DATA-MAC>,netdev=net1,bus=pci.0,addr=0x4,id=net1,rx_queue_size=1024,tx_queue_size=256,host_mtu=1500
Replace the two hardware address placeholders and adjust the tap interface names to match your virtual machine identifier. To obtain hardware addresses Proxmox considers valid for your host, create the virtual machine with two network devices first, record what Proxmox assigns, then delete the devices and use those values here.
Bus address 0x3 becomes mgmt0 inside the appliance. Bus address 0x4 becomes fabric0.
There must be no net0: or net1: lines in the configuration file. Proxmox-managed interfaces conflict with the raw definitions.
qm config 100 should show the args: line plus:
balloon: 0
boot: order=virtio0
cores: 16
cpu: host
machine: pc
memory: 65536
numa: 0
serial0: socket
virtio0: local-lvm:vm-100-disk-0,size=45G
virtio1: vmdata:vm-100-disk-0,size=500G
Start it and confirm the tap interfaces attached to the right bridges:
qm start 100
bridge link show | grep tap100The bootstrap wizard validates a Network Time Protocol server and a Domain Name System server. It will not proceed without both. On an isolated network the workstation provides them.
ip -br link
sudo ip link set eno1 up
sudo ip addr add 10.0.0.5/16 dev eno1Make it persistent with a netplan file:
network:
version: 2
ethernets:
eno1:
addresses:
- 10.0.0.5/16sudo chmod 600 /etc/netplan/99-lab.yaml
sudo netplan applyDo not set a gateway on this interface. A second default route breaks the workstation's own internet access.
sudo apt install chrony
sudo tee -a /etc/chrony/chrony.conf > /dev/null <<'CHRONY'
allow 10.0.0.0/16
local stratum 10
CHRONY
sudo systemctl restart chrony
chronyc sourcesThe local stratum 10 directive allows chrony to serve time when it has no upstream source of its own.
sudo apt install dnsmasq
sudo sed -i 's/#DNSStubListener=yes/DNSStubListener=no/' /etc/systemd/resolved.conf
sudo systemctl restart systemd-resolved
sudo tee /etc/dnsmasq.d/lab.conf > /dev/null <<'DNSMASQ'
listen-address=10.0.0.5
bind-interfaces
server=8.8.8.8
DNSMASQ
sudo systemctl restart dnsmasq
dig @10.0.0.5 cisco.com +shortReleasing port 53 from systemd-resolved is required on Ubuntu Desktop.
Required only when the switches sit on a different subnet reached through this workstation.
sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-lab-forward.confThe libvirt default network rejects inbound connections and masquerades outbound traffic. Two rules address both:
sudo iptables -t nat -I LIBVIRT_PRT 1 -s 192.168.122.0/24 -d 10.0.0.0/16 -j RETURN
sudo iptables -I LIBVIRT_FWI 1 -s 10.0.0.0/16 -d 192.168.122.0/24 -o virbr0 -j ACCEPTThe first prevents the switch from seeing all traffic as sourced from the workstation. The second permits Nexus Dashboard to open connections toward the switch.
libvirt rebuilds its chains whenever the network restarts, which flushes both rules. Make them persistent with a hook:
sudo mkdir -p /etc/libvirt/hooks
sudo tee /etc/libvirt/hooks/network > /dev/null <<'HOOK'
#!/bin/sh
if [ "$1" = "default" ] && [ "$2" = "started" ]; then
iptables -t nat -I LIBVIRT_PRT 1 -s 192.168.122.0/24 -d 10.0.0.0/16 -j RETURN
iptables -I LIBVIRT_FWI 1 -s 10.0.0.0/16 -d 192.168.122.0/24 -o virbr0 -j ACCEPT
fi
HOOK
sudo chmod +x /etc/libvirt/hooks/network
sudo systemctl restart libvirtdConfirm the network name with virsh net-list --all and adjust the comparison if it is not default. Test the hook by cycling the network:
sudo virsh net-destroy default && sudo virsh net-start default
sudo iptables -L LIBVIRT_FWI -n --line-numbers | head -3The accept rule should be at position 1 without running anything else.
Attach to the serial console:
qm terminal 100Exit with Control-O. The web interface console works as well.
First boot takes several minutes. Long static pauses are normal. Do not interrupt it. A hard stop partway through leaves the appliance in a state that takes longer to recover than waiting.
At the prompt to run first-boot setup:
- Set the administrator password. This becomes both the rescue user login and the initial browser password.
- Management address and mask:
10.0.0.10/16 - Gateway:
10.0.0.5 - Cluster leader: yes
- Review, then continue
Verify from the console once it settles:
ip addr
ping 10.0.0.5
acs healthInterfaces named mgmt0 and fabric0, with mgmt0 as a slave of bond1, confirm the bus addressing fix worked. Interfaces named ens18 and ens19 in a no-operation state mean it did not.
The appliance regenerates its Secure Shell host keys on every boot. Clear stale entries with ssh-keygen -f ~/.ssh/known_hosts -R 10.0.0.10.
Browse to https://10.0.0.10 and log in as admin with the console password.
Configuration
- Cluster name
- Network Time Protocol server:
10.0.0.5, entered as an address rather than a hostname - Domain Name System provider:
10.0.0.5
Node details
- Management:
10.0.0.10/16, gateway10.0.0.5 - Data:
10.1.0.2/24, gateway10.1.0.1 - Management persistent addresses:
10.0.1.10,10.0.1.11,10.0.1.12 - Data persistent addresses:
10.1.0.10,10.1.0.11
Three management persistent addresses are required. Two causes a silent failure documented under Troubleshooting.
Deployment mode
Fabric Controller. It is the only service supported on Kernel-based Virtual Machine deployments.
Save. Cluster formation takes up to thirty minutes. Refresh the browser manually rather than assuming the process has stalled. Do not stop the virtual machine during this stage.
Feature management
After bootstrap the left navigation may show only Overview and Admin. Go to Admin, then System Settings, then Feature Management, and select the deployment type. Fabric Discovery covers inventory and monitoring. Fabric Controller adds configuration management. The service restarts and the LAN menu appears.
Both settings are required before discovery works against a switch on another subnet.
LAN device management connectivity. Fabric Controller, then Admin, then System Settings, then Server Settings, then the Admin tab. Set to Management when the switches are reachable through the management interface. If it is set to Data, the service sources traffic from the data network and never reaches them.
Management network route. Admin Console, then System Resources, then Cluster Configuration, then Routes, then Management Network Routes. Add the switch subnet:
192.168.122.0/24
The service pods use this list to decide where they can send traffic. It is independent of the node's own routing table.
configure terminal
feature nxapi
snmp-server user <username> network-admin auth sha-256 <passphrase> priv aes-128 <passphrase>
vrf context management
ip route 0.0.0.0/0 192.168.122.1
exit
end
copy running-config startup-config
The Simple Network Management Protocol version 3 passphrase must be at least eight characters. Shorter values are rejected by the protocol without a useful error. NX-OS auto-creates a Simple Network Management Protocol user from each local login account, so a short login password produces an unusable Simple Network Management Protocol user. Create an explicit user instead.
The management Virtual Routing and Forwarding instance needs a default route. Without one the switch answers its own subnet but nothing beyond it.
Verify from a host on the switch's subnet before using the interface:
snmpwalk -v3 -l authPriv -u <username> -a SHA-256 -A '<passphrase>' \
-x AES -X '<passphrase>' 192.168.122.60 1.3.6.1.2.1.1.1.0Use the numeric object identifier unless the standard management information bases are installed locally. A successful query returns the system description string.
On a factory switch, Power On Auto Provisioning runs when the startup configuration is empty. Answer yes or skip at the abort prompt, then disable it with no boot poap enable and save the configuration.
Fabric Controller, then LAN, then Fabrics, then Create Fabric. LAN Classic suits existing switches you want to monitor. Choose a VXLAN EVPN template for configuration management.
Open the fabric, then Switches, then Add Switches:
| Field | Value |
|---|---|
| Seed address | 192.168.122.60 |
| Username | Simple Network Management Protocol user created in Part 7 |
| Password | Its passphrase |
| Authentication protocol | SHA2_256 |
| Maximum hops | 0 or 1 |
Use a single seed address rather than a range. Maximum hops counts Cisco Discovery Protocol neighbour depth, not Internet Protocol hops.
The switch appears in the discovery results and then transitions to Manageable.
Confirm on the wire while discovery runs:
sudo tcpdump -ni eno1 host 192.168.122.60A working exchange shows an ICMP echo request and reply, followed by a Transmission Control Protocol session to port 161 carrying the Simple Network Management Protocol traffic.
| Symptom | Cause | Where to look |
|---|---|---|
bond1 has no slaves; interfaces named ens18 and ens19 in a no-operation state |
Wrong bus addressing | acs system-config, where the uplinks list names mgmt0 and mgmt1 |
| Cluster reports healthy, discovery finds nothing, no packets on the wire | Fewer than three management persistent addresses | grep -i "external IP check" /app_logs/cisco-ndfc-dcnm-apps-logs/dcnm-fm/logfile-fm.log.log.1 |
| Appliance pings the switch but the service does not reach it | Missing management network route | Cluster Configuration, then Routes |
| Discovery reaches the switch but authentication fails | Passphrase under eight characters, or protocol mismatch | show snmp user on the switch; snmpwalk from the switch's subnet |
| Switch answers its own subnet only | No default route in the management Virtual Routing and Forwarding instance | show ip route vrf management |
| Connectivity works, then stops after a reboot | libvirt flushed the firewall rules | iptables -L LIBVIRT_FWI -n --line-numbers |
acs health
acs system-config
acs kubectl get pods -n cisco-ndfc
acs kubectl describe pod -n cisco-ndfc <pod-name> | tail -30acs health covers the platform, not the Fabric Controller service. It reports all components healthy while the service is unable to function. The pod list is the authoritative view. A pod showing 0/1 has a failing readiness probe.
Application logs are written to files rather than standard output, so kubectl logs returns nothing useful. Read them from the mounted paths instead:
ls -la /app_logs/cisco-ndfc-dcnm-apps-logs/
grep -iE "error|fail|refused" /app_logs/cisco-ndfc-dcnm-apps-logs/<component>/<logfile>Per node, Cisco's figures are requirements rather than recommendations:
- 16 virtual central processing units. Eight will not work.
- 64 gigabytes of memory
- 550 gigabytes of disk across two volumes
- Solid state or non-volatile memory express storage with input/output latency of 20 milliseconds or less
The imported boot volume is 45 gigabytes. Cisco documents 50, and at least one community report describes the installation hanging until it was increased to 100. If the bootstrap stalls with no other explanation, qm resize 100 virtio0 +55G is worth trying before rebuilding.
Nexus Dashboard on Proxmox, Cisco Community https://community.cisco.com/t5/cisco-software-discussions/nexus-dashboard-on-proxmox/td-p/5261138
Source of the bus addressing fix in section 2.4. Joseph Rinehart of Cisco posted the full procedure using raw QEMU arguments and bridge attachment scripts. Calin C. posted an alternative that renames the interfaces from inside the appliance through rescue mode at the boot loader. Contributors also report boot volume sizing issues and variations on the bridge scripts for virtual local area network aware bridges.
Cisco Nexus Dashboard 3.2.2, Deploying in Linux KVM https://www.cisco.com/c/en/us/td/docs/dcn/nd/3x/deployment/cisco-nexus-dashboard-and-services-deployment-guide-321/nd-deploy-kvm-322.html
The official procedure. Source of the resource requirements, the raw disk conversion, the virtio disk and network models, and the first-boot console sequence. It specifies virt-install, which places devices at different bus addresses than Proxmox does. That difference is the reason section 2.4 exists.
Proxmox Virtual Environment downloads https://www.proxmox.com/en/downloads/proxmox-virtual-environment/
Installer images and published checksums.