Over 18 months since the last post and I’m somehow still writing about basically the same thing! Weird!
In the last post I was writing about booting a Cisco Nexus 9000v image with Qemu, and I ran into some (definitely not self inflicted) challenges… Since that time I’ve been mostly running virtual devices via the fantastically useful vrnetlab – if you have not checked out vrnetlab I highly recommend it. Its kind of a weird thing, and it feels all sorts of wrong due to the fact that you are running a VM in a container (weird right??), but it is so very handy and a great way to package devices. There is a ton of cool stuff in the vrnetlab world worth checking out, but this post is not about that!
One of the challenges with vrnetlab is that it puts a Qemu virtual machine inside of the containers that it spawns, and in order to get connectivity to that virtual machine there is a bit of magic with socat and Qemu NAT. Despite being a bit of magic, this is totally cool, and it works really nicely – you basically just SSH/Netconf to the container IP and socat/NAT handles getting your connection plumbed into the container – due to the Qemu NAT return traffic is also all sorted out. This works great until you need to get from the router to something else. In my case I want to be able to easily spin up/down some routers for testing some streaming telemetry type things – that would of course include dial out… and so I’ve hit a bit of a road block, and I’m basically back to where I was 18 months ago at my last post – I need to spin up these devices “natively” (without vrnetlab).
All of my past experience with getting Qemu hosts fired up has been natively on Linux hosts – this makes sense because Qemu can use KVM for acceleration which makes everything much nicer! This time around, however, I wanted to run this natively on my mac, which of course presented a few interesting challenges!
Firstly, there is of course no KVM on Macos, but this is OK, because there is HVF! I’m positive there are meaningful differences between KVM and HVF, but for my purposes (and in testing thus far) I believe these can more or less be interchanged, so thats one problem down!
The next challenge is of course having to re-remember how to actually craft the Qemu command in order to get the routers to fire up. Thankfully, this is a relatively trivial task as we can boot up our devices in vrnetlab and snag the Qemu command that vrnetlab uses to launch the routers in the container. Here is an example for booting the csr1000v image:
qemu-system-x86_64 \ -M accel=hvf \ -display none \ -machine pc \ -monitor tcp:0.0.0.0:4001,server,nowait \ -m 8192 \ -serial telnet:0.0.0.0:5001,server,nowait \ -drive if=ide,file=csr1000v-universalk9.16.12.03-overlay.qcow2 \ -device pci-bridge,chassis_nr=1,id=pci.1 \ -device virtio-net-pci,netdev=p00,mac=52:54:00:01:01:00 -netdev tap,ifname=tap0,script=no,downscript=no,id=p00 \ -device virtio-net-pci,netdev=p01,mac=52:54:00:01:01:01,bus=pci.1,addr=0x2 -netdev socket,id=p01,listen=:10001 \ -device virtio-net-pci,netdev=p02,mac=52:54:00:01:01:02,bus=pci.1,addr=0x3 -netdev socket,id=p02,listen=:10002 \ -device virtio-net-pci,netdev=p03,mac=52:54:00:01:01:03,bus=pci.1,addr=0x4 -netdev socket,id=p03,listen=:10003 \ -device virtio-net-pci,netdev=p04,mac=52:54:00:01:01:04,bus=pci.1,addr=0x5 -netdev socket,id=p04,listen=:10004 \ -device virtio-net-pci,netdev=p05,mac=52:54:00:01:01:05,bus=pci.1,addr=0x6 -netdev socket,id=p05,listen=:10005 \ -device virtio-net-pci,netdev=p06,mac=52:54:00:01:01:06,bus=pci.1,addr=0x7 -netdev socket,id=p06,listen=:10006 \ -device virtio-net-pci,netdev=p07,mac=52:54:00:01:01:07,bus=pci.1,addr=0x8 -netdev socket,id=p07,listen=:10007 \ -device virtio-net-pci,netdev=p08,mac=52:54:00:01:01:08,bus=pci.1,addr=0x9 -netdev socket,id=p08,listen=:10008 \ -device virtio-net-pci,netdev=p09,mac=52:54:00:01:01:09,bus=pci.1,addr=0xa -netdev socket,id=p09,listen=:10009
A few things to note above:
- I’ve already replaced the
-enable-kvmthat I’ve historically used (and that vrnetlab uses) with
- This command will give us 10 interfaces on the device – interfaces
GigabitEthernet10are connected to sockets – for now this means they are basically just floating around doing nothing, but we can in the future connect them to other Qemu instances to build arbitrary topologies which is pretty snazzy!
- Lastly, but very critically – instead of the “normal” Qemu host NAT that vrnetlab uses for
GigabitEthernet1(what we use as the management interface) we’ll set this up to be a tap interface connected to
This last part is of course so that we can get this router talking out by just having it sit on the local LAN without any of the socat/NAT magic, and this is where things got a little crazy!
Everything was seeming to go as expected, however the interface would just not pick up a DHCP address! What gives I thought? Well maybe there is just something messed up with my DHCP server (really just my 3560CX) that is doing something that the virtual router is unhappy with… OK, so I attempted to provide a static address to this device…. same result… nothing working!
At this point I was getting pretty frustrated, and Google wasn’t really giving me anything helpful to go off of (or so I thought… more on that in a sec). OK, fine… I’m still a network person, let’s dig in a bit… The csr1000v was not getting any entries in the ARP table, so I gave it a static ARP entry for the default gateway to see if that would help out at all… no dice. OK, well what about the upstream switch? If memory serves, I did have an entry in the mac table for this virtual router, but of course nothing was working still…
Welp… time for
tcpdump of course! I didn’t snag any output, but basically I saw packets going to the router but nothing coming back out if I’m remembering this correctly. In any case, things were very much broken with no obvious reason as to why it was broken.
At this point I was feeling like I was going crazy and I had to just be missing something, or have an older version (or newer?) version of Qemu that was uncooperative or something… I built this out on a Linux host (of course setting hvf back to kvm) and things worked as expected pretty much right away. Well… that is fantastic, but not the goal, so this is obviously not ideal still!
I kept at the unhelpful Google searches until I came across a post that I must have seen 5 or 10 times but just thought it was maybe BS… the “fix” it seemed was to use not the
brew installed Qemu package, and also not Qemu compiled from source (which I had also tried at one point…), but instead the
macports installed Qemu. Are. You. Kidding. Me? This is crazy, right? I brushed this post off the first however many times I read it because it can’t be that crazy! I could see having the
brew version of Qemu have an issue for some reason or another, but ALSO compiling the latest Qemu from source gives the SAME ISSUE???? No way, I thought! At this point, however I was desperate and the post linked to a mailer that had at least two people saying that this indeed was the “fix”.
Sure enough, after installing
macports and Qemu via
macports and updating the command to “point to” that
macports Qemu install…. boom things worked exactly as you’d expect! In my case the only change to the launch command from above was to instead use the new Qemu binary:
Two last points before I forget – in order for the bridging to work you will need to have already setup the
bridge0 interface – I did this just in the
System Preferences pane on my Mac since this was a one-time deal for me. I set this up so my Ethernet interface was part of the bridge, and then can leave this alone for basically forever at this point. After that is sorted out and the Qemu VM is starting up you will need to add the
tap0 interface as part of this bridge:
ifconfig bridge0 addm tap0
I believe that the
downscript options can be fairly easily used to point to a script that can do the adding of the tap to the bridge (and even building the bridge and such if you needed), however for what I needed I didn’t feel the need to fiddle about with that, and have just been adding the taps to the bridge as needed.
Hopefully this helps somebody else out because… wow… that took me far longer than I would have liked to admit to figure out! I was chatting with my friend Kirk Byers about this and he had a great comment that sums up working with Qemu generally:
The arcane flags that I don't understand sounds like Standard qemu operating procedure :)
So true, Kirk, so true!