UEC/Eucalyptus – Debugging instances

I have had an issue with my MS-Windows 2003 server instance raised on one of the Node Controllers (NC) on my UEC setup on Lucid Lynx( Ubuntu 10.04 ). The instance came up moving from pending state to running state as expected and my hybrid fox showed public and private IPs. However, I was not able to ping or RDP into the instance. The Windows image was custom bundled and tried and tested on my Karmic UEC setup, so I was quite puzzled as to why I was unable to connect to the instance. I wished I had the luxury of connecting to the instance for troubleshooting using VNC, like I normally do when I start a VM on KVM or Xen. A little bit of probing of the Eucalyptus files on NC proved useful. Here is the hack that helped me attach a VNC display to the instance and ultimately I could identify and fix the actual problem.

It is not safe to leave this option enabled on a production setup.

Bringing up eucalyptus instance with VNC attached

On Node Controller, edit /usr/share/eucalyptus/gen_kvm_libvirt_xml (if KVM is the hypervisor) to change

<devices>
        <disk type='file'>
            <source file='BASEPATH/disk'/>
            <target dev='sda'/>
        </disk>
        <interface type='bridge'>
            <source bridge='BRIDGEDEV'/>
            <mac address='PRIVMACADDR'/>
            <model type='e1000'/>
        </interface>
        <serial type="file">
            <source path='BASEPATH/console.log'/>
            <target port='1'/>
        </serial>
    </devices>

to

<devices>
        <disk type='file'>
            <source file='BASEPATH/disk'/>
            <target dev='sda'/>
        </disk>
        <interface type='bridge'>
            <source bridge='BRIDGEDEV'/>
            <mac address='PRIVMACADDR'/>
            <model type='e1000'/>
        </interface>
        <serial type="file">
            <source path='BASEPATH/console.log'/>
            <target port='1'/>
        </serial>
        <graphics type='vnc' listen='A.B.C.D' port='5904'/>
    </devices>

where A.B.C.D is the ip address of the interface on NC that is reachable from the client machine where you want to run the vnc client. I assume that your NC has 2 interfaces and you are able to use the second interface to reach it without going through CC. Again, this is only for troubleshooting. In a production setup, you may have the NC connected to the outside world only through CC.

With these changes, KVM adds the “-vnc” option to the command with the specified NC’s ip address on display number 4.

To connect to the instance through VNC client from a different machine, use the NC’s ip address and display number 4 (Ex: A.B.C.D :4)

In the case of Xen, you would modify the relevant sections of /usr/share/eucalyptus/gen_libvirt_xml. I have not tested with Xen, though.

Linux Instances

You can use the same approach for Linux instances as well, with the caveat that you will not see any output with euca-get-console-output as console output does not get written to /var/lib/eucalyptus/instances/<user>/i-xxxxxxxx/console.log. The output will, of course, be displayed through VNC.

NOTE:

This works only if we are debugging one instance at a time. If you attempt to start another instance while one instance is already running, KVM will try to attach the same VNC display 4 to the second instance and will fail with the following error in /var/log/libvirtd/qemu/i-XXXXXXXX.log

inet_listen_opts: bind(ipv4,A.B.C.D,5904): Address already in use
inet_listen_opts:FAILED

More work to get more instances running with VNC attached. That is for some other day!