CSS Corp Open Source Services

[Eucalyptus Beginner’s Guide – UEC edition] Chapter 9 – Troubleshooting

leave a comment »

Chapter 8
Security
Chapter 10
Euca Commands

Eucalyptus Log files

The following log files help in debugging issues encountered while working with Eucalyptus:

Eucalyptus Component Log file(s)
CC cc.log, httpd-cc_error_log, cc-registration.log
NC nc.log, httpd-nc_error_log, euca_test_nc.log
CLC cloud-debug.log, cloud-error.log, cloud-output.log, axis2c.log
SC sc-stats.log, sc-registration.log
WS3 walrus-stats.log, walrus-registration.log

The logging level is controlled by the LOGLEVEL macro in eucalyptus.conf of the respective component. The log levels are DEBUG, INFO, WARN, ERROR, and FATAL (in descending order of verbosity). The default level is DEBUG (everything).

‘tail -f ‘ on the log files is a good way to understand what is happening with the components of Eucalyptus.

Linux commands

Linux comamnds like ps, top etc. run on CC and NC are helpful to understand the state of the processes related to Eucalyptus.

Libvirt tools

NC launches the instances by making libvirt calls to launch a VM. You can use libvirt tools like virsh for troubleshooting any hypervisor related issues.

Launching an instance with VNC attached(HACK):

If the output of euca-describe-instances shows the status of an instance as ‘running’, but if you are not able to SSH into it or RDP into it, it is is tough to find out where the failure is. In such cases, a VNC interface to the instance is generally of great help to troubleshoot further. But Eucalyptus does not provide a way of connecting to the instance through VNC. The following hack can help in setting up a VNC interface to the instance.
Windows Instances

On the NC, edit /usr/share/eucalyptus/gen_kvm_libvirt_xml (if KVM is the hypervisor) to change


<devices>
 <disk type='file'>
 <source file='BASEPATH/disk'/>
 <target dev='sda'/>
 </disk>
 <interface type='bridge'>
 <source bridge='BRIDGEDEV'/>
 <mac address='PRIVMACADDR'/>
 <model type='e1000'/>
 </interface>
 <serial type="file">
 <source path='BASEPATH/console.log'/>
 <target port='1'/>
 </serial>
 </devices>

to


<devices>
 <disk type='file'>
 <source file='BASEPATH/disk'/>
 <target dev='sda'/>
 </disk>
 <interface type='bridge'>
 <source bridge='BRIDGEDEV'/>
 <mac address='PRIVMACADDR'/>
 <model type='e1000'/>
 </interface>
 <serial type="file">
 <source path='BASEPATH/console.log'/>
 <target port='1'/>
 </serial>
 <graphics type='vnc' listen='A.B.C.D' port='5904'/>
 </devices>

where A.B.C.D is the IP address of the interface on NC that is reachable from the client machine where you want to run the vnc client. We assume that your NC has 2 interfaces and you are able to use the second interface to reach it without going through CC. Again, this is only for troubleshooting. In a production setup, you may have the NC connected to the outside world only through CC.

With these changes, KVM adds the “-vnc” option to the command with the specified NC’s ip address on display number 4.

To connect to the instance through VNC client from a different machine, use the NC’s ip address and display number 4.
Ex: A.B.C.D :4

In the case of Xen, you would modify the relevant sections of /usr/share/eucalyptus/gen_libvirt_xml. We have not tested with Xen, though.

Linux Instances

You can use the same approach for Linux instances as well, with the caveat that you will not see any output with euca-get-console-output as console output does not get written to /var/lib/eucalyptus/instances/<user>/i-xxxxxxxx/console.log. The output will, of course, be displayed though VNC.

NOTE: This works only if we are debugging one instance at a time. If you attempt to start another instance while one instance is already running, KVM will try to attach the same VNC display 4 to the second instance and will fail with the following error in /var/log/libvirtd/qemu/i-XXXXXXXX.log

inet_listen_opts: bind(ipv4,A.B.C.D,5904): Address already in use
inet_listen_opts:FAILED

Chapter 8
Security
Chapter 10
Euca Commands
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: