503 Errors and vCloud Director

In our lab set-up, I have been experimenting with vCloud Director configurations and high availability for vCloud Director cells – and more specifically what happens when the vCenters behind vCloud resources dissapear or have availability issues – then reconnect.

If a vCenter hosting the resources crashes and is subsequently restored, there may be odd or unexplained error codes from vCloud Director – specifically:

Unexpected Status Code: 503

…was seen a lot – when creating or deploying vApps within an Organisation that is backed by the resources managed by the vCenter that crashed. Interestingly, some vCloud functions continued to work whilst the continuity of vCenter was interupted, but this was due to the other resilient vCloud infrastructure (cells, database and AMPQ bus still responding).

The fix for these errors was to reconnect the vCenter server within the vCloud interface. It seems that simply allowing the vCenter to recover itself in sometimes not sufficient to allow the connectivity to recover automatically. To do this:

  1. Connect to the System vDC (the root or system Organisation for the cloud)
  2. Select the Administration tab
  3. From the left menu under vSphere resources, select vCenters
  4. Select the failed (and recovered) vCenter
  5. Right-click the vCenter and select ‘Reconnect’

A word or warning. While the reconnect operation only takes a very short time (a matter of seconds), this will affect other operations to the cloud consumers using resources from the effected vCenter, i.e. other errors may result from deploying or interacting with resources whilst the reconnect completes.

Jeremy loves all things technology! Has been in IT for years, loves Macs (but doesn't preach to others about their virtues), loves virtualization (and does shout about it's virtues), and sometimes skis, bikes and directs amateur plays!

Comments

  1. Jonathan Ross says

    First off wanted to thank you for this post. Came in handy the first time I came across this error.
    Our VC keeps crashing, however the vpxd process continues to run, so its hard to monitor for that.
    I’ve gone digging through the vCD logs, but am unable to find anything referencing this error. Do you by chance know if vCD logs this anywhere? Would make monitoring for this problem a lot easier if I can have BMC Patrol scrape the vCD logs.
    Thanks again,
    -Jon

Leave a Reply

Your email address will not be published. Required fields are marked *