If you were unlucky enough to download VMWare's ESX 3.5 update 2 in the past few weeks, you got a nasty surprise today when you tried to power on one of your virtuals:
"A general system error occurred: Internal Error."
And if you peeked into the logs, you might have seen:
http://msg.License.product.expired This product has expired.
(even though nothing was supposed to actually expire.)
Sorry - your VMs will no longer power on. Oops! Hope that isn't too important for you...
CEO Paul Maritz indicates that a leftover piece of pre-release code was to blame.
"Last night, we became aware of a code issue with the recently released update to ESX 3.5 and ESXi 3.5 (Update 2).When the time clock in a server running ESX 3.5 or ESXi 3.5 Update 2 hits 12:00AM on August 12th, 2008, the released code causes the product license to expire. The problem has also occurred with a recent patch to ESX 3.5 or ESXi 3.5 Update 2.
When an ESX or ESXi 3.5 server thinks its license has expired, the following can happen:
---- Virtual machines that are powered off cannot be turned on;
---- Virtual machines that have been suspended fail to leave suspend mode; and,
---- Virtual machines cannot be migrated using VMotion.
The issue was caused by a piece of code that was mistakenly left enabled for the final release of Update 2. This piece of code was left over from the pre-release versions of Update 2 and was designed to ensure that customers are running on the supported generally available version of Update 2.
In remedying the situation, we’ve already released an express patch for those customers that have installed/upgraded to ESX or ESXi 3.5 Update 2. Within the next 24 hours, we also expect to issue a full replacement for Update 2, which should be used by customers who want to perform fresh installs of ESX or ESXi.
I am sure you’re wondering how this could happen. We failed in two areas:
---- Not disabling the code in the final release of Update 2; and
---- Not catching it in our quality assurance process.
We are doing everything in our power to make sure this doesn’t happen again.
VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.
I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.
Thank You,
Paul Maritz
President and CEO "
From Gary Chen, a senior analyst with Yankee Group:
From the VMWare web site:"Software will always have bugs, but a widespread issue like this that affects all VMs is really damaging, especially at this point in time where virtualization is starting to take off. VMware is going to have to fix this fast, provide an explanation, and outline what they will do to strengthen their QA in the future."
"An issue with ESX/ESXi 3.5 Update 2 causes the product license to expire on August 12, 2008. VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13. These will be available from the page: http://www.vmware.com/download/vi. Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2."What if you've already upgraded to 3.5 Update 2?
Well - sorry about that.
Perhaps you'd like to set your clock back for a while?
Might I suggest January 1, 2000 as a handy target date?
See also:
http://it.slashdot.org/article.pl?no_d2=1&sid=08/08/12/1120235
http://www.theinquirer.net/gb/inquirer/news/2008/08/12/vmware-update-bombed
http://www.theregister.co.uk/2008/08/12/vmware_12_august_esx_cockup/
http://communities.vmware.com/thread/162377?tstart=0
http://kb2.vmware.com/kb/1006716.html
http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9112439&intsrc=news_ts_head
http://www.cio.com/article/443363/VMware_Bug_Shuts_Down_ESX_._Users
http://www.crn.com/software/210002793
http://www.networkworld.com/news/2008/081208-vmware-bug.html
http://www.theregister.co.uk/2008/08/13/vmware_apologises_12_august_code/
