Print View

How to create a self-signed certificate for VirtualCenter using OpenSSL

Issue

The default VirtualCenter certificate is only valid for 2 years from the date of install. If not using a dedicated Certificate Authority, a self signed certificate can suffice to secure the connection.

VMware have outlined the process of creating and using a root CA, rather than a self-signed certificate in the following document:

Replacing VirtualCenter Server Certificates
http://www.vmware.com/pdf/vi_vcserver_certificates.pdf

The instructions provided may not be complete and the following errors may be seen restarting the VirtualCenter Server service:

[YYYY-MM-DD HH:MM:SS.SSS 'App' 5944 error] [VpxKey::Decrypt] crypto failure: error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02
[YYYY-MM-DD HH:MM:SS.SSS 'App' 5944 error] [VpxdCert] Failed to decrypt password: applying key to encrypted data failed (likely the wrong key)
[YYYY-MM-DD HH:MM:SS.SSS 'App' 5944 error] ODBC error: () -
[YYYY-MM-DD HH:MM:SS.SSS 'App' 5944 error] Failed to intialize VMware VirtualCenter.
Shutting down...
[YYYY-MM-DD HH:MM:SS.SSS 'App' 5944 info] Forcing shutdown of VMware VirtualCenter now

Additional errors include:

vim.fault.InvalidLogon
"Login failed due to a bad username or password"

Resolution

The following process will create a self signed certificate valid for 10 years.

1) Create Self-Signed Certificate
1a) Generate the private key

- Run the following command:
openssl genrsa 1024 > rui.key

...................++++++
.............................................++++++
e is 65537 (0x10001)


1b) Generate the certificate
Ensure the Common Name below matches the DNS name of the VirtualCenter Server. Ideally use a DNS CNAME record, so the service can be moved to a different host for the duration of the certificate validity.

- Run the following command:
openssl req -new -x509 -nodes -sha1 -days 3650 -key rui.key > rui.crt

You are about to be asked to enter information that will be incorporated into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:WA
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]: The Ether
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:virtualcenter.domain.com
Email Address []:hostmaster@domain.com


1c) Genterate the PFX Certificate

- Run the following command:
openssl pkcs12 -export -in rui.crt -inkey rui.key -name rui -passout pass:testpassword -out rui.pfx


Note: The password needs to be set to testpassword to ensure that Customization Specifications continue to work.

2) Replace the VirtualCenter Certificates

- Stop the VirtualCenter Server service with the following command:
NET STOP VPXD

- Rename the existing rui.key, rui.crt and rui.pfx files in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL\
- Copy the rui.key, rui.crt and rui.pfx files created above to C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL\

3) Update the VirtualCenter Password

- Export "HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB" to allow the original certificates to be reused.

Note: The password below is for the user account defined in "HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter\DB\2"

- From a command prompt enter the following commands, password and verify password when prompted:
CD "C:\Program Files\VMware\Infrastructure\VirtualCenter Server"
vpxd -p

[YYYY-MM-DD HH:MM:SS.SSS 'App' 2548 info] Current working directory: C:\Program Files\VMware\Infrastructure\VirtualCenter Server
[YYYY-MM-DD HH:MM:SS.SSS 'Libs' 2548 info] HOSTINFO: Seeing Intel CPU, numCoresPerCPU 1 numThreadsPerCore 1.
[YYYY-MM-DD HH:MM:SS.SSS 'Libs' 2548 info] HOSTINFO: This machine has 1 physical CPUS, 1 total cores, and 1 logical CPUs.
[YYYY-MM-DD HH:MM:SS.SSS 'App' 2548 info] Log path: C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs
[YYYY-MM-DD HH:MM:SS.SSS 'App' 2548 info] Initializing SSL
[YYYY-MM-DD HH:MM:SS.SSS 'Libs' 2548 info] Using system libcrypto, version 90709F
[YYYY-MM-DD HH:MM:SS.SSS 'App' 2548 info] Vmacore::InitSSL: doVersionCheck = true, handshakeTimeoutUs = 120000000
Enter new DB password:
again:
[YYYY-MM-DD HH:MM:SS.SSS 'App' 2548 info] Reset DB password succeeded.


4) Start the VirtualCenter Server service

- Start the VirtualCenter Server service with the following command:
NET START VPXD


5) Reconnect ESX hosts
The ESX hosts need to be disconnected/reconnected to VirtualCenter. If this is not done the vim.fault.InvalidLogon dialogue may appear.

From the VMware Infrastructure Client perform the following for each ESX host:

- If ESX host is not disconnected right click on the ESX host and select Disconnect from the context menu
- Right click on the ESX host and select Connect from the context menu
- Enter the Username and Password and click Next
- If the "Duplicate Management: The host is already being managed by IP Address" warning appears click Yes
- Click Next to accept the summary
- Select the folder for the virtual machines and click Next
- Click Finish


The certificate will need to be imported as a Trusted Root Certificate Authority to be fully trusted and pass the following three test:

- The security certificate is from a trusted certifying authority
- The security certificate date is valid
- The name on the security certificate matches the name of the site

This can be done from the web browser, the CERTMGR.MSC console, or the MMC using Add/Remove Snap-ins | Certificates.

References

VirtualCenter Server Fails to Start After You Replace Default SSL Certificates with Custom SSL Certificates
http://kb.vmware.com/kb/1003070

Developer’s Setup Guide VMware Infrastructure SDK 2.5
http://www.vmware.com/support/developer/vc-sdk/visdk25pubs/visdk25setupguide.pdf

To modify the Web proxy service on VirtualCenter Server 2.5 to support HTTP:

- Log in to the VirtualCenter Server system as the Windows Administrator of the machine.
- Change to the directory containing the proxy.xml file: c:\Documents and Settings\AllUsers\Application Data\VMware VirtualCenter
- Use a text editor to open the proxy.xml file.
- Find the section of the file associated with the /sdk.
- Change the accessMode to httpAndHttps.
- Restart the VirtualCenter Service from a command line or from the Windows Services control panel.

Replacing or Regenerating an SSL Certificate for the Management Interface
http://kb.vmware.com/kb/1843

Products

VMware VirtualCenter 2.5
VMware vCenter Server 4.0 Update 1

Created: 16th December 2008
Updated: 14th April 2010

Print View

© 2005-2024 Jamie Morrison