PowerCLI for VM Encryption

Hi everyone,

I’m happy (ok, beyond happy!) to announce that our VM Encryption engineering team has released a PowerCLI module for VM Encryption! In case you weren’t aware, there’s a Github repository of VMware PowerShell modules. Check them out!

Included in there is the new PowerCLI Module for VM Encryption. It’s chock full of lots of great cmdlets and new VI Properties that make your day to day management of vSphere 6.5 VM Encryption easier to automate. The goal here is to help you operationalize security as easily as possible. If you can’t make security easy to incorporate into your day to day operations then people will find a way to not do it.

Encrypting a VM shouldn’t mean having to manage an encryption solution IN the VM. It should be as simple as “Get-VM” and piping that to “Enable-VMEncryption”, right? Well, with VM Encryption it IS! Let’s take a look.

New PowerCLI cmdlets

Here’s a list of the new cmdlets and what they do. Note: You will need your 6.5 vCenter connected to a KMS (Key Management Server) for many of these to work.

Enable-VMHostCryptoSafe

When a host runs an encrypted VM it goes into “CryptoSafe Mode”. This triggers things like ensuring that core dumps are encrypted. Learn more about Encrypted Core Dumps in the docs. When you encrypt a VM on a host that’s not in CryptoSafe Mode, the mode is enabled automatically. This cmdlet allows you to set a host into CryptoSafe Mode without running an encrypted VM.

Set-VMHostCryptoKey

When a host is in CryptoSafe Mode a key from the KMS is used to encrypt things like core dumps. This cmdlet allows you to change that key if the host is already in CryptoSafe Mode by getting a new key from the KMS and replacing the old one.

Enable-VMEncryption

Just like it says, this is the cmdlet that encrypts a virtual machine using the default encryption policy. You can pipe a get-VM <vmname> directly into Enable-VMEncryption.

Get-VM -Name win2012 | Enable-VMEncryption

One of the coolest things about this cmdlet is that it goes well beyond what’s available in the GUI. You can specify the encryption policy AND the KMS that you want to use. Yes, you can have multiple KMS servers and encrypt VM’s and disks using keys from each of them. For example, let’s use the example of two companies running on common infrastructure. The common one is “What if Coke and Pepsi are running on the same infrastructure?” In this example, I could encrypt all “Coke” VM’s with keys from the Coke KMS and all “Pepsi” VM’s with keys from the Pepsi KMS.

Example:

$EncryptionPolicy = Get-SpbmStoragePolicy -name "VM Encryption Policy
 Get-VM CokeVM  |Enable-VMEncryption -policy $EncryptionPolicy -KMSClusterID "CokeKMS"
 Get-VM PepsiVM |Enable-VMEncryption -policy $EncryptionPolicy -KMSClusterID "PepsiKMS"

You can incorporate this type of workflow into your provisioning process. Instead of Coke .vs. Pepsi, maybe you have two inter-related companies (think merged companies) or groups running on a common infrastructure and each of them have their own KMS for legal/regulatory/history reasons and you use a provisioning tool to build each companies VM’s. You can now easily add the ability to encrypt the VM’s and use the respective KMS servers within your existing provisioning workflow with little impact to day 2 operations.

Encryption Tasks

When you kick off the encryption of a VM each of these will kick off a task with a Task value. You can use that value to monitor the Reconfigure VM task that’s generated. Luc Dekens has a great little chunk of code that I used in the past. I wrote a PowerCLI script to load up a bunch of New-VM tasks into a hash table and start another one when a task was completed. In my script the intent was to create 500 VM’s as fast as possible for a demo.

Using something like this instead to encrypt VM’s would be relatively easy. The steps would look like this.

• Provide a list of VM’s
• Shut each one down
• Start the encryption
• Record the task ID and put it into the hash table
• Wait until the task completes
• Start the VM back up
• Loop back and start another encryption.

Add some logic to balance it across a cluster and you could theoretically encrypt a bunch of VM’s in a reasonable amount of time. (Of course, if you have a bunch of 6TB VM’s, they are going to take some time to encrypt depending on your environment!)

Enable-VMDiskEncryption

This cmdlet encrypts the VM disks with the default encryption policy. For example, some might like to only encrypt a specific disk on a VM. Or maybe, to re-use the Coke/Pepsi example, they want one disk encrypted with a key from the Coke KMS and one from the Pepsi KMS. I’ve actually had this use case come up from a customer!

Disable-VMEncryption

This cmdlet does exactly what it says. It will disable encryption on a VM and all its disks.

Disable-VMDiskEncryption

With this cmdlet , you can specify which disks you want to decrypt in a VM. For example, if you have a VM with an OS disk and a data disk and you want to disable encryption on the OS disk, you can specify just that disk.

Set-VMEncryptionKey

Ok, this is one of my favorites because the use case is very important to some customers. When you encrypt a VM and its disks, the key used to encrypt is generated by the ESXi host. It is an XTS-AES-256 key. Let’s call that the Data Encryption Key or DEK. That key is, in turn, encrypted with a key from the KMS. Let’s call that the Key Encryption Key or KEK.

Why do it this way? Why not just use the KMS key to encrypt all the data? How about a scenario where I want to change KMS vendors. Using this model, I only have to re-encrypt the KEK and not the whole VM and its disks. That’s called a “Shallow Rekey”.

If I’m the uber-paranoid type, then I might want to re-encrypt the whole VM and all of its disks. In that scenario, a new DEK is generated by ESXi and ALL the data is re-encrypted. This is called a “Deep Rekey”. This cmdlet supports both use cases and more!

So, let’s set things up. I have two KMS servers. “KMSold” and KMSnew”. To do a Shallow Rekey on a VM called “VM01” from the old to the new I would do the following:

Get-VM VM01 |Set-VMEncryptionKey -KMSClusterId "KMSnew"

You can do a Shallow Rekey on a running VM. This means you could change the KMS server on the fly with no downtime. Back to the case of the uber-paranoid, they may run multiple key managers. If one of the key managers becomes suspected of being compromised they could, with a few lines of PowerCLI, shallow rekey all their VM’s.

If they are beyond uber-paranoid (Uber²?) and wanted to do a Deep Rekey, then you would add “-Deep”. That would obviously take a LOT longer.

The VM needs to be powered off for this to happen. Because you are reading and writing encrypted data you will incur a significant operational overhead for this.

In a real-world scenario I don’t see Deep Rekey happening very often. If the DEK is secured with the KEK and the KEK hasn’t been compromised then the DEK should be ok. Your security guy’s opinion may vary.

If I just wanted to rekey the KEK used for the VM home files and leave the disks alone then I could do that as well by added “-SkipHardDisks”.

You will notice that you can’t rekey, shallow or deep, a VM that has snapshots. You will need to consolidate them before doing the rekey. If you try then the cmdlet will warn you. You are free to snapshot after the encryption has completed.

Set-VMDiskEncryptionKey

Like Set-VMEncryptionKey, if you want to rekey the encryption key for the disks on a per-KMS basis you can. Set-VMEncryptionKey can set both the VM Home Files and the disks. This cmdlet is just for the disks.

Why would I use this? Here’s a real world example I alluded to earlier. I talked with a customer that had SQL servers that his customers used “As A Service”. Each customer had their own VMDK that the SQL database lived on. He wondered if he could encrypt them with keys from the customers KMS. This cmdlet would allow that.

The VM would be encrypted with his KMS and then each VMDK would be encrypted with the individual customer KMS keys. Sure, it’s a bit of an edge-case scenario but what it demonstrates is the amount of flexibility vSphere VM Encryption and the underlying API brings to the table.

Like Set-VMEncryptionKey, the same use cases of Shallow and Deep Rekey are supported. You can Shallow Rekey a running VM but need to power off to do a Deep Rekey.

Get-VMEncryptionInfo

This cmdlet returns 5 items.

• Which Storage Profile is used
• The connected state
• The name of the VM
• The encrypted disks
• The Key ID

Get-VM "Windows10 -1" |Get-VMEncryptioninfo

 Name                           Value
 ----                           -----
 profile                        VM Encryption Policy
 connectState                   connected
 name                           Windows10 -1
 disks                          {}
 keyId                          VMware.Vim.CryptoKeyId

Now, the Key ID is the interesting one. That’s the key ID used by the KMS server. So when vCenter asks for a key, it requests it using the Key ID stored in the VM Settings. (i.e: the VMX file). You can see this in the Advanced Setting of “encryption.bundle” for the VM.

Warning: Don’t change this value! Ever! You could render your VM unbootable if you do. I’m showing it here for information purposes only.

Get-VM "Windows10 -1" |Get-AdvancedSetting -Name "encryption.bundle" |Select Value

Value
-----
vmware:key/list/(pair/(fqid/<VMWARE-NULL>/KMS01/10,HMAC%2dSHA%2d256,UVKKom81...

Ok, that’ s great, but what if I want to get a list of all VM’s and what Key Manager they are using? If Get-VMEncryptionInfo only returns the Key ID, how do you map that to the KMS server it’s coming from? Well, when you get the KeyId using Get-VMEncryptionInfo, you also get the “ProviderID” and it’s there that you can get the name of the KMS. I’ve created a new VI Property that I’ve fed back to the developers. As I was writing this blog the engineer who wrote the module accepted it and incorporated into the latest version of the module! Sweet!

Example:

New-VIProperty -Name KMSserver -ObjectType VirtualMachine -Value {
    Param ($VM)
    if ($VM.Encrypted) {
      $VM.EncryptionKeyId.ProviderId.Id
    }
    } -BasedOnExtensionProperty 'Config.KeyId' -Force | Out-Null

When you run it, you’ll get a list of the VM’s and what KMS server they are attached to.

Get-VM |Select Name,KMSserver

Name                                    KMSserver
----                                    ---------
Windows10 -1                            KMS01
Windows10
PyKMIPServer

Get-EntityByCryptoKey

So, I showed you how to get the KMS ID for a single VM but there’s another way to look at that problem. Get-EntityByCryptoKey can give you a bunch of information as well. If I want to see all VM’s that are encrypted with a key from a specific KMS, I do the following:

Get-EntityByCryptoKey -KMSClusterId kms01 -SearchVMs

Name                           Value
----                           -----
VMList                         Windows10 -1

If I want to get a lists of hosts that have keys used by a specific KMS I would do the following:

Get-EntityByCryptoKey -KMSClusterId kms01 -SearchVMHosts

Name                           Value
----                           -----
VMHostList                     {192.168.1.105, 192.168.1.109, 192.168.1.107,...

And if I want to get a list of VM’s that have disks that are encrypted then that’s pretty simple:

$disks = Get-EntityByCryptoKey -KMSClusterId kms01 -SearchDisks
 $disks.DiskList.Parent.name
 Windows10 -1

You can also get this information using the VI Properties.

 Get-VM | Get-HardDisk |Select Parent,Name,Encrypted

Parent                     Name                                       Encrypted
------                     ----                                       ---------
Windows10 -1               Hard disk 1                                     True
Windows10                  Hard disk 1                                    False
PyKMIPServer               Hard disk 1                                    False

Adding and removing KMS Servers are pretty straightforward.

 New-KMServer -KMServer 1.1.1.1 -KMSClusterId CompanyKMS -UserName "YourKMSUserName" -Password '***' -Name "KMServer01"

This adds the Key Management Server 1.1.1.1 into vCenter with the cluster name ‘CompanyKMS’ and KMS name ‘KMServer01’. If you have multiple KMS servers that replicate then you would add each one the same way, using the same KMS Cluster ID. vCenter uses the KMS ClusterID as a list of KMS servers to try. If the first one isn’t available, it will try the next.

To remove the server we just created from the KMS Cluster, you would use the following command.

Remove-KMServer -KMSClusterId “CompanyKMS" -KMSName "KMServer01"

Get-KMServerInfo

Name : KMS104
 Address : 192.168.1.104
 Port : 5696
 ProxyAddress :
 ProxyPort :
 Reconnect :
 Protocol :
 Nbio :
 Timeout :
 UserName :

This cmdlet will tell you which of your KMS servers are set to be the “Default” KMS. The Default KMS is the one used by the vCenter Web Client and if you use Enable-VMEncryption without specifying a KMS.

If you want to change which KMS Cluster is the default, this is the cmdlet for you. This could be helpful in a previously mentioned scenario where you are migrating from the old KMS to the new KMS. You would create the new KMS Cluster, Add KMS servers to it and then set the new Default KMS to be “KMSnew” to ensure any new VM’s are encrypted with the new KMS and then do a Shallow Rekey of all your VM’s so they pick up a new KEK from the new KMS.

 Set-DefaultKMSCluster -KMSClusterId ‘KMSnew'

Additional VIProperties

When you import the module you will gain additional VIProperties you can use in your code. You can see these new properties by searching the module for the New-VIProperty or by issuing a command such as Get-VM | Get-Member. You’ll see the new properties at the top of the output.

For virtual machines you’ll see the new properties of Encrypted, EncryptionKeyID & Locked.

Get-VM|Select Name,Encrypted,EncryptionKeyId,Locked

Name Encrypted EncryptionKeyId Locked
 ---- --------- --------------- ------
 Windows10 -1 True VMware.Vim.Crypt... False
 Windows10 False False
 PyKMIPServer False False

Now, you’re probably wondering what “Locked” means. An encrypted VM goes into locked (invalid) state when vCenter Server is restored to a previously backed up state. To clear this up you would un-register the VM from vCenter and then re-add it to inventory (registering the VM).

For hosts you will see the new properties of CryptoSafe, CryptoSafeSupported and AESNIStatus

Get-VMhost | Select Name,CryptoSafeSupported,CryptoSafe Name

Name CryptoSafeSupported CryptoSafe
 ---- ------------------- ----------
 192.168.1.105 True True
 192.168.1.109 True True
 192.168.1.107 True True
 192.168.1.106 True True
 192.168.1.108 True True

Most modern encryption solutions are using the AESNI instructions in the CPU. In every new model of Intel or AMD CPU’s these instructions are getting faster and faster. But some systems allow you to disable them in the BIOS. If that happens, your encryption operations will be done in software and that will have a serious impact on performance. So, ensure that if there is an option in your BIOS to enable AESNI, that it’s enable. If you want to check to see if AESNI is enabled on your host, that’s pretty simple as well as we expose that information via a VI Property.

Get-VMHost |Select Name,AESNIStatus

Name                                                                AESNIStatus
----                                                                -----------
192.168.1.105                                                              True
192.168.1.109                                                              True
192.168.1.107                                                              True
192.168.1.106                                                              True
192.168.1.108                                                              True

Wrap Up

In all of these great cmdlets there is logic to ensure you are doing things correct. For example, in Enable-VMEncryption, if you attempt to run that against a running VM the cmdlet will fail and tell you that you need to power off the VM in order to enable encryption or if you have snapshots the cmdlet will fail because you have to consolidate them for VM Encryption to proceed.

As soon as I got involved in this project one of the first things I mentioned was that we needed automation in order to better operationalize security. Selfishly, I pushed for PowerShell because most administrators use it. All of the things you see in these cmdlets leverage the vSphere API. If you are a Python or Perl or C++ person you can leverage the vSphere API to do all of these tasks.

If you have feedback or even want to contribute, all of this code is open sourced and hosted on GitHub. Go to https://github.com/vmware/PowerCLI-Example-Scripts to join up, download the module and provide your feedback and report any issues you may find in your use of the module.

Finally, a HUGE thank you for some really outstanding work by Baoyin Qiao, Jesse Pool, Swapneel Kekre and the rest of the VM Encryption team!

Thanks for reading,
mike