Automatic ACME SSL Certificate Rotation

Technology needs to be secure, but we also want to make it easy to use. This is the same for us engineers managing SSL Certificates and their rotation. You can get long life certificates, but why when you can get free ones generated via the Automated Certificate Management Environment (ACME protocol). This is normally due them expiring within 3 month, which you do not want to keep renewing and deploying every 3 months, especially when you have many services to maintain. Therefore, I have a pattern and design to renew certificates, which can also be adapted for any service or cloud provider.

This design uses specific technologies, but due to the makeup of it, each component can be swapped for whatever technology you are using. For example, where I use Azure Key Vault to store the certificate, this can easily be swapped for AWS Certificate Manager. This is also why it is a very good design as it can support multiple type of services, languages and providers.

For this article, I am using the following technologies

Azure DevOps -Deployment Software
https://azure.microsoft.com/en-us/services/devops/

Leys Encrypt- Certificate Provider
https://letsencrypt.org/docs/client-options/

Azure Key Vault -Certificate Store
https://docs.microsoft.com/en-us/azure/key-vault/general/basic-concepts

Azure Virtual Machine (Linux) -Application Host
https://azure.microsoft.com/en-us/services/virtual-machines/linux/

Azure DNS - DNS Provider
https://azure.microsoft.com/en-gb/services/dns/#overview

Posh-ACME- Automate Certificate Generation
https://poshac.me/docs/v4/Tutorial/

How it works

As you can see from the design it is the Azure DevOps that does the request of the certificate. This is so there is a single source that is doing the request per domain, instead of each of the resources doing it. This can save on the number of requests and number of certificates required per domain. You can request one certificate and all the resources using that domain can reap the benefits.

Request the Certificate

This section will explain the job of Azure DevOps to get the new certificate from Lets Encrypt and store it within the Azure Key Vault.

Get a Certificate Script

We start by setting the variables used within the script that will configure how it will be used.

env is the Environment, which is used later on to decide what Lets Encrypt Server to use. When using the Production Server, you are limited to how many requests you can make per domain per day, therefore for lower environments it makes sense to use the Staging Server where you might be requesting multiple time during deployments.

The acmeContact is an email contact that gets used for the Posh-ACME account but can be any email as long as it is formatted as an email.

The domain is the Fully Qualified Domain Name of the URL you will be requesting the certificate for.

Then finally you have the Azure Subscription name that holds the Azure DNS resource, which will be used later to get the Access Token for the request. This is if your resources are not hosted in the same Azure Subscription, but if they are then you can always just use the az cli to get the current Subscription.

$env="staging"
$acmeContact="me@email.com"
$domain="www.example.com"
$dnsSubscription="DNS-Subscription-Example"
if ($env -eq "production" -or $env -eq "staging") {
    $leServer="LE_PROD"
}else {
    $leServer="LE_STAGE"
}

We can then install the Posh-ACME PowerShell Module.

# Set Posh-ACME working directory
Write-Host "Install Module"
Install-Module -Name Posh-ACME -Scope CurrentUser -Force

Set the Lets Encrypt Server and install the Azure plugin for the script.

# Configure Posh-ACME server
Write-Host "Configure LE Server $leServer"
Set-PAServer $leServer
Get-PAPlugin Azure

When using the Posh-ACME you will need to setup an account attached to the domain for renewals, which can be auto-generated using the acmeContact email we setup earlier. The script below can also workout if you already have an account setup and if so, it will use the existing account.

# Configure Posh-ACME account
Write-Host "Setup Account"
$account = Get-PAAccount
if (-not $account) {
# New account
Write-Host "Create New Account"
$account = New-PAAccount -Contact $acmeContact -AcceptTOS
} elseif ($account.contact -ne "mailto:$acmeContact") {
# Update account contact
Write-Host "Set Existing Account $($account.id)"
Set-PAAccount -ID $account.id -Contact $acmeContact
}

We then need to get the Azure DNS resources Subscription ID and Access Token to pass into the certificate generation. If your DNS resource is not hosted within your current subscription, then you can use this script to get the Subscription details and then request the Access Token. If it does, then you can remove the part where it sets the subscription name and just show the current subscriptions context (`az account show — query ‘id’ -o tsv`).

# Acquire access token for Azure (as we want to leverage the existing connection)
Write-Host "Get Azure Details"
$azAccount = az account show -s $dnsSubscription -o json | ConvertFrom-Json
Write-Host "Azure DNS Sub $($azAccount.name)"
$token = (az account get-access-token --resource 'https://management.core.windows.net/' | ConvertFrom-Json).accessToken

You can now request the new certificate using the Post-ACME command and the obtained settings.

# Request certificate
$pArgs = @{
AZSubscriptionId = $azAccount.id
AZAccessToken = $token
}
New-PACertificate $domain -Plugin Azure -PluginArgs $pArgs -Verbose
$generatedCert=$(Get-PACertificate)
Write-Host($generatedCert)

Azure DevOps setup

Now we do not want to keep running this script every time, therefore we can add an extra script before to check this. It will check if the certificate exists and if so then it will check its expiry is within 14 days.

- task: AzureCLI@2
  displayName: 'Check if Cert expired in ${{ parameters.keyVaultName }}'
  name: cert
  inputs:
    azureSubscription: '${{ parameters.subscriptionName }}'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      $keyVaultName="${{ parameters.keyVaultName }}"
      $certName="${{ parameters.certName}}"
      $exportedCerts = az keyvault certificate list --vault-name $keyVaultName --query "[? name=='$certName']" -o json | ConvertFrom-Json
      $expired=$false
      if ($null -ne $exportedCerts -and $exportedCerts.length -gt 0){
      Write-Host "Certificate Found"
      $exportedCert = $exportedCerts[0]
      Write-Host "Certificate Expires $($exportedCert.attributes.expires)"
      $expiryDate=(get-date $exportedCert.attributes.expires).AddDays(-14)
      Write-Host "Certificate Forced Expiry is $expiryDate"
      if ($expiryDate -lt (get-date)){
        Write-Host "Certificate has expired"
        $expired=$true
      } else {
        Write-Host "Certificate has NOT expired"
      }
      } else {
        Write-Host "Certificate NOT Found"
       $expired=$true
     }
  Write-Host "##vso[task.setvariable variable=expired;isOutput=true]$expired"

This can then be used to decide if to run the certificate requesting script or not as part of the condition for the task.

- task: AzureCLI@2
  name: acmecert
  displayName: 'Request LE Cert for ${{ parameters.domain }}'
  condition: and(succeeded(), eq(variables['cert.expired'], 'True'))
  inputs:
    azureSubscription: '${{ parameters.subscriptionName }}'
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      $env="${{ parameters.environment }}"
      $acmeContact="me@email.com"
      $domain="${{ parameters.domain }}"
      $dnsSubscription="Reform-CFT-Mgmt"
      if ($env -eq "production" -or $env -eq "staging") {
        $leServer="LE_PROD"
      }else {
        $leServer="LE_STAGE"
      }
      # Set Posh-ACME working directory
      Write-Host "Install Module"
      Install-Module -Name Posh-ACME -Scope CurrentUser -Force
      # Configure Posh-ACME server
      Write-Host "Configure LE Server $leServer"
      Set-PAServer $leServer
      Get-PAPlugin Azure
      # Configure Posh-ACME account
      Write-Host "Setup Account"
      $account = Get-PAAccount
      if (-not $account) {
        # New account
        Write-Host "Create New Account"
        $account = New-PAAccount -Contact $acmeContact -AcceptTOS
      }
      elseif ($account.contact -ne "mailto:$acmeContact") {
        # Update account contact
        Write-Host "Set Existing Account $($account.id)"
        Set-PAAccount -ID $account.id -Contact $acmeContact
      }
      # Acquire access token for Azure (as we want to leverage the existing connection)
      Write-Host "Get Azure Details"
      $azAccount = az account show -s $dnsSubscription -o json | ConvertFrom-Json
      Write-Host "Azure DNS Sub $($azAccount.name)"
      $token = (az account get-access-token --resource 'https://management.core.windows.net/' | ConvertFrom-Json).accessToken
      # Request certificate
      $pArgs = @{
        AZSubscriptionId = $azAccount.id
        AZAccessToken = $token
      }
      New-PACertificate $domain -Plugin Azure -PluginArgs $pArgs -Verbose
      $generatedCert=$(Get-PACertificate)
      Write-Host($generatedCert)
Write-Host "##vso[task.setvariable variable=certPath;isOutput=true]$($generatedCert.CertFile)"Write-Host "##vso[task.setvariable variable=privateKeyPath;isOutput=true]$($generatedCert.KeyFile)"
Write-Host "##vso[task.setvariable variable=pfxPath;isOutput=true]$($generatedCert.PfxFullChain)"
Write-Host "##vso[task.setvariable variable=pfxPass;isOutput=true;issecret=true]$pfxPassword"

Store the Certificate

Finally, once we have the certificate generated we can export and put it within the Azure Key Vault by using the az cli to import the generated certificate.

- task: AzureCLI@2
displayName: 'Import Certificate into ${{ parameters.keyVaultName }}'
condition: and(succeeded(), eq(variables['cert.expired'], 'True'))
inputs:
  azureSubscription: '${{ parameters.subscriptionName }}'
  scriptType: 'pscore'
  scriptLocation: 'inlineScript'
  inlineScript: |
    $keyVaultName="${{ parameters.keyVaultName }}"
    $certName="${{ parameters.certName}}"
    $password="$(acmecert.pfxPass)"
    $pfxPath="$(acmecert.pfxPath)"
    az keyvault certificate import --vault-name $keyVaultName -n     $certName -f $pfxPath --password $password

Install Certificate

For this stage we are assuming the above steps have been followed, so the certificate is generated, valid and imported into the Azure Key Vault.

We will also assume for the Linux Virtual Machines (VMs) you have installed the Azure CLI and have a Azure Managed Identity (MI) attached to the VMs for authentication.

For the storing of certificates on the VMs we are also using Keytools, which is used with Java applications on Linux Machine.

In the script below we are getting all the variables and logging into the Azure CLI.

miClientId="${managedIdentityClientId}"
az login --identity --username $miClientId
keyVaultName="${keyVaultName}"
certName="${certName}"
domain="${domain}"
jksPath="/usr/local/conf/ssl.jks"
jksPass="${certPassword}"

We will then get the list of the certificates and generate the date we deem as expired, which is the certificates expiry date minus 14 days.

expiryDate=$(keytool -list -v -keystore $jksPath -storepass $jksPass | grep until | sed 's/.*until: //')
echo "Certificate Expires $expiryDate"
expiryDate="$(date -d "$expiryDate - 14 days" +%Y%m%d)"
echo "Certificate Forced Expiry is $expiryDate"
today=$(date +%Y%m%d)

If today’s date is less than the expiry date then we will not try get the new certificate, but if the certificate does not exist or its expiry date is less than today then we will renew.

To do this we will download the certificate from the Key Vault, but as it downloads it without a password, we are using the open SSL CLI to import/export the certificate with a password.

This then generates a new PFX, which we import into the Keytools after we delete the existing certificate.

if [[ $expiryDate -lt $today ]]; then
  echo "Certificate has expired"
  downloadedPfxPath="downloadedCert.pfx"
  signedPfxPath="signedCert.pfx"
  rm -rf $downloadedPfxPath || true
  az keyvault secret download --file $downloadedPfxPath --vault-name $keyVaultName --encoding base64 --name $certName
  rm -rf $signedPfxPath || true
  openssl pkcs12 -in $downloadedPfxPath -out tmpmycert.pem -passin pass: -passout pass:$jksPass
  openssl pkcs12 -export -out $signedPfxPath -in tmpmycert.pem -passin pass:$jksPass -passout pass:$jksPass
  keytool -delete -alias 1 -keystore $jksPath -storepass $jksPass
  keytool -importkeystore -srckeystore $signedPfxPath -srcstoretype pkcs12 -destkeystore $jksPath -deststoretype JKS -deststorepass $jksPass -srcstorepass $jksPass
else
  echo "Certificate has NOT expired"
fi

You can then put this on a daily cron job to check if the certificate is valid.

The only issue that this does come up against is the overlap of the pipeline schedule and the renewal script schedule from above. If you put them both on a daily schedule, one to renew the certificate and one to get the new certificate, you may be running the certificate pulling schedule run before it has been renewed. Although this is not ideal, as we renew 14 days in advance you would still have 13 days for it to catch up.

About the Author:
Christopher Pateman is a Senior Azure DevOps Engineer here at Version 1.