Do It, At Pace !

We use D365 CE and what used to be called the Common Data Service, and is now called the Dataverse.

Specifically, we want to send many thousands of updates through the API from a service bus. The Dataverse has various API quota and rate limiting which interfere with this plan.

As most of our D365 CE installation was created by contractors, apparently with no oversight whatsoever, it is boasts a horrific architecture.

Updating anything leads to a long chain of synchronous plugins and workflows, which then flows, synchronously, across to D365 Fin & Ops, and back again. This takes time, and very often hits every rate limit or quota its possible to hit with any kind of load over a couple of thousand updates.

Despite this, its necessary, for business reasons, for it to handle several thousand updates every night, within 15 mins. The update system provided by our contractors does not do this and also manages to lose a significant percentage of updates along the way.

I think it would be handy to have a mock system we can use that will replicate the rate limiting and quotas and produce the large number of errors produced by the real system.

It would be handy if it recorded what errors it produced, to tally against what the update system reported and the volume of updates processed along with the time it took.

With this, we can test alternative systems easily to ensure they are not unnecessarily slow, not losing updates and reporting all errors correctly.

Dataverse REST API

Th Dataverse has an ODATA 4.0 Rest API. There are probably already tools which can mock this, but as this is a lock down holiday project I decided not to look at them and make my own instead.

Tools

I wanted to use Azure Functions, since the system providing the updates is primarily a service bus and Azure Functions and I wanted to get acquainted with Azure Durable Functions, if I can find any way to crowbar them into my solution.

I was tempted to write this in Go, as I want to learn more about that and I might yet do this.

I have used Azure Functions for the web service and Terraform and a DevOps pipeline for deployment.

tbc …

We have an ever growing number of APIs hosted in Azure API Manager, in an ever growing number of environments.

Most of our APIs follow the same back end pattern, we have a number Azure Function Apps or Logic apps that talk to either our back end systems or those of a third party.

Deploying API’s manually is a real time sink, not to mention unutterably tedious and prone to transcription errors. I believe that out operations team has had one member of staff doing nothing other than configuring API-M for several weeks.

Every other part of our deployments are scripted and carried out by DevOps pipelines so it has been a constant source of irritation that we have not, until now, been able to eliminate this last manual step.

I have previously tried to use ARM templates to automate our deployments, but I ran out of time on that, towards the end it was just bogged down in the complexity and lack of clarity which ARM brings to the process.

Having recently introduced myself to Terraform, and noticing that it seemed to have a lot of support for API Manager I thought I would put some time aside to revisit APIM with Terraform.

In short, its far easier to use.

What I wanted was to be able to define an entire API, including its back end connections to logic apps or function apps, policy files, etc as a set of variables, hand that to Terraform and let it manage creating and updating the APIs from then on.

It looked like the best way of doing that was to create a Terraform module, this is what I did, and you can access the module here.

A lot of my work lately revolves around creating new applications that have a fairly similar structure in Azure.

I have a resource group to contain the new functionality, one or more function apps, to do some work, a key vault for the function apps to store their secrets and an application insights for monitoring.

We use Microsoft DevOps so the resource group and associated resources is generally contained within a single repo and has a single pipeline.

The pipeline for deploying the above with ARM Templates is tedious and long winded, I wondered if I could simplify it using Terraform.

I wanted to have the resource group and its resources defined purely in variables and I wanted as much of the gubbins to be fairly reusable, or capable of being used with further Terraform, or even ARM scripts for the cases when we need more stuff in the resource group.

On the whole it was fairly easy to build a Terraform module to achieve all of the above goals. Some parts, such as including Key Vault keys in the function apps configuration required a bit of thinking about.

The source code for the module is available here and also via the Terraform Registry.

Function App Group as Variables

I’ll start at the end, the following module configures my function app group.

module "terraform-functionapp-group" {
    source  = "JoeAtRest/functionapp-group/azurerm""

    subscription_prefix = "dev"
    location_prefix = "uks"
    location = "uksouth"
    app_name = "mytestgroup"

    keyvault_secrets = { "secret-name" : "the secret", "secret-name2" : "squirrel" }
    tags = { "Solution" : "Test" }

    functionapps = [{
        name              = "fa-1"    
        zip_path          = "local/fa1"
        ip_restrictions   = ["192.168.1.23","200.32.29.4"]
        settings          = { "NameInFuctionApp" = "https://some-url" ,  "OtherThingInFunctionApp" = "false" }
        key_settings      = [{ name = "mysecret", secret = "secret1" },{name = "myothersecret", secret = "secret2"}]
    }]

    access-policies = []
}

We have a naming convention for our Azure components to denote which subscription and region they’re in, the top few variables take care of those so that I only require one script for all regions and subscriptions.

The app_name is the name of the resource group.

When the key vault is created I want to preload it with the various secrets and what have you for the environment. In real life these would not be defined in the main.tf but would be in Azure Secure Files that I copy in the pipeline into the terraform folder.

Tags, its good to tag things.

   functionapps = [{
        name              = "fa-1"
        zip_path          = "local/fa1"
        ip_restrictions   = ["192.168.1.23","200.32.29.4"]
        settings          = { "NameInFuctionApp" = "https://some-url" ,  "OtherThingInFunctionApp" = "false" }
        key_settings      = [{ name = "mysecret", secret = "secret1" },{name = "myothersecret", secret = "secret2"}]
    }]

This is an array of function apps which I want to create, in this instance its just a single function app.

I am deploying the function app using the WEBSITE_RUN_FROM_PACKAGE setting, which means I build the code, zip it up and store the zip file in an Azure storage blob. I then use the SAS key in the function app settings to tell it where to run from.

This raised the first issue I faced with the Terraform process. If I always provide Terraform with a file that has the same name as the last version of the function app then it will not take any action, it thinks nothing has changed.

I suspect you can tell the resource not to behave like that and to always re-deploy, I didn’t know that 3 days ago when I made this module and pipeline.

Instead I always give my zip file a random name and pass that into the module in zip_path.

Our OPSEC team like us to restrict function apps to talk only to API-M, the list of ip_restrictions limit the function app to talk to only ip addresses in this list.

A function app needs settings, you see them in the Configuration blade in the Azure portal. There are a number of different kinds of settings, all lumped into the one place.

First of all ( my settings map ) are settings the function app is expecting to determine how it runs, flags, URLs of things its talking to, etc.

Secondly there are sensitive settings which you would rather store in key vault, you put a big long address into the configuration and Azure automatically provides your function app with the relevant value.

So that is enough configuration to create the resource group, a function app, a key vault and an application insights. The ARM equivalent is, it goes without saying, utterly hideous.

How Does That All Work ?

I am going to skip completely over the configuration of key vault and app insights, its a case of copying and pasting the relevant sections from the Terraform Azurerm pages.

Storage

The function app needs some storage, in which to keep its zip file.

resource "azurerm_storage_account" "function-storageaccount" {
  name                     = local.storage_account
  resource_group_name      = azurerm_resource_group.app-rg.name
  location                 = azurerm_resource_group.app-rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  tags                     = var.tags
}

resource "azurerm_storage_container" "function_storagecontainer" {
  name                  = local.storage_container
  storage_account_name  = azurerm_storage_account.function-storageaccount.name
  container_access_type = "private"
}

resource "azurerm_storage_blob" "function_storageblob" {
    for_each = { for functionapp in var.functionapps : functionapp.name => functionapp }

  name                   = each.value.zip_path
  storage_account_name   = azurerm_storage_account.function-storageaccount.name
  storage_container_name = azurerm_storage_container.function_storagecontainer.name
  type                   = "Block"
  source                 = each.value.zip_path
}

data "azurerm_storage_account_sas" "function_sas" {
  connection_string = azurerm_storage_account.function-storageaccount.primary_connection_string
  https_only        = false
  resource_types {
    service   = false
    container = false
    object    = true
  }
  services {
    blob  = true
    queue = false
    table = false
    file  = false
  }
  start  = "2018-03-21"
  expiry = "2028-03-21"
  permissions {
    read    = true
    write   = false
    delete  = false
    list    = false
    add     = false
    create  = false
    update  = false
    process = false
  }
}

Ta da ! Now it has storage.

The only bits of note here is the creation of the storage blob and loading the zip file into it and the generation of the SAS key which allows the function app to access its zip file.

First, the blob …

resource "azurerm_storage_blob" "function_storageblob" {
    for_each = { for functionapp in var.functionapps : functionapp.name => functionapp }

  name                   = each.value.zip_path
  storage_account_name   = azurerm_storage_account.function-storageaccount.name
  storage_container_name = azurerm_storage_container.function_storagecontainer.name
  type                   = "Block"
  source                 = each.value.zip_path
}

The cool part here is the for_each, it will create blob for each function app in my configuration ( which was a set, so can be multiple ). Each resource can then be identified elsewhere in the scripts like this

something = azurerm.storage_blob.function_storageblob["fa-1"].id

The source then instructs terraform to load the zip file into the blob.

Function App

The actual function app creation is very straight forward, which was just as well as it left plenty of time to work out how to get the settings set.

locals {
  app_settings = {
    "FUNCTIONS_WORKER_RUNTIME" : "dotnet",
    "FUNCTIONS_EXTENSION_VERSION" : "~3",
    "APPINSIGHTS_INSTRUMENTATIONKEY" : azurerm_application_insights.rg.instrumentation_key       
  }    
}

# App Service Plan
resource "azurerm_app_service_plan" "app_app_service_plan" {
  name                = local.app_service_plan
  location            = azurerm_resource_group.app-rg.location
  resource_group_name = azurerm_resource_group.app-rg.name
  kind                = "FunctionApp"
  sku {
    tier = "Dynamic"
    size = "Y1"
  }
  tags = var.tags
}

resource "azurerm_function_app" "app_functionapp" {
  for_each = { for functionapp in var.functionapps : functionapp.name => functionapp }

  name                      = "${var.subscription_prefix}-${var.location_prefix}-fa-${each.value.name}"
  location                  = azurerm_resource_group.app-rg.location
  resource_group_name       = azurerm_resource_group.app-rg.name
  app_service_plan_id       = azurerm_app_service_plan.app_app_service_plan.id
  storage_connection_string = azurerm_storage_account.function-storageaccount.primary_connection_string
  app_settings              = merge(
    each.value.settings,
    {"APPINSIGHTS_INSTRUMENTATIONKEY" : azurerm_application_insights.rg.instrumentation_key}, 
    {"WEBSITE_RUN_FROM_PACKAGE" : "https://${azurerm_storage_account.function-storageaccount.name}.blob.core.windows.net/${azurerm_storage_container.function_storagecontainer.name}/${azurerm_storage_blob.function_storageblob[each.value.name].name}${data.azurerm_storage_account_sas.function_sas.sas}"},    
    {"HASH" : filebase64sha256(each.value.zip_path) },
    zipmap(each.value.key_settings[*].name, [for s in each.value.key_settings[*].secret: "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.app_keyvault.vault_uri}secrets/${azurerm_key_vault_secret.app_secret[s].name}/${azurerm_key_vault_secret.app_secret[s].version})"]) 
    )
  version = "~3"
  identity {
    type = "SystemAssigned"
  }
  
  tags = var.tags
  
  site_config {
    dynamic "ip_restriction" {
      for_each = each.value.ip_restrictions
      
      content {
        ip_address  = "${ip_restriction.value}/32"        
      }
    }    
  }
}

The settings, so there are 3 categories of settings. There’s the boiler plate function app settings, which I have hard coded into the local variable.

locals {
  app_settings = {
    "FUNCTIONS_WORKER_RUNTIME" : "dotnet",
    "FUNCTIONS_EXTENSION_VERSION" : "~3",
    "APPINSIGHTS_INSTRUMENTATIONKEY" : azurerm_application_insights.rg.instrumentation_key       
  }    
}

Then there are the function app settings from the variables and finally the settings which I want to configure to use keyvault.

All these 3 things need to be brought together into the single settings map in the function app configuration.

Happily, terraform provides the merge function, to merge together many maps into a single map.

merge(
    each.value.settings,
    {"APPINSIGHTS_INSTRUMENTATIONKEY" : azurerm_application_insights.rg.instrumentation_key}, 
    {"WEBSITE_RUN_FROM_PACKAGE" : "https://${azurerm_storage_account.function-storageaccount.name}.blob.core.windows.net/${azurerm_storage_container.function_storagecontainer.name}/${azurerm_storage_blob.function_storageblob[each.value.name].name}${data.azurerm_storage_account_sas.function_sas.sas}"},    
    {"HASH" : filebase64sha256(each.value.zip_path) },
    zipmap(each.value.key_settings[*].name, [for s in each.value.key_settings[*].secret: "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.app_keyvault.vault_uri}secrets/${azurerm_key_vault_secret.app_secret[s].name}/${azurerm_key_vault_secret.app_secret[s].version})"]) 
    )

I don’t mind admitting, this bit took me way longer to work out than is feasible. Largely becuase I had messed up the data structure in each.value.settings whilst I was trying to get zipmap to work, the errors I though were telling me I had got the zipmap bit wrong were actually not about that at all.

Many hours later. I realised that. So, this may not be the best, or even most sensible way of doing this. It is the way which worked after many hours, so I am sticking with it.

zipmap(each.value.key_settings[*].name, [for s in each.value.key_settings[*].secret: "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.app_keyvault.vault_uri}secrets/${azurerm_key_vault_secret.app_secret[s].name}/${azurerm_key_vault_secret.app_secret[s].version})"]) 

So, I want the setting name as the maps key and then for that string to be generated using the secret name and the keyvault I created, invisibly so far as you are concerned, earlier as the value. Zipmap takes the first parameter as a list of keys, and the second as a list of values.

each.value.key_settings[*].name

key_settings is a set of objects. The object has the properties name and secret. The splat ( * ) generates a list of all of the names in the set of maps.

[for s in each.value.key_settings[*].secret: "@Microsoft.KeyVault(SecretUri=${azurerm_key_vault.app_keyvault.vault_uri}secrets/${azurerm_key_vault_secret.app_secret[s].name}/${azurerm_key_vault_secret.app_secret[s].version})"]

This bit, I have my doubts about this bit. But it works. So far as I can tell, it loops through each secret in the set of maps and creates a list where that secret has been transformed into the string after the :

Handily that is the exact string required for Azure to automatically provide the function app with the latest version of that secret.

Key Vault and Managed Identity

I’m not sure what the best strategy is with key vault, whether to have one key vault or lots of them. We seem to have opted to have lots of them, one per resource group.

The key vault is there to do two things

• Securely hold keys and secrets for the application
• Allow access to those people or things who should have access and prevent access for everything else

There are several things which require varying degrees of access to the key vault

• The devops pipeline needs to be able to add and remove keys and secrets and also, if I use terraform destroy the ability to delete the key vault
• The function apps need to be able to access their secrets
• Our operations teams needs to be able to see and change the keys or secrets
• In pre-production environments the development and test teams need to be able to access the keys and secrets

Access for the pipeline and access for the function apps, via a managed identity, is handled automatically by the module whilst the other cases are configurable using the access_policies parameter.

   access_policies = [{
             tenant_id           = "tenant id of azure subscription"
             object_id           = "object if of resource requiring access"
             key_permissions     = ["create","get"]
             secret_permissions  = ["set","get","delete"]
]}

There are various kinds of permissions you can assign to both keys and secrets, the example is only a few of them.

Managed Identity

The function app is given an Azure Managed Identity when its created through this property

identity {
    type = "SystemAssigned"
  }

Then the key vault assigns permissions to that managed identity like this

resource "azurerm_key_vault_access_policy" "app_keyvault_functionapps" {
  for_each = { for functionapp in var.functionapps : functionapp.name => functionapp }

  key_vault_id = azurerm_key_vault.app_keyvault.id

  tenant_id = azurerm_function_app.app_functionapp[each.value.name].identity[0].tenant_id
  object_id = azurerm_function_app.app_functionapp[each.value.name].identity[0].principal_id

  key_permissions = [    
    "get",
  ]

  secret_permissions = [    
    "get",    
  ]
}

First of all, there may be more than one function app so the for_each will apply this for all of them.

The tenant_id and object_id can then be retrieved from the resource.