Feature request: retrieve Azure Functions' secrets from Key Vault

[mmaitre314]

All our secrets are in Key Vault so we need a way for Azure Functions to retrieve them from there instead of looking them up in app settings. In function.json, connection strings could referenced using Key Vault URLs:

{
    "disabled": false,
    "bindings": [
        {
            "name": "myQueueItem",
            "queueName": "myqueue-items",
            "connection":"https://xxx.vault.azure.net:443/secrets/yyy",
            "type": "queueTrigger",
            "direction": "in"
        }
    ]
}

Connecting to Key Vault requires us to pass a client cert for app authentication, so the WEBSITE_LOAD_CERTIFICATES app setting will be needed.

[christopheranderson]

Is the problem that the settings themselves are exposed as environment variables or is it that you can access those settings via the portal? If App Service in general supported Key Vault for settings, would that satisfy you?

You likely don't want the URL inside the function.json since that would still service metadata you wouldn't want to check in. (Also this is probably a WebJobs Script issue, not core WebJobs)

[mmaitre314]

Handling that at the App Service level might work. I haven't tried but I heard something like that could be done via ARM template:

    "keyVaultId": {
      "value": "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.KeyVault/vaults/xxx"
    },
    "keyVaultSecretName": {
      "value": "xxx"
    },

(not sure about the exact syntax)

It's slightly more rigid than calling Key Vault at runtime: it won't allow automatic secret rotation without redeploying.

I was thinking about hooking the WebJob Script. If there is a point where custom initialization code can be run before any job runs, I could call Key Vault and set environment variables inside the process.

One advantage of Key Vault vs App Setting is that it works exactly the same during local testing. We just install a cert on the machine and don't need to copy/paste all the secrets as Functions' local testing currently required. One thing I haven't figured out in that setup: how to split Test vs Prod. When I start writing a function both sets of settings will likely be the same, but as code moves to Prod and becomes key in our services, separating Test and Prod settings will likely be needed.

[sjwaight]

Here is a workaround that allows you to retrieve secrets in a Function app: https://blog.siliconvalve.com/2016/11/15/azure-functions-access-keyvault-secrets-with-a-cert-secured-service-principal/

[parad0xchild]

@sjwaight There is another problem with Key Vault not being directly integrated to retrieve keys. While you can code the connection in via Cert, SAS, or other means, if you have a highly scalable Azure Function (thousands of instances at once) you will over load the Key Vault with requests from the same IP address (leaving it with no open ports to fulfill requests). This will result in a failure in your function and an error like "Only one usage of each socket address (protocol/network address/port) is normally permitted".

What really is needed is for a connection from the Azure Function to Key Vault so that you can define which secret or key you want, the orchestration to pull it out periodically and make it available to the function via an input variable. This will ensure you don't make tons of unnecessary calls to the key vault resulting in a failure.

[solvingj]

For reader who have not voted, there's an "under review" feature request here:

https://feedback.azure.com/forums/355860-azure-functions/suggestions/14634717-add-binding-to-key-vault

[SV-ZeroOne]

Can we please get a status update on this feature request?

I implemented the two workarounds that can be found below and then load tested them with loader.io to see if they could handle a few thousand request within 1 minute.

Workaround 1
and
Workdaround

Based on the load test the Key Vault could only handle around +-280 to 290 successful access attempts within a minute before it started to produce this error in the Azure functions logs

microsoft.threading.tasks: an error occurred while sending the request. system: unable to connect to the remote server. system: an attempt was made to access a socket in a way forbidden by its access permissions.

Please could this feature request get implemented soon as this is restricting the ability of the Azure functions to scale which require Key Vault access.

[rhythmnewt]

@SV-ZeroOne Have you considered using redis as the cache layer for key vault secrets? Granted, that's an additional dependency and expense, but it should improve the throughput. I am planning to do exactly that myself.

[sjwaight]

I ended up writing my own local cache layer right in the Function itself. No need for Redis for me :). If I have a change in Keyvault then my Function runtime just needs a restart to pick up the new item.

[akuzhan]

Hi @christopheranderson, is there any estimated deadline for this feature? Many thanks!

[ColbyTresness]

Hi all,

This is our highest upvoted User Voice item, and it's something we'll be cognizant of as we plan our timeline moving forward.

https://feedback.azure.com/forums/355860-azure-functions/suggestions/14634717-add-binding-to-key-vault

[solvingj]

It seems to me that the current beta for Azure Managed Service Identity gets us most of the way there: https://azure.microsoft.com/en-us/blog/keep-credentials-out-of-code-introducing-azure-ad-managed-service-identity/

I haven't had a chance to implement, but it seems the experience is effectively what we (and maybe most people?) have been needing. As usual I think the C# gets the most advantage, but I thought I read it was still pretty good for others. Anybody had chance to test?

[johnib]

where we at with this? I would really like to have it ready for use soon

[MV10]

@solvingj I've been using MSI for Functions accessing Key Vault as centralized storage of SQL connection strings, and you're right, it looks like most of the heavy lifting is done. Caching secrets (and the key client) is relatively trivial from there. The Azure folks have dreamed up a whole slew of weird new connection string patterns, it seems it ought to be easy to add another for key vault.

[ridhoq]

Bump on this. My use case is as follows: My Azure Function needs to be triggered when a message is published to a topic on Azure Service Bus. The connection string is a secret and should be saved in Azure Key Vault. However, since my function only fires upon message publication, I cannot retrieve the connection string during function execution from Key Vault - it has to happen before that for the Azure Function to even trigger.

The only work around I can think of is to inject the connection string from Key Vault during my build process (via Jenkins) into a local.settings.json file before I deploy it using the Zip deploy. Has anybody else run into a similar situation?

[paulbatum]

Regarding your workaround - I would recommend you inject the connection string (retrieved from key vault) into an ARM template you run that updates the app settings of the function app. I am not even sure if the local.settings.json approach will work in Azure, and even if it does it is not quite as clean from a security perspective (anyone with access to the storage account containing the zip would have access to the connection string).

[ridhoq]

@paulbatum thanks for the quick response! I was not aware of ARM templates as a deployment mechanism so I’ll look into using that as a work around. Just to clarify, does that mean there’s no other supported way of doing what I’m describing?

[paulbatum]

That is correct. This issue tracks adding the feature for the functions to support bindings that directly use connection strings from key vault.

[MBhatt01]

@ridhoq Have you resolved integrating ServiceBus connection string to local.settings.json using ARM? I am facing similar issue with EventHubTrigger and I have my connectionstring available in Key Vault, but I am not able to find any way to use the secret on function Run method trigger. Please guide me on this if you have achieved the same in any way.

[apedzko]

This request is very important for our team. Please implement it.

[CrazyTuna]

I've managed to get this working with Azure Function v2:

How to retrieve Azure Functions secrets from Key Vault

It uses the Microsoft.Extensions.Configuration.AzureKeyVault nuget package

I've added a startup script:

using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using System.Linq;

[assembly: WebJobsStartup(typeof(FunctionApp1.WebJobsExtensionStartup), "A Web Jobs Extension Sample")]
namespace FunctionApp1
{
    public class WebJobsExtensionStartup : IWebJobsStartup
    {
        public void Configure(IWebJobsBuilder builder)
        {
            // Gets the default configuration
            var serviceConfig = builder.Services.FirstOrDefault(s => s.ServiceType.Equals(typeof(IConfiguration)));
            var rootConfig = (IConfiguration)serviceConfig.ImplementationInstance;

            // Creates a new config based on the default one and adds the keyvault configuration builder
            var keyvaultName = rootConfig["keyVaultName"];
            var config = new ConfigurationBuilder()
                .AddConfiguration(rootConfig).AddAzureKeyVault($"https://{keyvaultName}.vault.azure.net/").Build();

            // Replace the existing config
            builder.Services.AddSingleton<IConfiguration>(config);
        }
    }
}

So:

• The function app need to use MSI
• The function app need to have at least Read/List secrets permissions in the key vault.
• The keyvaultName settings need to be presetn in your local.settings.json or app settings blade of the function app.

[BorisWilhelms]

@CrazyTuna looks like we had the same idea at the same time 😄

https://github.com/BorisWilhelms/azure-function-keyvault

[darewreck54]

Any idea of how to get this working locally in java? I tried the REST MSI method but it only works on the deployed instance on azure and not locally for development. I also tried using the azure function service principal (printed out the MSI_SECRET) and the principal ID to do an AAD authentication and it throws an exception stating that the principal ID dose not belong to the tenant ID which is strange because its stated in the resource files. I also associated the azure function as a policy for the azure key vault.

Any advice appreciated,
Thanks!

[apedzko]

Hi!

I'm trying to adjust the described approach (using WebJobsStartup) to dynamically fetch connection string to service bus from the keyvault for the function that is triggered by the service bus message. The function is running on the "Consumption" plan.

When i open the function on the azure portal, the function gets activated, the "WebJobsStartup" code successfully executes and the function can process the messages.

After some time of inactivity (around 30 minutes) the function instance gets disposed. Since then it is not triggered by the messages on the service bus.

If i move the connection string back into the config, everything works fine: the function is successfully triggered by a new service bus message after an inactivity timeout.

Could you please advise if i am missing something?

[CrazyTuna]

@apedzko, the code I've posted works with connectionstring. In your example you have a connectionstring named "manageSas". so you need a secret with name "manageSas" in your key vault. That's it.
The code I've posted add to the default configuration key vault secrets. So your connection can be either in a config, in env variables and/or key vault.

@alexjamesbrown This should work also for you. Just the name of your connectionstring need to match a secret in key vault

[georgiosd]

FYI all, this was just updated to say it's coming: https://feedback.azure.com/forums/355860-azure-functions/suggestions/14634717-add-binding-to-key-vault

[apedzko]

@CrazyTuna thank you for the reply.

Our example is simplified: in fact we need to fetch settings from the external REST API.
From what i could understand from your example, on startup i need to add a custom ConfigurationProvider that will expose settings from an external source.

If you look into the code that i have provided, this is exactly what my DummyConfigurationProvider is doing.

Am i missing something or is there a specific magic in the "Keyvault" implementation?

[myurashchyk]

@CrazyTuna

I deployed provided sample function using App Service plan / pricing tier ExampleFunctionPlan (Consumption)
After deploy function started successfully, loaded setting from key vault and logged info on existing / new BLOBS in specified container into app insights - as expected.
In 20-30 minutes of inactivity I see following logs in app insights:
Stopping JobHost
Job host stopped

Then I add some new BLOBS and nothing happens. Function is not started. If I have the same connection string in function app settings then function starts and processes new BLOBS.

In my case function processed new blobs in 2 days when I restarted chrome and it had open tab with function.

Is there any workaround for functions that use Consumption pricing tier to start on new BLOBS?

[CrazyTuna]

@myurashchyk, sorry I haven't tried. it seems @apedzko had the same kind of issue, any resolution from your side ?

[apedzko]

@CrazyTuna in fact we are on the same team with @myurashchyk :-). Thus the problem is the same. No resolution so far.

[rhythmnewt]

I am interested in this feature as well, but haven't had a chance to try it yet. @apedzko and @myurashchyk have you tried adding a timer triggered function to keep at least one instance alive?

[ebrucucen]

Apparently, this is in preview today: https://azure.microsoft.com/en-us/blog/simplifying-security-for-serverless-and-web-apps-with-azure-functions-and-app-service/

[apedzko]

It looks llike we still cannot offload the trigger settings for Azure Function into the keyvault - the function does not get activated in case the connection string to the service bus queue sits inside the keyvault.

[paulbatum]

@apedzko This should no longer be an issue. If you follow the steps outlined in the post linked above, the function app should still be activated appropriately. If you are finding that this is not the case, please share investigative information:

UTC Timestamp:
Function App name:
Function name(s) (as appropriate):
Region:

cc @cgillum @mattchenderson

[myurashchyk]

@paulbatum, @cgillum, @mattchenderson

2018-12-02 14:04:32.767 (UTC) - message was processed. I was monitoring function in Azure portal.
2018-12-02 14:30:16.442 (UTC) - Job host stopped (message in app insights)
2018-12-02 14:38:32 (UTC) - new message sent into queue
2018-12-02 14:48 (UTC) - message is still in queue
2018-12-02 14:49:06.067 (UTC) - I opened function in Azure portal and message was processed.

Function App name: MyPamTestFunc
Function name: TestFunc
Region: West US

Function logs received message and throws an exception to move message into dead letter queue:
[FunctionName("TestFunc")]
public static Task RunAsync([ServiceBusTrigger("testqueue", Connection = "manageSas")] string myQueueItem, ILogger logger)
{
logger.LogInformation($"{DateTime.UtcNow} : Received message: {myQueueItem}");
throw new System.Exception("move message to dead-letter queue");
return Task.CompletedTask;
}
function apps settings: manageSAS = @Microsoft.KeyVault(SecretUri=https://m***.vault.azure.net/secrets/manageSas/932b202eba844bf28811396f3bc1293c)

key vault settings:
key https://m***.vault.azure.net/secrets/manageSas/932b202eba844bf28811396f3bc1293c
value Endpoint=sb://m***.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=***=

queue settings:
testqueue - Shared access policies:
policy: manage
Claims: Manage, Send, Listen

[paulbatum]

@myurashchyk thanks for sharing these details. I was able to find our scale logs for your app, I can see some errors but I don't know what they mean yet. Have looped in a few folks. You should get an update from us within a day or two.

[cgillum]

This appears to be a case sensitivity issue: The function app setting is named manageSAS but in your code you bound your connection in the trigger to manageSas. Make the casing consistent and this problem should go away.

In a future service update (going out in the next few weeks), we're making these comparisons case insensitive.

[myurashchyk]

@paulbatum, @cgillum, thank you!
Making casing consistent fixed my issue.
We look forward to using the new feature when secret rotation support will be implemented.

[ebrucucen]

It looks like it does not work for Powershell v1 runtime (or I am missing something) on West Europe

2018-12-06T06:15:19.686 [Error] : Exception setting "ConnectionString": "The value's length for key 'password' exceeds it's limit of '128'."    
2018-12-06T06:15:20.350 [Error] Function completed (Failure, Id=d45a59be-65d8-4ac5-967a-8d76b4207ff4, Duration=795ms)```

[tvperez76]

I made some modifications to @apedzko code to append these secrets found in Azure KeyVault to Environment variables,
`
using Microsoft.Azure.KeyVault;
using Microsoft.Azure.Services.AppAuthentication;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
[assembly: WebJobsStartup(typeof(Functions.Startup), "A Web Job's Extension for KeyVault Configuration Retrieval")]
namespace Functions
{
public class Startup : IWebJobsStartup
{
public void Configure(IWebJobsBuilder builder)
{
builder.AddKVConfig();
}
}

///

/// Extension methods for registering with .
///

public static class KVConfigurationExtensions
{
public static void AddKVConfig(this IWebJobsBuilder builder)
{
if (builder == null)
{
throw new ArgumentNullException(nameof(builder));
}

        var tempProvider = builder.Services.BuildServiceProvider();
        var tempConfig = tempProvider.GetRequiredService<IConfiguration>();

        var configurationBuilder = new ConfigurationBuilder();

        var descriptor = builder.Services.FirstOrDefault(d => d.ServiceType == typeof(IConfiguration));
        if (descriptor?.ImplementationInstance is IConfigurationRoot configuration)
        {
            configurationBuilder.AddConfiguration(configuration);
        }

        string msiConnectionString = tempConfig["MSIConnectionString"];
        configurationBuilder.Add(new KVConfigurationSource(tempConfig.GetSection("KeyVaultBase"), msiConnectionString));
        var config = configurationBuilder.Build();

        builder.Services.Replace(ServiceDescriptor.Singleton(typeof(IConfiguration), config));
    }

    internal class KVConfigurationSource : IConfigurationSource
    {
        private readonly KeyVaultConfig keyVaultInfo;
        private readonly string msiConnectionString;

        public KVConfigurationSource(IConfigurationSection keyVaultInfoSection, string msiConnectionString)
        {
            this.keyVaultInfo = keyVaultInfoSection.Get<KeyVaultConfig>();
            this.msiConnectionString = msiConnectionString;
        }

        public IConfigurationProvider Build(IConfigurationBuilder builder)
        {
            var azureServiceTokenProvider = new AzureServiceTokenProvider();
            KeyVaultClient client = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback));
            return new KVConfigurationProvider(this.keyVaultInfo, client);
        }
    }

    internal class KVConfigurationProvider : ConfigurationProvider, IDisposable
    {
        private readonly KeyVaultConfig keyVaultInfo;
        private KeyVaultClient keyVaultClient;

        public KVConfigurationProvider(KeyVaultConfig keyVaultInfo, KeyVaultClient client)
        {
            this.keyVaultInfo = keyVaultInfo;
            this.keyVaultClient = client;
        }

        public override void Load()
        {
            var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
            string baseUrl = this.keyVaultInfo.BaseUrl;
            foreach (SecretMetadata secretKey in this.keyVaultInfo.SecretMetadata)
            {
                if (secretKey != null)
                {
                    string keyName = secretKey.KeyName;
                    string version = string.Empty;
                    if (!string.IsNullOrEmpty(secretKey.Version))
                    {
                        version = $"/{secretKey.Version}";
                    }

                    string url = $"{baseUrl}/secrets/{keyName}{version}";
                    var valueEntry = keyVaultClient.GetSecretAsync(url).Result;
                    if (valueEntry != null)
                    {
                        data[secretKey.Alias] = valueEntry.Value;
                        Environment.SetEnvironmentVariable(secretKey.Alias, valueEntry.Value, EnvironmentVariableTarget.Process);
                    }
                }
            }

            Data = data;
        }

        #region IDisposable Support
        private bool disposedValue = false; // To detect redundant calls

        protected virtual void Dispose(bool disposing)
        {
            if (!disposedValue)
            {
                if (disposing)
                {
                    this.keyVaultClient.Dispose();
                }

                disposedValue = true;
            }
        }

        // This code added to correctly implement the disposable pattern.
        public void Dispose()
        {
            Dispose(true);
        }
        #endregion
    }

    internal class KeyVaultConfig
    {
        public string BaseUrl { get; set; }
        public SecretMetadata[] SecretMetadata { get; set; }
    }

    internal class SecretMetadata
    {
        private string alias;
        public string KeyName { get; set; }
        public string Alias { get { if (string.IsNullOrEmpty(this.alias)) { return this.KeyName; } return this.alias; } set => this.alias = value; }
        public string Version { get; set; }
    }

}
`

The local.settings.json file would have the following entries:
{ "IsEncrypted": false, "Values": { "KeyVaultBase:BaseUrl": "https://xxx.vault.azure.net", "KeyVaultBase:SecretMetadata:0:KeyName": "SBDispatcher", "KeyVaultBase:SecretMetadata:0:Alias": "DispatcherConnectionString", "KeyVaultBase:SecretMetadata:1:KeyName": "SBProcessor", "KeyVaultBase:SecretMetadata:1:Alias": "ProcessorConnectionString", "KeyVaultBase:SecretMetadata:1:Version": "2adee0dde68d4a74b1f9e24653e1c1da", "MSIConnectionString": "RunAs=Developer; DeveloperTool=AzureCli" }, "ConnectionStrings": {}, "Host": { "LocalHttpPort": 7071, "CORS": "*" } }

[briandunnington]

@paulbatum Will this work for local debugging if I provide the Key Vault reference as a value in local.settings.json? I tested it and it just seemed to use the raw string. (I did grant my personal account the necessary rights to the key vault/etc but it didnt look like it even made it that far).

If I used this in appsettings.json in an ASP.NET Core app, would it work? Which part of Azure (App Service, Azure Application Settings, etc) is actually providing the feature?

[paulbatum]

I am not an expert in how this feature works, but my understanding is that it is in App Service feature that would not apply in either of the cases you mentioned (local development, settings references in appsettings.json). Pinging @mattchenderson to confirm.

[vishramendra]

Are there any limitation on this feature? I'm using this feature for a HTTP Triggered function app but rather than loading the actual value of the secret it is returning the command itself @Microsoft.KeyVault(SecretUri=https://demokeyvaultash.vault.azure.net/secrets/APIKey/).

[paulbatum]

@vishramendra If you look at the post, you'll see the format you enter needs to include the version e.g. https://myvault.vault.azure.net/secrets/mysecret/ec96f02080254f109c51a1f14cdb1931

[vishramendra]

@paulbatum. I'm using below format in Application settings. The function app service plan has managed identity enabled and the app has also been registered in the key vault. Please note that I'm using function app v1. Does this feature only work for function app v2? @Microsoft.KeyVault(SecretUri=https://demokeyvaultash.vault.azure.net/secrets/APIKey/8781ac7f930940bb823f2d0f9a38d62d)

[paulbatum]

@vishramendra as far as I know, this feature should work the same way for anything in app service, function app version shouldn't matter. I am not really sure how to troubleshoot problems like this, I will ask some other folks for help and point them here.

[mattchenderson]

@vishramendra Paul is correct - this works on anything in App Service. This is almost always an access policy issue, so I'd encourage you to double-check that. We have identified an issue for managed identities if you have the access policy set to "application-application" (meaning you set the "authorized application" flag in the access policy configuration). You just need to set the access policy principal to your managed identity, and it needs to have the "Get" permission for secrets.

[dpakgithub82]

Can you some put a example of using the keyvault for Azure Function blob connection string in a bindings,
where to put this code :"@Microsoft.KeyVaul" @mmaitre314 @christopheranderson @sjwaight @parad0xchild @solvingj

[vishramendra]

@mattchenderson You were right. Key Vault access policy doesn't need Application authorization as well as configure-template. My issue is resolved. Thanks to @paulbatum and you. :)

[dpakgithub82]

this option will not work in visual studio debugging:
@Microsoft.KeyVault(SecretUri=https://demokeyvaultash.vault.azure.net/secrets/APIKey/8781ac7f930940bb823f2d0f9a38d62d)

[Simon-Gregory-LG]

It seems @Microsoft.KeyVault doesn't work through the new VNET integration (using delegation) why would that be?
(i.e. if I integrate the appservice into a VNET, point the to the KeyVault using Service Endpoint and lock down the KeyVault to that vnet, this doesn't seem to work)

[rhythmnewt]

Any plans to make @Microsoft.KeyVault work from local.settings.json file?

[rhythmnewt]

It's been 1.5 years, any interest/update? Thanks.

[sjwaight]

Given these Key Vault references rely on Managed Identity you'd first need to solve enabling the local runtime environment to execute using Managed Identities. The same would apply to the App Configuration references you can use. I would recommend opening a new feature request if you need specific focus on this use case (note I don't work on the Product Group so I'm not speaking authoritatively here).