Hocon Secrets

Secret User and Password Variables

Usernames, passwords and other secrets should not be stored in your configuration files in clear text as these files are often stored directly in the version control system. They should also not be visible in logfiles after execution.

Secret providers are the secure solutions to work with credentials. All configuration items ending with ...Variable, like BasicAuthMode's userVariable and passwordVariable can be used with secret providers. The actual secret can be selected using the format: <SecretProviderId>#<SecretName>, e.g.

userVariable = "AZKV#JDBCuser"
passwordVariable = "AZKV#JDBCpw"

Secret Providers

Implemented secret providers are:

Provider	Pattern	Meaning
clear text	CLEAR#pd	The secret will be used literally (cleartext). This is only recommended for test environments.
environment variable	ENV#pd	The value for this secret will be read from the environment variable called "pd".
file	FILE#filename;secretName	read secretName from file filename. The file needs to be a property file with line format secretName=secretValue
Azure KeyVault	e.g. AZKV#secretName	Azure KeyVault element secretName. KeyValut specified in global section (see below)
Databricks Secret	e.g. DBSECRET#secretName	Databricks element secretName. Secret provider specified in global section (see below)

Additionaly, custom providers can be implemented, see Custom Secret Providers.

Azure KeyVault configuration

An Azure KeyVault would be specified as follows, here using an ID AZKV. The KeyVault name still needs to be specified.

global = {
  secretProviders = {
    AZKV = {
      className: io.smartdatalake.util.azure.AzureKeyVaultSecretProvider
      options = {
        keyVaultName: <azure-key-vault-name>
      }
    }
  }
 }

Databricks Secret configuration

Using Databricks credentials and other values can be stored encrypted as Databricks secrets. This can be used within SDLB by utilizing the class DatabricksSecretProvider. Here the ID DBSECRETS is used for further SDLB configuration references. Further, the secret scope (here scope=test) is required in Databricks. The configuration would look like:

global {
  secretProviders {
    DBSECRETS = {
     className = io.smartdatalake.util.secrets.DatabricksSecretProvider
     options = { scope = test }
    }
  }
}

Custom Secret Provider

Furthermore, a custom secret provider could be implemented as class using the trait SecretProvider and a constructor with parameter options: Map[String,String]. The specification in the global section would in general look like:

global {
  secretProviders {
    <secretProviderId> = {
     className = <fully qualified class name of SecretProvider>
     options = { <options as key/value> }
    }
  }
}