Using secrets to hide passwords and other confidential information in Databricks

A common something that you want to do in Databricks is hide your passwords and other confidential information in it, because else it will all be visible within your notebooks and logs.

Installing Databricks CLI

Luckily Databricks allows us just this. But before we can get started, we first have to configure our CLI tool of Databricks. For my example, I have decided to work with token authentication seeing that I login through AAD.

To configure a Token for your Databricks account, login to the portal and go to: User Settings -> Access Token -> Generate New Token and fill in a comment and a lifetime. (Note: you can leave lifetime empty, making a token for indefinite use).



We can now start installing our CLI and configure our account in it by using the token.

# Install the Databricks CLI
sudo pip install databricks-cli
databricks configure --token # Log in on our Databricks account with a token

# --- Details ---
# Host: <YOUR_HOST> # For Azure this could be: https://<REGION>.azuredatabricks.net e.g. https://westeurope.azuredatabricks.net
# Token: <TOKEN> # Our generated token in User Settings -> Access Token -> Generate New Token (Name = Databricks-CLI)

Creating a secret

Once we did that, we can actually start creating a secret. Open up your favorite terminal and run the following commands:

# Create our secret in a new scope
databricks secrets create-scope --scope <SECRET_SCOPE>
databricks secrets put --scope XAVIER --key <KEY_NAME> # Save the secret
# -> Input Password in the editor that opens and close it

# Show our scopes
databricks secrets list --scope <SECRET_SCOPE> # Shows Scopes
dbutils.secrets.get("<SECRET_SCOPE>", "<KEY_NAME>")

Congratulations! Your first secret has been created and can now be utilized in your notebooks with the dbutils command:

String password = dbutils.secrets.get("<SECRET_SCOPE>", "<KEY_NAME>") 



Xavier Geerinck

Xavier works as a Cloud Solution Architect at Microsoft, helping its customer unlock the full potential of the cloud. Even though he is still considered a young graduate, he achieved his first success at the age 16, by creating and selling his first startup. He then took this knowledge to create and help more startups in different markets such as technology, social media, philanthropy and home care. While in the meantime gaining more enterprise insights at renowned enterprises such as Nokia, Cisco and now Microsoft.

Read More