Git Secrets

Collaboration can be a powerful process. One of the best parts of developing for an open platform is the opportunity it brings to collectively build, learn, and share. It’s an opportunity we care about here at Sparkbox. Whenever appropriate, we share our work with the world. We make open source contributions where we can, and we aim for transparency as standard practice. However, there are times, like when we’re working with clients and sensitive information, that the collaboration bubble is much smaller and our code needs to be kept private.

It’s likely that somewhere, right at this very moment, someone is pushing private information to a public repository. Maybe it’s personal information, like a password, or even worse, someone else’s private information. It’s not an uncommon event. Github even has instructions for how to remove sensitive data from a repository. Our clients entrust us with their sensitive data, and in some instances ask us to keep any and all material related to their organizations completely out of public view. We take this responsibility seriously.

At Sparkbox, we’ve begun using a tool to help mitigate the risk of making these types of accidental commits. Git-secrets is a shell utility that can help us catch our mistakes.

What Is git-secrets?

It’s a get out of jail free card. More specifically, it’s a shell utility that can set up git hooks quickly and manage configuration of a scanning process that checks our commits before we go face-to-palm. Here are some highlights:

• It automates the creation of the necessary hooks - pre-commit, commit-msg, and prepare-commit-msg.

• It manages lists of prohibited patterns. These can be either literal strings or regular expressions à la egrep.

• It makes provisions for secret patterns to be configured both globally and locally (per repo).

• When the hooks run, it scans our commits and commit messages for occurrences of prohibited patterns.

• It has standalone --scan and --scan-history operations that can be run ad hoc.

Setup

Git-secrets can be installed globally on macOS using Homebrew - brew install git-secrets. Once the program is installed, you will have a new git secrets command available. Use it to set up git hooks in any repository. To do this, run git secrets --install from the root of a repository. At this point, you still need to tell git-secrets what words or patterns to look for.

• To add a regular expression: git secrets --add <my-prohibited-pattern>

• To add a string literal: git secrets --add --literal <my-prohibited-pattern>

Git-secrets will create an entry in the .git/config file local to the repository. It will use this local configuration to check for secret words within that repository. You can pass the --global flag to make additions that will run in every repo that has the proper hooks in place.

• To make a global addition: git secrets --add --global <my-prohibited-word>

Global additions will be added to your ~/.gitconfig file.

Providers

One problem with adding prohibited patterns directly to your global git config is that it puts sensitive information in a file that you might want to remain public. Lots of developers keep repos for their personal dotfiles, and many of us like to share our setup with the world. That usually means keeping dotfiles in a public repo. Adding private data to our global configuration is not a good option. To handle this, git-secrets uses something called a provider. A provider can be any command that writes to stdout. Each line written will be evaluated as a pattern for git-secrets to check during its scanning process. Assuming you have a file with your secret words at path/to/my/secrets. You could then add a provider using egrep like so... git secrets --add-provider --global -- egrep -v “(^#|^$)” path/to/my/secrets. The egrep command will ignore blank lines or those that begin with #.

If we have many secrets, we can take that a step further. Assume we have a directory of files, each with a specific grouping of patterns. This would allow us to keep our many secrets organized and more easily maintainable. Remember, any command that writes to stdout will work as a provider. A Node.js executable using process.stdout.write or console.log will work just fine. I certainly do a lot more JavaScripting than I do shell scripting, so I feel a lot more comfortable with the following approach. Here’s an example using JavaScript that reads a directory of files and outputs all the pertinent lines. To add this provider, the command would be git secrets --add-provider --global -- node path/to/provider.js, and the provider.js file looks something like this:

# !/usr/bin/env node

const path = require('path');
const { readdirSync, readFile } = require('fs');
const SECRETS = 'path/to/my/secrets/dir';

Promise.all( readdirSync(SECRETS).map(read) )
.then(removeBlanksAndComments)
.then(output)
.catch(console.error);

function read(filename) {
  return new Promise((resolve, reject) => {
    readFile(path.resolve(SECRETS, filename), 'utf-8', (err, contents) => {
      if (err) reject(err);
      resolve(contents);
    });
  });
}

function removeBlanksAndComments(files) {
  const clean = file =>
    file.split('\n')
    .filter(line => !/(^#|^$)/.test(line))
    .join('\n');

  return files.map(clean);
}

function output(secrets) {
  secrets.forEach(shh => {
    console.log(shh);
  });
}

Providers are a powerful feature that allow you the flexibility to provide secrets with simple programs like cat, complex programs like awk, or even a script written in your language of choice.

Further Automation

Git-secrets can only automate so much for us. We still need to make sure we run the --install operation in every repository in which we want to add hooks. This is another possible point of failure for us forgetful humans, so there are a couple things we can do to help limit the risk of forgetting. Git allows us to set up templates for our repos. Here’s how:

1. Make a directory for the template: mkdir ~/.git-template

2. Install the hooks in the template directory: git secrets --install ~/.git-template

3. Tell git to use it: git config --global init.templateDir ’~/.git-template’

Voila. Now every time you run git init or git clone, your hooks will be copied into the .git directory of your freshly created repo. If you don’t want to set the template globally, you can use it as needed with git init --template ’~/.git-template’.

That covers new repo creation, and cloning, but we haven’t addressed the problem of existing repos that weren’t created with the template. Here we have a couple options:

1. git init is a non-destructive operation, so feel free to run it in existing repos. It’s safe, and will retroactively apply the template you specify. OR

2. If you want to go “all in” and ensure that every repo has the proper hooks, here’s a script that will recursively walk a directory, such as ~/Projects and run git secrets --install in all repos.

Be Responsible

Git-secrets is a great tool that can help guard you from all-too-possible slip-ups and better secure your git workflow. However, it’s important to remember there are no silver bullets. As developers, we are often trusted with private information. It’s our responsibility to keep our repos clean and handle our clients’—as well as our personal—data with care.