Manage Secrets and Credentials¶

Secure configuration management for database credentials, storage access keys, and other sensitive settings.

Overview¶

DataJoint separates configuration into sensitive and non-sensitive components:

Component	Location	Purpose	Version Control
Non-sensitive	`datajoint.json`	Project settings, defaults	✅ Commit to git
Sensitive	`.secrets/` directory	Credentials, API keys	❌ Never commit
Dynamic	Environment variables	CI/CD, production	⚠️ Context-dependent

Configuration Priority¶

DataJoint loads configuration in this priority order (highest to lowest):

Programmatic settings — dj.config['key'] = value
Environment variables — DJ_HOST, DJ_USER, etc.
Secrets directory — .secrets/datajoint.json, .secrets/stores.*
Project configuration — datajoint.json
Default values — Built-in defaults

Higher priority sources override lower ones.

`.secrets/` Directory Structure¶

Create a .secrets/ directory in your project root:

project/
├── datajoint.json               # Non-sensitive settings (commit)
├── .gitignore                   # Must include .secrets/
├── .secrets/
│   ├── datajoint.json           # Database credentials
│   ├── stores.main.access_key   # S3/cloud storage credentials
│   ├── stores.main.secret_key
│   ├── stores.archive.access_key
│   └── stores.archive.secret_key
└── ...

Critical: Add .secrets/ to .gitignore:

# .gitignore
.secrets/

Database Credentials¶

Option 1: Secrets Directory (Recommended for Development)¶

Create .secrets/datajoint.json:

{
  "database.user": "myuser",
  "database.password": "mypassword"
}

Non-sensitive database settings go in datajoint.json:

{
  "database.host": "db.example.com",
  "database.port": 3306,
  "database.use_tls": true,
  "safemode": true
}

Option 2: Environment Variables (Recommended for Production)¶

For CI/CD and production environments:

export DJ_HOST=db.example.com
export DJ_USER=myuser
export DJ_PASS=mypassword
export DJ_PORT=3306
export DJ_TLS=true

Option 3: Programmatic Configuration¶

For scripts and applications:

import datajoint as dj

dj.config['database.host'] = 'localhost'
dj.config['database.user'] = 'myuser'
dj.config['database.password'] = 'mypassword'

Security note: Only use this when credentials come from secure sources (environment, vault, secrets manager).

Object Storage Credentials¶

File Storage (No Credentials)¶

Local or network-mounted file systems don't require credentials:

{
  "stores": {
    "default": "main",
    "main": {
      "protocol": "file",
      "location": "/data/my-project"
    }
  }
}

S3/MinIO Storage (With Credentials)¶

Config in `datajoint.json` (non-sensitive):¶

{
  "stores": {
    "default": "main",
    "main": {
      "protocol": "s3",
      "endpoint": "s3.amazonaws.com",
      "bucket": "my-bucket",
      "location": "my-project/data"
    },
    "archive": {
      "protocol": "s3",
      "endpoint": "s3.amazonaws.com",
      "bucket": "archive-bucket",
      "location": "my-project/archive"
    }
  }
}

Credentials in `.secrets/` directory:¶

Create separate files for each store's credentials:

.secrets/stores.main.access_key
.secrets/stores.main.secret_key
.secrets/stores.archive.access_key
.secrets/stores.archive.secret_key

File format: Plain text, one credential per file:

# .secrets/stores.main.access_key
AKIAIOSFODNN7EXAMPLE

# .secrets/stores.main.secret_key
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Alternative: Environment Variables¶

For cloud deployments:

export DJ_STORES_MAIN_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE
export DJ_STORES_MAIN_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Environment Variable Reference¶

Database Connections¶

Setting	Environment Variable	Description
`database.host`	`DJ_HOST`	Database hostname
`database.port`	`DJ_PORT`	Database port (default: 3306)
`database.user`	`DJ_USER`	Database username
`database.password`	`DJ_PASS`	Database password
`database.use_tls`	`DJ_TLS`	Use TLS encryption (true/false)

Object Stores¶

Pattern	Example	Description
`DJ_STORES_<NAME>_ACCESS_KEY`	`DJ_STORES_MAIN_ACCESS_KEY`	S3 access key ID
`DJ_STORES_<NAME>_SECRET_KEY`	`DJ_STORES_MAIN_SECRET_KEY`	S3 secret access key

Note: <NAME> is the uppercase store name with _ replacing special characters.

Security Best Practices¶

Development Environment¶

# 1. Initialize secrets directory
mkdir -p .secrets
chmod 700 .secrets  # Owner-only access

# 2. Create .gitignore
echo ".secrets/" >> .gitignore

# 3. Store credentials in .secrets/
cat > .secrets/datajoint.json <<EOF
{
  "database.user": "dev_user",
  "database.password": "dev_password"
}
EOF

# 4. Set restrictive permissions
chmod 600 .secrets/datajoint.json

Production Environment¶

# Use environment variables from secure sources
export DJ_USER=$(vault read -field=username secret/datajoint/db)
export DJ_PASS=$(vault read -field=password secret/datajoint/db)
export DJ_STORES_MAIN_ACCESS_KEY=$(vault read -field=access_key secret/datajoint/s3)
export DJ_STORES_MAIN_SECRET_KEY=$(vault read -field=secret_key secret/datajoint/s3)

CI/CD Environment¶

Use your CI system's secrets management:

GitHub Actions:

env:
  DJ_HOST: ${{ secrets.DJ_HOST }}
  DJ_USER: ${{ secrets.DJ_USER }}
  DJ_PASS: ${{ secrets.DJ_PASS }}

GitLab CI:

variables:
  DJ_HOST: $DJ_HOST
  DJ_USER: $DJ_USER
  DJ_PASS: $DJ_PASS

Docker Containers¶

Pass as environment variables:

docker run -e DJ_HOST=db.example.com \
           -e DJ_USER=myuser \
           -e DJ_PASS=mypassword \
           my-datajoint-app

Or mount secrets as read-only volume:

docker run -v $(pwd)/.secrets:/app/.secrets:ro \
           my-datajoint-app

Docker Compose:

services:
  app:
    image: my-datajoint-app
    env_file: .env
    volumes:
      - ./.secrets:/app/.secrets:ro

Cloud Platforms¶

AWS Lambda:

import os
import datajoint as dj

dj.config['database.user'] = os.environ['DJ_USER']
dj.config['database.password'] = os.environ['DJ_PASS']

Google Cloud Functions:

from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()
name = f"projects/{project_id}/secrets/dj-password/versions/latest"
response = client.access_secret_version(request={"name": name})
dj.config['database.password'] = response.payload.data.decode("UTF-8")

Azure Functions:

import os
import datajoint as dj

dj.config['database.user'] = os.environ['DJ_USER']
dj.config['database.password'] = os.environ['DJ_PASS']

Common Patterns¶

Local Development with Secrets Directory¶

import datajoint as dj

# Config loaded automatically from:
# 1. datajoint.json (project settings)
# 2. .secrets/datajoint.json (credentials)
conn = dj.conn()

Production with Environment Variables¶

import os
import datajoint as dj

# All configuration from environment
# No files needed
conn = dj.conn()

Multi-Environment Setup¶

Structure:

project/
├── datajoint.json               # Shared settings
├── .secrets/
│   ├── datajoint.dev.json      # Development credentials
│   ├── datajoint.staging.json  # Staging credentials
│   └── datajoint.prod.json     # Production credentials (if needed)
└── ...

Load by environment:

import os
import datajoint as dj

env = os.getenv('ENV', 'dev')
secrets_path = f'.secrets/datajoint.{env}.json'

# Load environment-specific secrets
import json
with open(secrets_path) as f:
    secrets = json.load(f)
    for key, value in secrets.items():
        dj.config[key] = value

conn = dj.conn()

Troubleshooting¶

Secrets Not Loading¶

Check configuration priority:

import datajoint as dj

# View current configuration
print(dj.config)

# Check specific setting
print(dj.config['database.user'])

# See where value came from
print(dj.config._config_sources)  # Not a real attribute, just conceptual

Permission Errors¶

# Fix .secrets/ permissions
chmod 700 .secrets
chmod 600 .secrets/*

Environment Variables Not Taking Effect¶

import os
import datajoint as dj

# Check environment variables are set
print(os.getenv('DJ_USER'))
print(os.getenv('DJ_PASS'))

# Force reload configuration
dj.config.clear()
conn = dj.conn(reset=True)

Accidentally Committed Secrets¶

Immediate actions:

Rotate credentials immediately
Remove from git history:

# Remove file from history
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch .secrets/datajoint.json" \
  --prune-empty --tag-name-filter cat -- --all

# Force push (coordinate with team!)
git push origin --force --all

Verify removal:

git log --all --full-history -- .secrets/datajoint.json

Configuration Templates¶

Minimal Development Setup¶

// datajoint.json
{
  "database.host": "localhost",
  "safemode": true
}

// .secrets/datajoint.json
{
  "database.user": "root",
  "database.password": "simple"
}

Production with S3 Storage¶

// datajoint.json
{
  "database.host": "db.example.com",
  "database.port": 3306,
  "database.use_tls": true,
  "safemode": false,
  "stores": {
    "default": "main",
    "main": {
      "protocol": "s3",
      "endpoint": "s3.amazonaws.com",
      "bucket": "my-bucket",
      "location": "my-project"
    }
  }
}

// .secrets/ directory
.secrets/datajoint.json                 # Database credentials
.secrets/stores.main.access_key         # S3 access key
.secrets/stores.main.secret_key         # S3 secret key