Type System¶

DataJoint's type system provides a three-layer architecture that balances database efficiency with Python convenience.

Three-Layer Architecture¶

graph TB
    subgraph "Layer 3: Codecs"
        blob["‹blob›"]
        blob_at["‹blob@›"]
        attach["‹attach›"]
        attach_at["‹attach@›"]
        npy["‹npy@›"]
        object["‹object@›"]
        filepath["‹filepath@›"]
        hash["‹hash@›"]
        plugin["‹plugin›"]
    end
    subgraph "Layer 2: Core Types"
        int32
        float64
        varchar
        json
        bytes
        uuid
    end
    subgraph "Layer 1: Native Types (MySQL / PostgreSQL)"
        INT["INT / INTEGER"]
        DOUBLE["DOUBLE / DOUBLE PRECISION"]
        VARCHAR_N["VARCHAR"]
        JSON_N["JSON"]
        BYTES_N["LONGBLOB / BYTEA"]
        UUID_N["BINARY(16) / UUID"]
    end

    blob --> bytes
    blob_at --> hash
    attach --> bytes
    attach_at --> hash
    hash --> json
    npy --> json
    object --> json
    filepath --> json

    bytes --> BYTES_N
    json --> JSON_N
    int32 --> INT
    float64 --> DOUBLE
    varchar --> VARCHAR_N
    uuid --> UUID_N

Core types provide portability — the same table definition works on both MySQL and PostgreSQL. For example, bytes maps to LONGBLOB on MySQL but BYTEA on PostgreSQL; uuid maps to BINARY(16) on MySQL but native UUID on PostgreSQL. Native types can be used directly but sacrifice cross-backend compatibility.

Layer 1: Native Database Types¶

Backend-specific types. Can be used directly at the cost of portability.

# Native types — work but not portable
column : TINYINT UNSIGNED   # MySQL only
column : MEDIUMBLOB         # MySQL only (use BYTEA on PostgreSQL)
column : SERIAL             # PostgreSQL only

Layer 2: Core DataJoint Types¶

Standardized, scientist-friendly types that work identically across backends.

Numeric Types¶

Unsigned integer types are not provided. Choose a signed type with sufficient range for your data.

String Types¶

Other Types¶

Layer 3: Codec Types¶

Codecs provide encode()/decode() semantics for complex Python objects.

Syntax¶

Codec types use angle bracket notation:

• Angle brackets: <blob>, <attach>, <object@store>
• @ indicates object store: <blob@> stores in the object store
• Store name: <blob@cold> uses named store "cold"

Built-in Codecs¶

Plugin Codecs¶

Additional schema-addressed codecs are available as separately installed packages. This ecosystem is actively expanding—new codecs are added as community needs arise.

Installation and discovery:

Plugin codecs use Python's entry point mechanism for automatic registration. Install the package and DataJoint discovers the codec automatically:

pip install dj-zarr-codecs

import datajoint as dj

# Codec is available immediately after install
@schema
class Analysis(dj.Computed):
    definition = """
    -> Recording
    ---
    data : <zarr@store>
    """

Packages declare their codecs in pyproject.toml under the datajoint.codecs entry point group:

[project.entry-points."datajoint.codecs"]
zarr = "dj_zarr_codecs:ZarrCodec"

DataJoint loads these entry points on first use, making third-party codecs indistinguishable from built-ins.

<blob> — Serialized Python Objects¶

Stores NumPy arrays, dicts, lists, and other Python objects using DataJoint's custom binary serialization format. The blob type has been in continuous use in DataJoint pipelines for 15+ years and maintains full backward compatibility. It provides an efficient way to serialize complex objects into an opaque binary string.

Serialization format:

• Protocol headers:
  • mYm — MATLAB-compatible format (see mYm on MATLAB FileExchange and mym on GitHub)
  • dj0 — Python-extended format supporting additional types
• Optional compression: zlib compression for data > 1KB
• Type-specific encoding: Each Python type has a specific serialization code
• Version detection: Protocol header embedded in blob enables format detection

Supported types:

• NumPy arrays (numeric, structured, recarrays)
• Collections (dict, list, tuple, set)
• Scalars (int, float, bool, complex, str, bytes)
• Date/time objects (datetime, date, time)
• UUID, Decimal

class Results(dj.Computed):
    definition = """
    -> Analysis
    ---
    spike_times : <blob>        # In database
    waveforms : <blob@>         # Object store, default store
    raw_data : <blob@archive>   # Object store, 'archive' store
    """

Storage modes:

• <blob> — Stored in database as LONGBLOB (up to ~1GB depending on MySQL config)
• <blob@> — Stored in object store via <hash@> with MD5 deduplication

<attach> — File Attachments¶

Stores files with filename preserved.

class Config(dj.Manual):
    definition = """
    config_id : int
    ---
    settings : <attach>         # Small config file in database
    data_file : <attach@>       # Large file in object store
    """

<npy@> — NumPy Arrays as .npy Files¶

Schema-addressed storage for NumPy arrays as standard .npy files. Returns NpyRef which provides metadata access (shape, dtype) without downloading.

class Recording(dj.Computed):
    definition = """
    -> Session
    ---
    waveform : <npy@>           # Default store
    spectrogram : <npy@archive> # Named store
    """

Lazy access:

ref = (Recording & key).fetch1('waveform')
ref.shape   # (1000, 32) — no download
ref.dtype   # float64 — no download

# Explicit load
arr = ref.load()

# Transparent numpy integration
result = np.mean(ref)  # Downloads automatically

Key features:

• Portable format: Standard .npy readable by NumPy, MATLAB, etc.
• Lazy loading: Shape/dtype available without I/O
• Safe bulk fetch: Fetching many rows doesn't download until needed
• Memory mapping: ref.load(mmap_mode='r') for random access to large arrays

<object@> — Schema-Addressed Storage¶

Schema-addressed storage for files and folders. Path mirrors the database structure: {schema}/{table}/{pk}/{attribute}.

class ProcessedData(dj.Computed):
    definition = """
    -> Recording
    ---
    results : <object@>       # Stored at {schema}/{table}/{pk}/results/
    """

Accepts files, folders, or bytes. Returns ObjectRef for lazy access.

<filepath@store> — Portable References¶

References to independently-managed files with portable paths.

class RawData(dj.Manual):
    definition = """
    session_id : int
    ---
    recording : <filepath@raw>  # Relative to 'raw' store
    """

Storage Modes¶

Object store codecs use one of two addressing schemes:

Hash-addressed — Path derived from content hash (e.g., _hash/ab/cd/abcd1234...). Provides automatic deduplication—identical content stored once. Used by <blob@>, <attach@>, <hash@>.

Schema-addressed — Path mirrors database structure: {schema}/{table}/{pk}/{attribute}. Human-readable, browsable paths that reflect your data organization. No deduplication. Used by <object@>, <npy@>, and plugin codecs (<zarr@>, <figpack@>, <photon@>).

Custom Codecs¶

Extend the type system for domain-specific data:

class GraphCodec(dj.Codec):
    """Store NetworkX graphs."""
    name = "graph"

    def get_dtype(self, is_store):
        return "<blob>"

    def encode(self, graph, *, key=None, store_name=None):
        return {
            'nodes': list(graph.nodes()),
            'edges': list(graph.edges())
        }

    def decode(self, stored, *, key=None):
        import networkx as nx
        G = nx.Graph()
        G.add_nodes_from(stored['nodes'])
        G.add_edges_from(stored['edges'])
        return G

Usage:

class Network(dj.Computed):
    definition = """
    -> Analysis
    ---
    connectivity : <graph>
    """

Choosing Types¶

Summary¶

1. Core types for simple data — int32, varchar, datetime
2. <blob> for Python objects — NumPy arrays, dicts
3. @ suffix for object store — <blob@>, <object@>
4. Custom codecs for domain-specific types

See Also¶

How-to Guides:

• Choose a Storage Type — Decision guide for selecting the right type
• Configure Object Storage — Setting up stores for external data
• Use Object Storage — Working with <blob@>, <attach@>, <object@>
• Use the npy Codec — Storing NumPy arrays as .npy files
• Use Plugin Codecs — Installing and using third-party codecs
• Create a Custom Codec — Building your own codec

Reference:

• Type System Specification — Complete type reference
• Codec API — Codec class interface
• npy Codec Specification — NpyRef and NpyCodec details
• Object Store Configuration — Store settings reference