Introduction to the workflow structure¶
This notebook gives a brief overview of the workflow structure and introduces some useful DataJoint tools to facilitate the exploration.
DataJoint needs to be pre-configured before running this notebook, if you haven't set up the configuration, refer to notebook 01-configure.
If you are familiar with DataJoint and the workflow structure, proceed directly to the next notebook 03-process to run the workflow.
For a more thorough introduction of DataJoint functions, please visit our general tutorial site - DataJoint CodeBook.
To load the local configuration, we will change the directory to the package root.
import os
os.chdir("..")
Schemas and tables¶
- The current workflow is composed of multiple database schemas, each of them corresponds to a module within
workflow_miniscope.pipeline
import datajoint as dj
from workflow_miniscope.pipeline import lab, subject, session, miniscope
- Each module contains a schema object that enables interaction with the schema in the database.
miniscope.schema
- The table classes in the module corresponds to a table in the schema in the database.
# preview columns and contents in a table
miniscope.Processing()
By importing the modules for the first time, the schemas and tables will be created inside the database.
Once created, importing modules will not create schemas and tables again, but the existing schemas/tables can be accessed and manipulated by the modules.
DataJoint tools to explore schemas and tables¶
dj.list_schemas(): list all schemas a user has access to in the current database
dj.list_schemas()
dj.Diagram(): plot tables and dependencies in a schema.
# plot diagram for all tables in a schema
dj.Diagram(miniscope)
Table tiers:
- Manual table
- Visually represented with a green box.
- Manually inserted table
- Expect new entries daily, e.g. Subject, Recording.
- Lookup table
- Visually represented with a gray box.
- Pre-inserted table
- Commonly used for general facts or parameters. e.g. Strain, ProcessingParamSet.
- Imported table
- Visually represented with a blue oval.
- Auto-processing table
- Processing depends on the importing of external files. e.g.
Processingrequires output files from CaImAn.
- Computed table
- Visually represented with a red circle.
- Auto-processing table
- Processing does not depend on files external to the database.
- Part table
- Visually represented with plain text.
- As an appendix to the master table, all the part entries of a given master entry represent a intact set of the master entry. e.g.
Maskof aSegmentation.
Dependencies:
- One-to-one primary
- Visually represented with a thick solid line.
- Share the exact same primary key, meaning the child table inherits all the primary key fields from the parent table as its own primary key.
- One-to-many primary
- Visually represented with a thin solid line.
- Inherit the primary key from the parent table, but have additional field(s) as part of the primary key as well.
- Secondary dependency
- Visually represented with a dashed line.
- The child table inherits the primary key fields from parent table as its own secondary attribute.
# plot diagram of tables in multiple schemas
dj.Diagram(subject) + dj.Diagram(session) + dj.Diagram(miniscope)
# plot diagram of selected tables and schemas
dj.Diagram(subject.Subject) + dj.Diagram(session.Session) + dj.Diagram(miniscope)
describe(): show table definition with foreign key references.
miniscope.Processing.describe()
heading: show attribute definitions regardless of foreign key references
miniscope.Processing.heading
dj.Diagram(lab)
subject: general animal information, such as User, Genetic background.
dj.Diagram(subject)
subject.Subject.describe()
session: General information of experimental sessions.
dj.Diagram(session)
session.Session.describe()
miniscope: miniscope raw recording and processed data
dj.Diagram(miniscope)
Summary and next step¶
This notebook introduced the overall structures of the schemas and tables in the workflow and relevant tools to explore the schema structure and table definitions.
In the next notebook 03-process, we will introduce the detailed steps to run through the workflow.