DataJoint U24 - Workflow Miniscope¶

Workflow Automation¶

In the previous notebook 03-Process, we ran through the workflow in detailed steps, manually adding each. The current notebook provides a more automated approach.

The commands here run a workflow using example data from the 00-DownloadData notebook, but note where placeholders could be changed for a different dataset.

In [1]:

                
                    Copied!
                    
                        
                        
                    
                    

            
import os
from pathlib import Path

# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd()) == "notebooks":
    os.chdir("..")
from workflow_miniscope.pipeline import session, miniscope
from workflow_miniscope import process
import os
from pathlib import Path

# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd()) == "notebooks":
    os.chdir("..")
from workflow_miniscope.pipeline import session, miniscope
from workflow_miniscope import process

[2023-01-05 14:40:41,466][WARNING]: lab.Project and related tables will be removed in a future version of Element Lab. Please use the project schema.
[2023-01-05 14:40:41,529][INFO]: Connecting cbroz@dss-db.datajoint.io:3306
[2023-01-05 14:40:41,893][INFO]: Connected cbroz@dss-db.datajoint.io:3306

We'll be using the process.py's run function automatically loop through all make functions, as a shortcut for calling each individually.

If you previously completed the 03-Process notebook, you may want to delete the contents ingested there, to avoid duplication errors.

In [3]:

                
                    Copied!
                    
                        
                        
                    
                    

            
safemode = True  # Set to false to turn off confirmation prompts
(session.Session & 'subject="subject1"').delete(safemode=safemode)
table_list = [
    miniscope.RecordingInfo,
    miniscope.Processing,
    miniscope.MotionCorrection,
    miniscope.Segmentation,
    miniscope.Fluorescence,
    miniscope.Activity,
]
for table in table_list:
    table.delete(safemode=safemode)
safemode = True  # Set to false to turn off confirmation prompts
(session.Session & 'subject="subject1"').delete(safemode=safemode)
table_list = [
    miniscope.RecordingInfo,
    miniscope.Processing,
    miniscope.MotionCorrection,
    miniscope.Segmentation,
    miniscope.Fluorescence,
    miniscope.Activity,
]
for table in table_list:
    table.delete(safemode=safemode)

[2023-01-05 14:42:32,975][INFO]: Deleting 0 rows from `u24_mini_session`.`session`
[2023-01-05 14:42:33,378][INFO]: Deleting 0 rows from `u24_mini_miniscope`.`_recording_info`
[2023-01-05 14:42:33,783][INFO]: Deleting 0 rows from `u24_mini_miniscope`.`__processing`
[2023-01-05 14:42:34,184][INFO]: Deleting 0 rows from `u24_mini_miniscope`.`_motion_correction`
[2023-01-05 14:42:34,586][INFO]: Deleting 0 rows from `u24_mini_miniscope`.`__segmentation`
[2023-01-05 14:42:34,988][INFO]: Deleting 0 rows from `u24_mini_miniscope`.`__fluorescence`
[2023-01-05 14:42:35,396][INFO]: Deleting 0 rows from `u24_mini_miniscope`.`__activity`

Ingestion of subjects, sessions¶

Refer to the user_data folder in the workflow. Fill subject and session information in files subjects.csv and sessions.csv. We can then use corresponding functions below to automatically ingest subject and session metadata.

In [4]:

                
                    Copied!
                    
from workflow_miniscope.ingest import ingest_subjects, ingest_sessions

ingest_subjects()
ingest_sessions()
from workflow_miniscope.ingest import ingest_subjects, ingest_sessions

ingest_subjects()
ingest_sessions()

[2023-01-05 14:43:22,668][INFO]: 
---- Inserting 0 entry(s) into Subject ----
[2023-01-05 14:43:22,669][INFO]: ---- Insert new `Session` and `Recording` ----
[2023-01-05 14:43:22,834][INFO]: ---- Inserting 0 entry(s) into reference.Device ----
[2023-01-05 14:43:23,128][INFO]: ---- Inserting 1 entry(s) into session.Session ----
[2023-01-05 14:43:23,377][INFO]: ---- Inserting 1 entry(s) into miniscope.Recording ----
[2023-01-05 14:43:23,379][INFO]: ---- Successfully completed ingest_sessions ----

Insert new ProcessingParamSet for CaImAn¶

This is not needed if you are using an existing ProcessingParamSet.

In [7]:

                
                    Copied!
                    
                        
                        
                    
                    

            
params_caiman = dict(
    decay_time=0.4,
    pw_rigid=False,
    max_shifts=(5, 5),
    gSig_filt=(3, 3),
    strides=(48, 48),
    overlaps=(24, 24),
    max_deviation_rigid=3,
    border_nan="copy",
    method_init="corr_pnr",
    K=None,
    gSig=(3, 3),
    gSiz=(13, 13),
    merge_thr=0.7,
    p=1,
    tsub=2,
    ssub=1,
    rf=40,
    stride=20,
    only_init=True,
    nb=0,
    nb_patch=0,
    method_deconvolution="oasis",
    low_rank_background=None,
    update_background_components=True,
    min_corr=0.8,
    min_pnr=10,
    normalize_init=False,
    center_psf=True,
    ssub_B=2,
    ring_size_factor=1.4,
    del_duplicates=True,
    border_pix=0,
    min_SNR=3,
    rval_thr=0.85,
    use_cnn=False,
)

params_dict = dict(
    processing_method="caiman",
    paramset_id=0,  # Change ID if changing parameters
    paramset_desc="Calcium imaging analysis with CaImAn using default parameters",
    params=params_caiman,
)

miniscope.ProcessingParamSet.insert_new_params(**params_dict)
params_caiman = dict(
    decay_time=0.4,
    pw_rigid=False,
    max_shifts=(5, 5),
    gSig_filt=(3, 3),
    strides=(48, 48),
    overlaps=(24, 24),
    max_deviation_rigid=3,
    border_nan="copy",
    method_init="corr_pnr",
    K=None,
    gSig=(3, 3),
    gSiz=(13, 13),
    merge_thr=0.7,
    p=1,
    tsub=2,
    ssub=1,
    rf=40,
    stride=20,
    only_init=True,
    nb=0,
    nb_patch=0,
    method_deconvolution="oasis",
    low_rank_background=None,
    update_background_components=True,
    min_corr=0.8,
    min_pnr=10,
    normalize_init=False,
    center_psf=True,
    ssub_B=2,
    ring_size_factor=1.4,
    del_duplicates=True,
    border_pix=0,
    min_SNR=3,
    rval_thr=0.85,
    use_cnn=False,
)

params_dict = dict(
    processing_method="caiman",
    paramset_id=0,  # Change ID if changing parameters
    paramset_desc="Calcium imaging analysis with CaImAn using default parameters",
    params=params_caiman,
)

miniscope.ProcessingParamSet.insert_new_params(**params_dict)

Trigger autoprocessing of the remaining calcium imaging workflow¶

The process.run() function in the workflow populates every auto-processing table in the workflow. If a table is dependent on a manual table upstream, it will not get populated until the manual table is inserted.
At this stage, process script populates through the table upstream of ProcessingTask (i.e. RecordingInfo)

In [5]:

                
                    Copied!
                    
process.run()
process.run()

[2023-01-05 14:44:40,146][INFO]: ---- Populating RecordingInfo ----
RecordingInfo: 100%|██████████| 1/1 [00:00<00:00,  1.05it/s]
[2023-01-05 14:44:41,759][INFO]: ---- Populating Processing ----
[2023-01-05 14:44:42,074][INFO]: ---- Populating MotionCorrection ----
[2023-01-05 14:44:42,366][INFO]: ---- Populating Segmentation ----
[2023-01-05 14:44:42,666][INFO]: ---- Populating Fluorescence ----
[2023-01-05 14:44:42,953][INFO]: ---- Populating Activity ----
[2023-01-05 14:44:43,246][INFO]: ---- Successfully completed miniscope/populate.py ----

We can then add a processing task as a combination of a scan key, processing parameters, and an output directory.

In [8]:

                
                    Copied!
                    
                        
                        
                    
                    

            
from element_interface.utils import find_full_path
from workflow_miniscope.pipeline import get_miniscope_root_data_dir

scan_key = (session.Session * miniscope.Recording).fetch("KEY", limit=1)[0]

scan_file = find_full_path(
    get_miniscope_root_data_dir(),
    (miniscope.RecordingInfo.File & scan_key).fetch("file_path", limit=1)[0],
)
caiman_dir = Path(scan_file.parent / "caiman")
miniscope.ProcessingTask.insert1(
    {**scan_key, "paramset_id": 0, "processing_output_dir": caiman_dir}
)
from element_interface.utils import find_full_path
from workflow_miniscope.pipeline import get_miniscope_root_data_dir

scan_key = (session.Session * miniscope.Recording).fetch("KEY", limit=1)[0]

scan_file = find_full_path(
    get_miniscope_root_data_dir(),
    (miniscope.RecordingInfo.File & scan_key).fetch("file_path", limit=1)[0],
)
caiman_dir = Path(scan_file.parent / "caiman")
miniscope.ProcessingTask.insert1(
    {**scan_key, "paramset_id": 0, "processing_output_dir": caiman_dir}
)

And trigger the processing

In [9]:

                
                    Copied!
                    
process.run()
process.run()

[2023-01-05 14:52:27,650][INFO]: ---- Populating RecordingInfo ----
[2023-01-05 14:52:27,819][INFO]: ---- Populating Processing ----
Processing: 100%|██████████| 1/1 [00:06<00:00,  6.69s/it]
[2023-01-05 14:52:34,673][INFO]: ---- Populating MotionCorrection ----
INFO:datajoint:---- Populating MotionCorrection ----
[2023-01-05 14:52:34,835][INFO]: ---- Populating Segmentation ----
INFO:datajoint:---- Populating Segmentation ----
[2023-01-05 14:52:34,995][INFO]: ---- Populating Fluorescence ----
INFO:datajoint:---- Populating Fluorescence ----
[2023-01-05 14:52:35,159][INFO]: ---- Populating Activity ----
INFO:datajoint:---- Populating Activity ----
[2023-01-05 14:52:35,201][INFO]: ---- Successfully completed miniscope/populate.py ----
INFO:datajoint:---- Successfully completed miniscope/populate.py ----

Next, we would select one of the results at the curation table.

In [12]:

                
                    Copied!
                    
key = miniscope.Processing.fetch("KEY")[0]
miniscope.Curation.create1_from_processing_task(key)
key = miniscope.Processing.fetch("KEY")[0]
miniscope.Curation.create1_from_processing_task(key)

And then we can continue processing.

In [13]:

                
                    Copied!
                    
process.run()
process.run()

[2023-01-05 14:57:55,980][INFO]: ---- Populating RecordingInfo ----
INFO:datajoint:---- Populating RecordingInfo ----
[2023-01-05 14:57:56,147][INFO]: ---- Populating Processing ----
INFO:datajoint:---- Populating Processing ----
[2023-01-05 14:57:56,314][INFO]: ---- Populating MotionCorrection ----
INFO:datajoint:---- Populating MotionCorrection ----
MotionCorrection: 100%|██████████| 1/1 [00:01<00:00,  1.65s/it]
[2023-01-05 14:57:58,128][INFO]: ---- Populating Segmentation ----
INFO:datajoint:---- Populating Segmentation ----
Segmentation:   0%|          | 0/1 [00:00<?, ?it/s][2023-01-05 14:57:58,721][WARNING]: Could not load all pixel values for at least one mask
WARNING:datajoint:Could not load all pixel values for at least one mask
Segmentation: 100%|██████████| 1/1 [00:01<00:00,  1.10s/it]
[2023-01-05 14:57:59,392][INFO]: ---- Populating Fluorescence ----
INFO:datajoint:---- Populating Fluorescence ----
Fluorescence: 100%|██████████| 1/1 [00:01<00:00,  1.15s/it]
[2023-01-05 14:58:00,711][INFO]: ---- Populating Activity ----
INFO:datajoint:---- Populating Activity ----
Activity: 100%|██████████| 2/2 [00:01<00:00,  1.24it/s]
[2023-01-05 14:58:02,362][INFO]: ---- Successfully completed miniscope/populate.py ----
INFO:datajoint:---- Successfully completed miniscope/populate.py ----

Summary and next step¶

This notebook runs through the workflow in an automatic manner.
The next notebook 05-Explore discussed the role of each table in more depth.