Instrument preprocessing

After a watcher uploads a raw file, Data Hub may run an instrument-specific preprocessor on the server. This page lists what each supported type expects, what metadata and files it writes, and what appears on run detail.

How preprocessing works

The server-side processor runs in the AWS Lambda that handles uploads to the raw storage bucket. For the full pipeline, see Architecture.

When a processor exists for the instrument's type:

1. A file upload to raw storage triggers the Lambda, or you click Reprocess on a file or run.
2. The processor downloads the file, extracts metadata, writes processed files when needed, then updates the run through the API.
3. The file status becomes processing, then completed or failed.

Set the instrument type from the instrument page: open the three-dot menu, choose Edit, and pick a supported type. That choice selects which processor runs. New instrument types need a code change; see Adding a new instrument in the developer docs.

Supported instruments at a glance

Use this table to find an instrument ID and what its processor produces. Details for each type follow.

Agilent 4150 TapeStation

Parses tape type from peak-table CSV filenames and shows TapeStation reports on run detail.

• Expected files: CSVs such as YYYY-MM-DD - HH-MM-SS-<TapeType>_peakTable.csv, plus matching PDFs when present
• Metadata: Tape Type (for example gDNA, HSD1000) when the CSV name matches the pattern
• Processed artifacts: none
• Run details page: tape-type badge, PDF carousel when PDFs exist, peak-table reports for CSVs

Azure 600 Gel Doc

The processor loads each gel TIFF and stretches intensities to the 1st to 99th percentile (per RGB channel, or as grayscale). It renders a figure of every TIFF page side by side and saves that as a 300 dpi .png.

It also reads imaging settings from the TIFF XPComment tag: capture_type, imaging_mode (true color, fluorescence, or chemiluminescence), wavelengths, and colors. Those fields drive run badges and filters on the instrument page.

• Expected files: .tif / .tiff from the Azure 600
• Metadata: capture_type, imaging_mode, wavelengths, colors
• Processed artifacts: contrast-enhanced .png (category: processed)
• Run details page: imaging badges, files table, image carousel of processed PNGs

Azure Cielo qPCR

Reads dye channel names from Cq Values CSVs so you can filter and badge runs by channel.

• Expected files: Cq Values .csv (other files in the run still upload; only CSVs contribute dye metadata)
• Metadata: dye_channels (list of unique names from the Fluorescence column)
• Processed artifacts: none
• Run details page: dye-channel badges and report tables for qualifying files

Epson V700 Scanner

The processor loads each scan TIFF and looks for agar plates inside gold 3D-printed frames. When it finds plates, it crops each one and runs colony detection: edge crop, contrast against the plate background, a difference-of-Gaussians filter, thresholding, then measurement of surviving colonies in physical units (area, centroid, eccentricity, mean RGB).

It writes a web-sized .jpg with plate outlines and colony boxes overlaid, and a _colonies.csv of those measurements when the pipeline ran. If no plates are found, it still exports a resized full-scan .jpg and skips colony detection. Run metadata includes TIFF tags, dpi, color_mode, plate_count, plate_boxes, and colony_detection summaries when colonies were measured.

• Expected files: .tif / .tiff plate scans
• Metadata: TIFF-derived fields such as dpi, color_mode, plate_count, plate_boxes, plus colony_detection summaries when colonies are found
• Processed artifacts: preview .jpg; when plates are detected, a _colonies.csv with colony measurements (both category: processed)
• Run details page: scanner badges, files table, and report section for processed CSVs and related files

Hina Microscope

The processor loads each Nikon .nd2 with Arcadia's microscopy tools, reduces every channel to a single 2D frame (max-projection over Z; first index over time or position), stretches intensities to the 1st to 99th percentile, and composites fluorescence onto a brightfield (or fallback) background using each channel's fluorophore color.

It uploads that RGB overlay as a .jpg for every file in the run. Channel layout is the same across a Hina run, so sizes, channels (name, excitation/emission nm, color), and dimensions are written once: the first file to finish processing sets run metadata; later files still get a JPG but skip the metadata update.

• Expected files: .nd2
• Metadata (written on the first file that arrives for the run): sizes, channels (name, excitation/emission nm, color), dimensions
• Processed artifacts: .jpg overlay (category: processed)
• Run details page: channel/size badges, files table, image carousel of processed JPGs

SpectraMax iD3 and iD5 plate readers

Both IDs share one processor. It turns SoftMax Pro .xls exports into a well-data CSV and measurement metadata for plate maps.

• Expected files: SoftMax .xls (UTF-16 text with a Plate: header)
• Metadata: measurement_mode (for example Absorbance), measurement_type (for example Endpoint, Kinetic), wavelengths (numeric strings without an nm suffix)
• Processed artifacts: {run_id}_raw_well_data.csv (category: processed)
• Run details page: wavelength/mode/type badges and interactive plate maps from the processed CSV

Instruments without a matching processor

If the instrument type is unset, generic, or otherwise has no preprocessor, Data Hub still stores raw uploads and lets you browse, claim, comment, and download. You do not get automatic metadata or processed artifacts until a processor exists and the type is set.

When processing fails

Open the run, find the file with status failed, and use Reprocess (or reprocess the whole run when that action is available). Details are in Browse and analyze runs.