Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions docs/deployments/2-bm/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ the power is the site's, neither is cut to 2-BM's shape.
Beyond the physical utilities, a run also draws on a compute pool (for reconstruction) and on data-transfer and
storage tiers. These are modelled through the `ComputePort` and `TransferPort` (a Method plus a port, not a new
deployment aggregate). The confirmed pipeline (DATA-1 through DATA-7): the detector writes to fast local NVMe on
`tomdet` (`/local1`), tomoscan auto-uploads each scan to the analysis tier (`/data2` or `/data3`), tomocupy
`tomdet` (`/local2`, corrected 2026-08-11 by a direct read of `DetectorTopDir`; this page previously said
`/local1`), tomoscan auto-uploads each scan to the analysis tier (`/data2` or `/data3`), tomocupy
reconstructs there (`..._rec/` beside the raw), and an operator copies the experiment to its canonical home on
Sojourner (`/gdata/dm/2BM/<yyyy-mm>/<exp>/{data,analysis,system}/`), shared to proposal and ESAF users through
the Globus collection `APS:DM:2BM` and archived to tape on a per-experiment timer (default one year). The
Expand All @@ -57,6 +58,12 @@ Read from the upstream source rather than assumed: [tomoscan](https://github.com
2-BM subclass and [dmagic](https://github.com/decarlof/dmagic). Every scan product is HDF5; no
acquisition path writes TIFF.

Confirmed 2026-08-11 against the live 2bmb deployment (DATA-9): the deployed `tomoscan_2bm.py`
and its siblings (`tomoscan_2bm_step.py`, `tomoscan_stream_2bm.py`, `tomoscan_fpga_2bm.py`, both
`.template` files) are byte-identical to `decarlof/tomoscan@ce86818`. The shared `tomoscan.py` base
class carries one uncommitted local edit, a camera-readout-timing tweak for one pixel format, nowhere
near `add_theta()` or the `ScanStatus` literals below.

The layout is Data Exchange. `tomoscan_2bm.py` addresses the datasets by name when it post-processes a
finished scan: `/exchange/data` (projections), `/exchange/data_white` (flats), `/exchange/data_dark`
(darks), and `/exchange/theta` (rotation angles). Frame bookkeeping lives in `/defaults/NDArrayUniqueId`
Expand All @@ -66,13 +73,18 @@ scan basename carries a three-digit counter, and `..._rec/` reconstruction direc
The file has a second author besides tomoscan: the areaDetector layout XML configured in the detector
IOC writes everything under `/process`, `/measurement`, and `/defaults` (public copy:
[`data-exchange/dxfile/doc/demo/areadetector/2-BM/`](https://github.com/data-exchange/dxfile/tree/master/doc/demo/areadetector/2-BM)).
Two of its facts matter to a reader. The commanded scan geometry
(`/process/acquisition/rotation/num_angles`, the flat and dark field mode-and-count groups) is written
`OnFileClose`, so it exists only in cleanly closed files, and it is what a shortfall check compares
captured frames against; the `/defaults` frame ids alone can never show tail truncation, because a
missed trigger never receives an id. The acquisition timestamp (`/process/acquisition/start_date`, from
the PV `S:IOC:timeOfDayISO8601`) is written `OnFileOpen`, so a crashed file keeps its timestamp while
losing its geometry.
The deployed file has no name match in that public tree (it lives on `tomdet` as `TomoScanLayout.xml`
at a beamline-local path, not one of the `2bma*` / `adimec2bmb*` names dxfile ships), so DATA-9 was
confirmed by reading its content on 2026-08-11 rather than by matching a filename: it writes the same
dataset paths and lifecycle timing this section assumes. Two of its facts matter to a reader. The
commanded scan geometry (`/process/acquisition/rotation/num_angles`, the flat and dark field
mode-and-count groups) is written `OnFileClose`, so it exists only in cleanly closed files, and it is
what a shortfall check compares captured frames against; the `/defaults` frame ids alone can never show
tail truncation, because a missed trigger never receives an id. The acquisition timestamp
(`/process/acquisition/start_date`, from the PV `S:IOC:timeOfDayISO8601`) is written `OnFileOpen`, so a
crashed file keeps its timestamp while losing its geometry. That PV carries an explicit UTC offset
(confirmed 2026-08-11, DATA-10: `2026-08-11T12:01:05-0500`), not naive local time, so a reader parses it
as an unambiguous instant with no site timezone rule needed.

**A finished capture is not a finished file, and this is the fact an ingest reader must respect.** The
end-of-scan sequence stops the file plugin (`FPCapture` to `Done`, then waits for `Capture_RBV` to reach
Expand All @@ -88,6 +100,10 @@ rather than trust the status message.
`add_theta()` also compares the frames actually written against the angles commanded, and logs a warning
naming the missing ones when they disagree. Dropped frames are therefore a known, detected, and
non-fatal condition: a reader that records only what landed will silently under-describe such a scan.
A genuine instance was found 2026-08-11 reading a real file from the beamline (`test_000.h5`, an early
smoke test rather than a production scan): 3601 angles commanded, one frame captured, `theta` absent.
`DataExchangeScanReader` handled it exactly as designed; DATA-8's frequency question (rare-and-alarming
versus routine for an actual production scan) is still open.

The experiment folder is computed, not conventional, which is what makes it derivable rather than
guessable. `dmagic`'s `dm.py` formats it as `{year_month}-{pi_last_name}-{gup_number}` from APS
Expand Down
15 changes: 8 additions & 7 deletions docs/deployments/2-bm/questions.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,14 +116,14 @@ Confirmed design basis (SCHED-1/2/3a, #269): a scheduled beamtime window is **im
The storage chain was settled by DATA-1 through DATA-7 (#270). What those answers did not cover, the file
itself, is now read from the upstream source instead of asked: tomoscan's 2-BM subclass and dmagic give the
Data Exchange layout, the file naming template, the end-of-scan ordering, and the experiment-folder
derivation. Those are recorded in [Operations](operations.md#inside-the-scan-file). Only what the source
cannot answer is left here.
derivation. Those are recorded in [Operations](operations.md#inside-the-scan-file). DATA-9 and DATA-10 were
settled the same way, by a direct read against the live 2bmb deployment on 2026-08-11 rather than a staff
reply; see [Operations](operations.md#inside-the-scan-file) for the evidence. Only what neither source can
answer is left here.

| ID | Priority | Question | CORA assumes | Already done? | Resolves |
| --- | --- | --- | --- | --- | --- |
| DATA-8 | `Nice-to-have` | How often do scans finish with dropped frames? `add_theta()` compares written frames against commanded angles and logs a warning when they disagree, so the condition is detected but not fatal. Knowing whether this is rare-and-alarming or routine decides whether a record of the scan should refuse to be written, or carry the shortfall as an ordinary recorded fact. | rare enough to treat as an exception worth surfacing, not a routine outcome to normalise | not yet | [Operations](operations.md) |
| DATA-9 | `Nice-to-have` | Is the tomoscan running at 2-BM the same `tomoscan_2bm.py` as the public repository HEAD, and is the detector IOC's areaDetector layout XML the same as the public copy in `data-exchange/dxfile`? The end-of-scan ordering, the Data Exchange dataset names, and the `/process` metadata paths (commanded angle counts, `start_date`) are all read from those public sources; a local divergence in either file would silently invalidate them. | the deployed code and layout match the public repositories | not yet | [Operations](operations.md) |
| DATA-10 | `Nice-to-have` | Does the timestamp PV `S:IOC:timeOfDayISO8601` (the source of `/process/acquisition/start_date` in every scan file) carry a UTC offset, or is it naive local time? A reader recording when a scan was captured must convert this string to an unambiguous instant; if it is naive, the site timezone rule becomes part of the file contract. | naive local time, US Central with DST | not yet | [Operations](operations.md) |
| DATA-8 | `Nice-to-have` | How often do scans finish with dropped frames? `add_theta()` compares written frames against commanded angles and logs a warning when they disagree, so the condition is detected but not fatal. Knowing whether this is rare-and-alarming or routine decides whether a record of the scan should refuse to be written, or carry the shortfall as an ordinary recorded fact. | rare enough to treat as an exception worth surfacing, not a routine outcome to normalise | not yet (a genuine instance was found 2026-08-11 reading a live 2bmb test file, `test_000.h5`: 3601 angles commanded, 1 frame captured, `theta` absent; `DataExchangeScanReader` handled it exactly as designed, but the file was an early smoke test, not evidence about routine production frequency) | [Operations](operations.md) |

## Where CORA runs

Expand All @@ -138,11 +138,12 @@ right person, or naming who that person is, is a complete answer to any row here

| ID | Priority | Question | CORA assumes | Already done? | Resolves |
| --- | --- | --- | --- | --- | --- |
| HOST-1 | `Blocks-go-live` | Is the host on the same network segment as the beamline IOCs, and if it is not, which Channel Access gateway or address list should it use? EPICS CA finds PVs by UDP broadcast on the local subnet by default, so a host one routed hop away sees nothing and reports it as a timeout rather than an error. Naming the gateway makes it a configuration line; leaving it unstated makes it a day of debugging that looks like a CORA fault. | the host can reach the IOCs, with an explicit address list rather than broadcast discovery | not yet | [Deployment](../../stack/deployment.md) |
| HOST-2 | `Blocks-go-live` | Can the host read the scan files directly, as a mount of the analysis tier (`/data2`, `/data3`) or of the Sojourner experiment tree, and read-only is sufficient? If no mount is possible, what is the supported way for an off-host reader to fetch a finished file? The answer decides whether CORA reads a dataset in place or has to copy it first, which is a different design and not a setting. | a read-only mount of at least one tier holding finished scan files | not yet | [Operations](operations.md#inside-the-scan-file) |
| HOST-1 | `Blocks-go-live` | Is the host on the same network segment as the beamline IOCs, and if it is not, which Channel Access gateway or address list should it use? EPICS CA finds PVs by UDP broadcast on the local subnet by default, so a host one routed hop away sees nothing and reports it as a timeout rather than an error. Naming the gateway makes it a configuration line; leaving it unstated makes it a day of debugging that looks like a CORA fault. | the host can reach the IOCs, with an explicit address list rather than broadcast discovery (confirmed 2026-08-11: `arcturus` itself uses `EPICS_CA_ADDR_LIST=164.54.113.168 10.54.113.24:16661`, not broadcast; whether CORA's host gets this same list, or needs its own, is asked of Francesco pending HOST-6) | not yet | [Deployment](../../stack/deployment.md) |
| HOST-2 | `Blocks-go-live` | Can the host read the scan files directly, as a mount of the analysis tier (`/data2`, `/data3`) or of the Sojourner experiment tree, and read-only is sufficient? If no mount is possible, what is the supported way for an off-host reader to fetch a finished file? The answer decides whether CORA reads a dataset in place or has to copy it first, which is a different design and not a setting. | a read-only mount of at least one tier holding finished scan files (confirmed 2026-08-11: none of `/local1`, `/data2`, `/data3`, `/gdata/dm/2BM` are visible from `arcturus`; finished files land on `tomdet` at `/local2/2BM/<experiment>/`, not `/local1` as Operations previously said; whether CORA's host gets a mount of that tier is asked of Francesco pending HOST-6) | not yet | [Operations](operations.md#inside-the-scan-file) |
| HOST-3 | `Blocks-go-live` | What durable storage can the host write backups to that is not the host's own disk, and does the host's own disk survive the host being lost or rebuilt? A backup written beside the database protects against operator error and corruption and against nothing else. This row also carries a deadline: backup-repository encryption is fixed when the repository is first created and cannot be added afterwards, so the target has to be known before that step, not after. | a facility share or object store is reachable; local disk is an interim posture only | not yet | [Deployment](../../stack/deployment.md) |
| HOST-4 | `Blocks-go-live` | Who needs to reach CORA's web interface, and from where: the beamline network only, anyone on the APS network, remote users over VPN, or remote users without one? This decides whether CORA sits behind an existing APS proxy or brings its own, and whether it needs a certificate and a resolvable name. | beamline and APS-network access, behind a facility-provided proxy that terminates TLS | not yet | [Deployment](../../stack/deployment.md) |
| HOST-5 | `Nice-to-have` | Who administers the host, and does the operating account have rights to install a scheduled system job? CORA needs a timer to run backups and expire old ones. If that is not permitted, the schedule has to live inside the application instead, which is a different and slightly worse design worth choosing deliberately. | beamline-administered, with rights to install a system timer | not yet | [Deployment](../../stack/deployment.md) |
| HOST-6 | `Blocks-go-live` | Which host does CORA actually run on: `arcturus`, `tomdet`, or a separate machine? `arcturus` and `tomdet` do not see the same filesystems or the same PVs (HOST-1, HOST-2), so this decides both. Found 2026-08-11: a `cora` conda env already exists on `tomdet` (Python 3.13.14, matching CORA's pinned version, `uv` installed, a couple of FastAPI-adjacent packages), but nothing is installed in it. Was that staged on purpose for this? | `tomdet`, since that is where the pre-staged environment and the scan files both are | not yet | [Deployment](../../stack/deployment.md) |

## Not on this page

Expand Down
Loading