Skip to content

Trigger cryoSIM processing#841

Merged
tieneupin merged 17 commits into
mainfrom
trigger-cryosim-processing
Jul 16, 2026
Merged

Trigger cryoSIM processing#841
tieneupin merged 17 commits into
mainfrom
trigger-cryosim-processing

Conversation

@tieneupin

@tieneupin tieneupin commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Adds more functionality to the cryoSIM workflow.

  • On the client side, Murfey will apply the cryoSIM context when it sees raw data files matching currently known criteria (raw data files have no suffix, and end with a certain set of patterns).
  • When a fluorescent data file is seen, it will submit a post to trigger data processing for the file.
  • Adds an endpoint to the backend server that will receive requests to trigger data processing from the client, and will forward the message to RabbitMQ's processing_recipe queue. The actual submission of the message will be deferred until the processing recipe and service have been set up.
  • Updates existing tests and adds new tests to reflect the updated behaviour.

Actual diff is about +170 lines, with the rest of the diff being due to tests.

@tieneupin tieneupin added the cryo-sim Part of the cryo-SIM pipeline extension label Jul 14, 2026
@tieneupin tieneupin self-assigned this Jul 14, 2026
@tieneupin tieneupin added server Relates to the server component client Relates to the client component enhancement New feature or request labels Jul 14, 2026
Comment thread src/murfey/server/api/workflow_sim.py Fixed
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.23810% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.47%. Comparing base (c93a69c) to head (41d4dc7).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #841      +/-   ##
==========================================
+ Coverage   53.29%   53.47%   +0.18%     
==========================================
  Files         102      103       +1     
  Lines       11030    11071      +41     
  Branches     1470     1476       +6     
==========================================
+ Hits         5878     5920      +42     
  Misses       4831     4831              
+ Partials      321      320       -1     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…nalyser should determine context using bright field files as well, whereas 'capture_post' should only be triggered on fluorescent ones
@tieneupin
tieneupin marked this pull request as ready for review July 14, 2026 13:31

@stephen-riggs stephen-riggs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues I can see, but questions about code duplication and return values

Comment thread src/murfey/client/contexts/sim.py Outdated
logger = logging.getLogger("murfey.client.contexts.sim")


def _get_source(file_path: Path, environment: MurfeyInstanceEnvironment) -> Path | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might be able to use the _get_source in the context.py as that should have the same behaviour. Looks like some contexts use that and some don't

@tieneupin tieneupin Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. This would be a good opportunity to make sure that the shared get_source works across workflows. I'll switch to it and test it for the SIM workflow.

Comment thread src/murfey/client/contexts/sim.py Outdated
return None


def _file_transferred_to(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, _file_transferred_to exists in the parent context.py with the same behaviour

Comment thread src/murfey/client/contexts/sim.py Outdated
source: Path,
file_path: Path,
rsync_basepath: Path,
) -> Path | None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would this return None?

@tieneupin tieneupin Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might have been oversight on my part back when I was learning how to set up the function for the CLEM workflow. I'll switch over to using the shared file_transferred_to function.

Comment thread src/murfey/client/contexts/sim.py Outdated
file_path=transferred_file,
rsync_basepath=Path(self._machine_config.get("rsync_basepath", "")),
)
if destination_file is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, what situation would the destination_file be None?

@tieneupin
tieneupin merged commit c85221f into main Jul 16, 2026
17 checks passed
@tieneupin
tieneupin deleted the trigger-cryosim-processing branch July 16, 2026 05:46
@tieneupin

tieneupin commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the quick review and helpful comments. I've taken your suggestions on board, and switched to using the shared _get_source and _file_transferred_to functions defined in murfey.client.context. The tests have passed, as has the test transfer on actual data in our dev setup, so I have gone ahead and merged this PR.

A future housekeeping-related PR would involve revisiting the other context files developed and switching them to using the shared _get_source and _file_transferred_to functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

client Relates to the client component cryo-sim Part of the cryo-SIM pipeline extension enhancement New feature or request server Relates to the server component

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants