Fault Tolerant Experiment Logic

[Am1r999]

Experiments should be stoppable and resumable leveraging the current model tracking feature.

• Options 1
• option 2
• ....

[pghiasia]

Completed:

1. Checkpoint model, optimizer, and RNG state to disk after every completed round
2. Handle stop signals gracefully so runs shut down cleanly instead of crashing mid-round
3. Resume a stopped run from its last checkpoint, reusing the original data and validating the config matches
4. Keep decentralized nodes in sync on resume by restarting from the earliest common completed round
5. Implement automated testing for experiment fault tolerance cases
6. Scope out distributed version of logging for latest round