Skip to content

Create new rsync workflow for '--from-files' flag

In an attempt to support more diverse rsync environments, we're adding support for the '--from-files' flag which lets us specify input to rsync.

Note that this workflow comes with a slew of work that we need to do for it so there's additional work beyond just spawning rsyncs.

  • Retrieve a file listing from the ingest server (tbd how this will look but most likely pre-split)
  • Create a new class for transferring partial Bags
    • The generic BagTransfer will run hashing and other operations which we don't want to do after each rsync
    • Needs to cleanup resources on success so we know what still needs to be transferred
  • Create a new class for Hashing a transferred Bag
    • Should be pretty simple, hashing is normally handled by the Bucket
    • Should also pass the computed hash to the ingest server
  • Tie everything together using completable futures
    • from retrieving the file listing to hashing, once everything is complete we want to continue with the rest of the flow