Updates to Tokenization Process

In order to help facilitate some of the Tokenization updates we need to make a few updates to how things currently work. We'll want a more centralized approach to keeping track of state, allowing both the ChronopolisTokenRequestBatch and TokenRegistrar to update a StateMachine with the state of a ManifestEntry. It will basically cycle between queued, processing, token_received, and completed. We could also have an error state to be safe.

I'm not sure if it would be best to go with a push or a pull model here when distributing work:

• doing a push keeps all threading a bit more centralized in the StateMachine but has impacts on the dependencies when creating beans
• doing a pull pushes the threading outside of the scope of the StateMachine and might create a cleaner dependency graph

To start off, we have a few new Interfaces (return values tbd):

StateMachine (name tbd)

void start(ManifestEntry entry); // Queue a ManifestEntry for Processing
void retry(ManifestEntry entry); // Requeue a ManifestEntry for Processing
void associate(ManifestEntry entry, TokenResponse response); // Mark a ManifestEntry as having received a Token

// Might be able to combine these two, not sure of the impact
void rm(ManifestEntry entry);
void complete(ManifestEntry entry);

// if doing pull
Set<ManifestEntry> getQueued(int size, long timeout, TimeUnit timeUnit);
Map<ManifestEntry, TokenResponse> getReceived(int size, long timeout, TimeUnit timeUnit);

Implemented By

• TokenStateMachine

Registrar

void submit(Map<ManifestEntry, TokenResponse> tokenResponses);

Implemented By

• HttpTokenRegistrar

ChronopolisRequestBatch

void fetchTokens(Set<ManifestEntry> entries);

Implemented By

• ChronopolisTokenRequestBatch

Filter

Replaced by Predicate<ManifestEntry>

Implemented By

• HttpFilter
• StateMachineFilter

---

There are also some minor changes we'll want to make to the BagProcessor, notably removing the ManifestTuple and creating ManifestEntry from the start and moving from a single Filter to a List<Filter>.