Maintain Cache of Tokens During Tokenization
Have the tokenization service maintain a local database (probably h2 as there is good support) of tokens which have been created.
- Should be optional
- If used, filters should be on the local database first, then go to the ingest server
- Might want to maintain a simple state table for bags so we know if they have started tokenizing or not
- Could build up a list of files for a bag, show their status (valid, token registered, etc)
- Remove tokens from db when communication with ingest is successful
- Probably want a join table for file → token