Skip to content

Maintain Cache of Tokens During Tokenization

Have the tokenization service maintain a local database (probably h2 as there is good support) of tokens which have been created.

  • Should be optional
  • If used, filters should be on the local database first, then go to the ingest server
  • Might want to maintain a simple state table for bags so we know if they have started tokenizing or not
    • Could build up a list of files for a bag, show their status (valid, token registered, etc)
  • Remove tokens from db when communication with ingest is successful
    • Probably want a join table for file → token