|
|
February development plan
|
|
|
|
|
|
To display progress we can use the moon emojis since at the moment we don't have many options.
|
|
|
|
|
|
* new_moon = not started
|
|
|
* first_quarter_moon = in progress
|
|
|
* full_moon = complete
|
|
|
|
|
|
|
|
|
### Replication
|
|
|
|
|
|
* Test automated pulling of content :new_moon:
|
|
|
* Migrate to use ingest.api.endpoints when setting up retrofit api - **Mike** :first_quarter_moon:
|
|
|
* Will need to support polling multiple endpoints as well, can just concentrate on one for now
|
|
|
* Update polling/replication of content from REST api
|
|
|
* If there are ongoing or queued replications, don't poll more
|
|
|
* Maybe we have a threshold of total replications (ex: 20)
|
|
|
* 5 ongoing + 15 queued = no poll
|
|
|
* 5 ongoing + 10 queued = poll w/ page size of 5
|
|
|
* 0 ongoing + 0 queued = poll w/ page size of 20
|
|
|
* Handle polling from the ingest API when results are paginated
|
|
|
|
|
|
### Ingest
|
|
|
|
|
|
* Add async method for putting bags :new_moon:
|
|
|
* Should return a 202 on success
|
|
|
* If the bag already exists, return a 302 and redirect to /api/staging/bag/{bag-id}?
|
|
|
* Could use spring batch for mangement of jobs
|
|
|
|
|
|
### Intake
|
|
|
|
|
|
* Bagging of content - **Kai** :first_quarter_moon:
|
|
|
* Add restful services to duraspace/dpn intake clients - **Mike** :new_moon:
|
|
|
|
|
|
### All
|
|
|
|
|
|
* Deprecation of amqp functions/classes - **Kai** :new_moon:
|
|
|
* Add unit tests for new models/services - **Mike & Kai** :new_moon:
|
|
|
|
|
|
-----------------------
|
|
|
|
|
|
### Future Tasks
|
|
|
|
|
|
* Expand Bag object
|
|
|
* Should be able to tell if a bag is staged
|
|
|
* Likewise for tokenization (incomplete/failed/success/ace_sync)
|
|
|
* May be put into its own table
|
|
|
* Add ACE sync to ingest
|
|
|
* Specify ACE endpoint to connect to
|
|
|
* Load all collections from ACE which are not in the ingest shell
|
|
|
* Handle pagination in replication shell
|
|
|
* The JSON returned is different if we use pages
|
|
|
* Eventually we'll want to have everything return the same content
|
|
|
(maybe do pagination by default)
|
|
|
* Update tokenization process
|
|
|
* Treat it like ACE and store tokens in the DB
|
|
|
* Allows us to start/stop tokenization based on where we are
|
|
|
* Allows us to rewrite token files for replicating nodes
|
|
|
* Will want a token sync as well from ACE |