|
|
|
April development plan
|
|
|
|
|
|
|
|
To display progress we can use the moon emojis since at the moment we don't have many options.
|
|
|
|
|
|
|
|
* new_moon (:new_moon:) = not started
|
|
|
|
* first_quarter_moon (:first_quarter_moon:) = in progress
|
|
|
|
* full_moon (:full_moon:) = complete
|
|
|
|
|
|
|
|
|
|
|
|
### Replication
|
|
|
|
|
|
|
|
* Test automated pulling of content
|
|
|
|
* Support polling from multiple endpoints
|
|
|
|
* Add injected value for Scheduled cron job
|
|
|
|
* Can use replication.cron
|
|
|
|
* Default to "0 0 * * * * *"
|
|
|
|
* Update polling/replication of content from REST api
|
|
|
|
* If there are ongoing or queued replications, don't poll more
|
|
|
|
* Maybe we have a threshold of total replications (ex: 20)
|
|
|
|
* 5 ongoing + 15 queued = no poll
|
|
|
|
* 5 ongoing + 10 queued = poll w/ page size of 5
|
|
|
|
* 0 ongoing + 0 queued = poll w/ page size of 20
|
|
|
|
* Look into the JobExplorer for querying all the jobs
|
|
|
|
* Only need to do these checks in the ReplicationQueryTask
|
|
|
|
* Handle polling from the ingest API when results are paginated
|
|
|
|
|
|
|
|
### Ingest
|
|
|
|
|
|
|
|
* Start basic UI
|
|
|
|
* Can use thymeleaf as our templating engine
|
|
|
|
* Display users/nodes, and their replications
|
|
|
|
* The [webapp][1] from the dpn project can serve as an example for getting started
|
|
|
|
* Reporting of tokenization
|
|
|
|
* If tokenization of a collection fails we want to know
|
|
|
|
* why it failed (the exception and stacktrace)
|
|
|
|
* what failed
|
|
|
|
|
|
|
|
### Intake
|
|
|
|
|
|
|
|
* Bagging of content - **Kai** :first_quarter_moon:
|
|
|
|
* Add restful services to duraspace/dpn intake clients - **Mike** :new_moon:
|
|
|
|
|
|
|
|
### All
|
|
|
|
|
|
|
|
* Deprecation of amqp functions/classes - **Kai** :new_moon:
|
|
|
|
* Add unit tests for new models/services - **Mike & Kai** :new_moon:
|
|
|
|
* Add file count and bag size to ingest request - **Kai**
|
|
|
|
* Update properties
|
|
|
|
* chron.storage.preservation -> storage.preservation
|
|
|
|
* chron.staging.bags -> storage.staging.bags
|
|
|
|
* chron.staging.tokens -> storage.staging.tokens
|
|
|
|
* chron.staging.restore -> storage.staging.restore
|
|
|
|
-----------------------
|
|
|
|
|
|
|
|
### Future Tasks
|
|
|
|
|
|
|
|
* Expand Bag object
|
|
|
|
* Should be able to tell if a bag is staged
|
|
|
|
* Likewise for tokenization (incomplete/failed/success/ace_sync)
|
|
|
|
* May be put into its own table
|
|
|
|
* token-location should be moved from the bag to a token table
|
|
|
|
* Add ACE sync to ingest
|
|
|
|
* Specify ACE endpoint to connect to
|
|
|
|
* Load all collections from ACE which are not in the ingest shell
|
|
|
|
* Handle pagination in replication shell
|
|
|
|
* The JSON returned is different if we use pages
|
|
|
|
* Eventually we'll want to have everything return the same content
|
|
|
|
(maybe do pagination by default)
|
|
|
|
* Token Sync
|
|
|
|
* From ACE endpoint
|
|
|
|
* May need API in ACE first
|
|
|
|
* Clean codebase
|
|
|
|
* Many unused properties can be cleaned out
|
|
|
|
* AMQP code can be removed
|
|
|
|
* chron-db not needed
|
|
|
|
* Support for multiple storage locations
|
|
|
|
* Could have multiple bag stages (in the event we pull from dpn/dura/etc)
|
|
|
|
* Use a password encoder in the ingest-rest server
|
|
|
|
* Don't want to store passwords as plaintext
|
|
|
|
|
|
|
|
[1]: https://gitlab.umiacs.umd.edu/shake/dpn/tree/develop/client/webapp |
|
|
|
\ No newline at end of file |