|
|
March development plan
|
|
|
|
|
|
To display progress we can use the moon emojis since at the moment we don't have many options.
|
|
|
|
|
|
* new_moon (:new_moon:) = not started
|
|
|
* first_quarter_moon (:first_quarter_moon:) = in progress
|
|
|
* full_moon (:full_moon:) = complete
|
|
|
|
|
|
|
|
|
### Replication
|
|
|
|
|
|
* Test automated pulling of content
|
|
|
* Migrate to use ingest.api.endpoints when setting up retrofit api - **Mike** :first_quarter_moon:
|
|
|
* Will need to support polling multiple endpoints as well, can just concentrate on one for now
|
|
|
* Add injected value for Scheduled cron job
|
|
|
* Can use replication.cron
|
|
|
* Default to "0 0 * * * * *"
|
|
|
* Update polling/replication of content from REST api
|
|
|
* If there are ongoing or queued replications, don't poll more
|
|
|
* Maybe we have a threshold of total replications (ex: 20)
|
|
|
* 5 ongoing + 15 queued = no poll
|
|
|
* 5 ongoing + 10 queued = poll w/ page size of 5
|
|
|
* 0 ongoing + 0 queued = poll w/ page size of 20
|
|
|
* Look into the JobExplorer for querying all the jobs
|
|
|
* Only need to do these checks in the ReplicationQueryTask
|
|
|
* Handle polling from the ingest API when results are paginated
|
|
|
* Daemonize
|
|
|
* Add a profile which runs in production for the daemon service
|
|
|
* This way we can still have the current setup as a backup
|
|
|
* See commons-daemon
|
|
|
|
|
|
### Ingest
|
|
|
|
|
|
* Add async method for putting bags
|
|
|
* Should return a 202 on success
|
|
|
* If the bag already exists, return a 302 and redirect to /api/staging/bag/{bag-id}?
|
|
|
* Could use spring batch for mangement of jobs
|
|
|
* With the changes to tokenization coming, we may want to hold off on this for now
|
|
|
* Add config with production profile
|
|
|
* Needed for DB initialization
|
|
|
* Don't want to execute schema and data scripts when we start the application
|
|
|
* Need something to ensure either an admin user gets created
|
|
|
* Grab all the users from the database and if the set is empty/null create an admin account
|
|
|
* Start basic UI
|
|
|
* Can use thymeleaf as our templating engine
|
|
|
* Display users/nodes, and their replications
|
|
|
* The [webapp][1] from the dpn project can serve as an example for getting started
|
|
|
* Update tokenization process - **Mike** :new_moon:
|
|
|
* Treat it like ACE and store tokens in the DB
|
|
|
* Allows us to start/stop tokenization based on where we are
|
|
|
* Allows us to rewrite token files for replicating nodes
|
|
|
* Will want a token sync as well from ACE
|
|
|
* Update how we create test data
|
|
|
* Already have a separate profile 'development' which creates some mock data
|
|
|
* We can further break this apart in case we only want to test some parts
|
|
|
* 'test-bags', 'test-tokens', etc. Each would get it's own bean.
|
|
|
|
|
|
### Intake
|
|
|
|
|
|
* Bagging of content - **Kai** :first_quarter_moon:
|
|
|
* Add restful services to duraspace/dpn intake clients - **Mike** :new_moon:
|
|
|
|
|
|
### All
|
|
|
|
|
|
* Deprecation of amqp functions/classes - **Kai** :new_moon:
|
|
|
* Add unit tests for new models/services - **Mike & Kai** :new_moon:
|
|
|
* Add file count and bag size to ingest request
|
|
|
|
|
|
-----------------------
|
|
|
|
|
|
### Future Tasks
|
|
|
|
|
|
* Expand Bag object
|
|
|
* Should be able to tell if a bag is staged
|
|
|
* Likewise for tokenization (incomplete/failed/success/ace_sync)
|
|
|
* May be put into its own table
|
|
|
* Add ACE sync to ingest
|
|
|
* Specify ACE endpoint to connect to
|
|
|
* Load all collections from ACE which are not in the ingest shell
|
|
|
* Handle pagination in replication shell
|
|
|
* The JSON returned is different if we use pages
|
|
|
* Eventually we'll want to have everything return the same content
|
|
|
(maybe do pagination by default)
|
|
|
* Token Sync
|
|
|
* From ACE endpoint
|
|
|
* May need API in ACE first
|
|
|
|
|
|
[1]: https://gitlab.umiacs.umd.edu/shake/dpn/tree/develop/client/webapp |
|
|
\ No newline at end of file |