ace issueshttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues2019-06-20T12:35:22-04:00https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/71TokenImportStatus redirect 4042019-06-20T12:35:22-04:00Ghost UserTokenImportStatus redirect 404When importing a token store through the UI, it is supposed to redirect to the TokenImportStatus servlet. However, the context path is being removed (i.e. /ace-am/TokenImportStatus is instead /TokenImportStatus) resulting in a 404.When importing a token store through the UI, it is supposed to redirect to the TokenImportStatus servlet. However, the context path is being removed (i.e. /ace-am/TokenImportStatus is instead /TokenImportStatus) resulting in a 404.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/68Show Duplicates vs Report Duplicates2019-05-13T15:05:02-04:00Ghost UserShow Duplicates vs Report DuplicatesThere's a ShowDuplicates servlet which appears to be an older version of display duplicate files in the Audit Manager. In addition, the ReportDuplicates is only one which is accessible in the UI, so the ShowDuplicates likely hasn't been ...There's a ShowDuplicates servlet which appears to be an older version of display duplicate files in the Audit Manager. In addition, the ReportDuplicates is only one which is accessible in the UI, so the ShowDuplicates likely hasn't been used in some time.
However, I'm not sure how the two servlets differentiate so we need to look at them in order to determine if there's any difference in functionality.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/67Report Duplicates 4042019-05-13T14:19:37-04:00Ghost UserReport Duplicates 404Querying ReportDuplicates with a collectionid /ReportDuplicates?collectionid=61&json=true will throw a 404
This is from duplicatereport-json not existing.
In my opinion it would be best to remove the optional json handling there in fav...Querying ReportDuplicates with a collectionid /ReportDuplicates?collectionid=61&json=true will throw a 404
This is from duplicatereport-json not existing.
In my opinion it would be best to remove the optional json handling there in favor of adding a duplicates api endpointhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/63Cut back on bootstrap/jquery usage2019-04-25T15:53:07-04:00Ghost UserCut back on bootstrap/jquery usageAs ACE does not get frequent updates, it might be good to move away from larger frameworks for CSS/javascript and instead stick to our own. As we do not rely on many features, it should not be much work to rewrite the pieces which we do ...As ACE does not get frequent updates, it might be good to move away from larger frameworks for CSS/javascript and instead stick to our own. As we do not rely on many features, it should not be much work to rewrite the pieces which we do use.
This way if there are security vulnerabilities in a framework, we won't need to push an update for a simple version fix.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/62AuditTokens Updates2019-04-23T13:43:35-04:00Ghost UserAuditTokens UpdatesWhen doing testing on #61 I noticed a few actions the AuditTokens class does which are inconsistent with the AuditThread:
* Log Start and Stop on IMS Failures: The AuditThread will log such events, showing that audits were attempted.
* ...When doing testing on #61 I noticed a few actions the AuditTokens class does which are inconsistent with the AuditThread:
* Log Start and Stop on IMS Failures: The AuditThread will log such events, showing that audits were attempted.
* Mark collections as interrupted on failures: connection to the IMS can be lost and this information will not be propagated through the collection state, only the event log.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/61Investigate Potential AuditThread Deadlock2019-04-24T14:06:24-04:00Ghost UserInvestigate Potential AuditThread DeadlockAt UCSD we saw an AuditThread which had been idle for 13 days, indicating that a deadlock might exist somewhere in the AuditThread. They have audit blocking enabled, with a maximum of a 5 minute block. From my testing, when going beyond ...At UCSD we saw an AuditThread which had been idle for 13 days, indicating that a deadlock might exist somewhere in the AuditThread. They have audit blocking enabled, with a maximum of a 5 minute block. From my testing, when going beyond the 5 minutes, a NullPointerException is thrown and the audit fails.
At the very least I think it might be time to rework how the blocking works now that we have better tools with Functions, Supplies, etc. In addition, it might be good to look into adding a number of retries available, instead of a maximum number of total time spent waiting. This way instead of checking against elapsed time, we only increment our attempt number and sleep for the specified amount of time.
### Info so far
* It occurs during an audit, so the beginning and end are ok
* RequestThread log entries are seen continuously, so the batch has not yet been closed
* ValidationThread log entries are not seen, a good candidate for where the blocking is occurring
* This would put our idle somewhere around `validator.add(item.getFileDigest(), token);`
* The TokenValidator itself seems sound, should be able to process without a lock
* TokenValidator::add does attempt to acquire a lock which can block, and is only blocked during TokenValidator::processBatch
* TokenValidator::processBatch only seems to block when communicating with the IMS; I don't see the database calls being what is causing us to block indefinitely (though maybe it's possible)
* It's possible that the old IMSService::blockUntil had a bug in it, though it seemed to be soundhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/60Servlet for Active Audits/Token Imports2019-04-04T14:23:38-04:00Ghost UserServlet for Active Audits/Token ImportsIn the ACE AMs for the Chronopolis Project, we've been ingesting large amounts of collections. We've noticed recently when trying to browse it is difficult to find out which collections are actively auditing, and it's almost impossible t...In the ACE AMs for the Chronopolis Project, we've been ingesting large amounts of collections. We've noticed recently when trying to browse it is difficult to find out which collections are actively auditing, and it's almost impossible to see what collections are ingesting tokens.
Given that these both use the same thread pool, as they are processes which modify collections, it would be nice to have a servlet which can show any active processes. We will probably want to show information from the audit/token ingest when browsing the collections in this servlet. Maybe retrieving information from the CollectionThreadPoolExecutor. As this is still in its conception, nothing is really set in stone yet.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/57Events API2018-10-24T14:39:56-04:00Ghost UserEvents APICreate an API for accessing events associated with a Collection (and possibly on MonitoredItems? that is more Logging rather than an Event though). For the purpose of this issue, an Event is defined as: File Audit, Token Audit, or Token ...Create an API for accessing events associated with a Collection (and possibly on MonitoredItems? that is more Logging rather than an Event though). For the purpose of this issue, an Event is defined as: File Audit, Token Audit, or Token Ingest. We should be able to see any past, current, and queued Events for a Collection.
Some brief prototyping:
`/rest/collections/<id>/events`
```
{
"current": {},
"queued": {"event_type": "file_audit", "status": "queued"},
"history": [{"event_type": "token_ingest", "status": "success", "session": 1},
{"event_type": "token_audit", "status": "success", "session":2}]
}
```
`/rest/collections/<id>/events/<session>`
```
{
"event_type": "token_ingest",
"status": "success",
"session": 1
}
```
`/rest/collections/<id>/events/<session>/log`
```
{
event_log entries
...
}
```
The main api for getting events on a collection is probably a bit misguided at the moment as we won't want to respond with all the history events ever on a collection, but it's a simple example for now. Likewise I'm not entirely sure what the data structure for an event should look like.1.14 - Extend REST APIhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/56UI Access for PartnerSite/TokenImportStatus/etc2018-05-24T15:46:53-04:00Ghost UserUI Access for PartnerSite/TokenImportStatus/etcWe have some Servlets which aren't really accessible through the main UI and only through other Servlets. We should first figure out which servlets we should provide access to from the main page and then figure out a good method of doing...We have some Servlets which aren't really accessible through the main UI and only through other Servlets. We should first figure out which servlets we should provide access to from the main page and then figure out a good method of doing so (the top bar is getting crowded - maybe a new nav bar is needed).
The two which stand out immediately are
* PartnerSite
* TokenImportStatushttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/55Deploy ace-ims-ear as part of release cycle2018-05-24T15:25:43-04:00Ghost UserDeploy ace-ims-ear as part of release cycleRelated to #53 - when creating an image for the IMS we'll want an artifact which we can download and deploy.
I'm not sure if we'll want to handle this within the ace-dist module as that currently handles the ace-am packaging. It would b...Related to #53 - when creating an image for the IMS we'll want an artifact which we can download and deploy.
I'm not sure if we'll want to handle this within the ace-dist module as that currently handles the ace-am packaging. It would be nice to have separate artifacts for each (am/ims) which we should keep in mind when completing this.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/54State Enum for MonitoredItems2018-05-22T12:57:10-04:00Ghost UserState Enum for MonitoredItemsInstead of using 'A', 'C', 'T', 'M', etc, we should have a state enum for MonitoredItems. This will require us to add an AttributeConverter so that we can maintain compatibility with what's in the database, and possibly some type of mapp...Instead of using 'A', 'C', 'T', 'M', etc, we should have a state enum for MonitoredItems. This will require us to add an AttributeConverter so that we can maintain compatibility with what's in the database, and possibly some type of mapping for when we get MonitoredItems through the rest apis.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/53Docker Images2018-05-24T15:25:43-04:00Ghost UserDocker ImagesIt would be nice to have Docker images for ease of deployment, or at least showing what a production ACE setup might look like.
For the AuditManager, an example Dockerfile can be seen on the [UA Libraries github](https://github.com/uali...It would be nice to have Docker images for ease of deployment, or at least showing what a production ACE setup might look like.
For the AuditManager, an example Dockerfile can be seen on the [UA Libraries github](https://github.com/ualibraries/ace-audit-manager)
The IMS will likely be trickier, as it will involve setting up a JEE server and various resources in order for the IMS to run.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/45Deploy to Obj from pipeline2017-12-06T11:42:16-05:00Ghost UserDeploy to Obj from pipelineWe can streamline our deploy process a bit more to include creating our release artifacts for us and pushing them to obj. Since this is all cli based, we should be able to update the .gitlab-ci.yml in order to accomplish this.
The files...We can streamline our deploy process a bit more to include creating our release artifacts for us and pushing them to obj. Since this is all cli based, we should be able to update the .gitlab-ci.yml in order to accomplish this.
The files needed to be uploaded are
* ace-$VERSION-src.tar.gz
* ace-dist-$VERSION-RELEASE-bin.tar.gz
* ace-dist-$VERSION-RELEASE-bin.ziphttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/37Audit threads not always cleaning up properly after system_errors2018-01-23T08:32:46-05:00Ghost UserAudit threads not always cleaning up properly after system_errorsWe've noticed when there are problems connecting to the IMS, the audit threads in the Audit Manager do not always terminate. This could be from extra validation/request batch threads going on, but it causes other audits to be unresponsiv...We've noticed when there are problems connecting to the IMS, the audit threads in the Audit Manager do not always terminate. This could be from extra validation/request batch threads going on, but it causes other audits to be unresponsive (as they can't be submitted) and unintuitive behavior in the ui.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/34Paginated pull of content in Browse Servlet2018-01-23T08:32:46-05:00Ghost UserPaginated pull of content in Browse ServletIn the browse servlet, folders with a large number of items can cause the Audit Manager to choke while it fetches everything from the database. We should aim to pull only a subset of items at a time.In the browse servlet, folders with a large number of items can cause the Audit Manager to choke while it fetches everything from the database. We should aim to pull only a subset of items at a time.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/26Improve Audit Schedule Timing2018-01-23T08:32:47-05:00Ghost UserImprove Audit Schedule TimingImprove audit policy granularity to allow for the following:
1. audits may only run during certain hours
2. Specify type of audit to run
3. Support multiple policies per collection
- ie, one policy may say, nightly scan for new files...Improve audit policy granularity to allow for the following:
1. audits may only run during certain hours
2. Specify type of audit to run
3. Support multiple policies per collection
- ie, one policy may say, nightly scan for new files while another policy runs a complete scan every three months
This will require the last audit time to be expanded. Possibly create a last audit table which references a policy.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/25Scan Only Mode2018-01-23T08:32:47-05:00Ghost UserScan Only ModeCreate an audit type that only scans for new objects.Create an audit type that only scans for new objects.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/24Cancel Compares2018-01-23T08:32:47-05:00Ghost UserCancel ComparesAdd an abort link on collection comparingAdd an abort link on collection comparinghttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/23Default Settings2018-01-23T08:32:47-05:00Ghost UserDefault SettingsAllow default settings to be configured through context.xml, allow drivers to be enabled/disabled as well.Allow default settings to be configured through context.xml, allow drivers to be enabled/disabled as well.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/22Read Only Collections2018-01-23T08:32:47-05:00Ghost UserRead Only CollectionsAllow collections to be marked as read-only, meaning no new files will be registered, any files appearing in the collection will be marked as error.
Allow collections to be marked as read-only, meaning no new files will be registered, any files appearing in the collection will be marked as error.