ace issueshttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues2019-06-25T12:56:40-04:00https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/72Token Import Odd Behavior2019-06-25T12:56:40-04:00Ghost UserToken Import Odd BehaviorWhen importing a large token store (~137k tokens) I'm seeing what appears to be a deadlock.
Upon inspecting the jvm, there are many threads waiting on acquiring the lock in MoniotredItemManager::getItemByPath, and one thread stuck when ...When importing a large token store (~137k tokens) I'm seeing what appears to be a deadlock.
Upon inspecting the jvm, there are many threads waiting on acquiring the lock in MoniotredItemManager::getItemByPath, and one thread stuck when getting the result list in MonitoredItemManager::getItemByPath.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/71TokenImportStatus redirect 4042019-06-20T12:35:22-04:00Ghost UserTokenImportStatus redirect 404When importing a token store through the UI, it is supposed to redirect to the TokenImportStatus servlet. However, the context path is being removed (i.e. /ace-am/TokenImportStatus is instead /TokenImportStatus) resulting in a 404.When importing a token store through the UI, it is supposed to redirect to the TokenImportStatus servlet. However, the context path is being removed (i.e. /ace-am/TokenImportStatus is instead /TokenImportStatus) resulting in a 404.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/70Remove stateEnum from API2019-06-05T12:45:23-04:00Ghost UserRemove stateEnum from APIIn various queries on Collections, there is an additional `stateEnum` field which is being displayed. We should either only display the `state` as a character, or replace it with the enum type.In various queries on Collections, there is an additional `stateEnum` field which is being displayed. We should either only display the `state` as a character, or replace it with the enum type.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/69ListItems exception2019-06-05T12:45:35-04:00Ghost UserListItems exceptionQuerying to /ListItem?json=true will trigger an exception. A collectionid is probably expected, but none is found.Querying to /ListItem?json=true will trigger an exception. A collectionid is probably expected, but none is found.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/68Show Duplicates vs Report Duplicates2019-05-13T15:05:02-04:00Ghost UserShow Duplicates vs Report DuplicatesThere's a ShowDuplicates servlet which appears to be an older version of display duplicate files in the Audit Manager. In addition, the ReportDuplicates is only one which is accessible in the UI, so the ShowDuplicates likely hasn't been ...There's a ShowDuplicates servlet which appears to be an older version of display duplicate files in the Audit Manager. In addition, the ReportDuplicates is only one which is accessible in the UI, so the ShowDuplicates likely hasn't been used in some time.
However, I'm not sure how the two servlets differentiate so we need to look at them in order to determine if there's any difference in functionality.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/67Report Duplicates 4042019-05-13T14:19:37-04:00Ghost UserReport Duplicates 404Querying ReportDuplicates with a collectionid /ReportDuplicates?collectionid=61&json=true will throw a 404
This is from duplicatereport-json not existing.
In my opinion it would be best to remove the optional json handling there in fav...Querying ReportDuplicates with a collectionid /ReportDuplicates?collectionid=61&json=true will throw a 404
This is from duplicatereport-json not existing.
In my opinion it would be best to remove the optional json handling there in favor of adding a duplicates api endpointhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/66Duplicates NPE2019-06-05T12:45:44-04:00Ghost UserDuplicates NPEQuerying on /ReportDuplicates?json=true causes a NPE
```
edu.umiacs.ace.monitor.access.CollectionCountContext.getFileCount(CollectionCountContext.java:195)
edu.umiacs.ace.monitor.compare.DuplicateReportServlet.processRequest(DuplicateRe...Querying on /ReportDuplicates?json=true causes a NPE
```
edu.umiacs.ace.monitor.access.CollectionCountContext.getFileCount(CollectionCountContext.java:195)
edu.umiacs.ace.monitor.compare.DuplicateReportServlet.processRequest(DuplicateReportServlet.java:125)
```https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/65Report Servlet Exception2019-06-05T12:45:53-04:00Ghost UserReport Servlet ExceptionWhen requesting `json=true` from the Report Servlet, a NoResultException can be thrown if the Report is emptyWhen requesting `json=true` from the Report Servlet, a NoResultException can be thrown if the Report is emptyhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/64ReportConfiguration servlet not removing report2019-06-05T12:46:01-04:00Ghost UserReportConfiguration servlet not removing reportA NullPointerException is being thrown, which prevents the report from being removed
```
[25/Apr/2019:15:55:51] [request /ReportConfiguration] Error removing report
java.lang.NullPointerException
at edu.umiacs.ace.monitor.reporti...A NullPointerException is being thrown, which prevents the report from being removed
```
[25/Apr/2019:15:55:51] [request /ReportConfiguration] Error removing report
java.lang.NullPointerException
at edu.umiacs.ace.monitor.reporting.SchedulerContextListener.removeJob(SchedulerContextListener.java:121)
at edu.umiacs.ace.monitor.reporting.ReportConfigurationServlet.processRequest(ReportConfigurationServlet.java:101)
at edu.umiacs.ace.util.EntityManagerServlet.wrapRequest(EntityManagerServlet.java:120)
at edu.umiacs.ace.util.EntityManagerServlet.doGet(EntityManagerServlet.java:244)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at edu.umiacs.ace.rest.CharsetFilter.doFilter(CharsetFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:615)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
```https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/63Cut back on bootstrap/jquery usage2019-04-25T15:53:07-04:00Ghost UserCut back on bootstrap/jquery usageAs ACE does not get frequent updates, it might be good to move away from larger frameworks for CSS/javascript and instead stick to our own. As we do not rely on many features, it should not be much work to rewrite the pieces which we do ...As ACE does not get frequent updates, it might be good to move away from larger frameworks for CSS/javascript and instead stick to our own. As we do not rely on many features, it should not be much work to rewrite the pieces which we do use.
This way if there are security vulnerabilities in a framework, we won't need to push an update for a simple version fix.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/62AuditTokens Updates2019-04-23T13:43:35-04:00Ghost UserAuditTokens UpdatesWhen doing testing on #61 I noticed a few actions the AuditTokens class does which are inconsistent with the AuditThread:
* Log Start and Stop on IMS Failures: The AuditThread will log such events, showing that audits were attempted.
* ...When doing testing on #61 I noticed a few actions the AuditTokens class does which are inconsistent with the AuditThread:
* Log Start and Stop on IMS Failures: The AuditThread will log such events, showing that audits were attempted.
* Mark collections as interrupted on failures: connection to the IMS can be lost and this information will not be propagated through the collection state, only the event log.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/61Investigate Potential AuditThread Deadlock2019-04-24T14:06:24-04:00Ghost UserInvestigate Potential AuditThread DeadlockAt UCSD we saw an AuditThread which had been idle for 13 days, indicating that a deadlock might exist somewhere in the AuditThread. They have audit blocking enabled, with a maximum of a 5 minute block. From my testing, when going beyond ...At UCSD we saw an AuditThread which had been idle for 13 days, indicating that a deadlock might exist somewhere in the AuditThread. They have audit blocking enabled, with a maximum of a 5 minute block. From my testing, when going beyond the 5 minutes, a NullPointerException is thrown and the audit fails.
At the very least I think it might be time to rework how the blocking works now that we have better tools with Functions, Supplies, etc. In addition, it might be good to look into adding a number of retries available, instead of a maximum number of total time spent waiting. This way instead of checking against elapsed time, we only increment our attempt number and sleep for the specified amount of time.
### Info so far
* It occurs during an audit, so the beginning and end are ok
* RequestThread log entries are seen continuously, so the batch has not yet been closed
* ValidationThread log entries are not seen, a good candidate for where the blocking is occurring
* This would put our idle somewhere around `validator.add(item.getFileDigest(), token);`
* The TokenValidator itself seems sound, should be able to process without a lock
* TokenValidator::add does attempt to acquire a lock which can block, and is only blocked during TokenValidator::processBatch
* TokenValidator::processBatch only seems to block when communicating with the IMS; I don't see the database calls being what is causing us to block indefinitely (though maybe it's possible)
* It's possible that the old IMSService::blockUntil had a bug in it, though it seemed to be soundhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/60Servlet for Active Audits/Token Imports2019-04-04T14:23:38-04:00Ghost UserServlet for Active Audits/Token ImportsIn the ACE AMs for the Chronopolis Project, we've been ingesting large amounts of collections. We've noticed recently when trying to browse it is difficult to find out which collections are actively auditing, and it's almost impossible t...In the ACE AMs for the Chronopolis Project, we've been ingesting large amounts of collections. We've noticed recently when trying to browse it is difficult to find out which collections are actively auditing, and it's almost impossible to see what collections are ingesting tokens.
Given that these both use the same thread pool, as they are processes which modify collections, it would be nice to have a servlet which can show any active processes. We will probably want to show information from the audit/token ingest when browsing the collections in this servlet. Maybe retrieving information from the CollectionThreadPoolExecutor. As this is still in its conception, nothing is really set in stone yet.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/59NPE in IngestThreadPool submitTokens2019-04-10T14:25:05-04:00Ghost UserNPE in IngestThreadPool submitTokensIf a token store is resubmitted for a collection before it finishes processing, a null pointer will be returned in place of a Future which will throw a NullPointerException when trying to use it.If a token store is resubmitted for a collection before it finishes processing, a null pointer will be returned in place of a Future which will throw a NullPointerException when trying to use it.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/58UI Display Incorrect Permissions2019-04-10T14:25:21-04:00Ghost UserUI Display Incorrect PermissionsIn the Report Servlet, a user might see options for removing/auditing a collection even if they do not have permission to do so. When attempting to use any of the links provided they will be met with a 403 as expected. We should remove t...In the Report Servlet, a user might see options for removing/auditing a collection even if they do not have permission to do so. When attempting to use any of the links provided they will be met with a 403 as expected. We should remove the errant ui elements when a user does not have the permissions for them.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/57Events API2018-10-24T14:39:56-04:00Ghost UserEvents APICreate an API for accessing events associated with a Collection (and possibly on MonitoredItems? that is more Logging rather than an Event though). For the purpose of this issue, an Event is defined as: File Audit, Token Audit, or Token ...Create an API for accessing events associated with a Collection (and possibly on MonitoredItems? that is more Logging rather than an Event though). For the purpose of this issue, an Event is defined as: File Audit, Token Audit, or Token Ingest. We should be able to see any past, current, and queued Events for a Collection.
Some brief prototyping:
`/rest/collections/<id>/events`
```
{
"current": {},
"queued": {"event_type": "file_audit", "status": "queued"},
"history": [{"event_type": "token_ingest", "status": "success", "session": 1},
{"event_type": "token_audit", "status": "success", "session":2}]
}
```
`/rest/collections/<id>/events/<session>`
```
{
"event_type": "token_ingest",
"status": "success",
"session": 1
}
```
`/rest/collections/<id>/events/<session>/log`
```
{
event_log entries
...
}
```
The main api for getting events on a collection is probably a bit misguided at the moment as we won't want to respond with all the history events ever on a collection, but it's a simple example for now. Likewise I'm not entirely sure what the data structure for an event should look like.1.14 - Extend REST APIhttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/56UI Access for PartnerSite/TokenImportStatus/etc2018-05-24T15:46:53-04:00Ghost UserUI Access for PartnerSite/TokenImportStatus/etcWe have some Servlets which aren't really accessible through the main UI and only through other Servlets. We should first figure out which servlets we should provide access to from the main page and then figure out a good method of doing...We have some Servlets which aren't really accessible through the main UI and only through other Servlets. We should first figure out which servlets we should provide access to from the main page and then figure out a good method of doing so (the top bar is getting crowded - maybe a new nav bar is needed).
The two which stand out immediately are
* PartnerSite
* TokenImportStatushttps://gitlab.umiacs.umd.edu/adapt/ace/-/issues/55Deploy ace-ims-ear as part of release cycle2018-05-24T15:25:43-04:00Ghost UserDeploy ace-ims-ear as part of release cycleRelated to #53 - when creating an image for the IMS we'll want an artifact which we can download and deploy.
I'm not sure if we'll want to handle this within the ace-dist module as that currently handles the ace-am packaging. It would b...Related to #53 - when creating an image for the IMS we'll want an artifact which we can download and deploy.
I'm not sure if we'll want to handle this within the ace-dist module as that currently handles the ace-am packaging. It would be nice to have separate artifacts for each (am/ims) which we should keep in mind when completing this.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/54State Enum for MonitoredItems2018-05-22T12:57:10-04:00Ghost UserState Enum for MonitoredItemsInstead of using 'A', 'C', 'T', 'M', etc, we should have a state enum for MonitoredItems. This will require us to add an AttributeConverter so that we can maintain compatibility with what's in the database, and possibly some type of mapp...Instead of using 'A', 'C', 'T', 'M', etc, we should have a state enum for MonitoredItems. This will require us to add an AttributeConverter so that we can maintain compatibility with what's in the database, and possibly some type of mapping for when we get MonitoredItems through the rest apis.https://gitlab.umiacs.umd.edu/adapt/ace/-/issues/53Docker Images2018-05-24T15:25:43-04:00Ghost UserDocker ImagesIt would be nice to have Docker images for ease of deployment, or at least showing what a production ACE setup might look like.
For the AuditManager, an example Dockerfile can be seen on the [UA Libraries github](https://github.com/uali...It would be nice to have Docker images for ease of deployment, or at least showing what a production ACE setup might look like.
For the AuditManager, an example Dockerfile can be seen on the [UA Libraries github](https://github.com/ualibraries/ace-audit-manager)
The IMS will likely be trickier, as it will involve setting up a JEE server and various resources in order for the IMS to run.