staff issueshttps://gitlab.umiacs.umd.edu/groups/staff/-/issues2021-11-30T10:44:04-05:00https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/73lsobj | bug with -l option (Show long listing including ACLs)2021-11-30T10:44:04-05:00Elena Zotkinalsobj | bug with -l option (Show long listing including ACLs)```
(env-umobj) bash-4.4$ lsobj lr_data_2015_2018 -l
-rwx--- 08/01/2018 03:29:19 0.0 b 2018FINRA_TRACE/
ACLs: ezotkina:FULL_CONTROL louiqa:FULL_CONTROL
-rwx--- 08/03/2018 01:11:16 192.3 MB 2018FINRA_TRACE/0009-CORP-2002-07-01-...```
(env-umobj) bash-4.4$ lsobj lr_data_2015_2018 -l
-rwx--- 08/01/2018 03:29:19 0.0 b 2018FINRA_TRACE/
ACLs: ezotkina:FULL_CONTROL louiqa:FULL_CONTROL
-rwx--- 08/03/2018 01:11:16 192.3 MB 2018FINRA_TRACE/0009-CORP-2002-07-01-2002-12-31.zip
ACLs: draghun1:FULL_CONTROL ezotkina:FULL_CONTROL louiqa:FULL_CONTROL
tharindu:FULL_CONTROL
.................
-rwx--- 08/16/2018 03:40:25 11.7 GB 2018FINRA_TRACE/TRACE.7z
ACLs: draghun1:FULL_CONTROL
Traceback (most recent call last):
File "/nfshomes/ezotkina/env-umobj/bin/lsobj", line 4, in <module>
__import__('pkg_resources').run_script('umobj==3.0.2', 'lsobj')
File "/nfshomes/ezotkina/env-umobj/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 654, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/nfshomes/ezotkina/env-umobj/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 1441, in run_script
exec(script_code, namespace, namespace)
File "/nfshomes/ezotkina/env-umobj/lib64/python3.6/site-packages/umobj-3.0.2-py3.6.egg/EGG-INFO/scripts/lsobj", line 152, in <module>
File "/nfshomes/ezotkina/env-umobj/lib64/python3.6/site-packages/umobj-3.0.2-py3.6.egg/EGG-INFO/scripts/lsobj", line 75, in list_bucket
AttributeError: 'dict' object has no attribute 'iteritems'
```
Looks like it fails when item has metadata.
Update: this is incompatibility issue between python2 and python3.Elena ZotkinaElena Zotkinahttps://gitlab.umiacs.umd.edu/staff/qav/-/issues/5Python3 Support2020-06-16T14:29:47-04:00Liam MonahanPython3 SupportUpgrade qav to be Python2 and 3 compatible.Upgrade qav to be Python2 and 3 compatible.Liam MonahanLiam Monahanhttps://gitlab.umiacs.umd.edu/staff/qav/-/issues/4Question validator with no choices silently doesn't ask the question2020-06-16T14:29:47-04:00Derek YarnellQuestion validator with no choices silently doesn't ask the questionhttps://gitlab.umiacs.umd.edu/staff/qav/blob/master/qav/questions.py#L107
I propose we log a warning level message that there are no choices.https://gitlab.umiacs.umd.edu/staff/qav/blob/master/qav/questions.py#L107
I propose we log a warning level message that there are no choices.https://gitlab.umiacs.umd.edu/staff/qav/-/issues/3URIValidator expansion2020-06-16T14:29:47-04:00Derek YarnellURIValidator expansionWe need to allow URIs to be more than ftp/http/https, since we can't easily enumerate these it should be a \w+ match.We need to allow URIs to be more than ftp/http/https, since we can't easily enumerate these it should be a \w+ match.https://gitlab.umiacs.umd.edu/staff/qav/-/issues/2dependencies2020-06-16T14:29:47-04:00Ghost Userdependencies*EDIT:* Updated patch to include README.md change
*EDIT2:* The tripple backpacks in the README.md patch causes markdown issues… reverting the patch here to the one just for setup.py, but emailing full patch to Liam
I think `netaddr...*EDIT:* Updated patch to include README.md change
*EDIT2:* The tripple backpacks in the README.md patch causes markdown issues… reverting the patch here to the one just for setup.py, but emailing full patch to Liam
I think `netaddr` is a dependency… [Line 6 in validators.py](https://gitlab.umiacs.umd.edu/staff/qav/blob/master/qav/validators.py#L6) I was getting ready to create a Homebrew formula, but they require a simple test. So I tried to create a script matching the one on the README.md and got the following error:
```
$ python test.py
Traceback (most recent call last):
File "test.py", line 1, in <module>
from qav.questions import Question
File "/usr/local/lib/python2.7/site-packages/qav/questions.py", line 1, in <module>
from qav.validators import Validator, CompactListValidator
File "/usr/local/lib/python2.7/site-packages/qav/validators.py", line 6, in <module>
from netaddr import IPAddress
ImportError: No module named netaddr
```
I then tried to fork `qav` but gitlab won’t let me due to an error. I’ll send a ticket to staff@umiacs with the details… So in lieu of a merge/pull request, here is a patch which I *think* should resolve the issue, however I have no personal experience publishing to PYPI.
```patch
From 771d92f83a97eaa449b537873ebe079013ebfa8e Mon Sep 17 00:00:00 2001
From: zbeekman <zbeekman@gmail.com>
Date: Wed, 8 Jul 2015 17:20:11 -0400
Subject: [PATCH] Added missing `netaddr` dependency
---
setup.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/setup.py b/setup.py
index d105353..99fb05a 100644
--- a/setup.py
+++ b/setup.py
@@ -21,6 +21,9 @@
author='Derek Yarnell',
author_email='derek@umiacs.umd.edu',
packages=['qav'],
+ install_requires=[
+ 'netaddr',
+ ],
url='https://github.com/UMIACS/qav',
license='MIT',
description='Question Answer Validation',
--
2.4.1
```
Put that in a file, say `qav.patch` and then `git apply qav.patch` or `git am --signoff qav.patch` to apply it and commit. After bumping the version number, and pushing to gitlab and github, I think you should be able to resubmit to PYPI, for this to take effect.https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/40syncobj recursive directory handling2020-05-12T22:37:14-04:00Ghost Usersyncobj recursive directory handlingWe may not be properly creating multiple levels of folder hierarchies when syncing data from S3.
```
(env)[rleimens@rhel7umdev umobj]$ bin/syncobj -r rleimens: /tmp/
100% |###########################################################...We may not be properly creating multiple levels of folder hierarchies when syncing data from S3.
```
(env)[rleimens@rhel7umdev umobj]$ bin/syncobj -r rleimens: /tmp/
100% |#######################################################################################################|
100% |#######################################################################################################|
100% |#######################################################################################################|
100% |#######################################################################################################|
Traceback (most recent call last): |
File "bin/syncobj", line 95, in <module>
args.multipart, True)
File "/srv/code/umobj_clean/umobj/umobj/transfer.py", line 136, in obj_download
download_file(key, filename)
File "/srv/code/umobj_clean/umobj/umobj/transfer.py", line 86, in download_file
os.makedirs(filename)
File "/srv/code/umobj_clean/env/lib64/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
<type 'type'>: [Errno 20] Not a directory: '/tmp/ldapreps/test/'
```Liam MonahanLiam Monahanhttps://gitlab.umiacs.umd.edu/staff/qav/-/issues/8Add Python Static Type Checking2020-03-31T15:26:03-04:00Ghost UserAdd Python Static Type CheckingNow that we are building off of python3 we can add static type checking.
https://docs.python.org/3/library/typing.htmlNow that we are building off of python3 we can add static type checking.
https://docs.python.org/3/library/typing.htmlhttps://gitlab.umiacs.umd.edu/staff/qav/-/issues/7Build RHEL7 RPMs for python32020-02-19T15:24:34-05:00Liam MonahanBuild RHEL7 RPMs for python3Build a version of qav called "python3-qav" via CI.Build a version of qav called "python3-qav" via CI.https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/71Update man page for chobj2020-02-03T15:32:15-05:00Liam MonahanUpdate man page for chobjUpdate chobj man page for recent behavior changes to chobj clear action.
https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/share/man/man1/chobj.1Update chobj man page for recent behavior changes to chobj clear action.
https://gitlab.umiacs.umd.edu/staff/umobj/blob/master/share/man/man1/chobj.1https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/50chobj clear on a bucket doesn't work2020-02-03T12:47:58-05:00Derek Yarnellchobj clear on a bucket doesn't work```
[derek@walrus:~ ] $ chobj -m clear test_umobj:
Traceback (most recent call last):
File "/usr/bin/chobj", line 169, in <module>
acl.add_user_grant('FULL_CONTROL', object.owner.id)
<type 'type'>: 'Bucket' object has no attribute ...```
[derek@walrus:~ ] $ chobj -m clear test_umobj:
Traceback (most recent call last):
File "/usr/bin/chobj", line 169, in <module>
acl.add_user_grant('FULL_CONTROL', object.owner.id)
<type 'type'>: 'Bucket' object has no attribute 'owner'
```https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/41Catch Invalid Character errors2020-01-27T17:59:19-05:00Ghost UserCatch Invalid Character errorsWe should catch XML invalid character values so that we don't throw an unseemly traceback similar to this:
```
$ rmobj -r crawled_genethn
Are you sure you want to remove all the contents of the bucket 'crawled_genethn'? [yes/no] y...We should catch XML invalid character values so that we don't throw an unseemly traceback similar to this:
```
$ rmobj -r crawled_genethn
Are you sure you want to remove all the contents of the bucket 'crawled_genethn'? [yes/no] yes
Traceback (most recent call last):
File "/scratch0/pyenv/bin/rmobj", line 65, in <module>
for key in bucket.list():
File "/scratch0/pyenv/lib/python2.7/site-packages/boto/s3/bucketlistresultset.py", line 34, in bucket_lister
encoding_type=encoding_type)
File "/scratch0/pyenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 473, in get_all_keys
'', headers, **params)
File "/scratch0/pyenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 407, in _get_all
xml.sax.parseString(body, h)
File "/usr/lib64/python2.7/xml/sax/__init__.py", line 49, in parseString
parser.parse(inpsrc)
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib64/python2.7/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib64/python2.7/xml/sax/expatreader.py", line 214, in feed
self._err_handler.fatalError(exc)
File "/usr/lib64/python2.7/xml/sax/handler.py", line 38, in fatalError
raise exception
<type 'type'>: <unknown>:1:62008: reference to invalid character number
```
We should give a CRITICAL error in this case. It may also be good to shoot off a DEBUG level error with the failing XML. https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/34error triggered by a logging message in obj_download2020-01-27T17:56:47-05:00Liam Monahanerror triggered by a logging message in obj_download```
Traceback (most recent call last):
File "/Users/liam/env/bin/cpobj", line 97, in <module>
args.multipart)
File "/Users/liam/env/lib/python2.7/site-packages/umobj/transfer.py", line 131, in obj_download
(key, key.size...```
Traceback (most recent call last):
File "/Users/liam/env/bin/cpobj", line 97, in <module>
args.multipart)
File "/Users/liam/env/lib/python2.7/site-packages/umobj/transfer.py", line 131, in obj_download
(key, key.size, filename))
<type 'type'>: 'ascii' codec can't decode byte 0xc2 in position 102: ordinal not in range(128)
```
We should alter/remove the logging message at line 131:
```python
logging.info("Downloading key %s (%d) to %s" %
(key, key.size, filename))
```https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/44Error during install attempt via pip2019-12-09T17:35:12-05:00Gregory KramidaError during install attempt via pipDistro: Ubuntu 16.04
Python: 3.5.2 (Also tried 2.7.12, same error)
Pip: 8.1.2
Pip installation fails with the following error:
```
~$ sudo -H pip3 install umobj
Collecting umobj
Using cached umobj-1.3.5.tar.gz
Collecting bo...Distro: Ubuntu 16.04
Python: 3.5.2 (Also tried 2.7.12, same error)
Pip: 8.1.2
Pip installation fails with the following error:
```
~$ sudo -H pip3 install umobj
Collecting umobj
Using cached umobj-1.3.5.tar.gz
Collecting boto (from umobj)
Using cached boto-2.42.0-py2.py3-none-any.whl
Collecting filechunkio (from umobj)
Using cached filechunkio-1.8.tar.gz
Collecting progressbar (from umobj)
Using cached progressbar-2.3.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-de2o42vu/progressbar/setup.py", line 5, in <module>
import progressbar
File "/tmp/pip-build-de2o42vu/progressbar/progressbar/__init__.py", line 59, in <module>
from progressbar.widgets import *
File "/tmp/pip-build-de2o42vu/progressbar/progressbar/widgets.py", line 121, in <module>
class FileTransferSpeed(Widget):
File "/usr/lib/python3.5/abc.py", line 133, in __new__
cls = super().__new__(mcls, name, bases, namespace)
ValueError: 'format' in __slots__ conflicts with class variable
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-de2o42vu/progressbar
```https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/67Python 3 support.2019-12-09T17:20:36-05:00Jason FanPython 3 support.Are there any plans to support Python 3? Also, umobj is installable with pip3, is this the desired behavior?Are there any plans to support Python 3? Also, umobj is installable with pip3, is this the desired behavior?https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/69Python3 Compatibility2019-12-09T14:25:38-05:00Ghost UserPython3 CompatibilityTo get ready for the move to python3 make sure the code base is compatible to run on both python2 and python3. This will help asses the work needed for boto/boto3To get ready for the move to python3 make sure the code base is compatible to run on both python2 and python3. This will help asses the work needed for boto/boto3https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/66Make a umobj docker image2019-10-28T13:12:55-04:00Liam MonahanMake a umobj docker imageMake a slim umobj docker image built off of python:2.7-slim to save us from having to pip install umobj on container runs.
Might want to also audit other projects upon completion for places that can be switched out to use this.Make a slim umobj docker image built off of python:2.7-slim to save us from having to pip install umobj on container runs.
Might want to also audit other projects upon completion for places that can be switched out to use this.https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/68Set the highest version of Python supported for pip installation2019-10-09T09:28:04-04:00Liam MonahanSet the highest version of Python supported for pip installationThis ticket was spawned from a user suggestion in #67.
We would like to limit which version of Python can install umobj. umobj only supports Python 2.6 and 2.7. We should set `python_requires`[0] in the project's `setup.py` to reflect...This ticket was spawned from a user suggestion in #67.
We would like to limit which version of Python can install umobj. umobj only supports Python 2.6 and 2.7. We should set `python_requires`[0] in the project's `setup.py` to reflect this.
*Ticket reserved for new Staffers to gain development experience*
[0] - https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requiresRishabh KaranthRishabh Karanthhttps://gitlab.umiacs.umd.edu/staff/umobj/-/issues/65Pacakge umobj for RHEL82019-07-29T10:42:39-04:00Liam MonahanPacakge umobj for RHEL8umobj is going to stay on python2 for now, os no immediate code changes. However, we need to build a version of the RPM on RHEL8 that asks for python2 versions of all its dependencies. I've already built all the dependencies into the r...umobj is going to stay on python2 for now, os no immediate code changes. However, we need to build a version of the RPM on RHEL8 that asks for python2 versions of all its dependencies. I've already built all the dependencies into the rhel8 repo tree with the `python2-` prefix.Liam MonahanLiam Monahanhttps://gitlab.umiacs.umd.edu/staff/umobj/-/issues/63webobj: SAXParseExceptions when calling get_website_configuration_obj()2019-07-28T12:53:16-04:00Liam Monahanwebobj: SAXParseExceptions when calling get_website_configuration_obj()I saw this crop up when adding python26 to the CI test suite.
https://gitlab.umiacs.umd.edu/staff/umobj/-/jobs/7471I saw this crop up when adding python26 to the CI test suite.
https://gitlab.umiacs.umd.edu/staff/umobj/-/jobs/7471https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/64mkobj: Prevent user from creating bucket with underscore2019-07-17T17:22:18-04:00Ghost Usermkobj: Prevent user from creating bucket with underscoreWe need to make sure our tooling doesn't allow users to create buckets that violate AWS's requirements.
Ceph seems to catch uppercase letters, but we need to start filtering out underscores.We need to make sure our tooling doesn't allow users to create buckets that violate AWS's requirements.
Ceph seems to catch uppercase letters, but we need to start filtering out underscores.Stephen EideStephen Eide