staff issueshttps://gitlab.umiacs.umd.edu/groups/staff/-/issues2021-03-24T13:05:50-04:00https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/72Using mvobj to rename large files slow and failed2021-03-24T13:05:50-04:00Ghost UserUsing mvobj to rename large files slow and failedI was trying to rename a 20GB file in a bucket using `mvobj` but it takes a very long time and failed.
```
mvobj scratch:transfer/AMI_exp/exp_conv_real_RIRs.tar scratch:transfer/AMI_exp/exp_conv_real_RIRs_0.tar
Traceback (most recent ca...I was trying to rename a 20GB file in a bucket using `mvobj` but it takes a very long time and failed.
```
mvobj scratch:transfer/AMI_exp/exp_conv_real_RIRs.tar scratch:transfer/AMI_exp/exp_conv_real_RIRs_0.tar
Traceback (most recent call last):
File "/home/zhy/.local/bin/mvobj", line 47, in <module>
preserve_acl=True)
File "/home/zhy/.local/lib/python3.5/site-packages/boto/s3/bucket.py", line 873, in copy_key
query_args=query_args)
File "/home/zhy/.local/lib/python3.5/site-packages/boto/s3/connection.py", line 671, in make_request
retry_handler=retry_handler
File "/home/zhy/.local/lib/python3.5/site-packages/boto/connection.py", line 1071, in make_request
retry_handler=retry_handler)
File "/home/zhy/.local/lib/python3.5/site-packages/boto/connection.py", line 1030, in _mexe
raise ex
File "/home/zhy/.local/lib/python3.5/site-packages/boto/connection.py", line 944, in _mexe
response = connection.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1242, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 311, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 272, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.5/ssl.py", line 791, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 575, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
```
And I opened the web interface at the same time to see a new file was first created and I thought the old file should be gone when it finishes. It feels like a `cp A B && rm A` operation (and `rm` failed) rather than a simple `mv`. Is this normal at all?https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/70Multipart Consistency issue on macOS2020-07-14T12:47:05-04:00Liam MonahanMultipart Consistency issue on macOSOn macOS, a file downloaded through multipart does not appear to have the right ETag, but the file does appear intact. The video file was the right size and it still played fine even though the md5 sum was wrong. On a UMIACS RHEL7 host...On macOS, a file downloaded through multipart does not appear to have the right ETag, but the file does appear intact. The video file was the right size and it still played fine even though the md5 sum was wrong. On a UMIACS RHEL7 host with umobj installed through RPM I was not able to recreate this problem.
* Python: 2.7
* Platform: macOS Mojave
In the following sequence the correct MD5 sum of the file used is `438506bc2706076990ac423648a27aa6`
```
(env27) hecuba:~ $ cpobj -Vm liam20191203:myvideo.mp4 .
INFO: Finished setting up logging.
INFO: Running ['/Users/liamwork/env27/bin/cpobj', '-Vm', 'liam20191203:myvideo.mp4', '.']
INFO: Download mode with bucket liam20191203.
INFO: Downloading with the key name/prefix myvideo.mp4.
INFO: Downloading to the local directory ..
INFO: Downloading key <Key: liam20191203,myvideo.mp4> (139820571) to .
INFO: Starting a multipart download for bucket liam20191203 and key myvideo.mp4
INFO: myvideo.mp4 : Size: 139820571 Chunk Size: 27075126 Number Chunks: 6
INFO: Starting downloading bytes 0 - 27075125.
INFO: Starting downloading bytes 27075126 - 54150251.
INFO: Starting downloading bytes 54150252 - 81225377.
INFO: Starting downloading bytes 81225378 - 108300503.
INFO: Starting downloading bytes 108300504 - 135375629.
INFO: Starting downloading bytes 135375630 - 139820570.
(env27) hecuba:~ $ md5 myvideo.mp4
MD5 (myvideo.mp4) = ec268a90658a40c0b95042f90c5839b4
(env27) hecuba:~ $ cpobj liam20191203:myvideo.mp4 myvideo_nomulti.mp4
100% |##########################################################################################|
(env27) hecuba:~ $ md5 myvideo_nomulti.mp4
MD5 (myvideo_nomulti.mp4) = 438506bc2706076990ac423648a27aa6
```
Even though I am reporting this problem on Python 2.7, it still happen on Python 3.6 on macOS as well.https://gitlab.umiacs.umd.edu/staff/umobj/-/issues/32upgrade umobj to use boto32021-04-01T14:09:14-04:00Liam Monahanupgrade umobj to use boto3The boto package is no longer being actively developed and all future development efforts are being directed towards boto3. I have been reading and I don't think there are many compelling features, but it will probably pay sooner rather...The boto package is no longer being actively developed and all future development efforts are being directed towards boto3. I have been reading and I don't think there are many compelling features, but it will probably pay sooner rather than later to be on the mainline version of the package. Both can be used side-by-side in the same package, however, since there's no namespacing issues.