README.md 11.4 KB
Newer Older
Derek Yarnell's avatar
Derek Yarnell committed
1
# umobj
2
3
4
5
6
7
8
9

[![PyPI](https://img.shields.io/pypi/v/umobj.svg)](https://pypi.python.org/pypi/umobj)
[![LGPL License badge](https://img.shields.io/badge/License-LGPL-blue.svg)](./LICENSE)
<!-- [![GitHub tag](https://img.shields.io/github/tag/umiacs/umobj.svg)](https://github.com/umiacs/umobj/releases)
     shields.io is having some serious problems with certain
	 services/badges -->


Liam Monahan's avatar
Liam Monahan committed
10
Command-line utilities for S3-compatible Object Storage
11

Liam Monahan's avatar
Liam Monahan committed
12
These utilities provide a command-line interface to an Amazon S3-compatible
Liam Monahan's avatar
Liam Monahan committed
13
file/object storage services, also known as an Object Store.
Derek Yarnell's avatar
Derek Yarnell committed
14

15
16
17
18
19
20
21
22
## Installation

```pip install umobj```

or

```python setup.py install```

Derek Yarnell's avatar
Derek Yarnell committed
23
24
## Setup

25
For convenience, consider setting a few environmental variables that umobj will use to authenticate you.
Derek Yarnell's avatar
Derek Yarnell committed
26

Liam Monahan's avatar
Liam Monahan committed
27
28
 * *OBJ_ACCESS_KEY_ID* - The user's access key
 * *OBJ_SECRET_ACCESS_KEY* - The user's secret key
Liam Monahan's avatar
Liam Monahan committed
29
 * *OBJ_SERVER* - The object server endpoint
30

Liam Monahan's avatar
Liam Monahan committed
31
For example, if you use the ```bash``` shell, you can add the following to your
Derek Yarnell's avatar
Derek Yarnell committed
32
33
```.bashrc``` or ```.bash_profile```.

Derek Yarnell's avatar
Derek Yarnell committed
34
```bash
35
36
export OBJ_ACCESS_KEY_ID="<YOUR_ACCESS_KEY>"
export OBJ_SECRET_ACCESS_KEY="<YOUR_SECRET_KEY>"
37
export OBJ_SERVER="obj.umiacs.umd.edu"
Derek Yarnell's avatar
Derek Yarnell committed
38
```
Derek Yarnell's avatar
Derek Yarnell committed
39

Liam Monahan's avatar
Liam Monahan committed
40
41
42
You can optionally pass the <code>--access_key</code>,
<code>--secret_key</code> and <code>--server</code> switches to the utilities
to get the same behavior.
Derek Yarnell's avatar
Derek Yarnell committed
43

Liam Monahan's avatar
Liam Monahan committed
44
## Utilities
Derek Yarnell's avatar
Derek Yarnell committed
45
46

These utilities use a common syntax for specifying the bucket and an optional
Liam Monahan's avatar
Liam Monahan committed
47
key or path.  The bucket and key are separated by a ```:```.  For example,
Liam Monahan's avatar
Liam Monahan committed
48
49
50
51
52
53
54
55
56
57
58
```my_bucket:bar.txt``` denotes the bucket ```my_bucket``` and the key
```bar.txt```.  For more information on a specific utility, you can pass the
flag ```-h``` or ```--help``` for help.

* [lsobj](#lsobj)
* [mkobj](#mkobj)
* [cpobj](#cpobj)
* [rmobj](#rmobj)
* [chobj](#chobj)
* [syncobj](#syncobj)
* [catobj](#catobj)
Raushan Alleyne's avatar
Raushan Alleyne committed
59
* [streamobj](#streamobj)
Raushan Alleyne's avatar
Raushan Alleyne committed
60
* [cmpobj](#cmpobj)
61
* [webobj](#webobj)
Liam Monahan's avatar
Liam Monahan committed
62
63

<a name="lsobj"></a>
Derek Yarnell's avatar
Derek Yarnell committed
64
65
### lsobj

Liam Monahan's avatar
Liam Monahan committed
66
67
To list buckets and keys in the Object Store you can use the <b>lsobj</b>
command.  If given without an argument, it will list your buckets.
Derek Yarnell's avatar
Derek Yarnell committed
68

Derek Yarnell's avatar
Derek Yarnell committed
69
```bash
Derek Yarnell's avatar
Derek Yarnell committed
70
71
72
73
74
75
$ lsobj
bob
test
zeta
```

Liam Monahan's avatar
Liam Monahan committed
76
You can then give it a bucket name to list the contents within that bucket.
Derek Yarnell's avatar
Derek Yarnell committed
77

Derek Yarnell's avatar
Derek Yarnell committed
78
```bash
Liam Monahan's avatar
Liam Monahan committed
79
% lsobj -l test
Derek Yarnell's avatar
Derek Yarnell committed
80
81
82
83
84
85
86
87
88
89
90
-rwx---	2013-10-04T15:25:09.000Z	    1.8 KB	UMIACSCA.pem
-rwx---	2013-10-04T15:25:24.000Z	  311.4 KB	cover.jpg
-rwx---	2013-10-04T15:27:39.000Z	    0.0 b 	foo/
-rwx---	2013-10-04T15:27:40.000Z	    0.0 b 	foo/bar
-rwx---	2013-10-04T15:25:32.000Z	   31.8 KB	screenshot.jpg
-rwx---	2013-10-04T15:26:48.000Z	    8.3 KB	thunderbird.xpm
================================================================================
		TOTAL:  	  353.3 KB 	6 Files
```

#### Directories
Liam Monahan's avatar
Liam Monahan committed
91

Liam Monahan's avatar
Liam Monahan committed
92
93
In an Object Store, there are only buckets and keys.  This means that any
apparent directory structure is only emulated using the UNIX ```/``` character
Liam Monahan's avatar
Liam Monahan committed
94
95
96
in a key name.  Any key in the bucket ending in a ```/``` will be interpreted
as a directory.  You can also list only subdirectories with the ```lsobj```
utility.
Derek Yarnell's avatar
Derek Yarnell committed
97

Derek Yarnell's avatar
Derek Yarnell committed
98
```bash
Liam Monahan's avatar
Liam Monahan committed
99
% lsobj -l test:foo/
Derek Yarnell's avatar
Derek Yarnell committed
100
101
102
103
104
foo/
-rwx---	2013-10-04T15:27:39.000Z	    0.0 b 	foo/
-rwx---	2013-10-04T15:27:40.000Z	    0.0 b 	foo/bar
================================================================================
		TOTAL:  	    0.0 b  	2 Files
Derek Yarnell's avatar
Derek Yarnell committed
105
```
Derek Yarnell's avatar
Derek Yarnell committed
106

Liam Monahan's avatar
Liam Monahan committed
107
<a name="mkobj"></a>
Derek Yarnell's avatar
Derek Yarnell committed
108
109
### mkobj

110
mkobj creates buckets and directories in the Object Store.
Derek Yarnell's avatar
Derek Yarnell committed
111
112

<b>Please note that bucket names are unique in the Object Store, so you may
Liam Monahan's avatar
Liam Monahan committed
113
very well get an error back that the name has already been taken.</b>
Derek Yarnell's avatar
Derek Yarnell committed
114

Derek Yarnell's avatar
Derek Yarnell committed
115
```bash
Liam Monahan's avatar
Liam Monahan committed
116
% mkobj foo
Derek Yarnell's avatar
Derek Yarnell committed
117
Created bucket foo.
Liam Monahan's avatar
Liam Monahan committed
118
% lsobj
Derek Yarnell's avatar
Derek Yarnell committed
119
120
121
122
123
124
bob
foo
test
zeta
```

Liam Monahan's avatar
Liam Monahan committed
125
You can also create directories to group your data.
Derek Yarnell's avatar
Derek Yarnell committed
126

Derek Yarnell's avatar
Derek Yarnell committed
127
```bash
Derek Yarnell's avatar
Derek Yarnell committed
128
% mkobj foo:bar/
Liam Monahan's avatar
Liam Monahan committed
129
% lsobj -l foo
Derek Yarnell's avatar
Derek Yarnell committed
130
131
-rwx---	2013-10-04T15:38:38.000Z	    0.0 b 	bar/
================================================================================
Liam Monahan's avatar
Liam Monahan committed
132
		TOTAL:  	    0.0 b  	1 File
Derek Yarnell's avatar
Derek Yarnell committed
133
134
```

Liam Monahan's avatar
Liam Monahan committed
135
<a name="mkobj"></a>
Derek Yarnell's avatar
Derek Yarnell committed
136
137
### cpobj

Liam Monahan's avatar
Liam Monahan committed
138
139
140
141
142
143
Copying files to the Object Store can be done per-file or recursively both to
and from the Object Store.

To copy a single file to the Object Store you can use <code>cpobj</code> and
specify a bucket with a trailing <code>:</code> (with an optional additional
path).
Derek Yarnell's avatar
Derek Yarnell committed
144

Derek Yarnell's avatar
Derek Yarnell committed
145
```bash
Derek Yarnell's avatar
Derek Yarnell committed
146
% cpobj test.png foo:
Liam Monahan's avatar
Liam Monahan committed
147
148
100% |#########################################################################|
% lsobj -l foo
Derek Yarnell's avatar
Derek Yarnell committed
149
150
151
152
153
154
-rwx---	2013-10-04T15:38:38.000Z	    0.0 b 	bar/
-rwx---	2013-10-07T20:06:48.000Z	   18.3 KB	test.png
================================================================================
		TOTAL:  	   18.3 KB 	2 Files
```

Liam Monahan's avatar
Liam Monahan committed
155
156
To copy a directory of files you will need to use the <code>-r</code> or
<code>--recursive</code> flags.
Derek Yarnell's avatar
Derek Yarnell committed
157

Derek Yarnell's avatar
Derek Yarnell committed
158
```bash
Derek Yarnell's avatar
Derek Yarnell committed
159
160
161
162
% lsobj foo
================================================================================
		TOTAL:  	    0.0 b  	0 Files
% cpobj -r /tmp/stuff foo:stuff
Liam Monahan's avatar
Liam Monahan committed
163
164
165
166
100% |#########################################################################|
100% |#########################################################################|
100% |#########################################################################|
% lsobj -l foo
Derek Yarnell's avatar
Derek Yarnell committed
167
168
169
170
171
172
173
174
-rwx---	2013-10-08T00:13:52.000Z	    0.0 b 	stuff/
-rwx---	2013-10-08T00:13:55.000Z	   26.0 KB	stuff/WindowsSecurity.png
-rwx---	2013-10-08T00:13:54.000Z	  226.5 KB	stuff/changepass.tiff
-rwx---	2013-10-08T00:13:55.000Z	   18.3 KB	stuff/test.png
================================================================================
		TOTAL:  	  270.8 KB 	9 Files
```

Liam Monahan's avatar
Liam Monahan committed
175
<a name="rmobj"></a>
Derek Yarnell's avatar
Derek Yarnell committed
176
### rmobj
Liam Monahan's avatar
Liam Monahan committed
177
178

You can delete your buckets and keys with ```rmobj```.  It can take a bucket
179
180
and work recursively to delete all the files and the bucket itself.
It can also just delete specific files in a bucket.
Derek Yarnell's avatar
Derek Yarnell committed
181

Derek Yarnell's avatar
Derek Yarnell committed
182
```bash
Derek Yarnell's avatar
Derek Yarnell committed
183
184
185
186
187
% rmobj -r foo
 Are you sure you want to remove all the contents of the bucket 'foo'? [yes/no] yes
 Do you want to remove the bucket 'foo'? [yes/no] no
```

Derek Yarnell's avatar
Derek Yarnell committed
188
```bash
Liam Monahan's avatar
Liam Monahan committed
189
% lsobj -l foo
Derek Yarnell's avatar
Derek Yarnell committed
190
191
192
193
194
195
-rwx---	2013-10-09T18:44:20.000Z	    1.0 KB	setup.cfg
-rwx---	2013-10-09T18:44:17.000Z	  781.0 b 	setup.py
-rwx---	2013-10-09T18:44:09.000Z	  289.0 b 	test-requirements.txt
================================================================================
		TOTAL:  	    2.0 KB 	3 Files
% rmobj foo:setup.cfg foo:setup.py
Liam Monahan's avatar
Liam Monahan committed
196
% lsobj -l foo
Derek Yarnell's avatar
Derek Yarnell committed
197
198
-rwx---	2013-10-09T18:44:09.000Z	  289.0 b 	test-requirements.txt
================================================================================
Liam Monahan's avatar
Liam Monahan committed
199
		TOTAL:  	  289.0 b  	1 File
Derek Yarnell's avatar
Derek Yarnell committed
200
201
```

Liam Monahan's avatar
Liam Monahan committed
202
You can also remove directories within a bucket.  To do so, you will need to
203
204
205
pass the <code>-r</code> flag.  This will prompt once for the removal of every
key under that directory unless the <code>-f</code> flag is passed as well.  
Specifying <code>-i</code> will prompt for every file.
Liam Monahan's avatar
Liam Monahan committed
206

Derek Yarnell's avatar
Derek Yarnell committed
207
```bash
Derek Yarnell's avatar
Derek Yarnell committed
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
% lsobj foo
-rwx---	2013-10-10T21:58:17.000Z	    0.0 b 	research/
-rwx---	2013-10-10T21:58:17.000Z	    0.0 b 	research/papers/
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/papers/paper1.tex
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/papers/paper2.tex
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/posters/
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/posters/poster1.tex
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/posters/poster2.tex
================================================================================
		TOTAL:  	    0.0 b  	7 Files
% rmobj -rf foo:research/posters/
% lsobj foo
-rwx---	2013-10-10T21:58:17.000Z	    0.0 b 	research/
-rwx---	2013-10-10T21:58:17.000Z	    0.0 b 	research/papers/
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/papers/paper1.tex
-rwx---	2013-10-10T21:58:18.000Z	    0.0 b 	research/papers/paper2.tex
================================================================================
		TOTAL:  	    0.0 b  	4 Files
```

228
229
230
231
232
233
234
235
236
237
You can use a glob character to delete all bucket contents, but not the
bucket itself.

```bash
% rmobj -rf foo:*
% lsobj foo
================================================================================
		TOTAL:  	    0.0 b  	0 Files
```

Liam Monahan's avatar
Liam Monahan committed
238
<a name="chobj"></a>
Derek Yarnell's avatar
Derek Yarnell committed
239
### chobj
Liam Monahan's avatar
Liam Monahan committed
240
241

Use ```chobj``` to set permissions/ACLs on buckets and keys.
Derek Yarnell's avatar
Derek Yarnell committed
242

Derek Yarnell's avatar
Derek Yarnell committed
243
```bash
Liam Monahan's avatar
Liam Monahan committed
244
% chobj -m clear -p liam:FULL_CONTROL -p derek:FULL_CONTROL foo:test-requirements.txt
Derek Yarnell's avatar
Derek Yarnell committed
245
246
```

Liam Monahan's avatar
Liam Monahan committed
247
<a name="syncobj"></a>
Derek Yarnell's avatar
Derek Yarnell committed
248
### syncobj
Liam Monahan's avatar
Liam Monahan committed
249
250
251
252
You can use ```syncobj``` to synchronize files and directories into the Object
Store.  It does this by comparing the checksum between the local and remote
versions of the key.  This does not follow symlinks and will warn when it
encounters files (character, block, fifo, etc...) that it can not archive.
Derek Yarnell's avatar
Derek Yarnell committed
253

Derek Yarnell's avatar
Derek Yarnell committed
254
```bash
Derek Yarnell's avatar
Derek Yarnell committed
255
256
257
258
% syncobj -r devel/puppet test:
100% |#########################################################################|
```

Liam Monahan's avatar
Liam Monahan committed
259
<a name="catobj"></a>
Liam Monahan's avatar
Liam Monahan committed
260
### catobj
Liam Monahan's avatar
Liam Monahan committed
261
262
263
264
265
266
267

Use ```catobj``` to print the contents of a key to stdout.  This may be useful
for streaming the content of a file to another command for some sort of
processing without having to write the key to an intermediary file.  If any
errors are printed in the process of running this command, you can be assured
that they will be printed to stderr, and should not interfere with what's being
printed to stdout.
Liam Monahan's avatar
Liam Monahan committed
268
269
270
271
272
273

```bash
% catobj foo:something.txt
Fallaces sunt rerum species.  Mutantur omnia nos et mutamur in illis.
```

Raushan Alleyne's avatar
Raushan Alleyne committed
274
275
276
277
<a name="streamobj"></a>
### streamobj

Use ```streamobj``` to read the contents of a data stream that has been piped from
278
stdout and upload it to a bucket. This allows you to send large amounts of data to
Raushan Alleyne's avatar
Raushan Alleyne committed
279
280
a bucket without saving it all on disk or in memory first. A name for the file
must be specified using the -n flag because a name can't be inferred from the
281
data stream.
Raushan Alleyne's avatar
Raushan Alleyne committed
282
283
284
285
286

```bash
% stream_source | streamobj -n filename foo:somedirectory
```

Raushan Alleyne's avatar
Raushan Alleyne committed
287
288
289
<a name="cmpobj"></a>
### cmpobj

Raushan Alleyne's avatar
Raushan Alleyne committed
290
Use ```cmpobj``` to check the md5 sum of a key in a bucket or compare the md5 sums of a local directory to a bucket. The key is downloaded in order for the correct md5 hash to be generated. Bagit archives can also be verified, which is done by retrieving the manifest and comparing the expected checksums to the ones computed by downloading each key.
Raushan Alleyne's avatar
Raushan Alleyne committed
291
292
293
294
295
296

```bash
% cmpobj mybucket:foo.txt
9029668a43dfa60f1f267eea59111bbc
```

297
298
299
300
301
302
303
304
<a name="webobj"></a>
### webobj

Use ```webobj``` to configure web configurations of a bucket. Currently, this
supports setting a S3 website configuration such that bucket can be served as a
static site.

```bash
305
306
% webobj -m create -c website --index=index.html --error=error.html mybucket
% webobj -m examine -c website mybucket
307
308
Index: index.html
Error Key: error.html
309
% webobj -m delete -c website mybucket
310
311
312

```

313
314
## Requirements

Liam Monahan's avatar
Liam Monahan committed
315
These utilities and libraries all work with Python 2.7+ and Python 3.6+
Derek Yarnell's avatar
Derek Yarnell committed
316

317
318
319
They require the following packages:

- Boto > 2.49.0
320
321
- FileChunkIO
- Progressbar
Liam Monahan's avatar
Liam Monahan committed
322
- qav
323

Ryan Leimenstoll's avatar
Ryan Leimenstoll committed
324
325
## Report an issue

Liam Monahan's avatar
Liam Monahan committed
326
The official repository for umobj is hosted in the [UMIACS Gitlab](https://gitlab.umiacs.umd.edu/staff/umobj) service. Please open new issues in the Gitlab tracker.
327
328
329

## License

Liam Monahan's avatar
Liam Monahan committed
330
    umobj - Command-line utilities for S3-compatible Object Storage
Ryan Leimenstoll's avatar
Ryan Leimenstoll committed
331
    Copyright (C) 2017  UMIACS
332

333
334
335
336
    This library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Lesser General Public
    License as published by the Free Software Foundation; either
    version 2.1 of the License, or (at your option) any later version.
337

338
339
340
341
    This library is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    Lesser General Public License for more details.
342

343
344
345
    You should have received a copy of the GNU Lesser General Public
    License along with this library; if not, write to the Free Software
    Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA
346

347
348
    Email:
        github@umiacs.umd.edu