Backup non-utf8 characters

[sstarcher]

I'm using vaultproject.io from Hashicorp that stores encrypted data in the Key/Value store of consul.

On backup I'm getting

Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.5.1', 'console_scripts', 'consulate')()
  File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 285, in main
  File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 134, in kv_backup
  File "build/bdist.linux-x86_64/egg/consulate/api/kv.py", line 236, in records
  File "build/bdist.linux-x86_64/egg/consulate/api/kv.py", line 303, in _get_all_items
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 77, in _get_list
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 61, in _get
  File "build/bdist.linux-x86_64/egg/consulate/adapters.py", line 77, in get
  File "build/bdist.linux-x86_64/egg/consulate/adapters.py", line 108, in _process_response
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 119, in __init__
  File "build/bdist.linux-x86_64/egg/consulate/api/base.py", line 147, in _demarshal
  File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 6: invalid start byte

[gmr]

What version of Python? Is the data binary?

[sstarcher]

The data is encrypted and that was python 2.7.9

[sstarcher]

I also just attempted on python 3.4

root@ef841a56c77d:/# python --version
Python 3.4.3
root@ef841a56c77d:/# ls /usr/local/lib/python3.4/site-packages/consulate-0.5.1-py3.4.egg
/usr/local/lib/python3.4/site-packages/consulate-0.5.1-py3.4.egg

with the same error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 6: invalid start byte

[gmr]

That's pretty interesting, I can put together a fix, but am concerned about the implications of binary data in the KV without any mime-type or what not to gleam what it is. Will pull something together, I think I'm not far behind you in using Vault.

[sstarcher]

Over concern with vault was if someone overwrote an important secret and having no way to recover it. I was going to use consulate to backup the K/V store in a versioned manner.

[gmr]

Makes sense. I'll try and get something out to address this today.

[gmr]

Sorry for the delay, this now works for me in Python 2 and 3. I'll be releasing 0.6.0 with this fix soon.

[sheldonh]

Looking forward to 0.6.0. :-)

[sheldonh]

@gmr Heads up. I've tested with 0e0fef2 and, although the backup gets further, printing out some of the data in the vault/ prefix, it still chokes with:

Traceback (most recent call last):
  File "/home/ubuntu/consulate/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 424, in main
  File "build/bdist.linux-x86_64/egg/consulate/cli.py", line 186, in kv_backup
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 5: invalid continuation byte

[sheldonh]

I have a consul-data-dir tarball for you to test with, if that'll help:

https://s3.amazonaws.com/starjuice/consul-data-dir.tar.gz

[gmr]

Thanks, I'm trying to get 0.6 out the door right now, testing against your data before doing so.

[gmr]

Hmm I see, so if it's binary data and trying to backup to JSON, it's blowing up.

[gmr]

Quickest fix is making a kv flag that allows backups using base64 encoded values.

[sheldonh]

Sounds great. For backups, readability is a nice-to-have. :-)
On 22 Jul 2015 5:00 pm, "Gavin M. Roy" [email protected] wrote:

Quickest fix is making a kv flag that allows backups using base64 encoded
values.

—
Reply to this email directly or view it on GitHub
#41 (comment).

[gmr]

I just committed this, will be releasing 0.6 shortly.

usage: consulate kv backup [-h] [-b] [-f [FILE]]

optional arguments:
  -h, --help            show this help message and exit
  -b, --base64          Base64 encode key values
  -f [FILE], --file [FILE]
                        JSON file to read instead of stdin

(restore has the same flag)

[gmr]

The tarball was very useful, btw. Here's the dump of it:

[["e0463ed4", 0, "bnVsbA=="], ["path/504d4c8b", 0, "YmFy"], ["vault/core/audit", 0, "bAAAAAQFqTcI/A5zgLSBBXUUbzKhQ8Gq/hKX8+TNay+vUFB4aZnTZRKj7A5jICyyjOw="], ["vault/core/auth", 0, "AAAAAQGljDhexOPvZhUJHO7C4yZ5rsxeZCR1l32hcWzYGU5VVc52fpCH7hJCwMOF7Kg63slvJUSM/7o04li3YZhvIGZG3bsuEk/r04uxfS1nNhiaX2EWERjr4uMm1yLswsIYukzw3ri7QJOUlrgRZdrN6r2o2BpGhzx7F6aJ26URLMIYvziHpR6g0hBJR06qo9O3fHow/+kBcS6aIkasdfWQzflcdw0J55dsRtvfqoHuWaog"], ["vault/core/keyring", 0, "AAAAAQGst6aFwXBuedBuRW6WCkwdxW7fu40emfM82H+sk3qhn4L6eECZscMmet5PAXDblHgTWGN8SW+dxsGVpkJxkh8Ioz+5k8NfNB9yF8iqhlNvMOi9FKNifBnPVuljaPqFSkgjDJ24FZupQeCAGADI7Q/QqnjUXt8ZiyodSvgSaLQ/GAohPy9pFnVVNJ0BF8iWeb54wOCjlLHhHdqPWoP+bI0BOdO5pG1gRGlmTaTDTer1xAOwUgjgFCSq1rke4AI1hr4BK4CrikYB/fFa5osQeNn951ftDamcrhprV+0+LfIgrHY53g=="], ["vault/core/leader/b00b2c56-6e4c-512e-208a-5c088a40e15c", 0, "AAAAAQG58/h3N8YZRX2pvZix9fB+WKAjRoCNl1MtkXgegB9fxX/iHsGddBpHBYcLRQmpbw2H"], ["vault/core/lock", 3304740253564472344, "YjAwYjJjNTYtNmU0Yy01MTJlLTIwOGEtNWMwODhhNDBlMTVj"], ["vault/core/master", 0, "AAAAAQFGSEayJOgzlrM87FNrppVpxRO8ASBKrq4l52QEgs6B/8vq9+sN0xDpj8wHWk7BQP7qsHRPjfoh1axxCXpO8KYcxiif7P6FidZbVSNfTv+KcS6MTbVqhZy5IVLltKqTogkHOhnlZ8DdfOFVnCAVa5jBd8Izf3APqA2Jr8kzfAjwj1SerUtxD9pRSMLIkwWm"], ["vault/core/mounts", 0, "AAAAAQEU6rgowv4HSaTpvFcZXL7WwGinMBL3rrhvqxRFI/dzWJi/QiW9s8AyI29Mt5Pq3UJ1EPMG/E1vLZke3IMZPMLpQLMMjKkDi8eDsyyTcf/+V8mZjj2hE58nklTxhsMsgHN+MAUdTjH/b3DNOXC40FVnEUHhud1zxoKtXZc97/0zHSz3VtXtd9hco1RhuM9rxksKBgS5gZ/XVWmRWu+sMa+qzBe6Zbrguny8II7c+Ydi6Aae+HXqfe8Yxk7aOqzUIL+ikD8si3/x6NeSeIPaT1LRFxLrMkMVzXfthNeoOi+3E9T/iScZtlEtta6axS7hzNOVSZZVzFsVIhjKOu9JF/jrwhHY0WgaLqZw/nLWsm+L7AWVonr5qAoV523+jCzeRUewaVQ4HtQh5in3Ws0d9xwSNnoUUitQxS/+WrWhf8lnbWyhseRBG9lDaArAs5qT/WomwTT2TNU="], ["vault/core/seal-config", 0, "eyJzZWNyZXRfc2hhcmVzIjo1LCJzZWNyZXRfdGhyZXNob2xkIjozfQ=="], ["vault/sys/token/id/8bf62cb0c237f7f7ef562e9bed921900c49499c0", 0, "AAAAAQEQWTmang0AESkIE42MH3RC+bB/e7RCoSBhbIbFPGAzfNY3Ea5gfsgwWi3rqbPrvPEKF5vI0vtlC51LmJr0SJEwpQq2DwVAuPVkqv/nc6UxD926cA9fGBvfWLmBnXSR1eQW5ddQMZlOawPenWPUjOTGRRYb4Wifx0h8yM1YWsLapTo76cu8s9VO8GmiooSZs6tEVsC6+8cd4D2G4bdgQSEiJlXap546ZONILefJ1i3l"], ["vault/sys/token/salt", 0, "AAAAAQFcm5UYRwErG1ztcKgVS6uGqjbZ3rQZr8WK4uTZXZTH4ZPJhFNgAtASlibjZoPgU4YQSt/qy+bmAz96G3MsjPAh"]]

[sheldonh]

Solid. Backup and restore of vault now work with --base64. Thank you so much, @gmr. I can sleep in peace now. :-)