Backing up binary data (Written by Hashicorp Vault) fails

[njsaunders]

I'm getting the following when attempting to run a consulate backup on my consul cluster:

(con)[Tue Nov 03 17:25:50] ~/r/temp/consulate$ consulate --api-host 10.1.15.125 --api-port 8500 kv backup --base64 -f consul_backup.json
Traceback (most recent call last):
  File "/Users/neil1/virtualenvs/con/bin/consulate", line 9, in <module>
    load_entry_point('consulate==0.6.0', 'console_scripts', 'consulate')()
  File "build/bdist.macosx-10.11-intel/egg/consulate/cli.py", line 469, in main
  File "build/bdist.macosx-10.11-intel/egg/consulate/cli.py", line 214, in kv_backup
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2014' in position 562: ordinal not in range(128)    

Looks like #41

Python 2.7.10
Consulate 0.6.0 (Both pip and master)

We've tracked the issue to Hashicorp Vault writing binary data.

Using python3.5 we instead get the error:

(test)[Tue Nov 03 17:41:37] ~/r/temp$ consulate --api-host 10.1.15.125 --api-port 8500 kv backup --base64 -f consul_backup.json
Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/Library/Python/2.7/site-packages/consulate/cli.py", line 214, in kv_backup
    records = [(k, f, base64.b64encode(v)) for k,f,v in records]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
TypeError: must be string or buffer, not None

If there's any other information we can provide please let us know!

[ben-nz]

Looks like I'm striking the same issue after pointing vault to our consul store:

[centos@consul-vault-i-ca11b50e ~]$ consulate kv backup
Traceback (most recent call last):
  File "/usr/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/usr/lib/python2.7/site-packages/consulate/cli.py", line 216, in kv_backup
    handle.write(json.dumps(records) + '\n')
  File "/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf9 in position 9: invalid start byte

Let me know if you'd like more info.

[gmr]

Can you provide a sample consul DB dir for me to try and backup?

[njsaunders]

Apologies for the delay - Consul DB containing Vault data attached.
consul-dir-vault-fail.tar.gz

[alexku-o]

@ben-nz - using -b like consulate kv backup -b worked for me (following what was suggested in #41)

[alexku-o]

@ben-nz - also, in case it wasn't obvious to you (as it wasn't to me), seems like you'd not use the -b option when you restore the file that you backed up with -b option (I tested that and that worked).

So, -b for backup, and no -b for restore.

[ben-nz]

@alexku-o Thanks mate I'll give that a try.

[tyx]

@alexku-o On my side I had to use -b (for base64) also to restire. If not, restore will succeed but vault would not be able to access its data anymore.

[alexku-o]

@tyx thanks. I actually tried restore with -b earlier but it was giving me some error. I ended up just using consul-backup because we have a simple backup/restore use case and it works perfectly well for us (and the fact that our test backup/restore worked with consul-backup but not with consulate).

edit: In case it came across that way, my comment here was certainly not in any way to discredit the incredible work that's been done on consulate. I appreciate a great deal of the work here, and am thankful.

[isavcic]

I'm hitting the same error. Value is binary data populated by consul-template for deduplication purposes, residing under consul-template/dedup/$HASH.

# consulate kv backup
Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 216, in kv_backup
    handle.write(json.dumps(records) + '\n')
  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 1: invalid start byte

# consulate kv backup -b
Traceback (most recent call last):
  File "/usr/local/bin/consulate", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 469, in main
    KV_ACTIONS[args.action](consul, args)
  File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 214, in kv_backup
    records = [(k, f, base64.b64encode(v)) for k,f,v in records]
  File "/usr/lib/python2.7/base64.py", line 53, in b64encode
    encoded = binascii.b2a_base64(s)[:-1]
TypeError: must be string or buffer, not None

[djenriquez]

Getting the same error as @isavcic when using -b, any progress for this?

[djenriquez]

In the meantime, I created a dockerized solution for backing up Consul's datastore that works with Vault: Consulidator

Backup:

CONSUL_ADDRESS=10.0.1.2
CONSUL_PORT=8500
BACKUP_FILE=my_backup

docker run --rm \
djenriquez/consulidator:v0.1.0 \
--backup \
--port $CONSUL_PORT $CONSUL_ADDRESS > $BACKUP_FILE

Restore:
Assuming that you are running the restore in the working directory that contains $BACKUP_FILE

CONSUL_ADDRESS=10.0.1.2
CONSUL_PORT=8500
BACKUP_FILE=my_backup

docker run --rm \
-v `pwd`:/restore \
djenriquez/consulidator:v0.1.0 \
--restore $BACKUP_FILE \
--port $CONSUL_PORT $CONSUL_ADDRESS

[gmr]

Closing this and tracking the forward direction in #109 and #110