Flux python-api kvs issue

[mgt16-LANL]

Hi all,

After playing around with kvs from the commandline:
After flux start , I ran:
flux kvs ls -R
which gives:

.:
admin       resource

admin:
eventlog

resource:
R           eventlog

Then if I run the line:

flux run flux kvs put COOL2=DUMB2
flux kvs ls -R now gives:

.:
admin       job         resource

admin:
eventlog

job:
0000

job.0000:
019d

job.0000.019d:
ff00

job.0000.019d.ff00:
0000

job.0000.019d.ff00.0000:
J           R           eventlog    guest       jobspec

job.0000.019d.ff00.0000.guest:
COOL2     exec      input     output

job.0000.019d.ff00.0000.guest.exec:
eventlog

resource:
R           eventlog

And if I run: flux kvs get job.0000.019d.ff00.0000.guest.COOL2
I get:
DUMB2

This all works great!

However, using the python API: If I run:

import flux
import flux.kvs

f = flux.Flux()
d = flux.kvs.KVSDir(f)

d['job']['0000']['019d']['ff00']['0000']['guest']['COOL2']

I get the following error:

Traceback (most recent call last):
  File "/global/u1/m/mgt16/kvs_testing/test.py", line 7, in <module>
    d['job']['0000']['019d']['ff00']['0000']['guest']['COOL2']
  File "/global/cfs/cdirs/m4369/mambaforge/lib/flux/python3.1/flux/kvs.py", line 175, in __getitem__
    return get(self.fhdl, self.key_at(key))
  File "/global/cfs/cdirs/m4369/mambaforge/lib/flux/python3.1/flux/kvs.py", line 71, in get
    return get_key_direct(flux_handle, key)
  File "/global/cfs/cdirs/m4369/mambaforge/lib/flux/python3.1/flux/kvs.py", line 38, in get_key_direct
    ret = json.loads(ffi.string(valp[0]).decode("utf-8"))
  File "/global/cfs/cdirs/m4369/mambaforge/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/global/cfs/cdirs/m4369/mambaforge/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/global/cfs/cdirs/m4369/mambaforge/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Am I misunderstanding the python API - or is this a bug?

[chu11]

Hi, I believe the python KVS API is quite out of the date for what the KVS is today. In the past, I believe all KVS data was stored in JSON, thus the json.loads() call in the trace above. Most users use the python "job" interfaces instead.

Could you explain your use case? Perhaps we need to determine a short term "make it work for now" fix or up the priority on actually making the python KVS API more functional than its older legacy state. Or if you can do what you need with the other API calls, perhaps it should be removed to remove future accidental usage.

(For issue linking, noted in #5037).

Edit: Sorry, I clicked submit before I was finished writing, hopefully you see the updated text here :-)

Edit2: For my own personal debugging, the json.loads() call has origins with 327f240, which is from Thu Jul 9 14:44:42 2015. That predates earliest treeobj work in kvs 7fc7ecf dated Wed Sep 7 15:39:07 2016

[mgt16-LANL]

We are looking to have something like a master flux process and submitted flux jobs add items to the KVS (or another non-filesystem-based store) that we can get out from the master process to allocate more resources/jobs where needed. Is this something that can be done from the python "job" interface?

edit: Thank you so much for the quick response!.

[chu11]

ok, lemme look to see if I can do something to the current api to make it work again. I'm actually wondering if the json.loads() call can just be removed, and the string returned instead. (possibly some corner cases to handle too)

[grondo]

For the time being you may be able to monkeypatch in a fixed version of flux.kvs.get_key_direct, e.g.:

import flux
from flux import kvs
from _flux._core import ffi, lib
from flux.wrapper import Wrapper

RAW = Wrapper(ffi, lib, prefixes=["flux_kvs", "flux_kvs_"])

def kvs_get(handle, key):
    valp = ffi.new("char *[1]")
    future = RAW.flux_kvs_lookup(handle, None, 0, key)
    RAW.flux_kvs_lookup_get(future, valp)
    if valp[0] == ffi.NULL:
        return None
    result = ffi.string(valp[0]).decode("utf-8")
    RAW.flux_future_destroy(future)
    return result

flux.kvs.get_key_direct = kvs_get

handle = flux.Flux()
d = flux.kvs.KVSDir(handle)

print(d["a"]["b"]["c"])

$ flux kvs put a.b.c="test"
$ flux python t.py
test

[chu11]

Playing around, I also thought of a very stupid workaround. Since json is expected, could just quote the string. It's stupid and only works for this specific case, not the general case depending on what data you want to actually put in the KVS.

import flux
import flux.kvs

f = flux.Flux()
d = flux.kvs.KVSDir(f)

print(d['a']['b']['c'])

>flux kvs put a.b.c=foo
>flux python ./test.py
Traceback (most recent call last):
  File "./test.py", line 7, in <module>
    print(d['a']['b']['c'])
  File "/g/g0/achu/chaos/git/flux-framework/flux-core/src/bindings/python/flux/kvs.py", line 175, in __getitem__
    return get(self.fhdl, self.key_at(key))
  File "/g/g0/achu/chaos/git/flux-framework/flux-core/src/bindings/python/flux/kvs.py", line 71, in get
    return get_key_direct(flux_handle, key)
  File "/g/g0/achu/chaos/git/flux-framework/flux-core/src/bindings/python/flux/kvs.py", line 38, in get_key_direct
    ret = json.loads(ffi.string(valp[0]).decode("utf-8"))
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

>flux kvs put a.b.c=\"foo\"
>flux python test.py
foo

[mgt16-LANL]

Thanks Mark and Al, at least for the time being I think this should work for us - I think we could represent pretty much any python object/datatype as strings so this SHOULD already let us write and read nearly any datatype to KVS.

[chu11]

Also, as an aside, the following are the same:

print(d['a']['b']['c'])                                                                                                                     
print(d['a.b.c'])  

IMO, the latter is preferred, just because that's what you'd do on the command line like flux kvs get a.b.c.

[chu11]

hopefully no one objects, I'm going to move this into discussions and leave #5198 as the issue for fixing.