Error while running examples on README

[skydiving94]

I am trying to run GNES locally on my Linux machine. However, when I run bash run.sh, I received the following errors.

E:EncoderService:[bas:run:305]:could not determine a constructor for the tag '!GPT2Encoder'
in "gpt2.yml", line 3, column 5

Namely, there is something wrong with gpt2.yml. I replaced the value of model_path with a real path to a downloaded GPT2 model, but it still did not work.

I installed gnes and also was running the docker service. Is there anything else we need to do to get it working? Thanks!

[hanxiao]

Hi, thanks again ❤️ for trying GNES and giving feedback at the early stage. I'd like to introduce you the new GNES Flow API (available since v0.0.46), enables a pythonic and intuitive way for building workflow in GNES. As an example, an indexing workflow can be simply defined as:

flow = (Flow(check_version=False, ctrl_with_ipc=True)
        .add_preprocessor(name='prep', yaml_path='yaml/prep.yml', replicas=3)
        .add_encoder(yaml_path='yaml/incep.yml', replicas=6)
        .add_indexer(name='vec_idx', yaml_path='yaml/vec.yml')
        .add_indexer(name='doc_idx', yaml_path='yaml/doc.yml', recv_from='prep')
        .add_router(name='sync', yaml_path='BaseReduceRouter', num_part=2, recv_from=['vec_idx', 'doc_idx']))

# then use it for indexing
with flow(backend='process') as fl:
    fl.index(bytes_gen=read_flowers(), batch_size=64)

🔰 You can find some resources here to help you getting started quickly:

• Image search demo using GNES Flow, Jupyter Notebook
• GNES Flow module documentation
• Getting started in GNES readme

🙇 Give it a try and we welcome your feedback and contribution.

[jageshmaharjan]

Hi @hanxiao , Thanks for this great contribution.

I was running the very initial sample (flower classification via query).
I did pip install only. I ran in terminal (not in jupyter-notebook).
I came up with this error,

>>> flow = (Flow(check_version=False)
...         .add_preprocessor(name='prep', yaml_path='yaml/prep.yml')
...         .add_encoder(yaml_path='yaml/incep.yml')
...         .add_indexer(name='vec_idx', yaml_path='yaml/vec.yml')
...         .add_indexer(name='doc_idx', yaml_path='yaml/doc.yml', recv_from='prep')
...         .add_router(name='sync', yaml_path='BaseReduceRouter', num_part=2, recv_from=['vec_idx', 'doc_idx']))
W:Flow:[__i:_po:173]:this object is not named ("name" is not found under "gnes_config" in YAML config), i will call it "Flow-e96c35ae". naming the object is important as it provides an unique identifier when serializing/deserializing this object.
usage:  [-h] [-v] [--verbose] [--port_in PORT_IN] [--port_out PORT_OUT]
        [--host_in HOST_IN] [--host_out HOST_OUT]
        [--socket_in {PULL_BIND,PULL_CONNECT,PUSH_BIND,PUSH_CONNECT,SUB_BIND,SUB_CONNECT,PUB_BIND,PUB_CONNECT,PAIR_BIND,PAIR_CONNECT}]
        [--socket_out {PULL_BIND,PULL_CONNECT,PUSH_BIND,PUSH_CONNECT,SUB_BIND,SUB_CONNECT,PUB_BIND,PUB_CONNECT,PAIR_BIND,PAIR_CONNECT}]
        [--port_ctrl PORT_CTRL] [--timeout TIMEOUT]
        [--dump_interval DUMP_INTERVAL] [--read_only]
        [--parallel_backend {thread,process}] [--num_parallel NUM_PARALLEL]
        [--parallel_type {PUSH_BLOCK,PUSH_NONBLOCK,PUB_BLOCK,PUB_NONBLOCK}]
        [--check_version] [--identity IDENTITY] [--route_table] [--squeeze_pb]
        [--ctrl_with_ipc] --yaml_path YAML_PATH
        [--py_path PY_PATH [PY_PATH ...]]
: error: argument --yaml_path: yaml/prep.yml can not be resolved, it should be a readable stream, or a valid file path, or a supported class name.
Traceback (most recent call last):
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2294, in _get_value
    result = type_func(arg_string)
  File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/cli/parser.py", line 63, in resolve_yaml_path
    ' or a valid file path, or a supported class name.' % path)
argparse.ArgumentTypeError: yaml/prep.yml can not be resolved, it should be a readable stream, or a valid file path, or a supported class name.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1766, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1972, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1912, in consume_optional
    take_action(action, args, option_string)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1824, in take_action
    argument_values = self._get_values(action, argument_strings)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2265, in _get_values
    value = self._get_value(action, arg_string)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2300, in _get_value
    raise ArgumentError(action, msg)
argparse.ArgumentError: argument --yaml_path: yaml/prep.yml can not be resolved, it should be a readable stream, or a valid file path, or a supported class name.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 707, in _get_parsed_args
    p_args, unknown_args = service_arg_parser().parse_known_args(args)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 1773, in parse_known_args
    self.error(str(err))
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2393, in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
  File "/data/anaconda/envs/gnes/lib/python3.6/argparse.py", line 2380, in exit
    _sys.exit(status)
SystemExit: 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 444, in add_preprocessor
    return self.add(Service.Preprocessor, *args, **kwargs)
  File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 638, in add
    args, p_args, unk_args = op_flow._get_parsed_args(op_flow, name, service_map[service]['parser'], kwargs)
  File "/data/anaconda/envs/gnes/lib/python3.6/site-packages/gnes/flow/__init__.py", line 712, in _get_parsed_args
    'you may want to double check your args "%s"' % (name, args))
ValueError: bad arguments for service "prep", you may want to double check your args "['--yaml_path', 'yaml/prep.yml', '--no_check_version']"

Some sort of arg error in yml file? is it that i missed to install some packages? because i just install with pip installl gnes but not pip install gnes[all]

[jageshmaharjan]

opps, my bad. I was running under different repo, so it couldn't find the yml files under yaml directory.
Can ignore the my above comment