From aa529c6cfd2293f1f56453e2d62000161c08de54 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 15:27:47 +0800
Subject: [PATCH 01/49] doc(readme): add quick start for readme
---
README.md | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index fe3dabc7..d9fdb90c 100644
--- a/README.md
+++ b/README.md
@@ -91,18 +91,18 @@ We provide GNES as a Docker image to simplify the installation. The Docker image
#### via [Docker cloud](https://cloud.docker.com/u/gnes/repository/list)
```bash
-docker pull gnes/gnes:latest
-docker run gnes/gnes:latest --help
+docker run gnes/gnes:latest
```
+This command downloads the latest GNES image and runs it in a container. When the container runs, it prints an informational message and exits.
+
#### via Tencent container service
We also provide a public mirror hosted on Tencent Cloud, from which Chinese mainland users can pull the image faster.
```bash
docker login --username=xxx ccr.ccs.tencentyun.com # login to Tencent Cloud so that we can pull from it
-docker pull ccr.ccs.tencentyun.com/gnes/gnes:latest
-docker run ccr.ccs.tencentyun.com/gnes/gnes:latest --help
+docker run ccr.ccs.tencentyun.com/gnes/gnes:latest
```
> π‘ Please note that version `latest` refers to the latest master of this repository, which is [mutable and may not always be a stable](./CONTRIBUTING.md#Merging-Process). Therefore, we recommend you to use an official release by changing the `latest` to a version tag, say `v0.0.24`.
@@ -143,6 +143,13 @@ pip install gnes[all]
+Either way, if you see the following message after `$gnes` or `$docker run gnes/gnes`, then you are ready to go!
+
+
+
+
+
+
Quick Start
As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. Currently, we support Kubernetes, Docker Swarm and a built-in solution. Click on one of the icons below to get started.
@@ -169,6 +176,8 @@ As a cloud-native application, GNES requires an **orchestration engine** to coor
+
+
### Using GNES with Kubernetes
TBA
From 44b7db2f113969cc0eea567fc8bf0e3d68023d11 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 15:28:27 +0800
Subject: [PATCH 02/49] doc(readme): add quick start for readme
---
.github/install-success.svg | 79 +++++++++++++++++++++++++++++++++++++
1 file changed, 79 insertions(+)
create mode 100644 .github/install-success.svg
diff --git a/.github/install-success.svg b/.github/install-success.svg
new file mode 100644
index 00000000..4412ab08
--- /dev/null
+++ b/.github/install-success.svg
@@ -0,0 +1,79 @@
+
\ No newline at end of file
From 0eb8c3f1d0b8f6b2d5569657222163c0a0664075 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 16:03:08 +0800
Subject: [PATCH 03/49] doc(readme): add quick start for readme
---
README.md | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index d9fdb90c..c84351c3 100644
--- a/README.md
+++ b/README.md
@@ -143,7 +143,7 @@ pip install gnes[all]
-Either way, if you see the following message after `$gnes` or `$docker run gnes/gnes`, then you are ready to go!
+Either way, if you see the following message after `$ gnes` or `$ docker run gnes/gnes`, then you are ready to go!
@@ -152,6 +152,22 @@ Either way, if you see the following message after `$gnes` or `$docker run gnes/
Quick Start
+## Preliminaries
+
+Before we start, let me first introduce two basic concepts serving as the backbone of GNES: **microservice** and **runtime**. For machine learning engineers and data scientists who are not familiar with the concept of *cloud-native* and *microservice*, one can picture a microservice as an app (on your smartphone). Each app runs independently, and an app may cooperate with other apps to accomplish a task. In GNES, we have four fundamental apps, aka. microservices, they are:
+
+- **Preprocessor**: transforming a real-world object to a list of workable semantic units;
+- **Encoder**: representing a semantic unit with vector representation;
+- **Indexer**: storing the vectors into memory/disk that allows fast-access;
+- **Router**: forwarding messages between microservices: e.g. batching, mapping, reducing.
+
+In GNES, we have implemented dozens of preprocessor, encoder, indexer to process different content forms, such as image, text, video. It is also super easy to plug in your own implementation, which we shall see an example in the sequel.
+
+Okay, now that we have a bunch of apps, what are we expecting them to do? In a typical search system, there are two fundamental tasks: **indexing** and **querying**. Indexing is storing the documents, querying is searching the documents, pretty straightforward. In a neural search system, one may also face another task: **training**, where one fine-tunes an encoder/preprocessor according to the data distribution in order to achieve better search relevance. These three tasks: indexing, querying and training are what we call three **runtimes** in GNES.
+
+
+
+
As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. Currently, we support Kubernetes, Docker Swarm and a built-in solution. Click on one of the icons below to get started.
From 8f61fb99e8b9166e8744d733569e261175c7f038 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 16:29:51 +0800
Subject: [PATCH 04/49] doc(readme): add quick start for readme
---
README.md | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/README.md b/README.md
index c84351c3..3a2a1fd6 100644
--- a/README.md
+++ b/README.md
@@ -84,7 +84,7 @@ There are two ways to get GNES, either as a Docker image or as a PyPi package.
β For cloud users, we **highly recommend using GNES via Docker image**.
-## Run GNES as a Docker Container
+### Run GNES as a Docker Container
We provide GNES as a Docker image to simplify the installation. The Docker image is built with GNES full dependencies, so you can run GNES out-of-the-box.
@@ -107,7 +107,7 @@ docker run ccr.ccs.tencentyun.com/gnes/gnes:latest
> π‘ Please note that version `latest` refers to the latest master of this repository, which is [mutable and may not always be a stable](./CONTRIBUTING.md#Merging-Process). Therefore, we recommend you to use an official release by changing the `latest` to a version tag, say `v0.0.24`.
-## Install GNES via `pip`
+### Install GNES via `pip`
You can also install GNES as a Python package via:
```bash
@@ -152,7 +152,7 @@ Either way, if you see the following message after `$ gnes` or `$ docker run gne
Quick Start
-## Preliminaries
+### Preliminaries
Before we start, let me first introduce two basic concepts serving as the backbone of GNES: **microservice** and **runtime**. For machine learning engineers and data scientists who are not familiar with the concept of *cloud-native* and *microservice*, one can picture a microservice as an app (on your smartphone). Each app runs independently, and an app may cooperate with other apps to accomplish a task. In GNES, we have four fundamental apps, aka. microservices, they are:
@@ -165,6 +165,9 @@ In GNES, we have implemented dozens of preprocessor, encoder, indexer to process
Okay, now that we have a bunch of apps, what are we expecting them to do? In a typical search system, there are two fundamental tasks: **indexing** and **querying**. Indexing is storing the documents, querying is searching the documents, pretty straightforward. In a neural search system, one may also face another task: **training**, where one fine-tunes an encoder/preprocessor according to the data distribution in order to achieve better search relevance. These three tasks: indexing, querying and training are what we call three **runtimes** in GNES.
+π‘ The key to understand GNES is to know *which runtime requires what microservices, and each of which does what*.
+
+### Build your first GNES network
From 92b358a494e3e9f52998627959abcd3f22b10289 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 16:31:20 +0800
Subject: [PATCH 05/49] doc(readme): add quick start for readme
---
README.md | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 3a2a1fd6..e8185b26 100644
--- a/README.md
+++ b/README.md
@@ -154,7 +154,11 @@ Either way, if you see the following message after `$ gnes` or `$ docker run gne
### Preliminaries
-Before we start, let me first introduce two basic concepts serving as the backbone of GNES: **microservice** and **runtime**. For machine learning engineers and data scientists who are not familiar with the concept of *cloud-native* and *microservice*, one can picture a microservice as an app (on your smartphone). Each app runs independently, and an app may cooperate with other apps to accomplish a task. In GNES, we have four fundamental apps, aka. microservices, they are:
+Before we start, let me first introduce two basic concepts serving as the backbone of GNES: **microservice** and **runtime**.
+
+#### Microservice
+
+For machine learning engineers and data scientists who are not familiar with the concept of *cloud-native* and *microservice*, one can picture a microservice as an app (on your smartphone). Each app runs independently, and an app may cooperate with other apps to accomplish a task. In GNES, we have four fundamental apps, aka. microservices, they are:
- **Preprocessor**: transforming a real-world object to a list of workable semantic units;
- **Encoder**: representing a semantic unit with vector representation;
@@ -163,6 +167,8 @@ Before we start, let me first introduce two basic concepts serving as the backbo
In GNES, we have implemented dozens of preprocessor, encoder, indexer to process different content forms, such as image, text, video. It is also super easy to plug in your own implementation, which we shall see an example in the sequel.
+#### Runtime
+
Okay, now that we have a bunch of apps, what are we expecting them to do? In a typical search system, there are two fundamental tasks: **indexing** and **querying**. Indexing is storing the documents, querying is searching the documents, pretty straightforward. In a neural search system, one may also face another task: **training**, where one fine-tunes an encoder/preprocessor according to the data distribution in order to achieve better search relevance. These three tasks: indexing, querying and training are what we call three **runtimes** in GNES.
π‘ The key to understand GNES is to know *which runtime requires what microservices, and each of which does what*.
From 4e77fd628454b90e256f2b6d5e6e0c238f1d2c80 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 16:53:38 +0800
Subject: [PATCH 06/49] doc(readme): add quick start for readme
---
.github/gnes-flow-1.svg | 399 ++++++++++++++++++++++++++++++++++++++++
README.md | 28 ++-
2 files changed, 426 insertions(+), 1 deletion(-)
create mode 100644 .github/gnes-flow-1.svg
diff --git a/.github/gnes-flow-1.svg b/.github/gnes-flow-1.svg
new file mode 100644
index 00000000..6eaf2eb7
--- /dev/null
+++ b/.github/gnes-flow-1.svg
@@ -0,0 +1,399 @@
+
push/pull
push/pull
push/pull
push/pull
gRPCFrontend
Preprocessor
Encoder
Indexer
\ No newline at end of file
diff --git a/README.md b/README.md
index e8185b26..3699c281 100644
--- a/README.md
+++ b/README.md
@@ -173,7 +173,33 @@ Okay, now that we have a bunch of apps, what are we expecting them to do? In a t
π‘ The key to understand GNES is to know *which runtime requires what microservices, and each of which does what*.
-### Build your first GNES network
+### Build your first GNES app
+
+Let's start with a typical indexing procedure:
+
+
\ No newline at end of file
diff --git a/README.md b/README.md
index 3699c281..26884572 100644
--- a/README.md
+++ b/README.md
@@ -195,7 +195,7 @@ Let's start with a typical indexing procedure:
From 1a914348d9129d02044b3a367e39f0b5c813fe34 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:16:18 +0800
Subject: [PATCH 11/49] doc(readme): add quick start for readme
---
README.md | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index b46fbe57..6d71a2a2 100644
--- a/README.md
+++ b/README.md
@@ -175,7 +175,7 @@ Okay, now that we have a bunch of apps, what are we expecting them to do? In a t
### Build your first GNES app
-Let's start with a typical indexing procedure:
+Let's start with a typical indexing procedure by writing a YAML config (see the left column of the table):
@@ -200,8 +200,33 @@ services:
+The YAML config should be pretty intuitive. It defines a pipeline workflow consists of preprocessing, encoding and indexing, where the output of the former component is the input of the next. For each component, we also associate it with a YAML config specifying how it should work. Right now they are not important for understanding the big picture, nonetheless curious readers can checkout how each YAML looks like by expanding the text below.
+
+ Encoder config: gpt2.yml (click to expand...)
+
+```yaml
+!PipelineEncoder
+component:
+ - !GPT2Encoder
+ parameter:
+ model_dir: $GPT2_CI_MODEL
+ pooling_stragy: REDUCE_MEAN
+ gnes_config:
+ is_trained: true
+ - !PCALocalEncoder
+ parameter:
+ output_dim: 32
+ num_locals: 8
+ gnes_config:
+ batch_size: 2048
+ - !PQEncoder
+ parameter:
+ cluster_per_byte: 8
+ num_bytes: 8
+```
+
As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. Currently, we support Kubernetes, Docker Swarm and a built-in solution. Click on one of the icons below to get started.
From b46f51a04be4dc8d9f0ceeb018f41e7a3999b3e7 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:22:10 +0800
Subject: [PATCH 12/49] doc(readme): add quick start for readme
---
README.md | 31 ++++++++++++++++++++++++++++++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 6d71a2a2..5ea26718 100644
--- a/README.md
+++ b/README.md
@@ -203,7 +203,20 @@ services:
The YAML config should be pretty intuitive. It defines a pipeline workflow consists of preprocessing, encoding and indexing, where the output of the former component is the input of the next. For each component, we also associate it with a YAML config specifying how it should work. Right now they are not important for understanding the big picture, nonetheless curious readers can checkout how each YAML looks like by expanding the text below.
- Encoder config: gpt2.yml (click to expand...)
+ Preprocessor config:
(click to expand...)
+
+```yaml
+!BIndexer
+parameter:
+ num_bytes: 8
+ data_path: /out_data/idx.binary
+gnes_config:
+ work_dir: ./
+ name: bindexer
+```
+
As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. Currently, we support Kubernetes, Docker Swarm and a built-in solution. Click on one of the icons below to get started.
From b34982b3d4503b89e68d9962a89d8da9ad3b9193 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:22:51 +0800
Subject: [PATCH 13/49] doc(readme): add quick start for readme
---
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.md b/README.md
index 5ea26718..35298150 100644
--- a/README.md
+++ b/README.md
@@ -216,7 +216,7 @@ gnes_config:
```
- Encoder config:
gpt2.yml
(click to expand...)
+ Encoder config: gpt2.yml (click to expand...)
```yaml
!PipelineEncoder
From 37a8b550670eb025aa694a021a9094a3f7510468 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:23:37 +0800
Subject: [PATCH 14/49] doc(readme): add quick start for readme
---
README.md | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index 35298150..acd7ae1d 100644
--- a/README.md
+++ b/README.md
@@ -203,7 +203,7 @@ services:
The YAML config should be pretty intuitive. It defines a pipeline workflow consists of preprocessing, encoding and indexing, where the output of the former component is the input of the next. For each component, we also associate it with a YAML config specifying how it should work. Right now they are not important for understanding the big picture, nonetheless curious readers can checkout how each YAML looks like by expanding the text below.
- Preprocessor config:
(click to expand...)
+ Indexer config: b-indexer.yml (click to expand...)
```yaml
!BIndexer
From b6db821a57d947b4433ff370035ad565d8b98a40 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:24:13 +0800
Subject: [PATCH 15/49] doc(readme): add quick start for readme
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index acd7ae1d..359c7b84 100644
--- a/README.md
+++ b/README.md
@@ -214,7 +214,7 @@ parameter:
gnes_config:
is_trained: true
```
-
+Encoder config: gpt2.yml (click to expand...)
@@ -257,7 +257,7 @@ gnes_config:
work_dir: ./
name: bindexer
```
-
+
As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. Currently, we support Kubernetes, Docker Swarm and a built-in solution. Click on one of the icons below to get started.
From 56783cdeef97d9178b655f535cd1e06543d2a58b Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:46:39 +0800
Subject: [PATCH 16/49] doc(readme): add quick start for readme
---
README.md | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 91 insertions(+), 5 deletions(-)
diff --git a/README.md b/README.md
index 359c7b84..8f74b609 100644
--- a/README.md
+++ b/README.md
@@ -32,7 +32,7 @@
Highlights β’
Overview β’
Install β’
- Quick Start β’
+ Getting Started β’
Documentation β’
Tutorial β’
Contributing β’
@@ -150,9 +150,9 @@ Either way, if you see the following message after `$ gnes` or `$ docker run gne
-
Quick Start
+
Getting Started
-### Preliminaries
+### π£ Preliminaries
Before we start, let me first introduce two basic concepts serving as the backbone of GNES: **microservice** and **runtime**.
@@ -200,7 +200,7 @@ services:
-The YAML config should be pretty intuitive. It defines a pipeline workflow consists of preprocessing, encoding and indexing, where the output of the former component is the input of the next. For each component, we also associate it with a YAML config specifying how it should work. Right now they are not important for understanding the big picture, nonetheless curious readers can checkout how each YAML looks like by expanding the text below.
+The YAML config should be pretty intuitive. It defines a pipeline workflow consists of preprocessing, encoding and indexing, where the output of the former component is the input of the next. For each component, we also associate it with a YAML config specifying how it should work. Right now they are not important for understanding the big picture, nonetheless curious readers can checkout how each YAML looks like by expanding the items below.
Preprocessor config: text-prep.yml (click to expand...)
@@ -259,7 +259,93 @@ gnes_config:
```
-As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. Currently, we support Kubernetes, Docker Swarm and a built-in solution. Click on one of the icons below to get started.
+On the right side of the above table, you can see how the actual data flow looks like. There is an additional component `gRPCFrontend` automatically added to the workflow, it allows you to feed the data and fetch the result via gRPC protocol.
+
+Now it's time to run! As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. We support Kubernetes, Docker Swarm and a shell-based multi-process solution. Moreover, [GNES board](https://board.gnes.ai) can automatically generate an orchestration config/script based on the YAML file you give. Let's see what the generated script looks like in this case.
+
+
+
+
Shell-based solution
DockerSwarm solution
+
+
+
+
+#!/usr/bin/env bash
+
+## Prerequirment of this script
+## You need to install GNES locally on this local machine
+## pip install gnes
+
+set -e
+
+trap 'kill $(jobs -p)' EXIT
+
+printf "starting service [32mgRPCFrontend[0m with [33m1[0m replicas...\n"
+gnes frontend --grpc_port 5566 --port_out 49668 --socket_out PUSH_BIND --port_in 60654 --socket_in PULL_CONNECT &
+printf "starting service [32mPreprocessor[0m with [33m1[0m replicas...\n"
+gnes preprocess --yaml_path text-prep.yaml --port_in 49668 --socket_in PULL_CONNECT --port_out 61911 --socket_out PUSH_BIND &
+printf "starting service [32mEncoder[0m with [33m1[0m replicas...\n"
+gnes encode --yaml_path gpt2.yml --port_in 61911 --socket_in PULL_CONNECT --port_out 49947 --socket_out PUSH_BIND &
+printf "starting service [32mIndexer[0m with [33m1[0m replicas...\n"
+gnes index --yaml_path b-indexer.yml --port_in 49947 --socket_in PULL_CONNECT --port_out 60654 --socket_out PUSH_BIND &
+
+wait
+
version: '3.4'
services:
From efe62ec4519e4b1ee1cbb0a51585ed8d14a2cbcb Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:54:04 +0800
Subject: [PATCH 19/49] doc(readme): add quick start for readme
---
README.md | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
diff --git a/README.md b/README.md
index d0763691..99621830 100644
--- a/README.md
+++ b/README.md
@@ -271,22 +271,17 @@ Now it's time to run! As a cloud-native application, GNES requires an **orchestr
#!/usr/bin/env bash
-
-## Prerequirment of this script
-## You need to install GNES locally on this local machine
-## pip install gnes
-
set -e
trap 'kill $(jobs -p)' EXIT
-printf "starting service [32mgRPCFrontend[0m with [33m1[0m replicas...\n"
+printf "starting service gRPCFrontend with 0 replicas...\n"
gnes frontend --grpc_port 5566 --port_out 49668 --socket_out PUSH_BIND --port_in 60654 --socket_in PULL_CONNECT &
-printf "starting service [32mPreprocessor[0m with [33m1[0m replicas...\n"
+printf "starting service Preprocessor with 0replicas...\n"
gnes preprocess --yaml_path text-prep.yaml --port_in 49668 --socket_in PULL_CONNECT --port_out 61911 --socket_out PUSH_BIND &
-printf "starting service [32mEncoder[0m with [33m1[0m replicas...\n"
+printf "starting service Encoder with 0 replicas...\n"
gnes encode --yaml_path gpt2.yml --port_in 61911 --socket_in PULL_CONNECT --port_out 49947 --socket_out PUSH_BIND &
-printf "starting service [32mIndexer[0m with [33m1[0m replicas...\n"
+printf "starting service Indexer with 0 replicas...\n"
gnes index --yaml_path b-indexer.yml --port_in 49947 --socket_in PULL_CONNECT --port_out 60654 --socket_out PUSH_BIND &
wait
From 1552e8d1a4848a75319c1820c86c14d1550b26de Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:55:23 +0800
Subject: [PATCH 20/49] doc(readme): add quick start for readme
---
README.md | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/README.md b/README.md
index 99621830..105017c8 100644
--- a/README.md
+++ b/README.md
@@ -275,13 +275,13 @@ set -e
trap 'kill $(jobs -p)' EXIT
-printf "starting service gRPCFrontend with 0 replicas...\n"
+printf "starting service gRPCFrontend with 0 replicas...\n"
gnes frontend --grpc_port 5566 --port_out 49668 --socket_out PUSH_BIND --port_in 60654 --socket_in PULL_CONNECT &
-printf "starting service Preprocessor with 0replicas...\n"
+printf "starting service Preprocessor with 0replicas...\n"
gnes preprocess --yaml_path text-prep.yaml --port_in 49668 --socket_in PULL_CONNECT --port_out 61911 --socket_out PUSH_BIND &
-printf "starting service Encoder with 0 replicas...\n"
+printf "starting service Encoder with 0 replicas...\n"
gnes encode --yaml_path gpt2.yml --port_in 61911 --socket_in PULL_CONNECT --port_out 49947 --socket_out PUSH_BIND &
-printf "starting service Indexer with 0 replicas...\n"
+printf "starting service Indexer with 0 replicas...\n"
gnes index --yaml_path b-indexer.yml --port_in 49947 --socket_in PULL_CONNECT --port_out 60654 --socket_out PUSH_BIND &
wait
@@ -335,7 +335,7 @@ configs:
file: gpt2.yml
Indexer30_yaml:
file: b-indexer.yml
-
+
From 08b2f8ed4df31d3d472affc56f420961fe5579d9 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 17:57:02 +0800
Subject: [PATCH 21/49] doc(readme): add quick start for readme
---
README.md | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index 105017c8..be9f5dbc 100644
--- a/README.md
+++ b/README.md
@@ -275,13 +275,13 @@ set -e
trap 'kill $(jobs -p)' EXIT
-printf "starting service gRPCFrontend with 0 replicas...\n"
+printf "starting service gRPCFrontend with 0 replicas...\n"
gnes frontend --grpc_port 5566 --port_out 49668 --socket_out PUSH_BIND --port_in 60654 --socket_in PULL_CONNECT &
-printf "starting service Preprocessor with 0replicas...\n"
+printf "starting service Preprocessor with 0replicas...\n"
gnes preprocess --yaml_path text-prep.yaml --port_in 49668 --socket_in PULL_CONNECT --port_out 61911 --socket_out PUSH_BIND &
-printf "starting service Encoder with 0 replicas...\n"
+printf "starting service Encoder with 0 replicas...\n"
gnes encode --yaml_path gpt2.yml --port_in 61911 --socket_in PULL_CONNECT --port_out 49947 --socket_out PUSH_BIND &
-printf "starting service Indexer with 0 replicas...\n"
+printf "starting service Indexer with 0 replicas...\n"
gnes index --yaml_path b-indexer.yml --port_in 49947 --socket_in PULL_CONNECT --port_out 60654 --socket_out PUSH_BIND &
wait
From bebfebbada9c4739bb828613c94a9b7dfc30fa44 Mon Sep 17 00:00:00 2001
From: hanhxiao
Date: Tue, 23 Jul 2019 18:01:59 +0800
Subject: [PATCH 22/49] doc(readme): add quick start for readme
---
README.md | 46 ++++++++++++++++++----------------------------
1 file changed, 18 insertions(+), 28 deletions(-)
diff --git a/README.md b/README.md
index be9f5dbc..66237d6f 100644
--- a/README.md
+++ b/README.md
@@ -259,17 +259,15 @@ gnes_config:
```
-On the right side of the above table, you can see how the actual data flow looks like. There is an additional component `gRPCFrontend` automatically added to the workflow, it allows you to feed the data and fetch the result via gRPC protocol.
+On the right side of the above table, you can see how the actual data flow looks like. There is an additional component `gRPCFrontend` automatically added to the workflow, it allows you to feed the data and fetch the result via gRPC protocol through port `5566`.
-Now it's time to run! As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. We support Kubernetes, Docker Swarm and a shell-based multi-process solution. Moreover, [GNES board](https://board.gnes.ai) can automatically generate an orchestration config/script based on the YAML file you give. Let's see what the generated script looks like in this case.
+Now it's time to run! [GNES board](https://board.gnes.ai) can automatically generate a starting script/config based on the YAML config you give. As a cloud-native application, GNES requires an **orchestration engine** to coordinate all micro-services. We support Kubernetes, Docker Swarm and shell-based multi-process. Let's see what the generated script looks like in this case.
-
-
-
Shell-based solution
-
-
-
-
+
+
+ Shell-based starting script (click to expand...)
+
+```bash
#!/usr/bin/env bash
set -e
@@ -277,7 +275,7 @@ trap 'kill $(jobs -p)' EXIT
printf "starting service gRPCFrontend with 0 replicas...\n"
gnes frontend --grpc_port 5566 --port_out 49668 --socket_out PUSH_BIND --port_in 60654 --socket_in PULL_CONNECT &
-printf "starting service Preprocessor with 0replicas...\n"
+printf "starting service Preprocessor with 0 replicas...\n"
gnes preprocess --yaml_path text-prep.yaml --port_in 49668 --socket_in PULL_CONNECT --port_out 61911 --socket_out PUSH_BIND &
printf "starting service Encoder with 0 replicas...\n"
gnes encode --yaml_path gpt2.yml --port_in 61911 --socket_in PULL_CONNECT --port_out 49947 --socket_out PUSH_BIND &
@@ -285,15 +283,13 @@ printf "starting service Indexer with 0 replicas...\n"
gnes index --yaml_path b-indexer.yml --port_in 49947 --socket_in PULL_CONNECT --port_out 60654 --socket_out PUSH_BIND &
wait
-