refactor: simplify job cli #944

frrist · 2022-04-27T02:42:21Z

Whats changed?

Remove the help command as the information lives in lilium.sh and the commands themselves. Furthermore maintaining docs on lilium.sh and in the CLI is unwanted overhead.
CLI commands for running watch, walk, index, find, fill, and survey jobs have changes
rename some Visor* structs to Lily*

All jobs are now grouped under the job command

$ lily job
NAME:
   lily job - Manage jobs being run by the daemon.

USAGE:
   lily job command [command options] [arguments...]

COMMANDS:
   run      run a job
   start    start a job.
   stop     stop a job.
   wait     wait on a job to complete.
   list     list all jobs and their status
   help, h  Shows a list of commands or help for one command

jobs are runnable via the `job run` command:

Since all jobs accept window, tasks, storage, name, restart-delay, restart-on-failure, and restart-on-completion options those flags now exist on the job run command

NAME:
   lily job run - run a job

USAGE:
   lily job run command [command options] [arguments...]

COMMANDS:
   walk           walk and index a range of the filecoin blockchain.
   watch          watch the head of the filecoin blockchain and index each new head as it becomes avaiable
   index          Index the state of a tipset from the filecoin blockchain.
   survey         Start a daemon job to survey the node and its environment.
   fill           fill gaps in the database for a given range and set of tasks.
   find           find gaps in the database for a given range and a set of tasks.
   tipset-worker  start a tipset-worker that consumes tasks from the provided queuing system and performs indexing
   help, h        Shows a list of commands or help for one command

OPTIONS:
   --window value           Duaration after which job execution will be canceled (default: 0s) [$LILY_JOB_WINDOW]
   --tasks value            Comma separated list of tasks to run in job. Each task is reported separately in the storage backend. [$LILY_JOB_TASKS]
   --storage value          Name of storage backend the job will write result to. [$LILY_JOB_STORAGE]
   --name value             Name of job for easy identification later. Will appear as 'reporter' in the visor_processing_reports table. [$LILY_JOB_NAME]
   --restart-delay value    Duration to wait before restarting job after it ends execution (default: 0s) [$LILY_JOB_RESTART_DELAY]
   --restart-on-failure     Restart the job if it fails. (default: false) [$LILY_JOB_RESTART_FAILURE]
   --restart-on-completion  Restart the job after it completes. (default: false) [$LILY_JOB_RESTART_COMPLETION]

`walk` job description

NAME:
   lily job run walk - walk and index a range of the filecoin blockchain.

USAGE:
   lily job run walk command [command options] [arguments...]

DESCRIPTION:
   The walk command will index state based on the list of tasks (--tasks) provided over the specified range (--from --to).
   Each epoch will be indexed serially starting from the heaviest tipset at the upper height (--to) to the lower height (--to).
   
   As and example, the below command:
     $ lily job run --tasks=block_headers,messages walk --from=10 --to=20
   walks epochs 20 through 10 (inclusive) executing the block_headers and messages task for each epoch.
   The status of each epoch and its set of tasks can be observed in the visor_processing_reports table.

COMMANDS:
   notify   notify the provided queueing system of epochs to index allowing tipset-workers to perform the indexing.
   help, h  Shows a list of commands or help for one command

OPTIONS:
   --from HEIGHT  Limit actor and message processing to tipsets at or above HEIGHT (default: 0) [$LILY_FROM]
   --to HEIGHT    Limit actor and message processing to tipsets at or below HEIGHT (default: 0) [$LILY_TO]

`watch` job description

NAME:
   lily job run watch - watch the head of the filecoin blockchain and index each new head as it becomes avaiable

USAGE:
   lily job run watch command [command options] [arguments...]

DESCRIPTION:
   The watch command subscribes to incoming tipsets from the filecoin blockchain and indexes them as the arrive.
   
   Since it may be the case that tipsets arrive at a rate greater than their rate of indexing the watch job maintains a
   queue of tipsets to index. Consumption of this queue can be configured via the --workers flag. Increasing the value provided
   to the --workers flag will allow the watch job to index tipsets simultaneously (Note: this will use a significant amount of system resources).
   
   Since it may be the case that lily experiences a reorg while the watch job is observing the head of the chain
   the --confidence flag may be used to buffed the amount of tipsets observed before it begins indexing - illustrated by the below diagram:
   
                *unshift*        *unshift*      *unshift*       *unshift*
                   │  │            │  │            │  │            │  │
                ┌──▼──▼──┐      ┌──▼──▼──┐      ┌──▼──▼──┐      ┌──▼──▼──┐
                │        │      │  ts10  │      │  ts11  │      │  ts12  │
      ...  ---> ├────────┤ ---> ├────────┤ ---> ├────────┤ ---> ├────────┤ --->  ...
                │  ts09  │      │  ts09  │      │  ts10  │      │  ts11  │
                ├────────┤      ├────────┤      ├────────┤      ├────────┤
                │  ts08  │      │  ts08  │      │  ts09  │      │  ts10  │
                ├────────┤      ├────────┤      ├────────┤      ├────────┤
                │  ...   │      │  ...   │      │  ...   │      │  ...   │
                ├────────┤      ├────────┤      ├────────┤      ├────────┤
                │  ts02  │      │  ts02  │      │  ts03  │      │  ts04  │
                ├────────┤      ├────────┤      ├────────┤      ├────────┤
                │  ts01  │      │  ts01  │      │  ts02  │      │  ts03  │
                ├────────┤      ├────────┤      ├────────┤      ├────────┤
                │  ts00  │      │  ts00  │      │  ts01  │      │  ts02  │
                └────────┘      └────────┘      └──│──│──┘      └──│──│──┘
                                                   ▼  ▼  *pop*     ▼  ▼  *pop*
                                                ┌────────┐      ┌────────┐
                 (confidence=10 :: length=10)   │  ts00  │      │  ts01  │
                                                └────────┘      └────────┘
                                                 (process)       (process)
   
   As and example, the below command:
     $ lily job run --tasks-block_headers,messages watch --confidence=10 --workers=2
   watches the chain head and only indexes a tipset after observing 10 subsequent tipsets indexing at most two tipset simultaneously.

COMMANDS:
   notify   notify the provided queueing system of epochs to index allowing tipset-workers to perform the indexing.
   help, h  Shows a list of commands or help for one command

OPTIONS:
   --confidence value   Sets the size of the cache used to hold tipsets for possible reversion before being committed to the database. (default: 2) [$LILY_CONFIDENCE]
   --workers value      Sets the number of tipsets that may be simultaneous indexed while watching. (default: 2) [$LILY_WATCH_WORKERS]
   --buffer-size value  Set the number of tipsets the watcher will buffer while waiting for a worker to accept the work. (default: 5) [$LILY_WATCH_BUFFER]
   --help, -h           show help (default: false)

`find` job description

NAME:
   lily job run find - find gaps in the database for a given range and a set of tasks.

USAGE:
   lily job run find [command options] [arguments...]

DESCRIPTION:
   The find job searches for gaps in a database storage system by executing the SQL gap_find() function over the visor_processing_reports table.
   find will query the database for gaps based on the list of tasks (--tasks) provided over the specified range (--from --to).
   An epoch is considered to have gaps iff:
   - a task specified by the --task flag is not present at each epoch within the specified range.
   - a task specified by the --task flag does not have status 'OK' at each epoch within the specified range.
   The results of the find job are written to the visor_gap_reports table with status 'GAP'.
   
   As an example, the below command:
    $ lily job run --tasks=block_headers,messages find --from=10 --to=20
   searches for gaps in block_headers and messages tasks from epoch 10 to 20 (inclusive). 
   
   Constraints:
   - the find job must NOT be executed against heights that were imported from historical data dumps: https://lilium.sh/data/dumps/ 
   since visor_processing_report entries will not be present for imported data (meaning the entire range will be considered to have gaps).
   - the find job must be executed BEFORE a fill job. These jobs must NOT be executed simultaneously.

OPTIONS:
   --from HEIGHT  Limit actor and message processing to tipsets at or above HEIGHT (default: 0) [$LILY_FROM]
   --to HEIGHT    Limit actor and message processing to tipsets at or below HEIGHT (default: 0) [$LILY_TO]
   --help, -h     show help (default: false)

`fill` job description

NAME:
   lily job run fill - fill gaps in the database for a given range and set of tasks.

USAGE:
   lily job run fill command [command options] [arguments...]

DESCRIPTION:
   The fill job queries the visor_gap_reports table for gaps to fill and indexes the data reported to have gaps.
   A gap in the visor_gap_reports table is any row with status 'GAP'.
   fill will index gaps based on the list of tasks (--tasks) provided over the specified range (--from --to).
   Each epoch and its corresponding list of tasks found in the visor_gap_reports table will be indexed independently.
   When the gap is successfully filled its corresponding entry in the visor_gap_reports table will be updated with status 'FILLED'.
   
   As an example, the below command:
     $ lily job run --tasks=block_headers,message fill --from=10 --to=20
   fills gaps for block_headers and messages tasks from epoch 10 to 20 (inclusive)
   
   Constraints:
   - the fill job must be executed AFTER a find job. These jobs must NOT be executed simultaneously.

COMMANDS:
   notify   notify the provided queueing system of gaps to index allowing tipset-workers to perform the indexing.
   help, h  Shows a list of commands or help for one command

OPTIONS:
   --from HEIGHT  Limit actor and message processing to tipsets at or above HEIGHT (default: 0) [$LILY_FROM]
   --to HEIGHT    Limit actor and message processing to tipsets at or below HEIGHT (default: 0) [$LILY_TO]

Jobs capable of notifying for distributed indexing now have a notify command

`watch notify`

NAME:
   lily job run watch notify - notify the provided queueing system of epochs to index allowing tipset-workers to perform the indexing.

USAGE:
   lily job run watch notify [command options] [arguments...]

DESCRIPTION:
   The notify command will insert tasks into the provided queueing system for consumption by tipset-workers.
   This command should be used when lily is configured to perform distributed indexing.

OPTIONS:
   --queue value  Name of queue system that job will notify. [$LILY_JOB_QUEUE]
   --help, -h     show help (default: false)

`walk notify`

NAME:
   lily job run walk notify - notify the provided queueing system of epochs to index allowing tipset-workers to perform the indexing.

USAGE:
   lily job run walk notify [command options] [arguments...]

DESCRIPTION:
   The notify command will insert tasks into the provided queueing system for consumption by tipset-workers.
   This command should be used when lily is configured to perform distributed indexing.

OPTIONS:
   --queue value  Name of queue system that job will notify. [$LILY_JOB_QUEUE]
   --help, -h     show help (default: false)

`fill notify`

NAME:
   lily job run fill notify - notify the provided queueing system of gaps to index allowing tipset-workers to perform the indexing.

USAGE:
   lily job run fill notify [command options] [arguments...]

DESCRIPTION:
   The notify command will insert tasks into the provided queueing system for consumption by tipset-workers.
   This command should be used when lily is configured to perform distributed indexing.

OPTIONS:
   --queue value  Name of queue system that job will notify. [$LILY_JOB_QUEUE]

`--api` and `--api-token` are now flags on the `lily` command only (previously each job took these flags, which was annoying)

NAME:
   lily - a tool for capturing on-chain state from the filecoin network

USAGE:
   lily [global options] command [command options] [arguments...]

COMMANDS:
   chain     Interact with filecoin blockchain
   daemon    Start a lily daemon process.
   export    
   help, h   Shows a list of commands or help for one command
   init      Initialise a lily repository.
   log       Manage logging
   migrate   Manage the schema version installed in a database.
   net       Manage P2P Network
   stop      Stop a running lily daemon
   sync      Inspect or interact with the chain syncer
   wait-api  Wait for lily api to come online
   job       Manage jobs being run by the daemon.

GLOBAL OPTIONS:
   --api value                   Address of lily api in multiaddr format. (default: "/ip4/127.0.0.1/tcp/1231") [$LILY_API]
   --api-token value             Authentication token for lily api. [$LILY_API_TOKEN]
   --log-level LEVEL             Set the default log level for all loggers to LEVEL (default: "info") [$GOLOG_LOG_LEVEL]
   --log-level-named value       A comma delimited list of named loggers and log levels formatted as name:level, for example 'logger1:debug,logger2:info' [$LILY_LOG_LEVEL_NAMED]
   --jaeger-tracing              (default: false) [$LILY_JAEGER_TRACING]
   --jaeger-service-name value   (default: "lily") [$LILY_JAEGER_SERVICE_NAME]
   --jaeger-provider-url value   (default: "http://localhost:14268/api/traces") [$LILY_JAEGER_PROVIDER_URL]
   --jaeger-sampler-ratio value  If less than 1 probabilistic metrics will be used. (default: 1) [$LILY_JAEGER_SAMPLER_RATIO]
   --prometheus-port value       (default: ":9911") [$LILY_PROMETHEUS_PORT]
   --redis-addr value            Redis server address in "host:port" format (default: "127.0.0.1:6379") [$LILY_REDIS_ADDR]
   --redis-username value        Username to authenticate the current connection when redis ACLs are used. [$LILY_REDIS_USERNAME]
   --redis-password value        Password to authenticate the current connection [$LILY_REDIS_PASSWORD]
   --redis-db value              Redis DB to select after connection to server (default: 0) [$LILY_REDIS_DB]
   --version, -v                 print the version (default: false)

TODO:

update CLI docs
update lilium.sh documentation
adjust helm

codecov-commenter · 2022-04-27T20:26:30Z

Codecov Report

Merging #944 (e3f12d6) into master (a35ae36) will not change coverage.
The diff coverage is 16.6%.

@@          Coverage Diff           @@
##           master    #944   +/-   ##
======================================
  Coverage    36.5%   36.5%           
======================================
  Files          27      27           
  Lines        1849    1849           
======================================
  Hits          675     675           
  Misses       1099    1099           
  Partials       75      75

frrist · 2022-04-28T00:53:28Z

commands/job/options.go

+	Name:        "tasks",
+	Usage:       "Comma separated list of tasks to run in job. Each task is reported separately in the storage backend.",
+	EnvVars:     []string{"LILY_JOB_TASKS"},
+	Value:       cli.NewStringSlice(tasktype.AllTableTasks...),


by default we enable all tasks.
This means calls to the survey command will error when passed the default values, and callers will need to explicit set tasks to its expected values - peeragents

That's not a huge deal. But ideally there's an override Flag which inherits the regular taskFlag fields but sets the Value appropriately. I leave this up to you.

Users can still do that with this flag: lily job run --tasks=peeragents survey overrides the default values of this flag, as does lily job run --task=block_header watch.

frrist · 2022-04-28T00:54:06Z

commands/job/options.go

+var RangeFromFlag = &cli.Int64Flag{
+	Name:        "from",
+	Usage:       "Limit actor and message processing to tipsets at or above `HEIGHT`",
+	EnvVars:     []string{"LILY_FROM"},
+	Destination: &rangeFlags.from,
+	Required:    true,
+}
+
+var RangeToFlag = &cli.Int64Flag{
+	Name:        "to",
+	Usage:       "Limit actor and message processing to tipsets at or below `HEIGHT`",
+	EnvVars:     []string{"LILY_TO"},
+	Destination: &rangeFlags.to,
+	Required:    true,
+}


reusable flags by all commands that accept a range: walk find and fill

frrist · 2022-04-28T00:55:11Z

commands/job/options.go

+
+var rangeFlags rangeOps
+
+func (r rangeOps) validate() error {


called in the before function on walk fill and find commands

frrist · 2022-04-28T00:56:04Z

commands/lily.go

@@ -14,10 +14,10 @@ import (
 	"github.com/filecoin-project/lily/lens/lily"
 )

-func GetAPI(ctx context.Context, addrStr string, token string) (lily.LilyAPI, jsonrpc.ClientCloser, error) {
-	addrStr = strings.TrimSpace(addrStr)
+func GetAPI(ctx context.Context) (lily.LilyAPI, jsonrpc.ClientCloser, error) {


looks at flags rather than accept them as arguments, decrease the opportunity for developer error.

frrist · 2022-04-28T01:06:00Z

chain/gap/fill.go

@@ -21,12 +21,12 @@ type Filler struct {
 	DB                   *storage.Database


changes here due to unify range flags (--to and --from) to use int64's. mechanical change

frrist · 2022-04-28T01:06:05Z

chain/gap/find.go

@@ -16,12 +16,12 @@ type Finder struct {
 	DB                   *storage.Database


changes here due to unify range flags (--to and --from) to use int64's. mechanical change

frrist · 2022-04-28T01:06:11Z

chain/gap/find_test.go

@@ -19,8 +19,8 @@ import (
 )


changes here due to unify range flags (--to and --from) to use int64's. mechanical change

frrist · 2022-04-28T01:06:16Z

chain/gap/notify.go

@@ -18,12 +18,12 @@ type Notifier struct {
 	queue                *queue.AsynQ


changes here due to unify range flags (--to and --from) to use int64's. mechanical change

frrist · 2022-04-28T01:06:54Z

commands/chain.go

@@ -9,14 +9,15 @@ import (



mechanical changes to remove client flags

frrist · 2022-04-28T01:08:00Z

commands/migrate.go

@@ -4,10 +4,11 @@ import (
 	"fmt"


mechanical changes to remove client flags and rename Visor* flags to Lily*

frrist · 2022-04-28T01:08:26Z

commands/net.go

@@ -29,12 +29,9 @@ var NetCmd = &cli.Command{
 var NetID = &cli.Command{


mechanical to remove client flags

frrist · 2022-04-28T01:08:43Z

commands/setup.go

@@ -29,23 +29,23 @@ import (



mechanical to rename Visor to lily

frrist · 2022-04-28T01:08:52Z

commands/stop.go

@@ -8,12 +8,9 @@ import (
 var StopCmd = &cli.Command{


mechanical to remove client flags

frrist · 2022-04-28T01:09:01Z

commands/sync.go

@@ -6,12 +6,13 @@ import (
 	"time"


mechanical to remove client flags

frrist · 2022-04-28T01:09:13Z

commands/wait.go

@@ -13,16 +13,13 @@ import (
 var WaitApiCmd = &cli.Command{
 	Name:  "wait-api",


mechanical to remove client flags

frrist · 2022-04-28T01:09:48Z

lens/lily/api.go

-	Name   string
-	Tasks  []string
-	Queue  string
+type LilyJobConfig struct {


Config used by all configs used to submit jobs

frrist · 2022-04-28T01:10:16Z

lens/lily/impl.go

@@ -75,13 +75,13 @@ func (m *LilyNodeAPI) ChainGetTipSetAfterHeight(ctx context.Context, epoch abi.C



mechanical change to use JobConfig struct

frrist · 2022-04-28T01:10:43Z

storage/sql.go

@@ -492,7 +492,7 @@ func GenerateUpsertStrings(model interface{}) (string, string) {
 }

 // returns a map of heights to missing tasks, and a list of heights to iterate the map in order with.


mechanical change to use int64's

placer14

A bunch of really small details. This is a great refactor. 🙌 Sticky approve to merge. 👍

placer14 · 2022-04-29T21:01:25Z

commands/job/options.go

+	}
+	if len(RunFlags.Tasks.Value()) == 0 {
+		// TODO don't panic
+		panic("need tasks")


nit: A more descriptive panic would be nice.

placer14 · 2022-04-29T21:28:00Z

lens/lily/api.go

+type LilyIndexNotifyConfig struct {
+	IndexConfig LilyIndexConfig
+
+	Queue string
 }


Hooray struct composition!

placer14 · 2022-04-29T21:40:41Z

commands/job/index.go

-	Usage: "Index the state of a tipset from the filecoin blockchain by height",
+	Usage: "Index the state of a tipset from the filecoin blockchain by height.",
+	Description: `
+	Index the state of a tipset from the filecoin blockchain by height. If the provided height is a null-round and error will be returned.


If the provided height is a null-round, an error will be returned.

placer14 · 2022-04-29T21:45:40Z

commands/job/watch.go

+to the --workers flag will allow the watch job to index tipsets simultaneously (Note: this will use a significant amount of system resources).
+
+Since it may be the case that lily experiences a reorg while the watch job is observing the head of the chain
+the --confidence flag may be used to buffed the amount of tipsets observed before it begins indexing - illustrated by the below diagram:


to buffer

placer14 · 2022-04-29T21:50:22Z

commands/help.go

-                        while 0 means none should be traced. No intermediate
-                        values are accepted. For a 'probabilistic' sampler
-                        the param indicates the fraction of function calls
-                        that should be sampled.


I think a lot of these hidden flags and envvars are not documented anywhere. If this is true, could we see about including them somewhere appropriate?

They are via:

./lily --help <omitted some details> GLOBAL OPTIONS: --api value Address of lily api in multiaddr format. (default: "/ip4/127.0.0.1/tcp/1231") [$LILY_API] --api-token value Authentication token for lily api. [$LILY_API_TOKEN] --log-level LEVEL Set the default log level for all loggers to LEVEL (default: "info") [$GOLOG_LOG_LEVEL] --log-level-named value A comma delimited list of named loggers and log levels formatted as name:level, for example 'logger1:debug,logger2:info' [$LILY_LOG_LEVEL_NAMED] --jaeger-tracing (default: false) [$LILY_JAEGER_TRACING] --jaeger-service-name value (default: "lily") [$LILY_JAEGER_SERVICE_NAME] --jaeger-provider-url value (default: "http://localhost:14268/api/traces") [$LILY_JAEGER_PROVIDER_URL] --jaeger-sampler-ratio value If less than 1 probabilistic metrics will be used. (default: 1) [$LILY_JAEGER_SAMPLER_RATIO] --prometheus-port value (default: ":9911") [$LILY_PROMETHEUS_PORT] --redis-addr value Redis server address in "host:port" format (default: "127.0.0.1:6379") [$LILY_REDIS_ADDR] --redis-username value Username to authenticate the current connection when redis ACLs are used. [$LILY_REDIS_USERNAME] --redis-password value Password to authenticate the current connection [$LILY_REDIS_PASSWORD] --redis-db value Redis DB to select after connection to server (default: 0) [$LILY_REDIS_DB] --version, -v print the version (default: false)

placer14 · 2022-04-29T21:52:53Z

commands/job/options.go

+	Name:        "tasks",
+	Usage:       "Comma separated list of tasks to run in job. Each task is reported separately in the storage backend.",
+	EnvVars:     []string{"LILY_JOB_TASKS"},
+	Value:       cli.NewStringSlice(tasktype.AllTableTasks...),


That's not a huge deal. But ideally there's an override Flag which inherits the regular taskFlag fields but sets the Value appropriately. I leave this up to you.

refactor: simplify job cli

799b6a9

frrist self-assigned this Apr 27, 2022

frrist linked an issue Apr 27, 2022 that may be closed by this pull request

Unify CLI #943

Closed

frrist added the kind/chore Technical debt due to be resolved label Apr 27, 2022

frrist added 3 commits April 27, 2022 11:51

fix tests

051fc5e

docs: add find description

4928cad

add docs

3a95eb7

remove help topics

c23834c

frrist marked this pull request as ready for review April 28, 2022 00:06

frrist requested review from placer14 and kasteph April 28, 2022 00:07

This was linked to issues Apr 28, 2022

lily help sync != lily sync --help #743

Closed

Mentions about Visor still around in CLI docs #742

Closed

fix: panics in job parse

d058508