[Bug]: cdk deploy start with multiple connectors in the same directory can fail

[digikata]

What happened

On Macos:

$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c1-connector.yaml
$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c2-connector.yaml

Results in one stopped connector if both deploys are run in the same directory

cdk deploy list
 NAME    STATUS
 c1           Running
 c2           Stopped

On linux:

$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c1-connector.yaml
Started connector `c1`
$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c2-connector.yaml
... checking package
Error: Failed to deploy from ipkg file

Caused by:
    Text file busy (os error 26)

Expected behavior

Both connectors should run

cdk deploy list
 NAME    STATUS
 c1           Running
 c2           Running

Describe the setup
Using connectors downloaded for locally running them.
cdk hub download infinyon/[email protected]

Create two configs with a different name:

c1-connector.yaml

apiVersion: 0.2.0
meta:
  version: 0.3.2
  name: c1
  type: http-source
  topic:
    version: 0.1.0
    meta:
      name: sentence 
http:
  endpoint: https://demo-data.infinyon.com/api/quote-stream
  method: GET
  stream: true
  delimiter: "\n\n"

c2-connector.yaml

apiVersion: 0.2.0
meta:
version: 0.3.2
name: c2
type: http-source
topic:
  version: 0.1.0
  meta:
    name: sentence 
http:
endpoint: https://demo-data.infinyon.com/api/quote-stream
method: GET
stream: true
delimiter: "\n\n"

How to reproduce it (as minimally and precisely as possible)

$ cdk hub download infinyon/[email protected]`
$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c1-connector.yaml
$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c2-connector.yaml
$ cdk deploy list

Workaround

A workaround to this issue is to deploy start connectors in separate directories.

$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c1-connector.yaml
$ cd ../somewhere-else
$ cdk deploy start --ipkg infinyon-http-source-0.3.3.ipkg -c c2-connector.yaml

[digikata]

Initial analysis looks like both cdk deploy calls generate logs to the same file http-source.log. Also an extraction from the ipkg of the connector binary http-source collides between the first and second invocation with macos showing slightly different symptoms than linux

One change suggested by this is to unpack the connector binary in a temp directory.

A second suggested change is to revise from using the package name (http-source in this case) to using the connector name (c1 or c2) for the log name. This would require changes in the cdk deploy start as well as cdk deploy log.

[sehz]

Ran into same issue. Main problem is that these artifacts are stored in the same directory. So here is possible solution:

• Download ipkg to fluvio hidden directory like: ~/.fluvio/cdk/ipkg. This also benefit of downloading only one time.
• All logs are related runtime files are stored in ~/.fluvio/cdk/deploy/<deploy_name..

[ajhunyady]

If we download to ~/.fluvio/cdk/ipkg we need to add new pkg cli command to view and manage local packages:

cdk pkg list
cdk pkg delete <pkg-name>

This assumes that packages will continue to be downloaded via cdk hub download.