Documentation on config.json inputs?

[aaronsteers]

We're looking to use this tap in one of our pipelines but we're having difficulty with the configuration. Does anyone have a config guide somewhere or a list of required/optional settings?

Happy to submit a PR to help expand the documentation, if someone is able to lead me in the right direction.

Thanks!

[ahardisty]

@aaronsteers

This passes the Discovery Mode and throws KeyError: 'tables'.

{
"host": "<ftp.hostname.com/">",
"sftp_root_dir": "<dir/foo/>",
"port": "22",
"username": "",
"password": "",
"private_key_file": ""
}

[aaronsteers]

Found this thread on Slack which confirms the need for either a tables list or a tables dict:

https://singer-io.slack.com/archives/C2TGFCZEV/p1581362621084000

---

It's not clear from context, but perhaps something like this inside the config.json file is worth a shot:

{
  "tables": ["filename.json"]
}

[aaronsteers]

@dmosorast and @cosimon - It looks like you 2 are the most recent contributors to PRs on this repo.
By chance, do you have any guidance for us on the above?

[rfa-mpatel]

Give this a try:

{
"host":"",
"port":22,
"username":"",
"password":"",
"private_key_file":null,
"tables":"[{\"table_name\":\"fx_spot_rates\",\"search_prefix\":\"/test/vv_data\",\"search_pattern\":\"fx_spot_rate.*.csv\",\"key_properties\":[\"ValDate\",\"Underlying\"],\"delimiter\":\",\"},{\"table_name\":\"fx_forward_rates\",\"search_prefix\":\"/test/vv_data\",\"search_pattern\":\"fx_forward_rate.*.csv\",\"key_properties\":[\"ValDate\",\"Underlying\",\"DelDate\"],\"delimiter\":\",\"},{\"table_name\":\"fx_hedging_cost\",\"search_prefix\":\"/test/vv_data\",\"search_pattern\":\"fx_hedging_cost.*.csv\",\"key_properties\":[\"ValDate\",\"Underlying\",\"Security\"],\"delimiter\":\",\"}]",
"start_date":"2020-01-01"
}

[aaronsteers]

@rfa-mpatel - Thanks very much! This points us in the right direction, for sure! 🚀

@ahardisty - I used this tool to unescape the json and then this tool to prettify the output.

Here's the underlying structure from above, prettified for readability:

[
   {
      "table_name":"fx_spot_rates",
      "search_prefix":"/test/vv_data",
      "search_pattern":"fx_spot_rate.*.csv",
      "key_properties":[
         "ValDate",
         "Underlying"
      ],
      "delimiter":","
   },
   {
      "table_name":"fx_forward_rates",
      "search_prefix":"/test/vv_data",
      "search_pattern":"fx_forward_rate.*.csv",
      "key_properties":[
         "ValDate",
         "Underlying",
         "DelDate"
      ],
      "delimiter":","
   },
   {
      "table_name":"fx_hedging_cost",
      "search_prefix":"/test/vv_data",
      "search_pattern":"fx_hedging_cost.*.csv",
      "key_properties":[
         "ValDate",
         "Underlying",
         "Security"
      ],
      "delimiter":","
   }
]