Repartitioning via nakadi SQL does not work. Key is in wrong place in payload.

[barry-hennessy]

Expected Behavior

Creating a nakadi SQL query with repartition_parameters is possible when repartitioning is set in the spec (within outputEventType).

Actual Behavior

The repartitioning dict is successfully parsed, but output in the wrong place in the body of the POST to create the SQL query.

According to the Nakadi SQL documentation repartition_parameters is a key within output_event_type, not a top level key itself.

I've hacked a patch in to make the change and can confirm that the following change in JSON output works:

 {
   "id": "event-name.multiple-partitions",
   "sql": "SELECT * FROM \"event-name\" AS a",
   "envelope": false,
   "output_event_type": {
     "name": "event-name.multiple-partitions",
     "owning_application": "app
     "category": "business",
     "audience": "component-internal",
     "cleanup_policy": "delete",
-    "retention_time": 86400000
-  },
-  "repartition_parameters": {
-    "number_of_partitions": 6,
-    "partition_strategy": "hash",
-    "partition_key_fields": [
-      "config_sku"
-    ]
+    "retention_time": 86400000,
+    "repartition_parameters": {
+      "number_of_partitions": 6,
+      "partition_strategy": "hash",
+      "partition_key_fields": [
+        "config_sku"
+      ]
+    }
   }
}

Steps to Reproduce the Problem

1. Add a repartitioning key to an event like so:

diff --git a/docs/examples/single/sql-query.yaml b/docs/examples/single/sql-query.yaml
index 5c2d0f2..7f5932b 100644
--- a/docs/examples/single/sql-query.yaml
+++ b/docs/examples/single/sql-query.yaml
@@ -13,6 +13,10 @@ spec:
    cleanup:
      policy: delete
      retentionTimeDays: 2
+    repartitioning:
+      partitionCount: 6
+      strategy: hash
+      keys: ["important_key"]
  auth:
    users:
      admins:

Side note: This may be good to add to the documentation/examples. It wasn't immediately clear how to specify this.

2. Apply the spec

clin apply -e ... -t ... docs/examples/single/sql-query.yaml -v -p -X

3. Note the location of repartition_parameters

4. The event will be created, but with the default number of parameters, not the ones you specified in repartitioning.

Specifications

• Version: clin, version 1.3.0
• Platform: mac osX 10.15.7
• Subsystem: -

[barry-hennessy]

The following change likely fixes the problem. Python is not my day-to-day language, so there may be a better/more pythonic way.

diff --git a/clin/clients/nakadi_sql.py b/clin/clients/nakadi_sql.py
index 46f7b80..8714a0a 100644
--- a/clin/clients/nakadi_sql.py
+++ b/clin/clients/nakadi_sql.py
@@ -120,7 +120,7 @@ def sql_query_to_payload(sql_query: SqlQuery) -> dict:
     }
 
     if sql_query.output_event_type.repartitioning:
-        payload["repartition_parameters"] = {
+        payload["output_event_type"]["repartition_parameters"] = {
             "number_of_partitions": sql_query.output_event_type.repartitioning.partition_count,
             "partition_strategy": str(
                 sql_query.output_event_type.repartitioning.strategy

[dlippok]

The bug should be fixed in version 1.4.1. In case of further problems, please feel free to reopen this issue again.