[druid] Superset cannot see realtime datasource with no segment

[zhiyuanouyang]

Make sure these boxes are checked before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if any
• I have reproduced the issue with at least the latest released version of superset
• I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

newest

Expected results

I use Druid Kafka-indexing to ingest realtime data into Druid datasource, and I can query using http://localhost:8082/druid/v2/datasources, the datasource shows there. I also do some query from command line and I can get responded results. But this dataset cannot be seen in superset even after I refresh the metadata of datasources. I do not know how to solve it

Actual results

Steps to reproduce

[mistercrunch]

Depending on your version of Druid, the "segment metadata" endpoint exposed by Druid that Superset needs needs for a physical segment to exist in order to report the metadata. In these earlier versions of Druid, it appears as though there was no other ways to get the schema and column type information we require. Normally once the first time granularity time period closes (say the first 24 hours) then you'll get a segment and then Superset will discorver it.

[zhiyuanouyang]

Thanks for your reply. But in the Kafka-Indexing-Service script, I specified the segment granularity to "MINUTE", but I still cannot see anything from superset after several minutes. Here is my script.

{
  "type": "kafka",
  "dataSchema": {
    "dataSource": "superset-kafka-realtime",
    "parser": {
      "type": "string",
      "parseSpec": {
        "format": "json",
        "timestampSpec": {
          "column": "timestamp",
          "format": "auto"
        },
        "dimensionsSpec": {
          "dimensions": [
            "gctype"
          ]
        }
      }
    },
    "metricsSpec": [
      {
        "name" : "count",
        "type" : "longSum",
        "fieldName" : "count"
      }
    ],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "MINUTE",
      "queryGranularity": "NONE"
    }
  },
  "tuningConfig": {
    "type": "kafka",
    "maxRowsPerSegment": 5000000
  },
  "ioConfig": {
    "topic": "test",
    "consumerProperties": {
      "bootstrap.servers": "localhost:9092"
    },
    "taskCount": 1,
    "replicas": 1,
    "taskDuration": "P30D"
  }
}

Is there any configuration to set to control the granularity you are talking about? Say I want to see the data update in every minute. Really appreciate!

[mistercrunch]

What is your Druid version? Do you have historical nodes setup? Does the segment metadata endpoint return column and type information?

Superset uses the segment metadata feature in Druid to provide the information it needs. The question of "how do I get version X of Druid to show segment metadata" really is a Druid question. The answer is probably to find a way for Druid to create the physical segment, or to upgrade Druid, though that's a question for the Druid community.

FYI at Airbnb we have a framework that makes it easy to load data from our Kafka topics into Druid that works along with our logging framework which uses Thrift to serialize and as a way to enforce schemas. In our case this framework creates the Superset datasources in an alternative way (not using the segment metadata endpoint) and that allows us to have more control over things with a script that is more specific.

[nipunb]

We are running into the same issue. The segment granularity is set to MINUTE. The druid/v2/datasources returns the data source in question (kafka) but it doesnt show up in Superset. What is segment metadata endpoint?

[nipunb]

I checked the druid metadata endpoint - druid/coordinator/v1/metadata/datasources
The datasource in question doesnt show up there, it does show up in the druid/v2/datasources but not in the metadata.

[mistercrunch]

Which version of Druid?

[nipunb]

It took ~1hr but it showed up after that.