StarRocks and Onetable.dev support

[alberttwong]

Onetable.dev is a tool that allows you to seamlessly interoperate between Apache Hudi, Delta Lake, and Apache Iceberg. It does this by converting the metadata from one format to another.

By default, onetable only generates the data and does not insert/update the data in Hive HMS. There is engineering work to make this happen.

[sagarlakshmipathy]

as of today there are 2 ways to sync OneTable tables in HMS.

1. Use tools like metasync for hudi, Iceberg's stored proc etc as highlighted here https://onetable.dev/docs/hms/#register-the-target-table-in-hive-metastore
2. Or as you are running sync, you can update HMS with the table like below.

val namespace = "demo" // requires that this db is already created in your HMS, in Trino you can run `create schema hive.demo`
val namespaceArray = Array(namespace)
val catalogOptions = new HashMap[String, String]();
catalogOptions.put("uri", "thrift://host.docker.internal:9083");
val icebergCatalogConfig = IcebergCatalogConfig.builder()
    .catalogImpl("org.apache.iceberg.hive.HiveCatalog")
    .catalogName("iceberg")
    .catalogOptions(catalogOptions)
    .build();
val perTableConfig = PerTableConfig.builder()
    .tableName(hudiTableName)
    .namespace(namespaceArray)
    .targetTableFormats(Arrays.asList(TableFormat.ICEBERG))
    .tableBasePath(hudiBasePath)
    .icebergCatalogConfig(icebergCatalogConfig)
    .syncMode(SyncMode.INCREMENTAL)
    .build()
val hudiSourceClientProvider = new HudiSourceClientProvider()
hudiSourceClientProvider.init(spark.sparkContext.hadoopConfiguration, Collections.emptyMap())
val oneTableClient = new OneTableClient(spark.sparkContext.hadoopConfiguration)
oneTableClient.sync(perTableConfig, hudiSourceClientProvider)

Note that this requires spark.

[alberttwong]

Moved to https://forum.starrocks.io/t/starrocks-and-onetable-dev-support/22