clickhouse_sinker

clickhouse_sinker is a sinker program that consumes kafka message and import them to ClickHouse.

Features

• Easy to use and deploy, you don't need write any hard code, just care about the configuration file
• Custom parser support.
• Support multiple sinker tasks, each runs on parallel.
• Support multiply kafka and ClickHouse clusters.
• Bulk insert (by config bufferSize and flushInterval).
• Loop write (when some node crashes, it will retry write the data to the other healthy node)
• Uses Native ClickHouse client-server TCP protocol, with higher performance than HTTP.

Install && Run

By binary files (suggested)

Download the binary files from release, choose the executable binary file according to your env, modify the conf files, then run ./clickhouse_sinker -conf conf

By source

• Install Golang

• Go Get

go get -u github.com/housepower/clickhouse_sinker/...

cd $GOPATH/src/github.com/housepower/clickhouse_sinker

go get -u github.com/kardianos/govendor

# may take a while
govendor sync

• Build && Run

go build -o clickhouse_sinker bin/main.go

## modify the config files, then run it
./clickhouse_sinker -conf conf

Support parsers

• Json

Supported data types

• UInt8, UInt16, UInt32, UInt64, Int8, Int16, Int32, Int64
• Float32, Float64
• String
• FixedString
• DateTime (UInt32), Date(UInt16)

Configuration

See config example

Custom metric parser

• You just need to implement the parser interface on your own

type Parser interface {
	Parse(bs []byte) model.Metric
}

See json parser