CI: kafka integration tests move_table meet panic: the commitTs of the emit row is less than the received row.

[asddongmen]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
   CI run integration tests.
2. What did you expect to see?
   pass all tests.
3. What did you see instead?
   move_table fails times and can find an panic in
   cdc_kafka_consumer.log

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/3939/pipeline/

[2021/09/24 14:46:49.037 +08:00] [PANIC] [common.go:87] ["the commitTs of the emit row is less than the received row"] [table=move_table.usertable] ["emit row startTs"=427941163863900200] ["emit row commitTs"=427941163863900200] ["last received row commitTs"=427941164073877740] [stack="github.com/pingcap/ticdc/cdc/sink/common.(*UnresolvedTxnCache).Append\n\tgithub.com/pingcap/ticdc/cdc/sink/common/common.go:87\ngithub.com/pingcap/ticdc/cdc/sink.(*mysqlSink).EmitRowChangedEvents\n\tgithub.com/pingcap/ticdc/cdc/sink/mysql.go:115\nmain.(*Consumer).ConsumeClaim\n\tcommand-line-arguments/main.go:442\ngithub.com/Shopify/sarama.(*consumerGroupSession).consume\n\tgithub.com/Shopify/[email protected]/consumer_group.go:690\ngithub.com/Shopify/sarama.newConsumerGroupSession.func2\n\tgithub.com/Shopify/[email protected]/consumer_group.go:615"]

4. Versions of the cluster

   • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

     (paste TiDB cluster version here)
     

   • TiCDC version (execute cdc version):

     https://github.com/pingcap/ticdc/pull/2880
     

[asddongmen]

another case:
cdc_kafka_consumer 2.log
https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/3853/pipeline/

[asddongmen]

another case:
https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/4070/pipeline

[asddongmen]

https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/4126/pipeline/

[asddongmen]

another case:
https://ci2.pingcap.net/blue/organizations/jenkins/cdc_ghpr_kafka_integration_test/detail/cdc_ghpr_kafka_integration_test/4487/pipeline/113
cdc_kafka_consumer.log

[asddongmen]

I change the severity from major to moderate because this bug is a test only bug.

[3AceShowHand]

[2022/03/31 19:21:31.211 +08:00] [INFO] [main.go:555] ["emit row"] [row="{\"start-ts\":432203542921216144,\"commit-ts\":432203542921216144,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODg1OTM4Njc0ODE2Mjc4NDg=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"WkNmVmVGYmpnU2RGRmZVeHJyT05KV0ZRZWpja2xuYnVxQW9ReWRvd3JpQ3pRUmRsdWhkY0Z1SEFCUWxYYXhldmNLWk95c0FObkpybGx0ZGVGWFN4SnVxaXNnQkhBd3BpR1ptVQ==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"aVdRZGhBTER2WFFqdktQZXBNQWVJR3l5a25sSU5xc0VPbHVITGdlWGhqUmpTTEdRc2VLbVhoRVRJdW90UmJwTXNxTFdmYkxpeHpmRGh4YWZoblJCdHJFbER1WnBaVkdnWllNUA==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"WkZmaUVUanFpam1CQkRyS25SYVp4ZkxSUnBac1FYeldrc2x3WlF1R3RMSUJueXhBdVNPaExYSXlXcEZvWHZac3ZyUnpNbE9vUEZhak9kd3d2UWJXRHl2ZUhsbm5CSmlZSVVsZw==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"QW5VRGFsRGZBakdzT2JZc3RRSmdkUWlaUW5VSUlhTW5tZ2dyaWVBTXFFV3RtbElRYVJCQVpnQWh0b3FOT2lTYk5PR1B5VFd2Uk9uSE9XU0JnUnRRa2tmYWJSdG5uaEF0V3R2eQ==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"SE1uRmViT2VxQWNqSnZYZk91aHhVc0hZYVp5d3pGVXNXaEdmdGlrcHZTUXFkblRlQWxkbGpXQmNTR2JlZGFDWkJ4VHFseU5jS2FJd1hQQnZUbnd5eVluRHVpdU5KUUlReHRBdg==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"eHJNcVR6bXpXTnBpaENHRnBObXFXU0ljaEJHZWV3UHJZaGlNUmNVWktiWEV2ZVZHWUt0cHZ0Y0pvc0d2V3dEV1pmanlubUpPV1RXc1hMakl4SlJLZ1FESllmdmdPUFhxSGFoWQ==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"ZG1LWWVUbnFZd2VERFN1Tm9Ea1ZpZFRrQ2JqZndWTkFQQkpoQ3V5TVhsc0VqZE1uZ2pkYnBhd0xiUFVSTWVVRnNtS2tnQm5JVk1taGtHcWtvZ3JLZkF3SFVQR1hzWm9MYlZZbA==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dXBEZ1pncFhueGhua2NES3VaZU1GaXhrYkpGUmVrY2VlUlh0VVF6RmlXRlZpWm5PeGtXT2duZkZyRmZWb01FUGZieHFoYlVQWmJOSHRhRENtSnpIZFRxb1h5R3lVUFVlSFF0VQ==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"aVVRa2VDcnBrZ1VXZFJGYVdCckdPaEpoTlBVUFNCUnBWdFZUcFdCRURidEJ3Qlh4SEpIUGluVXNXTmF6QlVibWdRWW5XR1lUSU9YQXNTblpGYXhxRUNtUHFpVW1Ia3hJSE9iRA==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dlJlbFV6ZGthbUdHUlBKeFZPdnNsdndXUlNTYnhhak16RXhUREVxdGJUSHRIbnB1b0pqUHFGb2NUUVRmWk5SeVZGT3RzV2R0cEdWZUF6QWJTVEJlenZrSGpySnJxc1pnUW1Veg==\"}],\"pre-columns\":null}"] [partition=2]
[2022/03/31 19:21:31.643 +08:00] [INFO] [main.go:555] ["emit row"] [row="{\"start-ts\":432203542960537688,\"commit-ts\":432203542960537688,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODY1MzAwODQxNTU4NTQyNTE=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"UHRxS0d2THdIZk51Q0dBdW1wbXdlbVJveEZlbUROWXZqTmtxRmZxZER3dHdaZ2xwWXFEWnJuYkx0Wnpob1N6VVlXRUhDVk1oT015UG1sSHFrbWNDS0RIQ1VPTHJFYW5RTWF6Yw==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"QmNGYVZYS3N4T0hVWXRjWGFPQWFrUUJnaU9FU3JiQ1J5eHZWY0RsRk5tQnVHQ2VxVVllTVp4U2xYRHRSeFJJQUFUR0FBcFJ4UExtQmJCTU9QdWhpQWpBRkllZVFBaGFaemdYbA==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"QkVLb3JCY0JSQm5icWtpS3hORHZiektZWVpBS21BclJOV3BXeENRQ25lRERUWmFaeWp1SkhjdGlxWk1ZRmZ2T3ZVcmlJWWFuTWVES1RxaUNocGRrck9jaUF6WUtNUnBWT2hSVQ==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"Tm1kc3RlVlp4dHNuQkdTZHRjQ1BndFBHWE5EcUR4b2huTFB4RlR6ZWJMR2FNeUdDc1NhUXp6UFRUQUxFRUppbFNUU3ljaGpyaG9zbGNQVUdsS2tQZUNCVUNPbEdtemtldFFrVA==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"YWNQcWpJQ3Z0aHJpUUpPSHpQd3VSaW9Za0VjUnJSeHFJR0RoaW91WnBYU3BhQUpSdndSaUtzYmVoU05XTXBXdmN1ZnhNS0VTckVaRGp1eWRvZ1NTTmpjR0twamFsbE9pWHJjYw==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"ZXJhcU5UQk9HbHZsaGh0TG5RS0R5c0dpcmNMaURNRFdWdEVLa0dId3lncFZ0WlVkcGxJQVpkaHF6QmN2Z3NBTnhEVnJzY3FEVVJ3a1h4QnJZeExEeUpTQ2V2cHBEc2VOeVZWYw==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"SEhZVHR4RVBsbGJtb2FzWW9HZmRWaVRXaUZpWmlSTHdrRHRsbUl4dlhrbmxlanpyWUVVWEZScExJUGlmVnhCV1hza1VEWHNmTUlTWXBKbXhTSFBZTmpabURsQXphS29MbUFkQQ==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"UHFkdFl4ZURmam1hcnN2Q014ZEpSbXJpZUNLZndHQVFBT1lvb05sWHFybWluV2pqUkNvalBSaFJxQWZ2a3JyWHZNWFp4SWlJcW5XbW9MZ1RrRktBTGliSGpUSElmem9UVVdoeA==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dkZyZnRVTWJ2QlJjWnZIek54eHdiVnpvT3NudVRoZW1ldGpCbnFDR0NMS3ZoTmRXVWxCQUVCYXlXRHJZQnNZQ3NoQllzQ0ZZYWtWdXJHWHR0cWFpTHJ5Z2Vvdm1xaWptYkZvQw==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"R21oSm5mSkhQR3ZScW5TbGtaaWp5cE95eGlvUFZKV2dyaEZJZ2VrWEpSRE9iYkhGZXRrQkVRZldMc1RHSHF0Z01lamFKTWN6ZVZCWlJydnRBcFl2V0piU3lJekdpdGJ2aXZlSQ==\"}],\"pre-columns\":null}"] [partition=2]
[2022/03/31 19:21:31.644 +08:00] [INFO] [main.go:555] ["emit row"] [row="{\"start-ts\":432203542960537688,\"commit-ts\":432203542960537688,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODY1MzExODM2Njc0ODI0NjI=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"Y0VEcWpQcnpueHdpbkFXekN2ZnNIWklBVFNoSkVSbmxXSmpQdmpTSmh3SWNoSUhReFFFTVFWZWx4dHdjQnhpWE5QWk10SGpvVlpZWVVjUVB2RkJMUmtaRkt5ZXhCdkhOTkxGdw==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dkZqUEFXU3JBbWVmdWtVc05wWkx2QmlMUml5bXNwZnltcVFKQ0FkYndWQkdUZVp5WnJFVVdlSGlGbXFmc2JUWVFMQnB1Rkp2SWlDZHFOUkxSTHdiT2dTZm5ySXBnQXh3UUlLWg==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"U1pvUlJOdkNRYmdwZVhHR2RyaWx3R0xZUUxQbUNJeldsWkh4VVFvREtGckV6THRnVHdjSGhZeWdXb29jbWhMcFpyVlBLem12ZUVZS2ZFbFlGWENNUEZKeWhLU2F2VEx2eGRhbA==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"U0ZNZmNlV2paRlZncFZMQUJKSFFwUnp1Y0puV0NVcHdUZWJlYU9QcVdXQ3VDd01lV1ZIdnFhc0xiaXB4Y1JCaWtxWk5RZnVWS01pQmVoYmRja0FZdmdrYkFCUUhwRlZwckdzUw==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dUpUaWdGaVRQSll5RG12Skt4YWl3aEJGcXBJTFJ2QUZMR2FkUkZVQ3Z3aENGbFJVSnZtUlBTQVJnTndtZ0pzQkl4SGxkeW9aZFFpQU5YUnlWbnVObEZrbmp2ZFZWWU5oUndBQw==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"ZmFQcWpsQnBJeHZKYkdIU2pSVkNwYUZZd1lFZG5ZdEt1eXJRZ3lUS3NnQ25EUEZFRWR4RnhFdUlWdXZCQld2Z09MRkdjT3hKc2VQY056UGhtRnN1ZUxsbEp2d3dJSWVNZXhncw==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"Vkh4SHN1YnlYaXJJZURCa1R5YUZPcW5DdVdIckxMd0NpdEZEZ2xZdEhWandLa3FValdZcXBZV0dnRVlTb0JwYVdCRVpKZEpkdFNQekZDZmN4ZXVsb094YUFtc3NIa0luSGdyQg==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"UFV2eWxST3lrbEtsclpTa3lac0dNWkdzY3lIZ09iUmVSbFNBc3NhSHBYRGVjeGlTbXFiS1ZDUnlVVm5EWm5OTFZETldPSUxsbWdNb0VJUVh0TlB4SHFJR3ZVZXpMZnpoeWt0TA==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"SXhVZ1RTY1hCb05RRlJiSEJJb2txc05EWWl4amNOYkNYZUNhaG5Rb09SZ0RwWkxxT0NrRG9ieHBTQ2hualh2bk1NSE1RWHlkb01pcU5QelZCV3l1YUx4RHJSa2phaW1OTXNKSw==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"RHd6S2h1bm9PZVlWWHZmRHZrdE55V05qVmJRWE94U29NWG1Xanljck13cWZwbmFvclFVQVpBUG9QcUpPQ2F5b0RHR0dqbXdRQk5CZERqZWN0Z3J4amtwVmh0SVJvUGpxVmdRdg==\"}],\"pre-columns\":null}"] [partition=2]
[2022/03/31 19:21:31.644 +08:00] [INFO] [main.go:555] ["emit row"] [row="{\"start-ts\":432203542960537688,\"commit-ts\":432203542960537688,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODY1Mjc4ODUxMzI1OTc4Mjk=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"eEhmc1RaZFRMS0xHV1BuQnNwRVVRUXlJa1NCS1dtbkdyYXJLWnR0RklTUVZkcWdHRVVWVWRKaVpNR25ld3JndXNMRG5VTlRCRWlQYkxTWW5hR2tLc2lGa3JETmNVVnRLRVBoZg==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"d1ZuamtaVU9zTEVIZ3dQcWZVcnVsS0ZueHR5WUt4QWhJVk9VQmRseGJDUlVrcFBHb1liTFp0TnFieHl4eHhaQ0NQcWRkSlpOc1FSYXprbFh5V25VTEVFTVRDb01XcXFhbVFCYg==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"S1R0dlFnVEZSQkV4ektyVUVqS0NoQnRBUm1PTHBXSmFFR2dOYWVtZUtTaUV1UmxCV1Ftc3Z6R2ZLbVJOTklmT1hQYWlvS1lhTk1KVmxkYkRCTlp0UmJWR2lFZWJqdllKU2JCQg==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"RVN0SkJYRE10R0pxZWFMWU1QVk1ObGdueEhxUld3QXlsS1ZqRERvV1llZ1ZKVE5PZEpTS0FZcFp6d2xBYWtHc3ZwZlpOdnB6dnhYQ1BSVXNpcFRSSmZKTEdPZGJiY3dYVURadw==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"emlrRVRRcnpCV2FvdHR5WEZhbklRdnlPZFpMZ01FRUlVVm1xR0FMY3hZWmpQSnVHVmZvb2pRbWRGUUNsZ3dRT050c2huR3p4ZXFyWUVrckdHc1lJek9jQk5qZ01xR2hCS3BxSA==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"ZGNHaE9Da2RrVWNScElkZE93cmlrSWNqY3Fwem9OR294RE9kTkttVFJPdVRKZXBnT0RLTHVhRmhqU1R1UklzZHpMRGlvVnFPS2htRkZuU0hKcGx4and3am1ocVlFcUhGdk92WQ==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"WVpFaFBhUkd1d2RLaEFsT1F6cmF1WGxvdlJ4RUNlWENBSUNTR2NVZ2diRENhaUJUZnRYR3VqaVBPUEpBUVhxTmFGeFdIcGJTdHVuSkR0SlVxVlpFcnJOcHFld1VQc0dPdWZPSw==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"QmNmQ1dCd1JPdWVmeGRPWVpIamZMZlNKRlZQUEJOcHFEcm9JdWxKeU5wdG9SdE11UXRZaXZWSlBVUXppdWF3S3dpSndIb0pMdGtEZmx2WWJ6VVB6UkhDZXJ4dHNMRmlnUnZPcw==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"b09sVUllVER6S1hkTHpNUVBEcE5maU1wRWpPeFR1ZFpaYUVtbmVoQk1xa1lYbW5PVW1kU3JjVVl4d0lpWkpwSEVtS0FqSElKSmNsQVFESVpHa3dkTllGYUpCWUNQcUhoRkJmeA==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"SFlhZndYcktUSU9rbHJFTk5wbUZnaElpbUtyd3JuYkp5YlVvc3BBc0F1RFd6RklSYVZmVU1MVUJ1QnprUnd6SExqdVV1SGNRaUNmWU9wb2FIYnlmaHJTbFVMQklFSmNlT1pncQ==\"}],\"pre-columns\":null}"] [partition=2]
[2022/03/31 19:21:31.644 +08:00] [INFO] [main.go:555] ["emit row"] [row="{\"start-ts\":432203542960537688,\"commit-ts\":432203542960537688,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODY1Mjg5ODQ2NDQyMjYwNDA=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"ZnJiQkxHTnBJWWVoQ2hnYklTSU9BSE5IRXBDdG1LTVNlWUZIelVTbVpoeFRkZm9GRkxKRHRDR05kZkhFRGJzSENxVFZFd0VjRGxNckNpRGpMeXV2dmhMd3Zna2pScVlDV2xwaQ==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"d3NmTHlnZlBnUUVRVkJTRXVaT1VsWXZDQk1qRElpZXZJbEdRU0dLQ21OanpnWmVkR3JjRVd5dGVuc0VYVWVlWVB3TFlYeUxaaHl1YU5sa3FUZkRmRmpKeFd4WkN4Rm5DWUlXRA==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"bVhUc2RqUWd2eW55YW55Wnhjb1NqcnRWZEhIZVZBREFldHlmcHdBR3RDWkhZZWFVVW13cW9JcldmWkREc2lPZFhoYkhwVUxxT1RyQlFKcW9JSVFpeVpHWmpEZXVvdmJ5QnNYQw==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"Z2luWXhseGRVZkhQZEhaWWxwVEthVG1XeEJVaEtkckJMZ3d1QWRWRGhTWVBmSUlpc1hJVklPb3lXTEhRZElPWEFIYnN0Y3VMcEF1VmNTendjTUpSVFVuVkRMeGNIeGRzb2lxUw==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"T3lGVHNZSk1NbUZpQ2t6eGtDZHZ1S3RTT2RSY1JTSFlUY1NHelVlVWVpRnVkc0hNUWxNSlJRU1V6YUhPV1RtbmFzeHNNVVNscXpmUm5CTXl5YXRtRGlFdklqaG1Qc1h3dlJKYQ==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"b09ydmpxb2RtWkdUdFNublBOd01VWEZKcmhBanpzR1Nsdk9uUlNQb3NmcWFrbmR4S01CQnV4SURkSXFrSXpsV3NYQkRGekxYSFpwWGlwS3NSZE5MYVdmd3FPR1NEaExyeFRPVQ==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"cUx5Ym14YW9jU2x4Z3JvUVpKSE5DTm16bExVV3FhdUlrZk51eVFQT2toSXBkdkFzZGxLa3JJeEhqQUpUUmxGd0xpd1BWakN1c0VkdUZ1RlZYdkloc1lnSGp5aGdBWVFGclhuTw==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"WFlyS2thVUpZYVd2bXloa1FwVlJqbWdkT3Nwa2JGZXJlcnlwZXRJRnZQQ0FSV3hzaWFQd3RXdlZwYkh0YVpBVXFnUEtrUE53cVBJVnlRZmJ5WW1paGxkdU96SlNKWWVYaklVZg==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"UUdHdmdMVlFKd3FhUE1HZVNCbkhVclVVa1RLeFVzZENmUHVYVnp4YUV2ZWhGVm9ManV3SFdWcFpWZkVzdHZVUkNyQ1N1dnJyRVdWRUNPVVR6a1JacXJhUENJYnNMdUJMZVhtZA==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"T0FlcHhwd3FOdG1XaklueUpLa0tpZFVSUnFNa1BCcmtGcnVtS0xxQkR3UU1xbFJTUE9zZ0p3V09RU2dBWk9MVnhZSmVFZUlIaXdYS0phQnZnT3VLRnNERWtFUWpRZFZZamt2VA==\"}],\"pre-columns\":null}"] [partition=2]
[2022/03/31 19:21:31.644 +08:00] [INFO] [main.go:555] ["emit row"] [row="{\"start-ts\":432203542960537688,\"commit-ts\":432203542960537688,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODY1MjU2ODYxMDkzNDE0MDc=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"Z21LdXRLY0hZVlRJQnJUSk11RnFSVXZYekhiTXZxZ2pPcExnblpZVFNiUEhSWFVjTkZackVKWnNxVnpTdEtvdlZYYmNzU1ZoVFZSU0toS2loVE9xcmxsbFZTYWpoRlN4aHZCUw==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"VGhCVkV3RnNtTndUbW1Oc3pmZUNvTHdvaVFNWlhOUFVCdFZxQUFUU3JPRVdKbWpZUkJkT3B2WnhEYkF5b29PaFFPQm5YbG5USmNVTWVhRXZhVHdDR1NXWGNialpaYVFYZVZxWA==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"aUJwQkJ4c3pVRU1Jb3B1RWJSQ01jd0d2cFd2QUtad2ZzdE9hRGZtSWlIYkdmTGtyWHJBUENZRVlOYmhDZWt0SVBoWXNoREJIUWluelNtRVVBd2RHdk5URHdRYnpveVl4TVJ1cw==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"cVprZkFGV0NUZWFZY2Raa1VHS0hMT29xSVZ5dGtBTENOckhjVXZEZ2tKRW9vWU9JaGd1a0lnckhQWXBkY3dheVBjcFZBdVBaYXdTSHR5aVJESE1paGxvWnRLQm9PbUNobXVuVg==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"RWF0cWFka0tPUXBhWkJ3S1dNdG1iVW9GU1dOcFVHQWZkenF1a1ppZlF0c2VVUkVrTWh0R0hLc1Z4d0djaGhpSWJIZUVJVlNTZFJ6ZGhldVFNVUdvbWhLVWhITnJub3pzRWVOTQ==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"aElmWWVzdHVNTGxCWXdqQXFzSGpOT0VoQlhmTVFSZkpTREVEUmZGalZwWGZSQ1RhWEl0bFRlc3JJUWx1Z05LbkpMUlNISmNHYmRzbnRBUXpzeU5UQndNUWlIeW9Nak9qVGNidA==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"eFd6VmV1RklmeERlWmlOTkNvd1Z2T0JpcmluTWZQSHVEdkJ2U1N2eElmS0FITmZzZWlOb2JScWNzZWlueklHWnJ2Tk5iR0dMbmZZa2NodVZvcVJpZnZDd3BNZ2NVUWRuT01mVg==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"YlNDd2pGd3V5U2tMWEN3b1ZDSGNIek9Sb1htbEdxV0pOTmZJZXBIUXpOdVl1V2dlUk5PUW1SRWh1ekFtbWRmSERYalVDeFhaeXl4c0NJUHJYUmhEcHJvQWdPS1RYa2NJSW9XTA==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dHFNZkFldERmVlp4VEFXdmJlQXJpdFBGcndLYnBOdkFwTmFQYnpoZ1pKdmFaQmRmbHNZaW5vQWpJWnZuYllpeUZVRG1mWmxBaG5JblB2bFFFTHJvRXp4Q0hneHhjYm10VHlRZg==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"eXlsaGFkdVRZdHh1YXRLZ0R1VWVjR0dIUllSWlVzZEZSd2Z2SWJqSmh1QkxuTXBhUGVxVVpkRk1Qa2hYU0tkY3JDTmtDU1NueEh0YXVGTFNRVHRHSGNXT2JDcXFuVEx5d3BjUA==\"}],\"pre-columns\":null}"] [partition=2]
[2022/03/31 19:21:33.548 +08:00] [PANIC] [common.go:85] ["the commitTs of the emit row is less than the received row"] ["last received row commitTs"=432203542960537688] [row="{\"start-ts\":432203542921216144,\"commit-ts\":432203542921216144,\"row-id\":0,\"table\":{\"db-name\":\"move_table\",\"tbl-name\":\"usertable\",\"tbl-id\":1,\"is-partition\":false},\"column-infos\":null,\"replica-id\":0,\"columns\":[{\"name\":\"YCSB_KEY\",\"type\":15,\"charset\":\"\",\"flag\":26,\"value\":\"dXNlcjYyODg1OTM4Njc0ODE2Mjc4NDg=\"},{\"name\":\"FIELD9\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"WkNmVmVGYmpnU2RGRmZVeHJyT05KV0ZRZWpja2xuYnVxQW9ReWRvd3JpQ3pRUmRsdWhkY0Z1SEFCUWxYYXhldmNLWk95c0FObkpybGx0ZGVGWFN4SnVxaXNnQkhBd3BpR1ptVQ==\"},{\"name\":\"FIELD8\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"aVdRZGhBTER2WFFqdktQZXBNQWVJR3l5a25sSU5xc0VPbHVITGdlWGhqUmpTTEdRc2VLbVhoRVRJdW90UmJwTXNxTFdmYkxpeHpmRGh4YWZoblJCdHJFbER1WnBaVkdnWllNUA==\"},{\"name\":\"FIELD7\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"WkZmaUVUanFpam1CQkRyS25SYVp4ZkxSUnBac1FYeldrc2x3WlF1R3RMSUJueXhBdVNPaExYSXlXcEZvWHZac3ZyUnpNbE9vUEZhak9kd3d2UWJXRHl2ZUhsbm5CSmlZSVVsZw==\"},{\"name\":\"FIELD6\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"QW5VRGFsRGZBakdzT2JZc3RRSmdkUWlaUW5VSUlhTW5tZ2dyaWVBTXFFV3RtbElRYVJCQVpnQWh0b3FOT2lTYk5PR1B5VFd2Uk9uSE9XU0JnUnRRa2tmYWJSdG5uaEF0V3R2eQ==\"},{\"name\":\"FIELD5\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"SE1uRmViT2VxQWNqSnZYZk91aHhVc0hZYVp5d3pGVXNXaEdmdGlrcHZTUXFkblRlQWxkbGpXQmNTR2JlZGFDWkJ4VHFseU5jS2FJd1hQQnZUbnd5eVluRHVpdU5KUUlReHRBdg==\"},{\"name\":\"FIELD4\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"eHJNcVR6bXpXTnBpaENHRnBObXFXU0ljaEJHZWV3UHJZaGlNUmNVWktiWEV2ZVZHWUt0cHZ0Y0pvc0d2V3dEV1pmanlubUpPV1RXc1hMakl4SlJLZ1FESllmdmdPUFhxSGFoWQ==\"},{\"name\":\"FIELD3\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"ZG1LWWVUbnFZd2VERFN1Tm9Ea1ZpZFRrQ2JqZndWTkFQQkpoQ3V5TVhsc0VqZE1uZ2pkYnBhd0xiUFVSTWVVRnNtS2tnQm5JVk1taGtHcWtvZ3JLZkF3SFVQR1hzWm9MYlZZbA==\"},{\"name\":\"FIELD2\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dXBEZ1pncFhueGhua2NES3VaZU1GaXhrYkpGUmVrY2VlUlh0VVF6RmlXRlZpWm5PeGtXT2duZkZyRmZWb01FUGZieHFoYlVQWmJOSHRhRENtSnpIZFRxb1h5R3lVUFVlSFF0VQ==\"},{\"name\":\"FIELD1\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"aVVRa2VDcnBrZ1VXZFJGYVdCckdPaEpoTlBVUFNCUnBWdFZUcFdCRURidEJ3Qlh4SEpIUGluVXNXTmF6QlVibWdRWW5XR1lUSU9YQXNTblpGYXhxRUNtUHFpVW1Ia3hJSE9iRA==\"},{\"name\":\"FIELD0\",\"type\":15,\"charset\":\"\",\"flag\":64,\"value\":\"dlJlbFV6ZGthbUdHUlBKeFZPdnNsdndXUlNTYnhhak16RXhUREVxdGJUSHRIbnB1b0pqUHFGb2NUUVRmWk5SeVZGT3RzV2R0cEdWZUF6QWJTVEJlenZrSGpySnJxc1pnUW1Veg==\"}],\"pre-columns\":null}"] [stack="github.com/pingcap/tiflow/cdc/sink/common.(*UnresolvedTxnCache).Append\n\tgithub.com/pingcap/tiflow/cdc/sink/common/common.go:85\ngithub.com/pingcap/tiflow/cdc/sink.(*mysqlSink).EmitRowChangedEvents\n\tgithub.com/pingcap/tiflow/cdc/sink/mysql.go:232\nmain.(*Consumer).ConsumeClaim\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/main.go:551\ngithub.com/Shopify/sarama.(*consumerGroupSession).consume\n\tgithub.com/Shopify/[email protected]/consumer_group.go:778\ngithub.com/Shopify/sarama.newConsumerGroupSession.func2\n\tgithub.com/Shopify/[email protected]/consumer_group.go:703"]

it tooks that consumer may meet duplicate messages from topic.