Processes/Deployments are not imported

[Benjoyo]

I am running simple monitor 2.3.0 with Zeebe 1.4.0-alpha2 and deploy my processes using the Python API (which works, because instances show up and respond to commands).

Problem: the Processes tab and diagram visualization under Instances are broken for me.

I debugged this and the reason is that there are no ProcessRecords consumed at all (by the Hazelcast connector). I am not sure what this record is supposed to be, because I can't seem to find a corresponding RecordValue in io.camunda.zeebe.protocol.record.value. What I found however, and also seems more natural to me given the ProcessEntity's fields, is a DeploymentRecord which I also saw the Hazelcast connector consume when I deploy a process.. So as a test I implemented an Importer like so:

ZeebeHazelcast.newBuilder(hazelcast)
    .addDeploymentListener(record -> 
        ifEvent(record, Schema.DeploymentRecord::getMetadata, processAndElementImporter::importDeployment))

public void importDeployment(final Schema.DeploymentRecord record) {
    final int partitionId = record.getMetadata().getPartitionId();

    if (partitionId != Protocol.DEPLOYMENT_PARTITION) {
      // ignore process event on other partitions to avoid duplicates
      return;
    }

    final Intent intent = DeploymentIntent.valueOf(record.getMetadata().getIntent());
    if (intent != DeploymentIntent.CREATED) {
      return;
    }

    for (int i = 0; i < record.getProcessMetadataCount(); i++) {
      var processMetadata = record.getProcessMetadata(i);
      var resource = record.getResources(i);

      final ProcessEntity entity = new ProcessEntity();
      entity.setKey(processMetadata.getProcessDefinitionKey());
      entity.setBpmnProcessId(processMetadata.getBpmnProcessId());
      entity.setVersion(processMetadata.getVersion());
      entity.setResource(resource.getResource().toStringUtf8());
      entity.setTimestamp(record.getMetadata().getTimestamp());
      processRepository.save(entity);
    }
  }

... and everything works as expected.

Do you have an explanation for this? Is it my Zeebe version? Why do you use ProcessRecord as opposed to DeploymentRecord and where is the counterpart in Zeebe?

[tokarenko]

I can confirm that Simple monitor 2.3.0 fails to integrate with Camunda 8.2.0 and shows empty list for the Processes tab. Camunda operate works as expected and shows deployed processes. I would be grateful if you could fix this.

I installed Simple monitor 2.3.0 in Docker with Camunda 8.2.0 composed from docker-compose.yaml augmented with the following lines to activate Hazelcast:

services:
  zeebe:
    image:
    container_name: zeebe
    ports:
      - "5701:5701"
    volumes:
      - ./zeebe-hazelcast-exporter-1.2.1-jar-with-dependencies.jar:/usr/local/zeebe/exporters/zeebe-hazelcast-exporter-jar-with-dependencies.jar
      - ./application.yaml:/usr/local/zeebe/config/application.yaml

  simple-monitor:
    image: ghcr.io/camunda-community-hub/zeebe-simple-monitor:2.3.0
    environment:
      - zeebe.client.broker.gateway-address=zeebe:26500
      - zeebe.client.worker.hazelcast.connection=zeebe:5701
    ports:
      - "8082:8082"
    networks:
      - camunda-platform
      - identity-network
    depends_on:
      - zeebe
      - identity

Docker log from Simple monitor container is provided below:

$ docker-compose up simple-monitor
keycloak is up-to-date
elasticsearch is up-to-date
identity is up-to-date
zeebe is up-to-date
Creating camunda-platform-hazelcast_simple-monitor_1 ... done
Attaching to camunda-platform-hazelcast_simple-monitor_1
simple-monitor_1  | 11:50:19.914 [Thread-0] DEBUG org.springframework.boot.devtools.restart.classloader.RestartClassLoader - Created RestartClassLoader org.springframework.boot.devtools.restart.classloader.RestartClassLoader@506a38ab
simple-monitor_1  | 
simple-monitor_1  | ___                 __
simple-monitor_1  |  _/  _  _ |_   _   (_  .  _   _  |  _   |\/|  _   _  . |_  _   _
simple-monitor_1  | /__ (- (- |_) (-   __) | ||| |_) | (-   |  | (_) | ) | |_ (_) |
simple-monitor_1  |                              |
simple-monitor_1  | 
simple-monitor_1  | ================================================================
simple-monitor_1  |                                 
simple-monitor_1  | 
simple-monitor_1  | 2022-07-11 11:50:21.092  INFO 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Starting ZeebeSimpleMonitorApp using Java 11.0.13 on c58706e7a647 with PID 1 (/app/classes started by root in /)
simple-monitor_1  | 2022-07-11 11:50:21.094 DEBUG 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Running with Spring Boot v2.6.3, Spring v5.3.15
simple-monitor_1  | 2022-07-11 11:50:21.095  INFO 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : No active profile set, falling back to default profiles: default
simple-monitor_1  | 2022-07-11 11:50:30.254  INFO 1 --- [  restartedMain] i.z.monitor.zeebe.ZeebeHazelcastService  : Connecting to Hazelcast 'zeebe:5701'
simple-monitor_1  | WARNING: An illegal reflective access operation has occurred
simple-monitor_1  | WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/app/libs/hazelcast-5.0.2.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
simple-monitor_1  | WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
simple-monitor_1  | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
simple-monitor_1  | WARNING: All illegal access operations will be denied in a future release
simple-monitor_1  | 2022-07-11 11:50:31.599  INFO 1 --- [  restartedMain] i.z.monitor.zeebe.ZeebeHazelcastService  : Importing records from Hazelcast...
simple-monitor_1  | 2022-07-11 11:50:31.791  INFO 1 --- [  restartedMain] i.z.h.connect.java.ZeebeHazelcast        : Read from ringbuffer 'zeebe' starting from sequence '0'
simple-monitor_1  | 2022-07-11 11:50:34.382  INFO 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Started ZeebeSimpleMonitorApp in 14.422 seconds (JVM running for 20.902)

[vinkenfl]

Same problem for us. New process definitions are imported into ZSM database.

[qume]

@Benjoyo We are having the same problem, and I would like to implement the fix you came up with.

Are you able to do a pull request? Or perhaps give us some pointers here on how and where to deploy the code in your comment.

We use docker to run Zeebe+Hazelcast connector. I'm assuming that's where the code changes are that you made, rather than in the simple_monitor?

Thank you

[saig0]

Importing the process records is correct. There is no need to import the deployment records. Previously, the simple monitor imported the deployment records but that was changed by #250.

Zeebe writes (and exports) a new process record when a new BPMN process is deployed. But it doesn't write a process record if the process didn't change compared to the previous version.

---

If a try to deploy a process with simple monitor version 2.3.0 and Zeebe 8.0.2 with Hazelcast 1.2.1 then the process is not visible in the simple monitor. But I see the following error on the log:

2022-08-08 03:55:05.518 ERROR 1 --- [pool-1-thread-1] i.z.h.connect.java.ZeebeHazelcast        : Failed to deserialize Protobuf message at sequence '1'

com.google.protobuf.InvalidProtocolBufferException: Protocol message had invalid UTF-8.
	at com.google.protobuf.InvalidProtocolBufferException.invalidUtf8(InvalidProtocolBufferException.java:164) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.Utf8$DecodeUtil.handleTwoBytes(Utf8.java:1929) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.Utf8$DecodeUtil.access$700(Utf8.java:1903) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.Utf8$UnsafeProcessor.decodeUtf8(Utf8.java:1421) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.Utf8.decodeUtf8(Utf8.java:340) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.CodedInputStream$ArrayDecoder.readStringRequireUtf8(CodedInputStream.java:815) ~[protobuf-java-3.19.1.jar:na]
	at io.zeebe.exporter.proto.Schema$ProcessRecord.<init>(Schema.java:24438) ~[zeebe-exporter-protobuf-1.1.0.jar:1.1.0]
	at io.zeebe.exporter.proto.Schema$ProcessRecord$1.parsePartialFrom(Schema.java:25485) ~[zeebe-exporter-protobuf-1.1.0.jar:1.1.0]
	at io.zeebe.exporter.proto.Schema$ProcessRecord$1.parsePartialFrom(Schema.java:25479) ~[zeebe-exporter-protobuf-1.1.0.jar:1.1.0]
	at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:100) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:120) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:125) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48) ~[protobuf-java-3.19.1.jar:na]
	at com.google.protobuf.Any.unpack(Any.java:231) ~[protobuf-java-3.19.1.jar:na]
	at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.handleRecord(ZeebeHazelcast.java:188) ~[zeebe-hazelcast-connector-1.1.0.jar:1.1.0]
	at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.handleRecord(ZeebeHazelcast.java:177) ~[zeebe-hazelcast-connector-1.1.0.jar:1.1.0]
	at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.readNext(ZeebeHazelcast.java:117) ~[zeebe-hazelcast-connector-1.1.0.jar:1.1.0]
	at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.readFromBuffer(ZeebeHazelcast.java:106) ~[zeebe-hazelcast-connector-1.1.0.jar:1.1.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
	at java.base/java.lang.Thread.run(Unknown Source) ~[na:an]

The issue seems to be related to the Hazelcast exporter/connector.

If I change the Hazelcast exporter version to 1.2.0 then the process is imported successfully and I see it in the simple monitor.

I guess that I need to release a new version of the simple monitor with an updated Hazelcast importer. Until then, I recommend using the Hazelcast version 1.2.0.

[saig0]

I built a new release: 2.4.0 🏗️

With the new version, I can't reproduce the issue anymore 🎉
So, I'm closing the issue. Please reopen the issue or comment if you still encounter this issue.

[qume]

Using 2.4.0 (I got confused there - in your comment it says 1.4.0, but it links to 2.4.0 so hopefully that is correct) with zeebe-with-hazelcast-exporter:8.0.5 docker image, I now get the following error:

Error: Diagram rendering: Error: required args <xml=string>

No errors show up in the console of the app itself

[saig0]

@qume I tried to reproduce the issue but it's working for me.

I used the following docker-compose.yaml:

version: "3"

networks:
  zeebe_network:
    driver: bridge

services:
  zeebe:
    container_name: zeebe_broker
    image: ghcr.io/camunda-community-hub/zeebe-with-hazelcast-exporter:8.0.5
    environment:
      - ZEEBE_LOG_LEVEL=debug
    ports:
      - "26500:26500"
      - "9600:9600"
      - "5701:5701"
    networks:
      - zeebe_network

  simple-monitor-in-memory:
    container_name: zeebe-simple-monitor-in-memory
    image: ghcr.io/camunda-community-hub/zeebe-simple-monitor:2.4.0
    environment:
      - zeebe.client.broker.gateway-address=zeebe:26500
      - zeebe.client.worker.hazelcast.connection=zeebe:5701
    ports:
      - "8082:8082"
    depends_on:
      - zeebe
    networks:
      - zeebe_network
    profiles:
      - in-memory

[qume]

Thanks for the docker-compose.

I had the same issue after using it, so dug deeper and found the issue was related to my bpmn model. Other models run fine. The model which fails loads fine with no errors in Camunda Modeller.

Perhaps if there is an error like the one displayed, there is more info available which could be logged to the console? I'm not familiar with the BPMN renderer so unsure if it's the case

[saig0]

Perhaps if there is an error like the one displayed, there is more info available which could be logged to the console? I'm not familiar with the BPMN renderer so unsure if it's the case

Yes. Currently, the errors from the BPMN renderer are only logged in the browser console. It could be nice to show these errors on the page 😅