UninitializedMessageException when repairing cluster

[TrevorBasinger]

I sent a POST request to /cluster/repair/start and my cassandra.dcos is continuously failing to start.

Here is the relevant portion of the stderr:

1927 sched.cpp:448] Framework registered with 20150625-151318-3212775596-5050-2156-0001
Exception in thread "Thread-2" com.google.protobuf.UninitializedMessageException: Message missing required fields: jobType, startedTimestamp
    at com.google.protobuf.AbstractMessage$Builder.newUninitializedMessageException(AbstractMessage.java:770)
    at io.mesosphere.mesos.frameworks.cassandra.CassandraFrameworkProtos$ClusterJobStatus$Builder.build(CassandraFrameworkProtos.java:11456)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.NodeTaskClusterJobHandler.rejectNode(NodeTaskClusterJobHandler.java:112)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.NodeTaskClusterJobHandler.handleTaskOffer(NodeTaskClusterJobHandler.java:78)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.CassandraCluster.handleClusterTask(CassandraCluster.java:689)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.CassandraCluster._getTasksForOffer(CassandraCluster.java:1154)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.CassandraCluster.getTasksForOffer(CassandraCluster.java:342)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.CassandraScheduler.evaluateOffer(CassandraScheduler.java:270)
    at io.mesosphere.mesos.frameworks.cassandra.scheduler.CassandraScheduler.resourceOffers(CassandraScheduler.java:93)

I haven't grokked the whole project, but if I had to guess, I think the problem might be here

I think the builder might need a jobType and startedTimestamp set the same way it was set here

I hope this is somewhat helpful. I'm still trying to figure out how to fix my cassandra.dcos service without uninstalling it all together. Please advise.

[BenWhitehead]

Thanks for the report @TrevorBasinger those two links you've provided look like good guesses. I'll try and spend some time putting a fix together.

In the mean time you can try running nodetool manually on each of the hosts that is running cassandra with the following command:

docker run -i -t --net=host cassandra:2.1.5 nodetool repair

[TrevorBasinger]

I appreciate you getting back with me. We've decided to run cassandra outside of DCOS for the time being. Thanks for your time!

[TrevorBasinger]

@BenWhitehead Wanted to let you know we're running cassandra inside mesosphere again. I'm willing to help test once you've got a fix ready.

[BenWhitehead]

great to hear, I hope to have a fix in soon.

[BenWhitehead]

@TrevorBasinger I just merged a change (in #119) that should help take care of the issue that is happening.

I managed to only get the exception once in my pre-fix testing, but didn't get it any times after applying the fix.

This will go out in the next release of the framework.