Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transition away from Java serialization for storing state on disk or in Zookeeper #497

Closed
wants to merge 13 commits into from

Commits on Jan 19, 2013

  1. Change StormTopology state serialization from Java to Thrift

    config and nimbus currently use the core Java serialization to store
    StormTopology instances. This commit will change this to use Thrift
    serialization instead. StormTopology is a Thrift struct so this
    basically involves finding all the places we call `Utils/serialize` and
    `Utils/deserialize` and replace them with a call to a method that
    serializes with Thrift instead.
    hausdorff committed Jan 19, 2013
    Configuration menu
    Copy the full SHA
    91a7bba View commit details
    Browse the repository at this point in the history
  2. Move serialization of Storm conf from Java default to Clojure default

    config.clj and nimbus serialize Storm configuration using the default
    Java implementation. This commit will phase out Java default
    serialization in favor of Clojure default serialization. There are
    obviously many reasons to phase out Java serialization, but the main
    rationale is that Java serialization will complain about version
    mismatch even if the change is semantically backward compatible.
    hausdorff committed Jan 19, 2013
    Configuration menu
    Copy the full SHA
    36de99e View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2013

  1. Fix serialization concurrency bug

    TSerializer is not threadsafe. In `Utils` we instantiate a static final
    Tserializer, but this can (and will) cause odd bugs if we start calling
    `serialize()` in different threads. Thus, every time we call
    `Utils/serializeTopology`, we create a new TSerializer.
    
    Another way to do this would be to lock it, which performance may or may
    not merit.
    hausdorff committed Jan 20, 2013
    Configuration menu
    Copy the full SHA
    527e886 View commit details
    Browse the repository at this point in the history
  2. Make serialization funcitons more generic

    Serialization of configuration is handled in config, but it is not
    different from a generic method for Clojure form serialization. This
    commit will move this method to utils so that we can use it for other
    things, like serialization in cluster.
    hausdorff committed Jan 20, 2013
    Configuration menu
    Copy the full SHA
    0656b60 View commit details
    Browse the repository at this point in the history
  3. Transition cluster state serialization to Clojure form serialization

    cluster.clj uses the stock Java serialization implementation. There are
    obviously many reason to not use standard Java serialization, but our
    main motivation is that Java will complain about serialized state
    when the versions don't match even if they're semantically backwards
    compatible.
    hausdorff committed Jan 20, 2013
    Configuration menu
    Copy the full SHA
    1b38e05 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2013

  1. Source LocalState serialization logic to state serialization interface

    Supervisor currently just uses stock jvm serialization to communicate
    LocalState. This is undesirable for many reasons, so this commit
    will introduce a serialization interface which makes code cleaner,
    letting us specify which type of serializer to use without populating
    Supervisor with unnecessary boxing/unboxing behavior, or LocalState
    with too much knowledge about what's happening in the Supervisor.
    
    Also this commit will introduce a basic implementation sketch for a
    serializer for LocalState (though it will just use the jvm serialization
    at this point).
    hausdorff committed Feb 22, 2013
    Configuration menu
    Copy the full SHA
    49c962f View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2013

  1. Source LocalState constants to Constants.java

    To build the serialization interface in Java, we need to put these
    constants in a Java file. This commit will put them in Constants.java
    hausdorff committed Feb 23, 2013
    Configuration menu
    Copy the full SHA
    ad36a5b View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2013

  1. Configuration menu
    Copy the full SHA
    73cd368 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2013

  1. Configuration menu
    Copy the full SHA
    b99656f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    8a2c9f0 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ff7a1b1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    2f840ef View commit details
    Browse the repository at this point in the history
  5. Merge branch 'master' of github.com:nathanmarz/storm into ser_dev

    Conflicts:
    	src/clj/backtype/storm/daemon/worker.clj
    	src/jvm/backtype/storm/Constants.java
    	src/jvm/backtype/storm/utils/LocalState.java
    hausdorff committed Mar 9, 2013
    Configuration menu
    Copy the full SHA
    6e9d8c1 View commit details
    Browse the repository at this point in the history