-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Transition away from Java serialization for storing state on disk or in Zookeeper #497
Commits on Jan 19, 2013
-
Change StormTopology state serialization from Java to Thrift
config and nimbus currently use the core Java serialization to store StormTopology instances. This commit will change this to use Thrift serialization instead. StormTopology is a Thrift struct so this basically involves finding all the places we call `Utils/serialize` and `Utils/deserialize` and replace them with a call to a method that serializes with Thrift instead.
Configuration menu - View commit details
-
Copy full SHA for 91a7bba - Browse repository at this point
Copy the full SHA 91a7bbaView commit details -
Move serialization of Storm conf from Java default to Clojure default
config.clj and nimbus serialize Storm configuration using the default Java implementation. This commit will phase out Java default serialization in favor of Clojure default serialization. There are obviously many reasons to phase out Java serialization, but the main rationale is that Java serialization will complain about version mismatch even if the change is semantically backward compatible.
Configuration menu - View commit details
-
Copy full SHA for 36de99e - Browse repository at this point
Copy the full SHA 36de99eView commit details
Commits on Jan 20, 2013
-
Fix serialization concurrency bug
TSerializer is not threadsafe. In `Utils` we instantiate a static final Tserializer, but this can (and will) cause odd bugs if we start calling `serialize()` in different threads. Thus, every time we call `Utils/serializeTopology`, we create a new TSerializer. Another way to do this would be to lock it, which performance may or may not merit.
Configuration menu - View commit details
-
Copy full SHA for 527e886 - Browse repository at this point
Copy the full SHA 527e886View commit details -
Make serialization funcitons more generic
Serialization of configuration is handled in config, but it is not different from a generic method for Clojure form serialization. This commit will move this method to utils so that we can use it for other things, like serialization in cluster.
Configuration menu - View commit details
-
Copy full SHA for 0656b60 - Browse repository at this point
Copy the full SHA 0656b60View commit details -
Transition cluster state serialization to Clojure form serialization
cluster.clj uses the stock Java serialization implementation. There are obviously many reason to not use standard Java serialization, but our main motivation is that Java will complain about serialized state when the versions don't match even if they're semantically backwards compatible.
Configuration menu - View commit details
-
Copy full SHA for 1b38e05 - Browse repository at this point
Copy the full SHA 1b38e05View commit details
Commits on Feb 22, 2013
-
Source LocalState serialization logic to state serialization interface
Supervisor currently just uses stock jvm serialization to communicate LocalState. This is undesirable for many reasons, so this commit will introduce a serialization interface which makes code cleaner, letting us specify which type of serializer to use without populating Supervisor with unnecessary boxing/unboxing behavior, or LocalState with too much knowledge about what's happening in the Supervisor. Also this commit will introduce a basic implementation sketch for a serializer for LocalState (though it will just use the jvm serialization at this point).
Configuration menu - View commit details
-
Copy full SHA for 49c962f - Browse repository at this point
Copy the full SHA 49c962fView commit details
Commits on Feb 23, 2013
-
Source LocalState constants to Constants.java
To build the serialization interface in Java, we need to put these constants in a Java file. This commit will put them in Constants.java
Configuration menu - View commit details
-
Copy full SHA for ad36a5b - Browse repository at this point
Copy the full SHA ad36a5bView commit details
Commits on Mar 4, 2013
-
Configuration menu - View commit details
-
Copy full SHA for 73cd368 - Browse repository at this point
Copy the full SHA 73cd368View commit details
Commits on Mar 9, 2013
-
Configuration menu - View commit details
-
Copy full SHA for b99656f - Browse repository at this point
Copy the full SHA b99656fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a2c9f0 - Browse repository at this point
Copy the full SHA 8a2c9f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for ff7a1b1 - Browse repository at this point
Copy the full SHA ff7a1b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f840ef - Browse repository at this point
Copy the full SHA 2f840efView commit details -
Merge branch 'master' of github.com:nathanmarz/storm into ser_dev
Conflicts: src/clj/backtype/storm/daemon/worker.clj src/jvm/backtype/storm/Constants.java src/jvm/backtype/storm/utils/LocalState.java
Configuration menu - View commit details
-
Copy full SHA for 6e9d8c1 - Browse repository at this point
Copy the full SHA 6e9d8c1View commit details