Make shell component serialisation plugable

CHANGELOG.md

@@ -1,3 +1,23 @@
+## 0.9.2-incubating (unreleased)
+ * STORM-173: Treat command line "-c" option number config values as such
+ * STORM-194: Support list of strings in *.worker.childopts, handle spaces
+ * STORM-288: Fixes version spelling in pom.xml
+ * STORM-208: Add storm-kafka as an external module
+ * STORM-285: Fix storm-core shade plugin config
+ * STORM-12: reduce thread usage of netty transport
+ * STORM-281: fix and issue with config parsing that could lead to leaking file descriptors
+ * STORM-196: When JVM_OPTS are set, storm jar fails to detect storm.jar from environment
+ * STORM-260: Fix a potential race condition with simulated time in Storm's unit tests
+ * STORM-258: Update commons-io version to 2.4
+ * STORM-270: don't package .clj files in release jars.
+ * STORM-273: Error while running storm topologies on Windows using "storm jar"
+ * STROM-247: Replace links to github resources in storm script
+ * STORM-263: Update Kryo version to 2.21+
+ * STORM-187: Fix Netty error "java.lang.IllegalArgumentException: timeout value is negative"
+ * STORM-186: fix float secs to millis long convertion
+ * STORM-70: Upgrade to ZK-3.4.5 and curator-1.3.3
+ * STORM-146: Unit test regression when storm is compiled with 3.4.5 zookeeper
+
 ## 0.9.1-incubating
 * Fix to prevent Nimbus from hanging if random data is sent to nimbus thrift port
 * Improved support for running on Windows platforms

NOTICE

@@ -1,13 +1,8 @@
 Apache Storm
-Copyright 2013 The Apache Software Foundation
+Copyright 2014 The Apache Software Foundation
 
 This product includes software developed at
 The Apache Software Foundation (http://www.apache.org/).
 
-
-This product includes software developed by Nathan Marz
-Copyright 2011-2013 Nathan Marz
-
-
 This product includes software developed by Yahoo! Inc. (www.yahoo.com)
-Copyright © 2012-2013 Yahoo! Inc.  All rights reserved.
+Copyright (c) 2012-2014 Yahoo! Inc.

README.markdown

@@ -71,6 +71,9 @@ under the License.
 * Flip Kromer ([@mrflip](https://github.com/mrflip))
 * David Lao ([@davidlao2k](https://github.com/davidlao2k))
 * P. Taylor Goetz ([@ptgoetz](https://github.com/ptgoetz))
+* Derek Dagit ([@d2r](https://github.com/d2r))
+* Robert Evans ([@revans2](https://github.com/revans2)) 
+
 
 ## Contributors
 
@@ -105,9 +108,7 @@ under the License.
 * Bryan Peterson ([@Lazyshot](https://github.com/Lazyshot))
 * Sam Ritchie ([@sritchie](https://github.com/sritchie))
 * Stuart Anderson ([@emblem](https://github.com/emblem))
-* Robert Evans ([@revans2](https://github.com/revans2)) 
 * Lorcan Coyle ([@lorcan](https://github.com/lorcan))
-* Derek Dagit ([@d2r](https://github.com/d2r))
 * Andrew Olson ([@noslowerdna](https://github.com/noslowerdna))
 * Gavin Li ([@lyogavin](https://github.com/lyogavin))
 * Tudor Scurtu ([@tscurtu](https://github.com/tscurtu))

bin/storm

@@ -22,6 +22,7 @@ import random
 import subprocess as sub
 import getopt
 import re
+import shlex
 
 def identity(x):
     return x
@@ -45,7 +46,7 @@ if (not os.path.isfile(USER_CONF_DIR + "/storm.yaml")):
     USER_CONF_DIR = CLUSTER_CONF_DIR
 CONFIG_OPTS = []
 CONFFILE = ""
-JAR_JVM_OPTS = os.getenv('STORM_JAR_JVM_OPTS', '')
+JAR_JVM_OPTS = shlex.split(os.getenv('STORM_JAR_JVM_OPTS', ''))
 
 
 def get_config_opts():
@@ -147,15 +148,15 @@ def jar(jarfile, klass, *args):
     Runs the main method of class with the specified arguments. 
     The storm jars and configs in ~/.storm are put on the classpath. 
     The process is configured so that StormSubmitter 
-    (http://nathanmarz.github.com/storm/doc/backtype/storm/StormSubmitter.html)
+    (http://storm.incubator.apache.org/apidocs/backtype/storm/StormSubmitter.html)
     will upload the jar at topology-jar-path when the topology is submitted.
     """
     exec_storm_class(
         klass,
         jvmtype="-client",
         extrajars=[jarfile, USER_CONF_DIR, STORM_DIR + "/bin"],
         args=args,
-        jvmopts=[' '.join(filter(None, [JAR_JVM_OPTS, "-Dstorm.jar=" + jarfile]))])
+        jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile])
 
 def kill(*args):
     """Syntax: [storm kill topology-name [-w wait-time-secs]]
@@ -262,7 +263,7 @@ def nimbus(klass="backtype.storm.daemon.nimbus"):
     supervision with a tool like daemontools or monit. 
 
     See Setting up a Storm cluster for more information.
-    (https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
+    (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
     """
     cppaths = [CLUSTER_CONF_DIR]
     jvmopts = parse_args(confvalue("nimbus.childopts", cppaths)) + [
@@ -282,7 +283,7 @@ def supervisor(klass="backtype.storm.daemon.supervisor"):
     under supervision with a tool like daemontools or monit. 
 
     See Setting up a Storm cluster for more information.
-    (https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
+    (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
     """
     cppaths = [CLUSTER_CONF_DIR]
     jvmopts = parse_args(confvalue("supervisor.childopts", cppaths)) + [
@@ -303,7 +304,7 @@ def ui():
     should be run under supervision with a tool like daemontools or monit. 
 
     See Setting up a Storm cluster for more information.
-    (https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
+    (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
     """
     cppaths = [CLUSTER_CONF_DIR]
     jvmopts = parse_args(confvalue("ui.childopts", cppaths)) + [
@@ -324,7 +325,7 @@ def logviewer():
     tool like daemontools or monit. 
 
     See Setting up a Storm cluster for more information.
-    (https://github.com/nathanmarz/storm/wiki/Setting-up-a-Storm-cluster)
+    (http://storm.incubator.apache.org/documentation/Setting-up-a-Storm-cluster)
     """
     cppaths = [CLUSTER_CONF_DIR]
     jvmopts = parse_args(confvalue("logviewer.childopts", cppaths)) + [
@@ -344,7 +345,7 @@ def drpc():
     with a tool like daemontools or monit. 
 
     See Distributed RPC for more information.
-    (https://github.com/nathanmarz/storm/wiki/Distributed-RPC)
+    (http://storm.incubator.apache.org/documentation/Distributed-RPC)
     """
     cppaths = [CLUSTER_CONF_DIR]
     jvmopts = parse_args(confvalue("drpc.childopts", cppaths)) + [
@@ -392,7 +393,7 @@ def print_commands():
     """Print all client commands and link to documentation"""
     print "Commands:\n\t",  "\n\t".join(sorted(COMMANDS.keys()))
     print "\nHelp:", "\n\thelp", "\n\thelp <command>"
-    print "\nDocumentation for the storm client can be found at https://github.com/nathanmarz/storm/wiki/Command-line-client\n"
+    print "\nDocumentation for the storm client can be found at http://storm.incubator.apache.org/documentation/Command-line-client.html\n"
     print "Configs can be overridden using one or more -c flags, e.g. \"storm list -c nimbus.host=nimbus.mycompany.com\"\n"
 
 def print_usage(command=None):

conf/defaults.yaml

@@ -35,6 +35,7 @@ storm.zookeeper.retry.intervalceiling.millis: 30000
 storm.cluster.mode: "distributed" # can be distributed or local
 storm.local.mode.zmq: false
 storm.thrift.transport: "backtype.storm.security.auth.SimpleTransportPlugin"
+storm.multilang.serializer: "backtype.storm.multilang.JsonSerializer"
 storm.messaging.transport: "backtype.storm.messaging.netty.Context"
 
 ### nimbus.* configs are for the master

examples/storm-starter/README.markdown

@@ -0,0 +1,108 @@
+# Example Storm Topologies
+
+Learn to use Storm!
+
+---
+
+Table of Contents
+
+* <a href="#getting-started">Getting started</a>
+* <a href="#maven">Using storm-starter with Maven</a>
+* <a href="#intellij-idea">Using storm-starter with IntelliJ IDEA</a>
+
+---
+
+
+<a name="getting-started"></a>
+
+# Getting started
+
+## Prerequisites
+
+First, you need `java` and `git` installed and in your user's `PATH`.  Also, two of the examples in storm-starter
+require Python and Ruby.
+
+Next, make sure you have the storm-starter code available on your machine.  Git/GitHub beginners may want to use the
+following command to download the latest storm-starter code and change to the new directory that contains the downloaded
+code.
+
+    $ git clone git://github.com/apache/incubator-storm.git && cd incubator-storm/examples/storm-starter
+
+
+## storm-starter overview
+
+storm-starter contains a variety of examples of using Storm.  If this is your first time working with Storm, check out
+these topologies first:
+
+1. [ExclamationTopology](src/jvm/storm/starter/ExclamationTopology.java):  Basic topology written in all Java
+2. [WordCountTopology](src/jvm/storm/starter/WordCountTopology.java):  Basic topology that makes use of multilang by
+   implementing one bolt in Python
+3. [ReachTopology](src/jvm/storm/starter/ReachTopology.java): Example of complex DRPC on top of Storm
+
+After you have familiarized yourself with these topologies, take a look at the other topopologies in
+[src/jvm/storm/starter/](src/jvm/storm/starter/) such as [RollingTopWords](src/jvm/storm/starter/RollingTopWords.java)
+for more advanced implementations.
+
+If you want to learn more about how Storm works, please head over to the
+[Storm project page](http://storm.incubator.apache.org).
+
+
+<a name="maven"></a>
+
+# Using storm-starter with Maven
+
+## Install Maven
+
+Install [Maven](http://maven.apache.org/) (preferably version 3.x) by following
+the [Maven installation instructions](http://maven.apache.org/download.cgi).
+
+
+## Running topologies with Maven
+
+storm-starter topologies can be run with the maven-exec-plugin. For example, to
+compile and run `WordCountTopology` in local mode, use the command:
+
+    $ mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology
+
+You can also run clojure topologies with Maven:
+
+    $ mvn compile exec:java -Dstorm.topology=storm.starter.clj.word_count
+
+## Packaging storm-starter for use on a Storm cluster
+
+You can package a jar suitable for submitting to a Storm cluster with the command:
+
+    $ mvn package
+
+This will package your code and all the non-Storm dependencies into a single "uberjar" at the path
+`target/storm-starter-{version}-jar-with-dependencies.jar`.
+
+
+## Running unit tests
+
+Use the following Maven command to run the unit tests that ship with storm-starter.  Unfortunately `lein test` does not
+yet run the included unit tests.
+
+    $ mvn test
+
+
+<a name="intellij-idea"></a>
+
+# Using storm-starter with IntelliJ IDEA
+
+## Importing storm-starter as a project in IDEA
+
+The following instructions will import storm-starter as a new project in IntelliJ IDEA.
+
+
+* Open _File > Import Project..._ and navigate to the storm-starter directory of your storm clone (e.g.
+  `~/git/incubator-storm/examples/storm-starter`).
+* Select _Import project from external model_, select "Maven", and click _Next_.
+* In the following screen, enable the checkbox _Import Maven projects automatically_.  Leave all other values at their
+  defaults.  Click _Next_.
+* Click _Next_ on the following screen about selecting Maven projects to import.
+* Select the JDK to be used by IDEA for storm-starter, then click _Next_.
+    * At the time of this writing you should use JDK 6.
+    * It is strongly recommended to use Sun/Oracle JDK 6 rather than OpenJDK 6.
+* You may now optionally change the name of the project in IDEA.  The default name suggested by IDEA is "storm-starter".
+  Click _Finish_ once you are done.

examples/storm-starter/multilang/resources/splitsentence.py

@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import storm
+
+class SplitSentenceBolt(storm.BasicBolt):
+    def process(self, tup):
+        words = tup.values[0].split(" ")
+        for word in words:
+          storm.emit([word])
+
+SplitSentenceBolt().run()

examples/storm-starter/multilang/resources/splitsentence.rb

@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+require "./storm"
+
+class SplitSentenceBolt < Storm::Bolt
+  def process(tup)
+    tup.values[0].split(" ").each do |word|
+      emit([word])
+    end
+  end
+end
+
+SplitSentenceBolt.new.run