Release 6.5.2 (#303)

* Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2
splunk · Oct 8, 2019 · 8f9a1c5 · 8f9a1c5
1 parent 0a9c727
commit 8f9a1c5
Show file tree

Hide file tree

Showing 23 changed files with 296 additions and 180 deletions.
diff --git a/docs/CONFIGURE.md b/docs/CONFIGURE.md
@@ -382,8 +382,10 @@ specifically be supported by all plugins. Plugins that write to files like spool
     * Set event sourcetype in Splunk to <source> Defaults to 'eventgen' if none specified.
 
     host = <host>
-    * ONLY VALID WITH outputMode SPLUNKSTREAM
-    * Set event host in Splunk to <host>. Defaults to 127.0.0.1 if none specified.
+    * When outputMode is splunkstream, set event host in Splunk to <host>.
+    * When outputMode is syslogout and syslogAddHeader is set to true, add initial header with hostname <host>,
+      see syslogAddHeader for details.
+    * Defaults to 127.0.0.1 if none specified.
 
     host.token = <regular expression>
     * PCRE expression used to identify the host name (or partial name) for replacement.
@@ -420,6 +422,14 @@ specifically be supported by all plugins. Plugins that write to files like spool
     * Only supports UDP ports
     * Required
 
+    syslogAddHeader = true | false
+    * Controls whether syslog messages should be prefixed with an RFC3164 compliant header
+      including the host value defined for the sample.
+    * Useful in situations where you want to output generated events to syslog and make it
+      possible for the receiving syslog server to use the sample's defined host value instead of
+      the hostname of the host that eventgen is running on.
+    * Defaults to false
+
 ###### tcpout
     tcpDestinationHost = <host>
     * Defaults to 127.0.0.1

diff --git a/docs/CONTRIBUTE_CODE.md b/docs/CONTRIBUTE_CODE.md
@@ -5,18 +5,26 @@ If you want to contribute code to eventgen, please read over the following guide
 
 ## Pull request guidelines
 
-
 If you want to contribute to an eventgen repo, please use a GitHub pull request. This is the fastest way for us to evaluate your code and to merge it into the code base. Please don’t file an issue with snippets of code. Doing so means that we need to manually merge the changes in and update any appropriate tests. That decreases the likelihood that your code is going to get included in a timely manner. Please use pull requests.
 
+
+## Release versioning guidelines
+
+Major Release — Increment the first digit by 1 if the new features break backwards compatibility/current features
+
+Minor Release — Increment the middle digit by 1 if the new features don’t break any existing features and are compatible with the app in it’s current state
+
+Patch Release — Increment the last digit by 1 if you’re publishing bug/patch fixes to your app
+
 ### Get started
 
 If you’d like to work on a pull request and you’ve never submitted code before, follow these steps:
 1. fork eventgen to your github workspace
 2. If you want to fix bugs or make enhancement, please make sure there is a issue in eventgen project. Refer [this guide](FILE_ISSUES.md) to create a issue.
 
-
 After that, you’re ready to start working on code.
 
+
 ### Working on the code
 
 The process of submitting a pull request is fairly straightforward and generally follows the same pattern each time:
@@ -75,6 +83,7 @@ The message summary should be a one-sentence description of the change, and it m
 
 **Note**: please squash you changes in one commit before firing the pull request. One commit in one PR keeps the git history clean.
 
+
 #### Step 3: Rebase onto upstream
 
 Before you send the pull request, be sure to rebase onto the upstream source. This ensures your code is running on the latest available code. We prefer rebase instead of merge when upstream changes. Rebase keeps the git history clearer.
@@ -83,6 +92,7 @@ git fetch upstream
 git rebase upstream/master
 ```
 
+
 #### Step 4: Run the tests
 
 The is a place holder as well. We should write about 
@@ -101,6 +111,7 @@ Next, push your changes to your clone:
 git push origin fix/issue123
 ```
 
+
 #### Step 6: Submit the pull request
 
 Before creating a pull request, here are some recommended **check points**. 
@@ -118,7 +129,6 @@ Next, create a pull request from your branch to the eventgen develop branch.
 Mark @lephino , @arctan5x , @jmeixensperger , @li-wu , @GordonWang as the reviewers.
 
 
-
 ## Code style and formatting tools
 
 Since Eventgen is written in python, we apply a coding style based on [PEP8](https://www.python.org/dev/peps/pep-0008/).

diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock
@@ -1,12 +1,12 @@
 GEM
   remote: https://rubygems.org/
   specs:
-    activesupport (4.2.10)
+    activesupport (4.2.11.1)
       i18n (~> 0.7)
       minitest (~> 5.1)
       thread_safe (~> 0.3, >= 0.3.4)
       tzinfo (~> 1.1)
-    addressable (2.5.2)
+    addressable (2.6.0)
       public_suffix (>= 2.0.2, < 4.0)
     coffee-script (2.4.1)
       coffee-script-source
@@ -16,7 +16,7 @@ GEM
     commonmarker (0.17.13)
       ruby-enum (~> 0.5)
     concurrent-ruby (1.1.5)
-    dnsruby (1.61.2)
+    dnsruby (1.61.3)
       addressable (~> 2.5)
     em-websocket (0.5.1)
       eventmachine (>= 0.12.9)
@@ -27,13 +27,13 @@ GEM
     execjs (2.7.0)
     faraday (0.15.4)
       multipart-post (>= 1.2, < 3)
-    ffi (1.10.0)
+    ffi (1.11.1)
     forwardable-extended (2.6.0)
-    gemoji (3.0.0)
-    github-pages (197)
-      activesupport (= 4.2.10)
+    gemoji (3.0.1)
+    github-pages (198)
+      activesupport (= 4.2.11.1)
       github-pages-health-check (= 1.16.1)
-      jekyll (= 3.7.4)
+      jekyll (= 3.8.5)
       jekyll-avatar (= 0.6.0)
       jekyll-coffeescript (= 1.1.1)
       jekyll-commonmark-ghpages (= 0.1.5)
@@ -81,13 +81,13 @@ GEM
       octokit (~> 4.0)
       public_suffix (~> 3.0)
       typhoeus (~> 1.3)
-    html-pipeline (2.10.0)
+    html-pipeline (2.12.0)
       activesupport (>= 2)
       nokogiri (>= 1.4)
     http_parser.rb (0.6.0)
     i18n (0.9.5)
       concurrent-ruby (~> 1.0)
-    jekyll (3.7.4)
+    jekyll (3.8.5)
       addressable (~> 2.4)
       colorator (~> 1.0)
       em-websocket (~> 0.5)
@@ -204,39 +204,39 @@ GEM
       jekyll-feed (~> 0.9)
       jekyll-seo-tag (~> 2.1)
     minitest (5.11.3)
-    multipart-post (2.0.0)
-    nokogiri (1.10.2)
+    multipart-post (2.1.1)
+    nokogiri (1.10.4)
       mini_portile2 (~> 2.4.0)
     octokit (4.14.0)
       sawyer (~> 0.8.0, >= 0.5.3)
     pathutil (0.16.2)
       forwardable-extended (~> 2.6)
-    public_suffix (3.0.3)
+    public_suffix (3.1.1)
     rb-fsevent (0.10.3)
     rb-inotify (0.10.0)
       ffi (~> 1.0)
     rouge (2.2.1)
     ruby-enum (0.7.2)
       i18n
     ruby_dep (1.5.0)
-    rubyzip (1.2.2)
+    rubyzip (1.2.3)
     safe_yaml (1.0.5)
-    sass (3.7.3)
+    sass (3.7.4)
       sass-listen (~> 4.0.0)
     sass-listen (4.0.0)
       rb-fsevent (~> 0.9, >= 0.9.4)
       rb-inotify (~> 0.9, >= 0.9.7)
-    sawyer (0.8.1)
-      addressable (>= 2.3.5, < 2.6)
-      faraday (~> 0.8, < 1.0)
+    sawyer (0.8.2)
+      addressable (>= 2.3.5)
+      faraday (> 0.8, < 2.0)
     terminal-table (1.8.0)
       unicode-display_width (~> 1.1, >= 1.1.1)
     thread_safe (0.3.6)
     typhoeus (1.3.1)
       ethon (>= 0.9.0)
     tzinfo (1.2.5)
       thread_safe (~> 0.1)
-    unicode-display_width (1.5.0)
+    unicode-display_width (1.6.0)
 
 PLATFORMS
   ruby
@@ -245,4 +245,4 @@ DEPENDENCIES
   github-pages
 
 BUNDLED WITH
-   2.0.1
+   2.0.2
diff --git a/docs/PLUGINS.md b/docs/PLUGINS.md
@@ -16,29 +16,37 @@ Plugins inherit from a base plugin class and are placed in their appropriate dir
 Let's take a look at the simplest plugin available to us, the Devnull output plugin:
 
 ```python
-from __future__ import division
 from outputplugin import OutputPlugin
-import sys
+from logging_config import logger
+
 
 class DevNullOutputPlugin(OutputPlugin):
+    name = 'devnull'
     MAXQUEUELENGTH = 1000
+    useOutputQueue = True
 
-    def __init__(self, sample):
-        OutputPlugin.__init__(self, sample)
+    def __init__(self, sample, output_counter=None):
+        OutputPlugin.__init__(self, sample, output_counter)
         self.firsttime = True
 
     def flush(self, q):
+        logger.info('flush data to devnull')
         if self.firsttime:
             self.f = open('/dev/null', 'w')
+            self.firsttime = False
         buf = '\n'.join(x['_raw'].rstrip() for x in q)
         self.f.write(buf)
 
+
 def load():
     """Returns an instance of the plugin"""
     return DevNullOutputPlugin
+
 ```
 
-First, we import the OutputPlugin superclass. For output plugins, they define a constant MAXQUEUELENGTH to determine the maximum amount of items in queue before forcing a queue flush.
+First, we import the OutputPlugin superclass. For output plugins, they define a constant `MAXQUEUELENGTH` to determine the maximum amount of items in queue before forcing a queue flush.
+
+`useOutputQueue` is set to `True` here to use the output queue which functions as a reduce step when you need to maintain a single thread or a limited number of threads outputting data
 
 ``__init__()`` is very simple. It calls its superclass init and sets one variable, firsttime. ``flush()`` is also very simple.
 If it's the first time, open the file /dev/null, otherwise, output the queue by writing it to the already open file.
@@ -56,34 +64,34 @@ class SplunkStreamOutputPlugin(OutputPlugin):
     intSettings = [ 'splunkPort' ]
 ```
 
-MAXQUEUELENGTH should look normal, but these other class variables need a little explanation.
+`MAXQUEUELENGTH` should look normal, but these other class variables need a little explanation.
 
 ### Configuration Validation
 Config validation is a modular system in Eventgen, and plugins must be allowed to specify additional configuration parameters that the main Eventgen will consider valid and store.
-*Note that eventgen.conf.spec generation is not yet automated, which means plugins must ship with the default distribution and eventgen.conf.spec must be maintained manually.*
+> Note that `eventgen.conf.spec` generation is not yet automated, which means plugins must ship with the default distribution and eventgen.conf.spec must be maintained manually.
 Eventually spec file generation will be automated as well.
 
 The main configuration of Eventgen validates itself by a list of configuration parameters assigned by type, and each of the configuration parameters is validated by that type.
 The settings list is required:
 
-* validSettings 				|  Defines the list of valid settings for this plugin
+* validSettings: Defines the list of valid settings for this plugin
 
 The following lists are optional and likely to be used by many plugins:
 
-* intSettings			|   Will validate the settings as integers
-* floatSettings			|   Will validate the settings as floating point numbers
-* boolSettings			|   Will validate the settings as booleans
-* jsonSettings			|   Will validate the settings as a JSON string
-* defaultableSettings	|   Settings which can be specified in the [global] stanza and will pass down to individual stanzas
-* complexSettings       |   A dictionary of lists or function callbacks, containing a setting name with list of valid options or a callback function to validate the setting.
+* intSettings: Will validate the settings as integers
+* floatSettings: Will validate the settings as floating point numbers
+* boolSettings: Will validate the settings as booleans
+* jsonSettings: Will validate the settings as a JSON string
+* defaultableSettings: Settings which can be specified in the [global] stanza and will pass down to individual stanzas
+* complexSettings: A dictionary of lists or function callbacks, containing a setting name with list of valid options or a callback function to validate the setting.
 
 ## Methods required per plugin type
 
 Each plugin type will define a different method required.
 
 **Plugin Type** | **Method** | **Returns** | **Notes**
 --- | --- | --- | ---
-Rater | ``rate()`` | Integer count of events to generate | n/a
+Rater | ``rate()`` | Integer count of events to generate | N/A
 Generator | ``gen(count, earliest, latest) `` | Success (0) | Events get put into an output queue by calling the Sample's ``send()`` or ``bulksend()`` methods in the output object.
 Output | ``flush(q)`` | Success (0) | Gets a deque list q to operate upon and output as configured.
 
@@ -92,48 +100,46 @@ Output | ``flush(q)`` | Success (0) | Gets a deque list q to operate upon and ou
 We reviewed a simple Output Plugin earlier, let's look at a simple Generator Plugin:
 
 ```python
-from __future__ import division
+import datetime
+from datetime import timedelta
+
 from generatorplugin import GeneratorPlugin
-import os
-import logging
-import datetime, time
-import itertools
-from collections import deque
+from logging_config import logger
+
 
 class WindbagGenerator(GeneratorPlugin):
     def __init__(self, sample):
         GeneratorPlugin.__init__(self, sample)
 
-        # Logger already setup by config, just get an instance
-        logger = logging.getLogger('eventgen')
-        globals()['logger'] = logger
-
-        from eventgenconfig import Config
-        globals()['c'] = Config()
-
-    def gen(self, count, earliest, latest):
-        l = [ {'_raw': '2014-01-05 23:07:08 WINDBAG Event 1 of 100000'} for i in xrange(count) ]
-
-        self._out.bulksend(l)
+    def gen(self, count, earliest, latest, samplename=None):
+        if count < 0:
+            logger.warning('Sample size not found for count=-1 and generator=windbag, defaulting to count=60')
+            count = 60
+        time_interval = timedelta.total_seconds((latest - earliest)) / count
+        for i in xrange(count):
+            current_time_object = earliest + datetime.timedelta(0, time_interval * (i + 1))
+            msg = '{0} -0700 WINDBAG Event {1} of {2}'.format(current_time_object, (i + 1), count)
+            self._out.send(msg)
         return 0
 
+
 def load():
     return WindbagGenerator
+
 ```
 
-For this generator plugin, notice we inherit from GeneratorPlugin instead of OutputPlugin. This plugin is also quite simple.
-In its ``__init__()`` method, it calls the superclass ``__init__()`` and it sets up two global variables, c, which holds the config
-(and is a Singleton pattern which can be instantiated many times) and a copy of the logger which we'll use for logging in most plugins.
+For this generator plugin, notice we inherit from `GeneratorPlugin` instead of `OutputPlugin`. This plugin is also quite simple.
 
-Secondly, it defines a gen() method, which generates ``count`` events between ``earliest`` and ``latest`` time. In this case, we ignore the timestamp and return just event text.
-Then we call bulksend. This plugin has several performance optimizations: using a list constructor instead of a loop and using bulksend instead of send.
+Secondly, it defines a `gen()` method, which generates ``count`` events between ``earliest`` and ``latest`` time. In this case, we ignore the timestamp and return just event text.
+Then we call `bulksend`. This plugin has several performance optimizations: using a list constructor instead of a loop and using bulksend instead of send.
 Let's see how this could be implemented in a slightly less performant but easier to understand way:
 
 ```python
-    def gen(self, count, earliest, latest):
-        for x in xrange(count):
-            self._sample.send({ '_raw': '2014-01-05 23:07:08 WINDBAG Event 1 of 100000' })
-
+    def gen(self, count, earliest, latest, samplename=None):
+        for i in xrange(count):
+            current_time_object = earliest + datetime.timedelta(0, time_interval * (i + 1))
+            msg = '{0} -0700 WINDBAG Event {1} of {2}'.format(current_time_object, (i + 1), count)
+            self._out.send(msg)
         return 0
 ```
 
@@ -142,4 +148,4 @@ Here, we use ``send()`` instead of ``bulksend()`` and a loop to make it easier t
 # Shipping a Plugin
 
 When you've developed a plugin that you want to use in your app, shipping it with your app is easy.
-Place any Eventgen plugin in your Splunk app's ``bin/`` directory and we'll search for and find any plugins referenced by a ``outputMode``, ``generator`` or ``rater`` config statement.
+Place any Eventgen plugin in your Splunk app's ``bin/`` directory and we'll search for and find any plugins referenced by a ``outputMode``, ``generator`` or ``rater`` config statement.
diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md
@@ -139,6 +139,9 @@ syslogDestinationPort = <port>
     * Defaults to port 1514
     * Only supports UDP ports
 
+syslogAddHeader = true | false
+    * Defaults to false
+
 tcpDestinationHost = <host>
     * Defaults to 127.0.0.1
 

diff --git a/splunk_eventgen/eventgen_api_server/eventgen_controller_api.py b/splunk_eventgen/eventgen_api_server/eventgen_controller_api.py
@@ -35,7 +35,7 @@ def gather_response(response_number_target=0):
             if not response_number_target:
                 response_number_target = int(self.redis_connector.message_connection.pubsub_numsub(self.redis_connector.servers_channel)[0][1])
             response_num = 0
-            countdown = 1.5 / self.interval
+            countdown = 60 / self.interval
             for i in range(0, int(countdown)):
                 if response_num == response_number_target:
                     break