Server revamp (#266)

* initial file structure

* Added main

* Hello world

* Initial core object

* Initialized blueprint

* index, conf, status working

* get and post volume working

* adding mode var

* Stop, Start, and Restart endpoints working

* Reset endpoint

* wip setup rabbitmq

getting error creating vhost

* Working bundle with zip and basic clean up

* Setup working with error logging

* Added threaded flask execution

* PerDayVolume bug fix and doc update

* Reset bug fix and document update

* Added logging to bundle

* Adding a forcible stop

* wip using http instead of rabbitmq

* Dockerfile fix

* wip getting register and status working

* wip tries to fix some stuff, tries to get logs working

* Added controller logging

* wip getting registering working with docker

* Dockerfile fix for stdout

* finishes functionality

* comments code for local running

* Put conf working

* fixes stop with no body failing

* cleans some stuff up

* fixes error messages

* delete more prints

* Test fix

* Clean up

* fixes missing import

* fixes stop not having optional body

* Redis queue with controller rewrite

* Retry logic for redis calls

* Removing egbundle files

* mode fix

* Small test fix

* Changing test check

* BundleFix

* Setup and PUT conf fix

* Adding sudo

* SSH support

* SSH fix

* processing

* multiprocess fix

* Multiprocess error case fixing

* regex stanza volume fix

* Global index fix

* Introducing tail

* dependency upgrade

* fix sftp location for alpine image

* Removing constants

* Refactor

* doc update

* self forcing fix

* Process issue

* outputMode fix

* Doc update

* Fixing workerpool loop

* Error catching forcible stop errors

* Increasing generator queue size

* Test fix from Lynch

* import error

* Test fix

* Test fix

* dependency clean up

* Fixing moar tests

* CircleCI error reporting fix

* Test case test_server_stop fixed

* Restart fix

* docs typo

* docs typo

* Test scenario fix

* last test to fix

* Bring back the docker-compose install

* Fix bug #275

* blueprint class cleanup

* duplicate code removal

* Removing unused import

* multiprocess args fix

.circleci/config.yml

@@ -16,6 +16,7 @@ jobs:
       - run:
           name: Run Tests
           command: |
+            set -e
             make test
           no_output_timeout: 30m
       - store_test_results:

.gitignore

@@ -27,6 +27,7 @@ dist
 _book
 *.result
 venv/*
+eventgenEnv/*
 *.log.*
 splunk_eventgen-*/
 .env

LICENSE

@@ -220,7 +220,6 @@ The following components are provided under the Apache License 2.0. See project
     (Apache License 2.0) pyOpenSSL (https://github.com/pyca/pyopenssl/blob/master/LICENSE)
     (Apache License 2.0) docker (https://github.com/docker/docker-py/blob/master/LICENSE)
     (Apache License 2.0) requests-futures (https://github.com/ross/requests-futures/blob/master/LICENSE)
-    (Apache License 2.0) nameko (https://github.com/nameko/nameko/blob/master/LICENSE.txt)
 
 ========================================================================
 MIT licenses
@@ -244,7 +243,6 @@ BSD-style licenses
 The following components are provided under a BSD-style license. See project link for details.
 
     (BSD 2-Clause "Simplified" License) mock (https://github.com/testing-cabal/mock/blob/master/LICENSE.txt)
-    (BSD 3-Clause) pyrabbit (https://github.com/bkjones/pyrabbit/blob/master/LICENSE)
     (BSD 3-Clause) logutils (https://opensource.org/licenses/BSD-3-Clause)
     (BSD 3-Clause) jinja2 (https://github.com/pallets/jinja/blob/master/LICENSE)
     (BSD 3-Clause) ujson(https://github.com/esnme/ultrajson/blob/master/LICENSE.txt)

Makefile

@@ -107,12 +107,20 @@ eg_network:
 run_server: eg_network
 	docker kill eg_server || true
 	docker rm eg_server || true
-	docker run --network eg_network --name eg_server -e EVENTGEN_AMQP_HOST="eg_controller" -d -p 9501:9500 eventgen:latest server
+	docker run --network eg_network --name eg_server -e REDIS_HOST=eg_controller -d -p 9501:9500 eventgen:latest server
 
 run_controller: eg_network
 	docker kill eg_controller || true
 	docker rm eg_controller || true
-	docker run --network eg_network --name eg_controller  -d -p 5672:5672 -p 15672:15672 -p 9500:9500 eventgen:latest controller
+	docker run --network eg_network --name eg_controller  -d -p 6379:6379 -p 9500:9500 eventgen:latest controller
+
+run_standalone:
+	docker kill eg_standalone || true
+	docker rm eg_standalone || true
+	docker run --name eg_standalone  -d -p 9500:9500 eventgen:latest standalone
+
+run_local_standalone:
+	python -m splunk_eventgen service -r standalone
 
 docs:
 	cd docs/; bundle install; bundle exec jekyll serve

dockerfiles/Dockerfile

@@ -1,4 +1,4 @@
-FROM rabbitmq:3.7.15-management-alpine
+FROM redis:5.0.5-alpine
 
 RUN apk --no-cache upgrade && \
 	apk add --no-cache --update \
@@ -10,17 +10,31 @@ RUN apk --no-cache upgrade && \
 	openssl-dev \
 	libxml2-dev \
 	libxslt-dev \
+	bash \
+	sudo \
+	openssh \
+	tar \
+	acl \
+	g++ \
+	git \
 	curl && \
 	pip install --upgrade pip && \
 	rm -rf /tmp/* && \
-	rm -rf /var/cache/apk/*
+	rm -rf /var/cache/apk/* && \
+	ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa && \
+	mkdir -p /var/run/sshd && \
+	mkdir -p /root/.ssh && \
+	chmod 0700 /root/.ssh && \
+	passwd -u root && \
+	pip install git+git://github.com/esnme/ultrajson.git
 
+COPY dockerfiles/sshd_config /etc/ssh/sshd_config
 COPY dockerfiles/entrypoint.sh /sbin/entrypoint.sh
 COPY dist/* /root/splunk_eventgen.tgz
 RUN pip install /root/splunk_eventgen.tgz && \
-	rm /root/splunk_eventgen.tgz && \
-	echo "[{rabbit, [{loopback_users, []}]}]." >> /etc/rabbitmq/rabbitmq.config
+	rm /root/splunk_eventgen.tgz
 
-EXPOSE 5672 15672 9500
-WORKDIR  /usr/lib/python2.7/site-packages/splunk_eventgen
+EXPOSE 2222 6379 9500
+RUN chmod a+x /sbin/entrypoint.sh
+WORKDIR /usr/lib/python2.7/site-packages/splunk_eventgen
 ENTRYPOINT ["/sbin/entrypoint.sh"]

dockerfiles/entrypoint.sh

@@ -1,11 +1,12 @@
 #!/bin/bash
 
 set -e
+/usr/sbin/sshd
 
 if [ "$#" = 0 ]; then
 	tail -F -n0 /etc/hosts && wait
 elif [ "$1" = "controller" ]; then
-	rabbitmq-server &
+	redis-server &
 	splunk_eventgen service --role controller &
 	tail -F -n0 /etc/hosts && wait
 elif [ "$1" = "server" ]; then
@@ -16,4 +17,4 @@ elif [ "$1" = "standalone" ]; then
 	tail -F -n0 /etc/hosts && wait
 else
 	"$@"
-fi
+fi

dockerfiles/sshd_config

@@ -0,0 +1,35 @@
+Port 2222
+AcceptEnv LANG LANGUAGE XMODIFIERS LC_* RD_*
+AddressFamily any
+AllowAgentForwarding yes
+AllowTcpForwarding yes
+AuthorizedKeysFile %h/.ssh/authorized_keys
+ChallengeResponseAuthentication no
+Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
+ClientAliveInterval 180
+Compression delayed
+HostBasedAuthentication no
+HostKey /etc/ssh/ssh_host_rsa_key
+IgnoreRhosts yes
+IgnoreUserKnownHosts yes
+KexAlgorithms [email protected],ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
+LogLevel INFO
+LoginGraceTime 30s
+MACs [email protected],[email protected],[email protected],hmac-sha2-512,hmac-sha2-256,[email protected]
+MaxAuthTries 5
+MaxSessions 128
+PasswordAuthentication no
+PermitEmptyPasswords no
+PermitRootLogin yes
+PermitTunnel yes
+PermitUserEnvironment no
+PidFile /var/run/sshd.pid
+PrintMotd no
+Protocol 2
+PubKeyAuthentication yes
+StrictModes no
+Subsystem sftp /usr/lib/ssh/sftp-server
+SyslogFacility AUTH
+TcpKeepalive yes
+UseDns no
+X11Forwarding yes

docs/ARCHITECTURE.md

@@ -46,11 +46,16 @@ Given the complexity and the reimplementation of a number of features during ref
 
     python -m splunk_eventgen generate tests/<dir>/<test>.conf
 
-# Server-Controller Architecture
+# Controller-Server Architecture
 
-This is a new feature included in version >= 6.0 Traditionally, it has been difficult to configure multiple Eventgen instances at the same time. The performance of a single Eventgen instance is often limited in its architecture and compute power of the host machine.
-Therefore, it is inevitable that we will need to use more than one Eventgen instance for larger data generation. We introduce server-controller architecture to do this in a more user friendly way.
+This is a new feature included in version >= 6.0 Traditionally, it has been difficult to configure multiple Eventgen instances at the same time. The performance of a single Eventgen instance is often limited in its architecture and compute power of the host machine. We introduce Controller-Server architecture to support scalable Eventgen deployment.
 
-In Server-Controller Architecture, we may have one or multiple servers (Eventgen instances), one controller and one instance of RabbitMQ. It is ok to run RabbitMQ locally with the controller.
-The Servers and controller are communicating via RabbitMQ, where the controller has a capability to broadcast incoming requests to the servers and aggregate the results to output to the users.
+In this Architecture, we may have one or multiple servers (Eventgen instances), one controller and one instance of Redis (which lives inside of the same container as the controller). You don't have to worry about setting up Redis yourself. Redis is used for communication between Controller and Servers.
+
+In order to get setup, I would recommend building a Docker image locally using ``make image`` command.
+Then, start a single instance of Controller (using ``controller`` as the container argument). You may start any number of servers (using ``server`` as the container argument) with env var ``REDIS_HOST`` and ``REDIS_PORT`` that points to the controller's host address and Redis port (by default ``6379``).
+
+# Standalone Architecture
+
+Users can also set up Standalone Eventgen servers without a controller (using ``standalone`` as the container argument). Standalone mode allows users to use the same backend API calls.
 

docs/REFERENCE.md

@@ -605,8 +605,12 @@ Note, "TARGET_NAME" is a variable that should be replaced by the hostname of Eve
     * Starts target Eventgen instance's data generation
 * ```POST /stop```
     * Stops all Eventgen instances' data generation
+    * body is optional; default is false. Setting force to true will destroy current Queues and trying to hard stop a running Eventgen object by causing errors.
+        * Format: ```{"force": true}``` or ```{"force": false}```
 * ```POST /stop/<TARGET_NAME>```
     * Stops target Eventgen instance's data generation
+    * body is optional; default is false. Setting force to true will destroy current Queues and trying to hard stop a running Eventgen object by causing errors.
+        * Format: ```{"force": true}``` or ```{"force": false}```
 * ```POST /restart```
     * Restarts all Eventgen instances' data generation
 * ```POST /restart/<TARGET_NAME>```
@@ -651,6 +655,7 @@ Note, "TARGET_NAME" is a variable that should be replaced by the hostname of Eve
             * Default values
                 * mode: "roundrobin"
                 * hostname_template: "idx{0}"
+                * hosts: [] # list of host addresses
                 * protocol: "https"
                 * key: "00000000-0000-0000-0000-000000000000"
                 * key_name: "eventgen"
@@ -664,6 +669,7 @@ Note, "TARGET_NAME" is a variable that should be replaced by the hostname of Eve
             * Default values
                 * mode: "roundrobin"
                 * hostname_template: "idx{0}"
+                * hosts: [] # list of host addresses
                 * protocol: "https"
                 * key: "00000000-0000-0000-0000-000000000000"
                 * key_name: "eventgen"
@@ -703,3 +709,17 @@ Note, "TARGET_NAME" is a variable that should be replaced by the hostname of Eve
         ```
         $ curl http://localhost:9500/volume/egx1 -X POST -d '{"perDayVolume": 200}'
         ```
+* ```POST /reset```
+    * Stops a running Eventgen run, reset the Eventgen Core Object, and reconfigure the server.
+    * Example:
+        ```
+        $ curl http://localhost:9500/reset -X POST
+        ```
+
+* ```POST /reset/<TARGET_NAME>```
+    * Stops a running Eventgen run, reset the Eventgen Core Object, and reconfigure the server.
+    * Example:
+        ```
+        $ curl http://localhost:9500/reset/egx1 -X POST
+        ```
+

docs/SETUP.md

@@ -101,18 +101,15 @@ $ python -m splunk_eventgen -v generate tests/sample_eventgen_conf/replay/eventg
 $ splunk_eventgen -v generate path/to/eventgen.conf
 ```
 
-##### Controller/Server Cluster ###
-
-A quick preface on this mode of operation: due to its complexity, this is only recommended if you're developing or comfortable with technical setups. Having said that, you can follow these instructions:
-
-1. Install and run [RabbitMQ](https://www.rabbitmq.com/download.html) locally
-2. Install [Eventgen PyPI module](SETUP.md#pypi-setup)
-3. To set up a controller, run `splunk_eventgen service --role controller`
-4. To set up a server, run `splunk_eventgen service --role server`
-5. By default, the controller and server will try to locate RabbitMQ on pyamqp://localhost:5672 using credentials guest/guest and RabbitMQ's web UI at http://localhost:15672.  If you're running another rabbitMQ server, you may error out.
-6. You can change any of those parameters using the CLI - for instance, if your RabbitMQ is accessible on rabbit-mq.company.com with credentials admin/changeme you should run `splunk_eventgen service --role controller --amqp-host rabbit-mq.company.com --amqp-user admin --amqp-pass changeme`
-7. Please see `splunk_eventgen service --help` for additional CLI options
-8. **NOTE:** Running the controller and server on the same machine will cause port collisions for Eventgen web server. To mitigate this, you can tell the server to run on a separate port using `splunk_eventgen service --web-server-address 0.0.0.0:9501`
+##### Controller-Server Cluster ###
+
+Please follow these instructions to run an Eventgen cluster on your Docker environment:
+
+1. `make image`
+2. Create a Docker network: `docker network create --attchable --driver bridge eg_network`
+3. To set up a controller, run `docker run --network eg_network --name eg_controller  -d -p 6379:6379 -p 9500:9500 eventgen:latest controller`
+4. To set up a server, run `docker run --network eg_network --name eg_server -e REDIS_HOST=eg_controller -d -p 9501:9500 eventgen:latest server`
+* Note that REDIS_HOST needs to be a resolvable host address to the controller. Also, --name should be used to differientiate a server from another.
 
 ---
 

requirements.txt

@@ -3,7 +3,6 @@ pytest-xdist
 mock
 pytest-cov
 docker==2.7.0
-nameko
 pyOpenSSL
 lxml==4.3.4
 pytest-mock>=1.10.4
@@ -16,10 +15,11 @@ ujson>=1.35
 pyyaml
 httplib2
 jinja2
-pyrabbit==1.1.0
 urllib3==1.24.2
 pyOpenSSL
 flake8>=3.7.7
 yapf>=0.26.0
 isort>=4.3.15
-structlog==19.1.0
+Flask>=1.0.3
+redis==3.2.1
+structlog==19.1.0