Adding support for fixed-sourcetype licenses #173

paul-hoff · 2018-09-19T20:43:58Z

Adding some logic to support a fixed-sourcetype license

recipes/license_server.rb

acharlieh · 2018-09-24T03:57:54Z

This isn't the only change to support fixed-sourcetype licenses... the licenses are put into a different directory, and the stack_ids are generated. For the single sourcetype case, it seems to be fixed-sourcetype_UPPERCASE(SHA256(sourcetype)) but a question is what happens with multiple.

acharlieh

We also need to make changes in the creation of the pools in server.conf, each pool has a stack_id associated with it, and that pool maps to the type we have here.

Unless we can figure out a way to handle multiple types of pools gracefully, we should short-circuit and fail if we have multiple types of licenses set.

acharlieh · 2018-09-29T00:56:44Z

recipes/license_server.rb

    hash[type] ||= {}
    hash[type][key] = value
  end
  hash
 end

+unless sourcetype_name.nil?


Instead of breaking this out... we should just do the hash while we're parsing the licenses groups, and set the fixed-sourcetype_SHA as the type key... in that manner, we get rid of the this hash.

(lest we forget that we could have multiple stacks...)

But also, the seeding of that hash on line 29 should be taken out...

acharlieh · 2018-09-29T01:01:44Z

recipes/license_server.rb

 unless node.run_state['cerner_splunk']['total_allotted_pool_size'].nil?
  total_allotted_pool_size = node.run_state['cerner_splunk']['total_allotted_pool_size']
  fail "Sum of pool sizes is #{CernerSplunk.human_readable_size total_allotted_pool_size}. Exceeds total available pool size of #{CernerSplunk.human_readable_size total_available_license_quota}." if total_allotted_pool_size > total_available_license_quota
 end

 license_groups.each do |type, keys|
-  prefix = "#{node['splunk']['home']}/etc/licenses/#{type}"
+  prefix = if type == 'fixed-sourcetype'


If we put the hash into the type above then we get rid of this pattern, and also make use of the type variable in the block below.

recipes/license_server.rb

acharlieh · 2018-09-29T01:47:41Z

recipes/license_server.rb

    quota = doc.at_xpath('/license/payload/quota/text()').to_s.to_i
    expiration_time = doc.at_xpath('/license/payload/expiration_time/text()').to_s.to_i
-    total_available_license_quota += quota if type == 'enterprise' && expiration_time > Time.now.to_i
+    total_available_license_quota += quota if (type == 'enterprise' || type == 'fixed-sourcetype') && expiration_time > Time.now.to_i


I wonder if we should restructure the license_groups hash, to have license_quota in there along with the map of licenses... That way we're not worried about the specific license type but also just maintain a quota for each set...

We probably should push the set of types into the run_state, that way we can use them for building out the pools in server.conf, since we need to set stack_id on all those as well.

recipes/license_server.rb

nmattam

Verify that you can successfully converge the license server.

vagrant up chef s_license

recipes/license_server.rb

paul-hoff · 2018-10-03T19:20:38Z

I verified that I can successfully converge the license server

==> s_license: Chef Client finished, 36/52 resources updated in 11 minutes 58 seconds

nmattam · 2018-10-04T03:29:19Z

recipes/_configure_server.rb

@@ -249,11 +249,11 @@

  license_pools['pools'].each do |pool, pool_config|
    pool_max_size = CernerSplunk.convert_to_bytes pool_config['size']
-    server_stanzas["lmpool:#{pool}"] = {
+    server_stanzas["lmpool:#{node.run_state['type']}"] = {


Won't this value be empty since we are evaluating this at the compile time? Can you show what the server.conf looks like on the license server?

Also, are we grouping pools by license type?

Talking with Charlie, he mentioned that for the fixed-sourcetype license the folder and pool are the same, which is fixed-sourcetype_Uppercase(sha256(sourcetypeName)). So, I was trying to account for that here

Gemfile

.rubocop.yml

recipes/license_server.rb

gravesb · 2018-10-09T22:23:12Z

recipes/_configure_server.rb

+          'slaves' => pool_config['GUIDs'].join(','),
+          'stack_id' => node.run_state['type']
+        }
+        allotted_pool_size += pool_max_size


Since this moves the allotted_pool_size summation from compile time to execution time, I believe that makes the run state for the total_allotted_pool_size incorrect (since the node.run_state is set below, outside the ruby_block at compile time). Then the error checking on line 48-51 of the license_server recipe is no longer accurate.

If one part of it moves to execution time, then all the parts have to move. But then it won't fail the chef run until part way through, which makes that error checking considerably less useful since the bad server conf would have already been written out. Will Splunk fail to start if the pools configured in server.conf are greater than the available licenses?

I checked and Splunk doesn't seem to care if the total allocation of the pools is greater than the amount of the license stack. The service will still start fine. Although if you create or edit a pool via the UI and try to make it larger, something in the web console will set the pool size equal to the amount of available license when it saves it.

I think the easiest fix is to remove the ruby block here and simply re-order the operations in the license_server recipe so the data bag is processed prior to including this recipe. Just move line 18 down between lines 46 and 48 in the license_server recipe.

gravesb

lines 73-87 of the license-server.rb recipe don't quite work any more. If you have one type of license and attempt to switch to a different type, the original license type is not cleaned up. For example, if I converge a node with an enterprise license and later converge it with a fixed-sourcetype license, the enterprise license remains.

recipes/license_server.rb

paul-hoff · 2018-10-24T18:40:13Z

Should I use some sort of regular expression on line 76 of license-server.rb instead of the type variable to fix lines 73-87?

gravesb · 2018-10-24T20:30:17Z

Yeah, tinkering with that regex may work along with a bit more complex code to exclude the files we want to keep.

Another option would be to add something like this to that ruby block:

Dir.glob("#{node['splunk']['home']}/etc/licenses/*") { |dir| FileUtils.rm_rf(dir) unless dir.end_with?(type) }

may also need to require 'fileutils' somewhere in there along with similar logging to what's there for the other files we delete.

paul-hoff · 2018-10-26T18:51:43Z

I have added this code to the ruby block, and was able to successfully converge the license server. I am unsure how to test its actual functionality since only a single demo license is being used. Is this kind of what you had in mind?

      existing_directory = Dir.glob("#{node['splunk']['home']}/etc/licenses/*")
      delete_directory = existing_directory.delete_if { |dir| FileUtils.rm_rf(dir) unless dir.end_with?(type) }
      delete_directory.each do |directory|
        Chef::Log.info("ruby_block[license cleanup] deleted unconfigured license directory #{directory}")
      end

gravesb · 2018-10-26T19:55:38Z

yeah, something like that. Given that you're working on a PR for fixed-sourcetype license support I assume you have access to other licenses that you could use to test locally (though obviously you wouldn't want to commit those changes as part of this PR).

acharlieh · 2018-10-31T14:04:05Z

recipes/license_server.rb

@@ -66,6 +72,11 @@
 b = ruby_block 'license cleanup' do
  block do
    license_groups.each do |type, licenses|
+      existing_directory = Dir.glob("#{node['splunk']['home']}/etc/licenses/*")


I know that we only support one type now, but this logically should be outside of the license_groups.each loop

acharlieh · 2018-11-01T13:02:06Z

recipes/license_server.rb

    license_groups.each do |type, licenses|
+      existing_directory.each do |dir|


Pull this loop out of the license_groups loop too...

acharlieh · 2018-11-01T14:53:26Z

recipes/license_server.rb

@@ -65,6 +71,11 @@

 b = ruby_block 'license cleanup' do
  block do
+    existing_directory = Dir.glob("#{node['splunk']['home']}/etc/licenses/*")
+    existing_directory.each do |dir|
+      FileUtils.rm_rf(dir) unless dir.end_with?(type)


Alrighty now just need to fix the logic a bit... as we want to only delete those directories that we expect there to be no licenses, so something like: (double check me on this... this is rather haphazard, but should give you the general idea). (also because type is now not defined in this scope )

next if license_groups.keys.include? File.basename(dir) Chef::Log.info( "... blah..." ) FileUtils.rm_rf(dir)

This should be fixed with d4c6810

acharlieh · 2018-11-01T21:08:51Z

recipes/license_server.rb

    hash[type][key] = value
  end
+  node.run_state['type'] = type


This should either move as is to right after the fail ... if on line 37
OR
This should be outside the inject loop after the end on line 42, slightly modified like this:

node.run_state['type'] = license_groups.keys.first

That type variable defined on line 31 feels like it's really intended to only be valid within the unless block (line 28 -> 39),

If you happen to be retrieving a level where skipping the keys 'id', 'chef_type' and 'data_bag' mattered, and one of those happened to be traversed prior to your actual key that you're interested in, you'll have a NameError: undefined local variable or method `type' on your hands

nit: should 'type' be 'license_type' or 'stack_id' ... something a bit more descriptive as a key?

that makes sense, these changes are reflected in faaab86

* Add support for fixed-sourcetype licenses

gravesb · 2018-11-05T17:31:33Z

Merged in b426f47.

Added logic for fixed-sourcetype

f9e8f4a

paul-hoff mentioned this pull request Sep 19, 2018

Support for fixed-sourcetype license #172

Closed

fixed operator

fb0b87a

gravesb reviewed Sep 20, 2018

View reviewed changes

recipes/license_server.rb Outdated Show resolved Hide resolved

Added ( ) for clarity

17db974

gravesb reviewed Sep 20, 2018

View reviewed changes

recipes/license_server.rb Outdated Show resolved Hide resolved

paul-hoff and others added 3 commits September 25, 2018 09:55

fixed parentheses

ab5d1fa

Additional changes to support fixed-sourcetype

282845f

Fixed rubocop offenses

5939ca7

paul-hoff changed the title ~~Added logic for fixed-sourcetype~~ Adding support for fixed-sourcetype licenses Sep 27, 2018

acharlieh suggested changes Sep 29, 2018

View reviewed changes

paul-hoff added 4 commits October 2, 2018 13:31

various recommended changes

fd76fe6

added run_state

c0c8d26

lock cucumber-core to ~> 3.2

fab0bfb

alphabetize

e5e5bd4

nmattam suggested changes Oct 2, 2018

View reviewed changes

recipes/license_server.rb Outdated Show resolved Hide resolved

recipes/license_server.rb Outdated Show resolved Hide resolved

replaced type variable

4b42747

minor fix

88b6707

nmattam reviewed Oct 4, 2018

View reviewed changes

Gemfile Show resolved Hide resolved

nmattam reviewed Oct 4, 2018

View reviewed changes

.rubocop.yml Outdated Show resolved Hide resolved

paul-hoff added 2 commits October 5, 2018 16:23

Short-circuit if multiple license type/ push type to run_state

7fc33e5

madde change to ruby block

992d7e6

nmattam reviewed Oct 8, 2018

View reviewed changes

recipes/license_server.rb Outdated Show resolved Hide resolved

fix short-circuit check for mulitple licenses

9b224c5

gravesb reviewed Oct 9, 2018

View reviewed changes

re-order operations

c6e299e

gravesb suggested changes Oct 23, 2018

View reviewed changes

recipes/license_server.rb Outdated Show resolved Hide resolved

paul-hoff added 2 commits October 30, 2018 10:46

license directory cleanup

647c229

newline

f6868b1

gravesb approved these changes Oct 30, 2018

View reviewed changes

nmattam approved these changes Oct 30, 2018

View reviewed changes

acharlieh reviewed Oct 31, 2018

View reviewed changes

paul-hoff added 2 commits October 31, 2018 12:08

moved existing_directory delartion outside of loop

1ab4564

fixed newline

4b91db5

acharlieh reviewed Nov 1, 2018

View reviewed changes

moved loop

2ac8f99

acharlieh reviewed Nov 1, 2018

View reviewed changes

fixed logic

d4c6810

acharlieh reviewed Nov 1, 2018

View reviewed changes

node.run_state['type'] changed to node.run_state['license_type']

faaab86

acharlieh approved these changes Nov 2, 2018

View reviewed changes

gravesb pushed a commit that referenced this pull request Nov 5, 2018

Merge pull request #173 from paul-hoff/support-fixed-sourcetype

b426f47

* Add support for fixed-sourcetype licenses

gravesb closed this Nov 5, 2018

gravesb added this to the v2.23.0 milestone Feb 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for fixed-sourcetype licenses #173

Adding support for fixed-sourcetype licenses #173

paul-hoff commented Sep 19, 2018

acharlieh commented Sep 24, 2018

acharlieh left a comment

acharlieh Sep 29, 2018

acharlieh Sep 29, 2018

acharlieh Sep 29, 2018

nmattam left a comment

paul-hoff commented Oct 3, 2018

nmattam Oct 4, 2018 •

edited

Loading

nmattam Oct 4, 2018

paul-hoff Oct 4, 2018

gravesb Oct 9, 2018

gravesb Oct 10, 2018 •

edited

Loading

gravesb Oct 11, 2018 •

edited

Loading

gravesb left a comment •

edited

Loading

paul-hoff commented Oct 24, 2018

gravesb commented Oct 24, 2018 •

edited

Loading

paul-hoff commented Oct 26, 2018

gravesb commented Oct 26, 2018

acharlieh Oct 31, 2018

acharlieh Nov 1, 2018

acharlieh Nov 1, 2018 •

edited

Loading

paul-hoff Nov 1, 2018

acharlieh Nov 1, 2018

acharlieh Nov 1, 2018

paul-hoff Nov 2, 2018

gravesb commented Nov 5, 2018 •

edited

Loading

		license_groups.each do \|type, licenses\|
		existing_directory.each do \|dir\|

Adding support for fixed-sourcetype licenses #173

Adding support for fixed-sourcetype licenses #173

Conversation

paul-hoff commented Sep 19, 2018

acharlieh commented Sep 24, 2018

acharlieh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmattam left a comment

Choose a reason for hiding this comment

paul-hoff commented Oct 3, 2018

nmattam Oct 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gravesb Oct 10, 2018 • edited Loading

Choose a reason for hiding this comment

gravesb Oct 11, 2018 • edited Loading

Choose a reason for hiding this comment

gravesb left a comment • edited Loading

Choose a reason for hiding this comment

paul-hoff commented Oct 24, 2018

gravesb commented Oct 24, 2018 • edited Loading

paul-hoff commented Oct 26, 2018

gravesb commented Oct 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acharlieh Nov 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gravesb commented Nov 5, 2018 • edited Loading

nmattam Oct 4, 2018 •

edited

Loading

gravesb Oct 10, 2018 •

edited

Loading

gravesb Oct 11, 2018 •

edited

Loading

gravesb left a comment •

edited

Loading

gravesb commented Oct 24, 2018 •

edited

Loading

acharlieh Nov 1, 2018 •

edited

Loading

gravesb commented Nov 5, 2018 •

edited

Loading