Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sidekiq errors and exits during startup when using datadog and prometheus exporter #212

Closed
dorongutman opened this issue Jan 28, 2022 · 3 comments
Closed

Sidekiq errors and exits during startup when using datadog and prometheus exporter #212

dorongutman opened this issue Jan 28, 2022 · 3 comments

Comments

@dorongutman
Copy link

dorongutman commented Jan 28, 2022 •
edited
Loading

Ruby version: 2.7.1
Rails version: 6.0.4.4

I am cross posting this in the other gems repos as I don't know who/what is the culprit.
When trying to run sidekiq with both prometheus exporter and datadog's sdk (ddtrace), sidekiq exits upon start (after a few seconds of waiting) with the following error:
stack level too deep

Full traceback log:

Traceback (most recent call last):
	8741: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `<main>'
	8740: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `eval'
	8739: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `<main>'
	8738: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `load'
	8737: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/bin/sidekiq:31:in `<top (required)>'
	8736: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq/cli.rb:70:in `run'
	8735: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:117:in `redis_info'
	8734: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:97:in `redis'
	 ... 8729 levels...
	   4: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/extensions.rb:106:in `[]'
	   3: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/configurable.rb:51:in `resolve'
	   2: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:14:in `resolve'
	   1: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:22:in `parse_matcher'
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/vendor/resolver.rb:39:in `resolve': stack level too deep (SystemStackError)

This issue doesn't happen without the datadog gem or the prometheus gem.

I have created a reproduction repo with as little changes as possible compared to a fresh rails new installation - https://github.com/dorongutman/rails-sidekiq-datadog-prometheus-issue

The readme explains everything but in short:

  • the only changes made are the addition of the three gems, and 2 initializers - promethus.rb and sidekiq.rb - both only have prometheus-exporter relevant configuration, as described in the prometheus exporter docs.
  • removing the middleware configuration of the prometheus exporter makes sidekiq load fine without the error

For convenience, here are the two initializers:
sidekiq.rb

require 'sidekiq/api'
require 'prometheus_exporter/instrumentation'

$LOAD_PATH << Rails.root

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add PrometheusExporter::Instrumentation::Sidekiq
  end
  config.death_handlers << PrometheusExporter::Instrumentation::Sidekiq.death_handler
  config.on :startup do
    PrometheusExporter::Instrumentation::SidekiqQueue.start
    PrometheusExporter::Instrumentation::Process.start type: 'sidekiq'
  end
  at_exit do
    PrometheusExporter::Client.default.stop(wait_timeout_seconds: 10)
  end
end

prometheus.rb

  require 'prometheus_exporter/middleware'
  Rails.application.middleware.unshift PrometheusExporter::Middleware

Here's the full log from running sidekiq to its death (this specific run was with RAILS_ENV=production and DD_TRACE_ENABLED=false to have fewer lines and potential issues but it happens in any combination):

 bundle exec sidekiq
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:115: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:119: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:123: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:146: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:150: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:154: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:84: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:32: warning: The called method `initialize' is defined here
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Booted Rails 6.0.4.4 application in production environment
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Running in ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [-darwin21]
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Booting Sidekiq 6.4.0 with redis options {}
Traceback (most recent call last):
	8741: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `<main>'
	8740: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `eval'
	8739: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `<main>'
	8738: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `load'
	8737: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/bin/sidekiq:31:in `<top (required)>'
	8736: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq/cli.rb:70:in `run'
	8735: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:117:in `redis_info'
	8734: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:97:in `redis'
	 ... 8729 levels...
	   4: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/extensions.rb:106:in `[]'
	   3: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/configurable.rb:51:in `resolve'
	   2: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:14:in `resolve'
	   1: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:22:in `parse_matcher'
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/vendor/resolver.rb:39:in `resolve': stack level too deep (SystemStackError)
@SamSaffron
Copy link
Member

This is the old method patching problem. You can pick aliasing method chains or prepending modules. Start mixing them together and trouble starts happening.

Open to a PR that allows you to opt for a style of method patching as long as we keep the current default.

@SamSaffron SamSaffron reopened this Jan 31, 2022
@inner-whisper
Copy link

@SamSaffron I think this issue can be closed by same reason as #159 (comment)

Style of method patching can be chosen now - https://github.com/discourse/prometheus_exporter#choosing-the-style-of-method-patching

@SamSaffron
Copy link
Member

Thanks @inner-whisper !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SamSaffron @inner-whisper @dorongutman