Unable to remove subscriptions

[ValkyrieOps]

Hello,
I am using this in conjunction with the Sensu Path Discovery asset. The handler works great when adding new subscriptions but subsequent deleted paths (and therefore subscriptions passed to the handler) do not get removed from the entity. If this is desired behavior what is the expected workflow for dynamically removing subscriptions? Running latest agent/backend cluster as well. Please let me know if you would like more information. Cheers

[calebhailey]

Great question! Can you elaborate on the scenario in which a path would be discovered (and the corresponding subscription added) but then later the path is no longer found? What is the desired service discovery workflow? Are you attempting to monitor something about a system or service that is only intermittently present? Please advise.

[ValkyrieOps]

Thanks for getting back to me. My workflow is directed at physical servers wherein the applications deployed to them to monitor can shift around for various reasons (proprietary hardware change testing/fallback, DR solutions etc). Historically in Core, this management was accomplished by additional subscription files being present in the conf.d directory, with a bare bones client config. I could simply add/remove those files as part of the deployment/decommission of an application, restart the client and be up and running again. With the recent change to Go to permit agent management of an entity again I could probably replicate this with some bash/regexing to populate the agent.yml file, but that doesn't seem to be very idiomatic with how entities are intended to be managed with Sensu Go. Ideally the output of this handler could be configured to be authoritative for an entity but perhaps there is a better workflow for subscription management I'm not considering? It's not the end of the world to have to prune subscriptions manually with the CLI I just don't see how it would function at scale without operating under the assumption that all of your hardware/instances are always ephemeral. Let me know if there is some more information or examples I can provide to help. Thanks again!

[calebhailey]

I see. This helps. My immediate reaction is to take a closer look at your previous/existing workflow for clues:

Historically in [Sensu] Core, this management was accomplished by additional subscription files being present in the conf.d directory, with a bare bones client config. I could simply add/remove those files as part of the deployment/decommission of an application, restart the client and be up and running again.

If your management workflow is decentralized (configured at the edge, e.g. via Puppet or Ansible), you could retain that workflow.

Some potential approaches that are already supported in Sensu Go:

• The same process that would manage deleting a file and restarting the agent could interact with the Sensu Entities API and only modify the subscriptions field; see: https://docs.sensu.io/sensu-go/latest/api/entities/#update-an-entity-with-patch

• The same process that would manage deleting a file and restarting the agent could modify the agent configuration and restart the agent, as "agent-managed entities" were added in Sensu Go 6.2. This is very close to the legacy workflow, but with a few subtle improvements (the control plane now understand if an entity is agent-managed, or managed via the backend/API). See the --agent-managed-entity flag, here: https://docs.sensu.io/sensu-go/latest/observability-pipeline/observe-schedule/agent/#configuration-via-flags

The first two approaches are very similar, and mostly a matter of preference - does a decentralized agent/entity management workflow work better, or does a centralized agent/entity management workflow work better?

Regardless if one of these solutions works for now, I'm also interesting in identifying a pattern and adding logic for removing (pruning?) subscriptions via the entity manager handler.

[ValkyrieOps]

If your management workflow is decentralized (configured at the edge, e.g. via Puppet or Ansible), you could retain that workflow.

Indeed. Chef is the primary tooling here and subscriptions are generated via assigned node roles as part of application deployments (along with your standard server metrics). The benefit to individual subscription files though is that when stepping through a particular deployment (piecemeal, intraday, debugging etc) the total list of subscriptions already in production doesn't get squashed. This is 100% a Chef problem though.

Some potential approaches that are already supported in Sensu Go:

• The same process that would manage deleting a file and restarting the agent could interact with the Sensu Entities API and only modify the subscriptions field; see: https://docs.sensu.io/sensu-go/latest/api/entities/#update-an-entity-with-patch

This is likely the move, either with the API directly or leveraging the sensu_entity resource from the sensu-go-chef cookbook (which has been fantastic btw). Just trying to evaluate which will be the least path of resistance for us for translating all of the logic from Core.

Regardless if one of these solutions works for now, I'm also interesting in identifying a pattern and adding logic for removing (pruning?) subscriptions via the entity manager handler.

It would certainly save us time in the conversion process but there are bigger fish to fry so to speak. I really appreciate you taking the time to discuss here. Thanks again

[calebhailey]

@ValkyrieOps thanks for the feedback! I'm going to keep this issue open for future consideration and/or external contributions (insert obligatory "PRs are always welcome" comment here)... meanwhile, I'm hoping to tackle a few other features such as support for managing entity labels and annotations.

Please feel free to bump this issue if you have any additional feedback to share! 🍻