REST API: Write config in a parametrized package

[DamianoChini]

Is your feature request related to a problem? Please describe.

We want to apply "hot configuration changes" in Icinga 2 without losing configuration objects changed by the API in between deployments when doing a "cold deploy" with the Icinga Director. We need this because in our big installations the deploy takes 2-3 minutes to perform a complete restart.
In other words, we want to create (and update) Icinga objects (hosts, services, hostgroups, etc.) via Icinga 2 APIs, but at the same time we want to create these objects with the Icinga Director. Currently this is not possible because the configuration of the two Icinga 2 packages (Director package and _api package) would go in conflict.

Describe the solution you'd like

We'd like to be able to configure the name of the package that the Icinga APIs will use to write the configurations. In this way, we will simply ask the Icinga API to write directly in our desired package.

[Al2Klimov]

we want to create (and update) Icinga objects (hosts, services, hostgroups, etc.) via Icinga 2 APIs, but at the same time we want to create these objects with the Icinga Director

In other words, you'd have to maintain the same set of monitored objects twice.

@lippserd Wouldn’t it be better for the Director (and the core, of course) to support "hot updates" by itself?

[Thomas-Gelf]

@Al2Klimov: I see no reason why in future this couldn't also be used by the Director: deploy small changes immediately, and silently flush those single object creations and/or modifications with the next full deployment. Activity Log would then distinct undeployed, live-deployed and deployed modifications. @lippserd and I have been involved in a brainstorming related to this feature request. OPs main motivation: reloading Icinga takes way too long in large environments, so this could be a nice feature allowing one to mix up partial and full deployments.

As you correctly pointed out, it's then up to whoever uses this feature to track those objects externally. Could this be the Director? Absolutely. Documentation should point out that this parameter is NOT what should normally be used, and that nobody would clean up your package if you do not care.

Long time ago we tried a similar approach in the Director with live object creation (and hot updates) via the console API. Development on this feature stopped as such objects are not replicated in the cluster. Parts of this attempt are still to be found in the Director source tree and can be triggered with some hidden flag. However, as long as the cluster doesn't forward live object modifications, there is no way to finalize this.

Working through the "single object API" would result in config files stored to disk, sometimes tricky to clean up. In this case the cleanup would take place with every full deployment, as the former package would be flushed. At least this is the base assumption behind this feature request, as far as I understood.

If you prefer introducing a third method for changing objects: as long as it is replication-safe and it allows one to wipe such "runtime changes" with the next full deployment: count me all in. But it's probably better to extend one of the two possibilities we have right now. The focus is on getting the functionality implemented in one way or the other. Guess there is no hard preference for either way, as long as the targeted issue could somehow be tackled.

Cheers,
Thomas

[Al2Klimov]

I doubt we should extend the regular per-object config management API endpoints. #8640 clearly demonstrates the loose of safety checks due to a such way. I think we (team core) should get to the whiteboard to discuss how we should get this done... in CW45. 😉

[Al2Klimov]

Just got an idea: yes, the Director writes whole config files to Icinga. But it also has the separate config objects. It also has the individual attributes. It even has diffs of all the objects and attributes between every single deployment. Couldn’t the director by itself (optionally?) just manage the individual objects (of course not the apply rules) by the Icinga API and not the packages?

I.e. (as the user wishes?) apply rules (if the user has such at all) stay in the Director package. On a new deployment if anything in the director package changes, we’re very sorry and still have to trigger an Icinga reload. Independent of that if any individual objects change, the Director just creates/modifies/deletes them as regular API objects (effectively in the _api package) and – of course – tracks the changes by itself.

If you decide to do that, please do me a favor: modify attributes and custom vars only at root level no matter how small the change is. I.e. if just vars.lolcat.foo.bar.x.y.z changes, update the whole vars.lolcat.

[lippserd]

#8640 clearly demonstrates the loose of safety checks due to a such way.

@Al2Klimov Can you explain this further?

Couldn’t the director by itself (optionally?) just manage the individual objects (of course not the apply rules) by the Icinga API and not the packages?

Sure it could. But wouldn't that lead to a lot of files?
Also I believe that this means more work than implementing the package parameter in Icinga.
The cleanup as @Thomas-Gelf mentioned would be "built in" then.

[Al2Klimov]

#8640 clearly demonstrates the loose of safety checks due to a such way.

@Al2Klimov Can you explain this further?

Have a look at that PR. It removes several ifs which prevent deletion of config you shall not delete.

Couldn’t the director by itself (optionally?) just manage the individual objects (of course not the apply rules) by the Icinga API and not the packages?

Sure it could. But wouldn't that lead to a lot of files?

Yes... ?

Also I believe that this means more work than implementing the package parameter in Icinga.

We wouldn’t need the package parameter in Icinga.

The cleanup as @Thomas-Gelf mentioned would be "built in" then.

Which cleanup? #8639 (comment) wouldn’t need any cleanup, would it?

[DamianoChini]

As discussed with @lippserd , this issue is no more needed and can be discarded.

[tbauriedel]

@lippserd
We have a similar request that would build on that exact functionality. I'm just curious what the reason was for discontinuing this effort. In conjunction with Director issue 2153, this sounds promising.

[julianbrost]

The current mix of API/JSON-created and file/DSL-created (including deployed from Director) objects is already quite complex and fragile in some places. With #8640, there already was a PR that attempted this, but it had to remove many safety checks in the API code so everything would just become more fragile in this way. I think the main motivation for this feature request is not that anyone actually wants this very specific feature but rather as a workaround for slow reload times. With the 2.14 release, there will come significant improvements (but also, don't expect wonders, we're not yet where we'd like to be in the end) and I believe that further improvements there are the way forward. I'd prefer to spend time on that compared to fixing bugs that the other change would be prone to introduce.