[processor/sumologicschema]: add nesting processor #877
Conversation
github-actions
bot
added
documentation
aboguszewski-sumo
force-pushed
the
json-deflattener
branch
from
c352124 to
9712e25
Compare
aboguszewski-sumo
force-pushed
the
json-deflattener
branch
2 times, most recently
from
ffa6f9c to
9048ad6
Compare
| #### Known issues | ||
|
|
||
| This feature has undefined behavior when in input there are various keys that will be mapped to the same nested structure. | ||
| For example, given the following attributes: | ||
|
|
||
| ```json | ||
| { | ||
| "a.b.c.": "d", | ||
| "a": { | ||
| "b": { | ||
| "c": "e" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| It is not possible to predict, what will be the result. It will have the following structure: | ||
|
|
||
| ```json | ||
| { | ||
| "a": { | ||
| "b": { | ||
| "c": ... | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| However, it is not possible to know a priori if the value under key `c` will be equal to `d` or `e`. |
There was a problem hiding this comment.
There is one issue that I encountered and I think that we should definitely make it a UB.
There was a problem hiding this comment.
what is happening in current implementation?
IMHO, We shouldn't override existing keys
There was a problem hiding this comment.
In general, keys are overwritten. If there was a map, the maps are merged (for example "a.b": "c" and "a": {"d":"e"} get merged into "a": {"b": "c", "d":"e"}).
The problem is that we can't easily decide whether we will be overwriting a key, because it depends on the order in which Range method of pcommon.Map traverses the attributes. We would have to make some kind of preprocessing to check all existing keys.
There was a problem hiding this comment.
makes sense, I think for the first iteration it should be fine. Let's follow the KISS principle
aboguszewski-sumo
force-pushed
the
json-deflattener
branch
from
9048ad6 to
d8212dc
Compare
|
|
||
| # Specifies if attributes should be nested, basing on their keys. | ||
| # See "Nesting attributes" documentation chapter from this document. | ||
| nest_attributes: | ||
| # Defines whether attributes should be nested. | ||
| # default = false | ||
| enabled: {true, false} | ||
|
|
||
| # Defines the string used to separate key names in attributes that are to be nested. | ||
| # default = "." | ||
| separator: <separator> |
There was a problem hiding this comment.
In my opinion, we could have some pattern matching here. So in case customer wants to leave some keys which has separator in it he can do i, but current behavior seems to be reasonable for first iteration
There was a problem hiding this comment.
This has been already commented this on Slack. IMO we should add this in another PR (it is also arguable what should the filters be: we can do an allowlist or a denylist, with advanced matching options or not... imo it's the best to just add a denylist of prefixes)
aboguszewski-sumo
force-pushed
the
json-deflattener
branch
from
c90568c to
bb97d04
Compare
Updates #865
We're waiting for a response on the issue, but besides that, this is finished.