Datajson v2.0 #12175
Closed
Datajson v2.0 #12175
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch introduces a new keyword datajson that is similar
to dataset with a twist. Where dataset allows match from sets,
datajson allows the same but also adds JSON data to the alert
event. This data is comint from the set definition it self.
For example, an ipv4 set will look like:
10.16.1.11,{"test": "success","context":3}
The syntax is value and json data separated by a comma.
The syntax of the keyword is the following:
datajson:isset,src_ip,type ip,load src.lst,key src_ip;
Compare to dataset, it just have a supplementary option key
that is used to indicate in which subobject the JSON value
should be added.
The information is added in the even under the alert.extra
subobject:
"alert": {
"extra": {
"src_ip": {
"test": "success",
"context": 3
},
The main interest of the feature is to be able to contextualize
a match. For example, if you have an IOC source, you can do
value1,{"actor":"APT28","Country":"FR"}
value2,{"actor":"APT32","Country":"NL"}
This way, a single dataset is able to produce context to the
event where it was not possible before and multiple signatures
had to be used.
Ticket: OISF#7372
Previous code was using an array and introducing a limit in the number of datajson keywords that can be used in a signature. This patch uses a linked list instead to overcome the limit. By using a first element of the list that is part of the structure we limit the cost of the feature to a structure member added to PacketAlert structure. Only the PacketAlertFree function is impacted as we need to iterate to find potential allocation. Ticket: OISF#7372
It was not handling correctly the json values with space as they were seen as multiple arguments. Ticket: OISF#7372
regit
requested review from
jasonish,
inashivb,
jufajardini,
victorjulien and
a team
as code owners
|
I still have to play with this a little more, and datasets in general, but a few quick observations, not all related to this PR, but make things confusing. If my IP dataset input file is simply: then
Should |
|
Information: ERROR: QA failed on SURI_TLPW2_autofp_suri_time. ERROR: QA failed on SURI_TLPR1_suri_time.
Pipeline 23600 |
OK, I need to fix the parsing for here. :/
I more in favor of building the set from the CTI tool at the correct cut before exporting it to Suricata. I think it corresponds to most of the usage I've seen. But potentially we could open the issue and try to see if someone is interested by it. |
|
Closing in favor of #12289. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update of #12102
Contribution style:
https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html
Our Contribution agreements:
https://suricata.io/about/contribution-agreement/ (note: this is only required once)
Changes (if applicable):
(including schema descriptions)
https://redmine.openinfosecfoundation.org/projects/suricata/issues
Link to ticket: https://redmine.openinfosecfoundation.org/issues/7372
Describe changes:
SV_REPO=OISF/suricata-verify#2123