Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQL: add MSSQL support and allow storing only a subset of event fields #2172

Merged
merged 6 commits into from
Jul 25, 2022

Conversation

creideiki
Copy link
Contributor

@creideiki creideiki commented May 4, 2022

This pull request enhances IntelMQ's SQL database support in three ways:

  1. Add support for MSSQL servers in the SQLMixin class, requiring the package pymssql.
  2. Add a parameter fields to SQLOutputBot, containing a list of which event fields to store in the database. If this is not specified, store all fields as before.
  3. Add a parameter reconnect_delay to SQLMixin, making it wait that many seconds before attempting to reconnect after a connection error.
    • We found out the need for this the hard way, when the DB server rebooted and IntelMQ's tight reconnect loop filled the entire disk with error logs before the DB server came back.

@codecov-commenter
Copy link

codecov-commenter commented May 4, 2022

Codecov Report

Merging #2172 (0b45ea6) into develop (80a149a) will decrease coverage by 0.02%.
The diff coverage is 53.57%.

@@             Coverage Diff             @@
##           develop    #2172      +/-   ##
===========================================
- Coverage    76.33%   76.31%   -0.03%     
===========================================
  Files          441      441              
  Lines        23686    23710      +24     
  Branches      3747     3753       +6     
===========================================
+ Hits         18081    18094      +13     
- Misses        4865     4876      +11     
  Partials       740      740              
Impacted Files Coverage Δ
intelmq/lib/mixins/sql.py 51.25% <25.00%> (-5.68%) ⬇️
intelmq/bots/outputs/sql/output.py 94.28% <91.66%> (-1.87%) ⬇️
intelmq/bots/experts/reverse_dns/expert.py 83.87% <0.00%> (+1.61%) ⬆️

@sebix sebix added feature Indicates new feature requests or new features component: bots labels Jul 25, 2022
Copy link
Member

@sebix sebix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't test it, but looks good overall.

Entry in changelog is missing

intelmq/lib/mixins/sql.py Show resolved Hide resolved
docs/user/bots.rst Show resolved Hide resolved
docs/user/bots.rst Show resolved Hide resolved
@@ -29,16 +35,21 @@ class SQLOutputBot(OutputBot, SQLMixin):
sslmode = "require"
table = 'events'
user = "intelmq"
fields = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fields = None
fields: Optional[list] = None

intelmq/bots/outputs/sql/output.py Show resolved Hide resolved
@sebix sebix merged commit c5eecac into certtools:develop Jul 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: bots feature Indicates new feature requests or new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants