Flow: add loki.source.azure_event_hubs component #3412

akselleirv · 2023-04-03T08:39:14Z

PR Description

Add a new component loki.source.azure_event_hubs which read messages from Azure Event Hub using Kafka and forwards them to other loki components. It is a port of this PR: grafana/loki#8787 with the addition of OAUTHBEARER authentication.

The main motivation behind this PR is adding support for collecting logs from Azure resources.

This authentication mechanism is also added to the loki.source.kafka component as this component depends on it.

Which issue(s) this PR fixes

Not related to an issue, but was created based on this conversation: https://grafana.slack.com/archives/C01050C3D8F/p1680105638704069

Notes to the Reviewer

PR Checklist

CHANGELOG updated
Documentation added
Tests updated

CLAassistant · 2023-04-03T08:39:19Z

All committers have signed the CLA.

rfratto · 2023-04-03T14:19:27Z

❤️ Thanks for the PR! We'll review this as soon as we can.

tpaschalis

Nice one! 🙌 Comments are mostly around documentation and style, this is looking good!

component/all/all.go

component/loki/source/azureeventhubs/azureeventhubs_test.go

docs/sources/flow/reference/components/loki.source.azure_event_hubs.md

component/loki/source/azureeventhubs/azureeventhubs.go

component/loki/source/azureeventhubs/internal/parser/parser.go

docs/sources/flow/reference/components/loki.source.azure_event_hubs.md

tpaschalis · 2023-04-04T06:21:46Z

component/loki/source/azureeventhubs/azureeventhubs.go

+		case <-ctx.Done():
+			return nil
+		case entry := <-c.handler:
+			c.mut.RLock()


for @tpaschalis: we might be susceptible to something like #3391 here, both for this and the loki.source.kafka component.

Will check it out to see if there's a need for a fix after this is merged.

tpaschalis · 2023-04-04T06:23:52Z

component/loki/source/azureeventhubs/azureeventhubs.go

+	t, err := kt.NewSyncer(c.opts.Registerer, c.opts.Logger, cfg, entryHandler, &parser.AzureEventHubsTargetMessageParser{
+		DisallowCustomMessages: newArgs.DisallowCustomMessages,
+	})


For other reviewers, here's the crucial bit: the shared kafkatarget package was updated in Promtail to expect a Parser implementation, and they've added one for Azure Event Hubs.

akselleirv · 2023-04-04T08:48:47Z

Nice one! 🙌 Comments are mostly around documentation and style, this is looking good!

Thank you for the review @tpaschalis! I've updated the PR based on your feedback

rfratto

🎉 Nice! I reviewed this purely from the documentation perspective, and I'll let @tpaschalis sign off on the code side.

CHANGELOG.md

docs/sources/flow/reference/components/loki.source.azure_event_hubs.md

akselleirv · 2023-04-04T15:50:18Z

🎉 Nice! I reviewed this purely from the documentation perspective, and I'll let @tpaschalis sign off on the code side.

Thank you! Updated the changes now :)

Remove non-recommended additional header as pointed out here: #3412 (comment)

rfratto

LGTM! @tpaschalis I'll leave final approval / merging to you :)

Remove non-recommended additional header as pointed out here: #3412 (comment)

Remove non-recommended additional header as pointed out here: #3412 (comment) (cherry picked from commit e2c8f54)

Remove non-recommended additional header as pointed out here: #3412 (comment) (cherry picked from commit e2c8f54) Co-authored-by: Robert Fratto <[email protected]>

tpaschalis · 2023-04-05T20:11:28Z

@akselleirv apologies for the hassle, but we recently got a PR that mass updates files for typos, grammatical errors etc. Could you please rebase, and I think we're good to go!

Co-authored-by: Paschalis Tsilias <[email protected]>

Co-authored-by: Robert Fratto <[email protected]>

akselleirv · 2023-04-05T20:26:45Z

@tpaschalis rebased now and the CI is all green 🚀

tpaschalis

LGTM, thank you Aksel!

leelax22 · 2023-05-24T01:02:31Z

I have a question about azure event hub.

grafana/loki#8788

In the following link
"Messages contain a few(one or more) log lines. We need to be able to split them into separate entries."

This is the part I was struggling with, so I tried using loki.source.azure_event_hubs, hoping that it would allow me to split them into separate logs.

The logs I collected through my azure aks diagnostics setup come into loki via event hub and grafana agent flow as follows.

Looking at the code change history on github, it seems that tests were made through log data from function app or logic app, etc. but is the seperate entry applied only to those items? Or is there something I'm missing?

leelax22 · 2023-05-24T06:54:44Z

I solved the problem. The problem was loki api address is not 'url = "loki:3100/api/v1/push" in the docs (https://grafana.com/docs/agent/latest/flow/reference/components/loki.source.azure_event_hubs/#example). It should be "loki:3100/loki/api/v1/push" accroding to the docs(https://grafana.com/docs/loki/latest/api/)

Remove non-recommended additional header as pointed out here: #3412 (comment)

akselleirv force-pushed the feat/azureeventhubs branch 6 times, most recently from eeb9da8 to bd6a9f9 Compare April 3, 2023 09:54

tpaschalis reviewed Apr 4, 2023

View reviewed changes

akselleirv force-pushed the feat/azureeventhubs branch from 9ce3acf to 55c5406 Compare April 4, 2023 08:46

rfratto reviewed Apr 4, 2023

View reviewed changes

rfratto added a commit that referenced this pull request Apr 4, 2023

Update loki.source.kafka.md

570fd68

Remove non-recommended additional header as pointed out here: #3412 (comment)

rfratto mentioned this pull request Apr 4, 2023

Update loki.source.kafka.md #3448

Merged

rfratto approved these changes Apr 4, 2023

View reviewed changes

rfratto added a commit that referenced this pull request Apr 4, 2023

Update loki.source.kafka.md (#3448)

e2c8f54

Remove non-recommended additional header as pointed out here: #3412 (comment)

grafanabot pushed a commit that referenced this pull request Apr 4, 2023

Update loki.source.kafka.md (#3448)

d64825b

Remove non-recommended additional header as pointed out here: #3412 (comment) (cherry picked from commit e2c8f54)

rfratto added a commit that referenced this pull request Apr 4, 2023

Update loki.source.kafka.md (#3448) (#3449)

2af0d70

Remove non-recommended additional header as pointed out here: #3412 (comment) (cherry picked from commit e2c8f54) Co-authored-by: Robert Fratto <[email protected]>

akselleirv force-pushed the feat/azureeventhubs branch from 7169060 to 809b6ca Compare April 5, 2023 07:48

akselleirv and others added 9 commits April 5, 2023 22:15

loki.source.azure_event_hubs component

a7bbe6f

fix: rename package and filenames to reflect component name

e43088f

format river example in test

910acbc

docs: use component name and remove block terminology

e28c9bd

Co-authored-by: Paschalis Tsilias <[email protected]>

docs: document usage of multiple event hub components

692ffcf

docs: improve disallow_custom_messages description

5d554f0

Co-authored-by: Paschalis Tsilias <[email protected]>

parser: inform about copy from promtail

fa94370

added validation of assignor value

7c79b1e

docs: add info about __azure_event_hubs_category label

a94dc0c

akselleirv and others added 8 commits April 5, 2023 22:15

docs: fix bad grammar

627794f

Co-authored-by: Robert Fratto <[email protected]>

docs: improve link name

5e84dcb

Co-authored-by: Robert Fratto <[email protected]>

docs: remove code text from basic pricing plan text

a0ee4b4

Co-authored-by: Robert Fratto <[email protected]>

docs: cleanup table description

5fdec57

Co-authored-by: Robert Fratto <[email protected]>

docs: add fully_qualified_namespace outside of table

0d24f56

Co-authored-by: Robert Fratto <[email protected]>

docs: added missing period

7e89d82

Co-authored-by: Robert Fratto <[email protected]>

docs: improve sentence about assignor

4ed6eb9

Co-authored-by: Robert Fratto <[email protected]>

docs: moved labels header inside arguments header

f53652d

akselleirv force-pushed the feat/azureeventhubs branch from 809b6ca to f53652d Compare April 5, 2023 20:19

akselleirv requested a review from tpaschalis April 5, 2023 20:33

tpaschalis approved these changes Apr 6, 2023

View reviewed changes

tpaschalis merged commit c4ce55a into grafana:main Apr 6, 2023

akselleirv deleted the feat/azureeventhubs branch April 6, 2023 06:35

clayton-cornell pushed a commit that referenced this pull request Aug 14, 2023

Update loki.source.kafka.md (#3448)

8bca019

Remove non-recommended additional header as pointed out here: #3412 (comment)

clayton-cornell pushed a commit that referenced this pull request Aug 14, 2023

Flow: add loki.source.azure_event_hubs component (#3412)

5fd2639

clayton-cornell pushed a commit that referenced this pull request Aug 14, 2023

Update loki.source.kafka.md (#3448)

4d87445

Remove non-recommended additional header as pointed out here: #3412 (comment)

clayton-cornell pushed a commit that referenced this pull request Aug 14, 2023

Flow: add loki.source.azure_event_hubs component (#3412)

1ba117e

wildum mentioned this pull request Oct 26, 2023

Promtail sync loki.source.kafka and loki.source.azure_event_hubs #5617

Merged

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Feb 24, 2024

github-actions bot locked as resolved and limited conversation to collaborators Feb 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flow: add loki.source.azure_event_hubs component #3412

Flow: add loki.source.azure_event_hubs component #3412

akselleirv commented Apr 3, 2023 •

edited

Loading

CLAassistant commented Apr 3, 2023 •

edited

Loading

rfratto commented Apr 3, 2023

tpaschalis left a comment

tpaschalis Apr 4, 2023

tpaschalis Apr 4, 2023

akselleirv commented Apr 4, 2023

rfratto left a comment

akselleirv commented Apr 4, 2023

rfratto left a comment

tpaschalis commented Apr 5, 2023 •

edited

Loading

akselleirv commented Apr 5, 2023

tpaschalis left a comment

leelax22 commented May 24, 2023

leelax22 commented May 24, 2023

Flow: add loki.source.azure_event_hubs component #3412

Flow: add loki.source.azure_event_hubs component #3412

Conversation

akselleirv commented Apr 3, 2023 • edited Loading

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CLAassistant commented Apr 3, 2023 • edited Loading

rfratto commented Apr 3, 2023

tpaschalis left a comment

Choose a reason for hiding this comment

tpaschalis Apr 4, 2023

Choose a reason for hiding this comment

tpaschalis Apr 4, 2023

Choose a reason for hiding this comment

akselleirv commented Apr 4, 2023

rfratto left a comment

Choose a reason for hiding this comment

akselleirv commented Apr 4, 2023

rfratto left a comment

Choose a reason for hiding this comment

tpaschalis commented Apr 5, 2023 • edited Loading

akselleirv commented Apr 5, 2023

tpaschalis left a comment

Choose a reason for hiding this comment

leelax22 commented May 24, 2023

leelax22 commented May 24, 2023

akselleirv commented Apr 3, 2023 •

edited

Loading

CLAassistant commented Apr 3, 2023 •

edited

Loading

tpaschalis commented Apr 5, 2023 •

edited

Loading