refactor(events): populate object identifiers in outgoing webhooks analytics events during retries

[SanchithHegde]

Type of Change

• Bugfix
• New feature
• Enhancement
• Refactoring
• Dependency updates
• Documentation
• CI/CD

Description

As of now, the analytics events raised during outgoing webhook retries do not include object identifiers such as the payment ID, refund ID, etc., and in turn, they are not populated in ClickHouse. The object identifiers are only available in the events raised for the initial webhook delivery attempts.

This PR updates the outgoing webhooks code to include the object identifiers in the analytics events for all retries. This is being done by storing the object identifiers in a new column in the events table, called metadata.

In addition, this PR extracts out common code to implement FromSql and ToSql traits on types converted to/from JSON/JSONB into a declarative macro.

Additional Changes

• This PR modifies the API contract
• This PR modifies the database schema
• This PR modifies application configuration/environment variables

This PR adds a metadata column to the events table.

Motivation and Context

Ensuring that all webhook events including retries are correctly associated with the object in question (payment, refund, etc.), and that they are correctly displayed on the control center when viewing events for that resource.

How did you test it?

1. Configure an incorrect webhook URL in the business profile so that outgoing webhooks are retried.

2. Perform an action that triggers an outgoing webhook, such as creating a successful payment.

3. Monitor the events raised for the payment ID in Kafka and ClickHouse:

   Kafka: Searching for messages under the hyperswitch-outgoing-webhook-events topic with the payment ID has multiple events, including retries:

   

   ClickHouse: Searching for rows by payment ID in the outgoing_webhook_events table returns multiple rows, all having the payment_id populated, and only the initial attempt having the content field populated.

   SELECT * FROM outgoing_webhook_events WHERE payment_id = 'pay_123' ORDER BY created_at ASC;

   

4. The analytics API to obtain outgoing webhook event logs returns all of the events, and each of them contain the payment ID:

   curl '{{baseUrl}}/analytics/v1/outgoing_webhook_event_logs?payment_id={{payment_id}}' -H 'Authorization: Bearer <JWT_TOKEN>'

   The control center should also be ideally displaying all the events in the payment details page, that does not seem to be case however in control center version v1.30.1.

5. Verify similar behavior for the other webhooks with their corresponding identifiers: refunds, payouts, etc.

Checklist

• I formatted the code cargo +nightly fmt --all
• I addressed lints thrown by cargo clippy
• I reviewed the submitted code
• I added unit tests for my changes where possible

[hrithikesh026]

Suggested change

	ALTER TABLE events DROP COLUMN metadata;
	ALTER TABLE events DROP COLUMN IF EXISTS metadata;

[hrithikesh026]

Suggested change

	ALTER TABLE events ADD COLUMN metadata JSONB DEFAULT NULL;
	ALTER TABLE events ADD COLUMN IF NOT EXISTS metadata JSONB DEFAULT NULL;