Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/generalize schema validation #6

Merged
merged 3 commits into from
Feb 29, 2024
Merged

Feature/generalize schema validation #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
de50788
feat: (DHEI-15120) - Introduce clean schemastore usage
chrisingenhaag Feb 28, 2024
b9d57c5
feat: (DHEI-15120) - Finalize open-source preparations
chrisingenhaag Feb 29, 2024
1d72d9b
feat: (DHEI-15120) - Add gradle property option to inject schemaStore…
chrisingenhaag Feb 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,33 @@ For the optimal setup, ensure you have:

- A running instance of Kafka
- Access to a Kubernetes cluster on which the `Subscription` (subscriber.horizon.telekom.de) custom resource definition has been registered
- A running instance of ENI API, which has access to the resources of the Telekom Integration Platform control plane

## Configuration
Starlight configuration is managed through environment variables. Check the [complete list](docs/environment-variables.md) of supported environment variables for setup instructions.

### Schema validation
Starlight basically supports schema validation for incoming events. Unfortunately this part of starlight is not yet ready for open-source.

If you can´t wait for this, you´re able to provide your own implementation. Basically this is done via a valid bean instance of the [SchemaStore](https://github.com/telekom/pubsub-horizon-spring-parent/blob/main/horizon-core/src/main/java/de/telekom/eni/pandora/horizon/schema/SchemaStore.java) interface from [horizon-pubsub-spring-parent](https://github.com/telekom/pubsub-horizon-spring-parent) module.
You are able to create your own spring-boot-starter with a custom implementation and use the environment variable `ADDITIONAL_SCHEMASTORE_IMPL` to build with your custom artifact and use the following configuration or make use of the available environment variables for these properties.

```yaml
starlight:
features:
schemaValidation: true

# only used for schema validation (not yet possible as OSS for starlight)
eniapi:
baseurl: https://api.example.com/
refreshInterval: 60000

# only used for schema validation (not yet possible as OSS for starlight)
oidc:
issuerUrl: https://oauth.example.com/
clientId: foo
clientSecret: bar
```

## Running Starlight
### Locally
Before you can run Starlight locally you must have a running instance of Kafka and ENI API locally or forwarded from a remote cluster.
Expand Down
35 changes: 14 additions & 21 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,9 @@
import org.springframework.boot.gradle.plugin.SpringBootPlugin

plugins {
id 'org.springframework.boot' version '3.2.2'
id 'org.springframework.boot' version '3.2.3'
id 'io.spring.dependency-management' version '1.1.4'
id 'java'
id "org.sonarqube" version "4.4.1.3373"
id "jacoco"
}

Expand Down Expand Up @@ -44,27 +43,13 @@ configurations {
}
}

ext {
repoBase = System.getenv("PANDORA_ARTIFACTORY_GRADLE_CONTEXT") ?: "$artifactory_contextUrl"
repositories {
mavenCentral()
}

repositories {
maven {
credentials {
username System.getenv('PANDORA_ARTIFACTORY_GRADLE_USERNAME') ?: "$artifactory_user"
password System.getenv('PANDORA_ARTIFACTORY_GRADLE_PASSWORD') ?: "$artifactory_password"
}
url "$repoBase/maven-central"
}
maven {
credentials {
username System.getenv('PANDORA_ARTIFACTORY_GRADLE_USERNAME') ?: "$artifactory_user"
password System.getenv('PANDORA_ARTIFACTORY_GRADLE_PASSWORD') ?: "$artifactory_password"
}
url "$repoBase/dhei-pandora-gradle"
}
maven { url 'https://repo.spring.io/milestone' }
mavenLocal()
ext {
// this is internal, overwrite with your specific implementation if you want
schemalValidationImplDependency = System.getenv('ADDITIONAL_SCHEMASTORE_IMPL') ?: System.getProperty("schemaStoreImplMavenDependency") ?: ""
}

dependencies {
Expand All @@ -82,6 +67,9 @@ dependencies {
testImplementation 'org.springframework.kafka:spring-kafka-test'
testImplementation 'org.springframework.security:spring-security-test'

testImplementation 'commons-codec:commons-codec:1.16.1'


// 3rd party
implementation 'com.fasterxml.jackson.core:jackson-databind'
implementation ("com.github.erosb:everit-json-schema:${everitJsonVersion}") {
Expand All @@ -97,6 +85,11 @@ dependencies {
// Telekom Integration Platform
implementation "de.telekom.eni:horizon-spring-boot-starter:${horizonParentVersion}"

// optional
if (!schemalValidationImplDependency.allWhitespace) {
implementation "${schemalValidationImplDependency}"
}

testImplementation 'org.springframework.boot:spring-boot-testcontainers'
testImplementation 'org.testcontainers:junit-jupiter'
}
Expand Down
20 changes: 7 additions & 13 deletions docs/environment-variables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,28 +15,22 @@ Starlight is configured using environment variables. The following are supported
| ZIPKIN_SAMPLER_PROBABILITY | 1.0 | |
| STARLIGHT_INFORMER_NAMESPACE | playground | The Kubernetes namespace from which the EventSubscription CRD is being polled |
| STARLIGHT_FEATURE_PUBLISHER_CHECK | true | Enable ownership verification for published events |
| STARLIGHT_FEATURE_SCHEMA_VALIDATION | true | Enable schema validation for published events |
| STARLIGHT_HEADER_PROPAGATION_BLACKLIST | x-spacegate-token,authorization,content-length,host,accept.*,x-forwarded.*,cookie | A list of headers that will not be forwarded in the published event |
| STARLIGHT_ISSUER_URL | http://localhost:8080/auth/realms/default | The issuer(s) that are trusted by Starlight |
| STARLIGHT_DEFAULT_ENVIRONMENT | integration | The default environment that is used for multi-tenancy |
| STARLIGHT_PUBLISHING_TOPIC | published | The Kafka topic where events will be published |
| STARLIGHT_PUBLISHING_TIMEOUT_MS | 5000 | The timeout used when publishing events to Kafka |
| ENIAPI_BASEURL | localhost:8080 | Base URL of the ENI-API (used for polling event schemas) |
| ENIAPI_REFRESHINTERVAL | 30000 | How often new schemas will be polled from the ENI-API |
| IRIS_ISSUER_URL | https://iris.example.com/auth/realms/default/protocol/openid-connect/token | The issuer that is used to retrieve a token when calling ENI-API |
| CLIENT_ID | foo | Client ID that is used to retrieve a token when calling ENI-API |
| CLIENT_SECRET | bar | Client secret that is used to retrieve a token when calling ENI-API |
| VICTORIALOG_ENABLED | false | |
| VICTORIALOG_COLLECTOR_URL | http://localhost:8428 | |
| VICTORIALOG_CLIENT_ID | starlight | |
| VICTORIALOG_BATCH_SIZE | 128 | |
| VICTORIALOG_OBSERVATION_FLUSH_INTERVAL | 30000 | |
| VICTORIALOG_COUNT_EVENTS_INTERVAL | 1000 | |
| VICTORIALOG_SAMPLING_RATE | 1.0 | |
| STARLIGHT_KAFKA_BROKERS | kafka:9092 | The Kafka broker that is used for publishing events |
| STARLIGHT_KAFKA_TRANSACTION_PREFIX | starlight | The transaction-prefix that is used for publishing events |
| STARLIGHT_KAFKA_GROUP_ID | starlight | The Kafka consumer group that is used for publishing events |
| STARLIGHT_KAFKA_LINGER_MS | 5 | How long the Kafka waits for other records before transmissing the batch ([Reference](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#linger-ms)) |
| STARLIGHT_KAFKA_ACKS | 1 | How often the events needs to be acknowledge by Kafka |
| STARLIGHT_KAFKA_COMPRESSION_ENABLED | false | If events send to Kafka should be compressed |
| STARLIGHT_KAFKA_COMPRESSION_TYPE | none | The compression type used to compress events |
| STARLIGHT_FEATURE_SCHEMA_VALIDATION | false | Enable schema validation for published events |
| ENIAPI_BASEURL | localhost:8080 | Base URL of the SchemaStore endpoint (used for polling event schemas) |
| ENIAPI_REFRESHINTERVAL | 30000 | How often new schemas will be polled from the SchemaStore |
| IRIS_ISSUER_URL | https://iris.example.com/auth/realms/default/protocol/openid-connect/token | The issuer that is used to retrieve a token when calling SchemaStore endpoint |
chrisingenhaag marked this conversation as resolved.
Show resolved Hide resolved
| CLIENT_ID | foo | Client ID that is used to retrieve a token when calling SchemaStore endpoint |
| CLIENT_SECRET | bar | Client secret that is used to retrieve a token when calling SchemaStore endpoint |

2 changes: 1 addition & 1 deletion gradle.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
everitJsonVersion=1.14.4
horizonParentVersion=3.2.1
horizonParentVersion=4.0.3
18 changes: 1 addition & 17 deletions settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,8 @@
// SPDX-License-Identifier: Apache-2.0

pluginManagement {
ext {
repoBase = System.getenv("PANDORA_ARTIFACTORY_GRADLE_CONTEXT") ?: "$artifactory_contextUrl"
}
repositories {
maven {
credentials {
username System.getenv('PANDORA_ARTIFACTORY_GRADLE_USERNAME') ?: "$artifactory_user"
password System.getenv('PANDORA_ARTIFACTORY_GRADLE_PASSWORD') ?: "$artifactory_password"
}
url "$repoBase/maven-central"
}
maven {
credentials {
username System.getenv('PANDORA_ARTIFACTORY_GRADLE_USERNAME') ?: "$artifactory_user"
password System.getenv('PANDORA_ARTIFACTORY_GRADLE_PASSWORD') ?: "$artifactory_password"
}
url "$repoBase/gradle-plugins-cache/"
}
mavenCentral()
}
}

Expand Down
28 changes: 0 additions & 28 deletions src/main/java/de/telekom/horizon/starlight/api/EventController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,6 @@

import de.telekom.eni.pandora.horizon.model.event.Event;
import de.telekom.eni.pandora.horizon.tracing.HorizonTracer;
import de.telekom.eni.pandora.horizon.victorialog.client.VictoriaLogClient;
import de.telekom.eni.pandora.horizon.victorialog.model.HTTPHeader;
import de.telekom.eni.pandora.horizon.victorialog.model.Observation;
import de.telekom.horizon.starlight.exception.HorizonStarlightException;
import de.telekom.horizon.starlight.service.PublisherService;
import de.telekom.horizon.starlight.service.TokenService;
Expand Down Expand Up @@ -37,20 +34,16 @@ public class EventController {

private final HorizonTracer tracer;

private final VictoriaLogClient victoriaLogClient;

private final ReportingService reportingService;

@Autowired
EventController(TokenService tokenService,
PublisherService publisherService,
HorizonTracer tracer,
VictoriaLogClient victoriaLogClient,
ReportingService reportingService) {
this.tokenService = tokenService;
this.publisherService = publisherService;
this.tracer = tracer;
this.victoriaLogClient = victoriaLogClient;
this.reportingService = reportingService;
}

Expand All @@ -63,12 +56,6 @@ public ResponseEntity<Void> headRequest(@PathVariable String environment) {
public ResponseEntity<Event> publishEvent(@RequestBody Event event,
@PathVariable String environment,
@RequestHeader MultiValueMap<String, String> httpHeaders) throws HorizonStarlightException {


var observation = createObservationFromEvent(event);

httpHeaders.add(HTTPHeader.TRACK_LATENCY.getValue(), observation.isEmpty() ? "0": "1");

addTracingTags(event);

publisherService.checkRealm(tokenService.getRealm(), environment);
Expand All @@ -78,24 +65,9 @@ public ResponseEntity<Event> publishEvent(@RequestBody Event event,

reportingService.markEventProduced(event);

observation.ifPresent(v -> {
victoriaLogClient.finishAndAddObservation(v);
victoriaLogClient.countEvent(event);
});

return ResponseEntity.status(HttpStatus.CREATED).body(null);
}

private Optional<Observation> createObservationFromEvent(Event event) {
Observation observation = null;
if (victoriaLogClient.shouldStartTrackingLatency()) {
observation = victoriaLogClient.startObservationFromEvent(event);

}

return Optional.ofNullable(observation);
}

private void addTracingTags(Event event) {
var currentSpan = Optional.ofNullable(tracer.getCurrentSpan());

Expand Down
Loading
Loading