Skip to content

Commit

Permalink
feat: (DHEI-15120) - Finalize open-source preparations
Browse files Browse the repository at this point in the history
  • Loading branch information
@chrisingenhaag
chrisingenhaag committed Feb 29, 2024
1 parent de50788 commit b9d57c5
Show file tree
Hide file tree
Showing 6 changed files with 38 additions and 19 deletions.
24 changes: 23 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,33 @@ For the optimal setup, ensure you have:

- A running instance of Kafka
- Access to a Kubernetes cluster on which the `Subscription` (subscriber.horizon.telekom.de) custom resource definition has been registered
- A running instance of ENI API, which has access to the resources of the Telekom Integration Platform control plane

## Configuration
Starlight configuration is managed through environment variables. Check the [complete list](docs/environment-variables.md) of supported environment variables for setup instructions.

### Schema validation
Starlight basically supports schema validation for incoming events. Unfortunately this part of starlight is not yet ready for open-source.

If you can´t wait for this, you´re able to provide your own implementation. Basically this is done via a valid bean instance of the [SchemaStore](https://github.com/telekom/pubsub-horizon-spring-parent/blob/main/horizon-core/src/main/java/de/telekom/eni/pandora/horizon/schema/SchemaStore.java) interface from [horizon-pubsub-spring-parent](https://github.com/telekom/pubsub-horizon-spring-parent) module.
You are able to create your own spring-boot-starter with a custom implementation and use the environment variable `ADDITIONAL_SCHEMASTORE_IMPL` to build with your custom artifact and use the following configuration or make use of the available environment variables for these properties.

```yaml
starlight:
features:
schemaValidation: true

# only used for schema validation (not yet possible as OSS for starlight)
eniapi:
baseurl: https://api.example.com/
refreshInterval: 60000

# only used for schema validation (not yet possible as OSS for starlight)
oidc:
issuerUrl: https://oauth.example.com/
clientId: foo
clientSecret: bar
```
## Running Starlight
### Locally
Before you can run Starlight locally you must have a running instance of Kafka and ENI API locally or forwarded from a remote cluster.
Expand Down
3 changes: 0 additions & 3 deletions build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,6 @@ dependencies {
dependencyManagement {
imports {
mavenBom SpringBootPlugin.BOM_COORDINATES
if (!schemalValidationImplDependency.allWhitespace) {
mavenBom "${schemalValidationImplDependency}"
}
}
}

Expand Down
20 changes: 7 additions & 13 deletions docs/environment-variables.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,28 +15,22 @@ Starlight is configured using environment variables. The following are supported
| ZIPKIN_SAMPLER_PROBABILITY | 1.0 | |
| STARLIGHT_INFORMER_NAMESPACE | playground | The Kubernetes namespace from which the EventSubscription CRD is being polled |
| STARLIGHT_FEATURE_PUBLISHER_CHECK | true | Enable ownership verification for published events |
| STARLIGHT_FEATURE_SCHEMA_VALIDATION | true | Enable schema validation for published events |
| STARLIGHT_HEADER_PROPAGATION_BLACKLIST | x-spacegate-token,authorization,content-length,host,accept.*,x-forwarded.*,cookie | A list of headers that will not be forwarded in the published event |
| STARLIGHT_ISSUER_URL | http://localhost:8080/auth/realms/default | The issuer(s) that are trusted by Starlight |
| STARLIGHT_DEFAULT_ENVIRONMENT | integration | The default environment that is used for multi-tenancy |
| STARLIGHT_PUBLISHING_TOPIC | published | The Kafka topic where events will be published |
| STARLIGHT_PUBLISHING_TIMEOUT_MS | 5000 | The timeout used when publishing events to Kafka |
| ENIAPI_BASEURL | localhost:8080 | Base URL of the ENI-API (used for polling event schemas) |
| ENIAPI_REFRESHINTERVAL | 30000 | How often new schemas will be polled from the ENI-API |
| IRIS_ISSUER_URL | https://iris.example.com/auth/realms/default/protocol/openid-connect/token | The issuer that is used to retrieve a token when calling ENI-API |
| CLIENT_ID | foo | Client ID that is used to retrieve a token when calling ENI-API |
| CLIENT_SECRET | bar | Client secret that is used to retrieve a token when calling ENI-API |
| VICTORIALOG_ENABLED | false | |
| VICTORIALOG_COLLECTOR_URL | http://localhost:8428 | |
| VICTORIALOG_CLIENT_ID | starlight | |
| VICTORIALOG_BATCH_SIZE | 128 | |
| VICTORIALOG_OBSERVATION_FLUSH_INTERVAL | 30000 | |
| VICTORIALOG_COUNT_EVENTS_INTERVAL | 1000 | |
| VICTORIALOG_SAMPLING_RATE | 1.0 | |
| STARLIGHT_KAFKA_BROKERS | kafka:9092 | The Kafka broker that is used for publishing events |
| STARLIGHT_KAFKA_TRANSACTION_PREFIX | starlight | The transaction-prefix that is used for publishing events |
| STARLIGHT_KAFKA_GROUP_ID | starlight | The Kafka consumer group that is used for publishing events |
| STARLIGHT_KAFKA_LINGER_MS | 5 | How long the Kafka waits for other records before transmissing the batch ([Reference](https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#linger-ms)) |
| STARLIGHT_KAFKA_ACKS | 1 | How often the events needs to be acknowledge by Kafka |
| STARLIGHT_KAFKA_COMPRESSION_ENABLED | false | If events send to Kafka should be compressed |
| STARLIGHT_KAFKA_COMPRESSION_TYPE | none | The compression type used to compress events |
| STARLIGHT_FEATURE_SCHEMA_VALIDATION | false | Enable schema validation for published events |
| ENIAPI_BASEURL | localhost:8080 | Base URL of the SchemaStore endpoint (used for polling event schemas) |
| ENIAPI_REFRESHINTERVAL | 30000 | How often new schemas will be polled from the SchemaStore |
| IRIS_ISSUER_URL | https://iris.example.com/auth/realms/default/protocol/openid-connect/token | The issuer that is used to retrieve a token when calling SchemaStore endpoint |
| CLIENT_ID | foo | Client ID that is used to retrieve a token when calling SchemaStore endpoint |
| CLIENT_SECRET | bar | Client secret that is used to retrieve a token when calling SchemaStore endpoint |

2 changes: 1 addition & 1 deletion gradle.properties
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
everitJsonVersion=1.14.4
horizonParentVersion=0.0.0-main-SNAPSHOT
horizonParentVersion=4.0.3
2 changes: 2 additions & 0 deletions src/main/java/de/telekom/horizon/starlight/cache/SchemaCacheUpdateService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@
import de.telekom.horizon.starlight.config.StarlightConfig;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
@ConditionalOnProperty(name = "starlight.features.schemaValidation", havingValue = "true")
@Slf4j
public class SchemaCacheUpdateService {

Expand Down
6 changes: 5 additions & 1 deletion src/main/resources/application.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ spring:
data:
redis:
url: ${STARLIGHT_REPORTING_REDIS_URL}
main:
banner-mode: off
logging:
level:
root: ${LOG_LEVEL:INFO}
Expand Down Expand Up @@ -48,7 +50,7 @@ kubernetes:
starlight:
features:
publisherCheck: ${STARLIGHT_FEATURE_PUBLISHER_CHECK:true}
schemaValidation: ${STARLIGHT_FEATURE_SCHEMA_VALIDATION:true}
schemaValidation: ${STARLIGHT_FEATURE_SCHEMA_VALIDATION:false}
security:
# Must be lower-case and comma-separated, can be regex
headerPropagationBlacklist: ${STARLIGHT_HEADER_PROPAGATION_BLACKLIST:x-spacegate-token,authorization,content-length,host,accept.*,x-forwarded.*,cookie}
Expand All @@ -64,10 +66,12 @@ starlight:
redis:
enabled: ${STARLIGHT_REPORTING_REDIS_ENABLED:false}

# only used for schema validation (not yet possible as OSS for starlight)
eniapi:
baseurl: ${ENIAPI_BASEURL:localhost:8080}
refreshInterval: ${ENIAPI_REFRESHINTERVAL:30000}

# only used for schema validation (not yet possible as OSS for starlight)
oidc:
issuerUrl: ${IRIS_ISSUER_URL:https://iris.example.com/auth/realms/default/protocol/openid-connect/token}
clientId: ${CLIENT_ID:foo}
Expand Down

0 comments on commit b9d57c5

Please sign in to comment.