Feature Request: Support for Configurable Liveness and Readiness Probes in Astarte Custom Resources (CRs)

[jingwenzz]

Description It would be highly beneficial to have the ability to configure livenessProbe and readinessProbe settings directly within the Astarte Custom Resource (CR). Currently, these settings are managed internally by the operator, with no option for customization. This limitation can lead to challenges in tailoring health check configurations to suit specific deployment environments and workloads.

Use Case Customizing probes can:

• Allow fine-tuning of initialDelaySeconds, timeoutSeconds, failureThreshold, etc., for components running in diverse infrastructure conditions.
• Help avoid false positives or unnecessary restarts in environments with slower response times or transient load spikes.
• Improve observability and operational control by aligning health checks with application-specific requirements.

Proposed Solution Introduce support for customizable livenessProbe and readinessProbe settings in the Astarte CR. This could be implemented as optional fields in the CR for each Astarte component (e.g., appengineApi, housekeepingApi, etc.).

Example:

components:
  appengineApi:
    livenessProbe:
      httpGet:
        path: /health
        port: 4000
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 20
      timeoutSeconds: 10
      failureThreshold: 5
    readinessProbe:
      httpGet:
        path: /health
        port: 4000
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 20
      timeoutSeconds: 10
      failureThreshold: 5

Impact This feature would:

• Provide greater flexibility in managing health checks for Astarte deployments.
• Reduce operator intervention by eliminating the need to manually patch deployments post-deployment.
• Improve the stability and reliability of Astarte clusters in diverse environments.
• Additional Context This request aligns with best practices for Kubernetes deployments, where configurable probes are commonly used to adapt applications to their specific operational contexts.

Thank you for considering this feature request. Please let me know if any additional details or use cases are required.

[Annopaolo]

Hello @jingwenzz, thank you for your suggestion! It is indeed a nice feature to have.
If you would like to contribute we can provide guidance and support, otherwise we'll keep you updated when we add it!

[jingwenzz]

Thank you for the response @Annopaolo, I’d be happy to contribute to this feature. Please let me know how I can get started or any guidelines I should follow. Looking forward to your guidance and support!

[matt-mazzucato]

Hi @jingwenzz and thanks for being open to contributing!

Here you are some suggestions to get the job done.

1. Add new fields to the Astarte CRD. You can find its definition in astarte_types.go (both in the v1alpha2 and v1alpha3 packages)

• To allow users to customize probes, we recommend adding a livenessProbe, a readinessProbe and a startupProbe to the AstarteGenericClusteredResource;
• Each new probe shall be of type *v1.Probe.

2. Adapt the reconciliation logic defined in EnsureAstarteGenericBackend.
   Here, the signature of the EnsureAstarteGenericBackendWithCustomProbe function is not well defined and we suggest to:

• Move the overall content of EnsureAstarteGenericBackendWithCustomProbe into the EnsureAstarteGenericBackend function. The EnsureAstarteGenericBackendWithCustomProbe function shall be removed;
• What is currently called customProbe shall be replaced by the liveness, readiness and startup probes as defined in point 1;
• In case no custom probes are defined, the current behavior must be used: i.e. all the pods must have a readiness and liveness probe as defined in getAstarteBackendProbe (a startup probe is not currently defined).

Please, use the master branch as the target for your changes and do not forget to add an entry to the changelog.

Hope it helps. Of course do not hesitate to ask for any questions if needed. 😃