Improve configuration for remaining provider proxies (#54)

Includes the following changes:
- Improves configuration and adds overrides for other (non-GRPC) provider proxies
- Improves the config strategy overall with some things moved to common places
- Minor fix: removes an unnecessary `Arc` usage in the provider proxies

Part of #17

Cargo.toml

@@ -22,6 +22,7 @@ members = [
   "mocks/mock_mapping_service",
   "proc_macros",
   "provider_proxies/grpc/v1",
+  "provider_proxies/http_mock_provider_proxy",
   "provider_proxies/in_memory_mock_provider_proxy",
   "provider_proxy_selector",
 ]

common/src/config_utils.rs

@@ -8,6 +8,8 @@ use config::{ConfigError, File};
 use home::home_dir;
 use serde::Deserialize;
 
+pub const JSON_EXT: &str = "json";
+
 const CONFIG_DIR: &str = "config";
 const DOT_FREYJA_DIR: &str = ".freyja";
 const FREYJA_HOME: &str = "FREYJA_HOME";

docs/config-overrides.md

@@ -0,0 +1,11 @@
+# Config Overrides
+
+Many Freyja components support configuration overrides to enable users to override default settings and provide custom configuration. This is achieved with configuration layering. Components using this configuration strategy will define a default config, which is often suitable for basic scenarios or getting started quickly. This default config can be overridden at runtime using custom values. When loading configuration, a component will probe for and unify config in the following order, with values near the end of the list taking higher precedence:
+
+- The default config
+- A config file in the working directory of the executable (for example, the directory you were in when you ran the `cargo run` command)
+- `$FREYJA_HOME/config/{config_name}.json`. If you have not set a `$FREYJA_HOME` directory, this defaults to:
+  - Unix: `$HOME/.freyja/config/{config_name}.json`
+  - Windows: `%USERPROFILE%\.freyja\config\{config_name}.json` (note that Windows support is not guaranteed by Freyja)
+
+Because the config is layered, the overrides can be partially defined and only specify the top-level configuration fields that should be overridden. Anything not specified in an override file will use the default value.

mapping_clients/in_memory_mock_mapping_client/README.md

@@ -4,24 +4,19 @@ The In-Memory Mock Mapping Client mocks the behavior of a mapping service from w
 
 ## Configuration
 
-The adapter's default config is located at `res/mock_mapping_config.default.json` and will be copied to the build output automatically. This file is a list of config objects with the following properties:
-
-- `begin`: an integer indicating when to enable the mapping value below
-- `end`: an optional integer indicating when to disable the mapping value below. Set to `null` if you never want the value to "turn off"
-- `value`: a mapping that should be emitted at some point during the application's lifetime. This has the following properties:
-  - `source`: the ID of the entity that will be used as the source for this mapping. This should match something that's retrievable with the `find_by_id` API of the digital twin adapter that you're using.
-  - `target`: a set of key-value pairs that will be passed to the cloud adapter. This is completely free-form, and will potentially be used by the cloud adapter to help with addressing the correct digital twin instance and/or properties for upstream data emissions.
-  - `interval_ms`: the interval at which the entity should be queried for changes
-  - `emit_on_change`: a value indicating whether data emission should be skipped if the value hasn't changed since the last emission. Set to true to enable this behavior.
-  - `conversion`: a conversion that should be applied. Set to null if no conversion is needed. Otherwise the conversion is configured with the `mul` and `offset` properties, and the value `y` that is emitted is calculated as `y = mul * x + offset`. Note that conversions are only supported for signal values which can be parsed as `f64`.
-
-You can override the default values by defining your own `mock_mapping_config.json`. The adapter will probe for and unify config in this order, with values near the end of the list taking higher precedence:
-
-- The default config
-- A `mock_mapping_config.json` file in the working directory of the executable (for example, the directory you were in when you ran the `cargo run` command)
-- `$FREYJA_HOME/config/mock_mapping_config.json`. If you have not set a `$FREYJA_HOME` directory, this defaults to:
-  - Unix: `$HOME/.freyja/config/mock_mapping_config.json`
-  - Windows: `%USERPROFILE%\.freyja\config\mock_mapping_config.json` (note that Windows support is not guaranteed by Freyja or this adapter)
+This adapter supports the following configuration settings:
+
+- `values`: a list of mappings to use. Each entry in the list is an object with the following properties:
+  - `begin`: an integer indicating when to enable the `value`
+  - `end`: an optional integer indicating when to disable the `value`. Set to `null` if you never want the value to "turn off"
+  - `value`: a mapping that should be emitted at some point during the application's lifetime. This has the following properties:
+    - `source`: the ID of the entity that will be used as the source for this mapping. This should match something that's retrievable with the `find_by_id` API of the digital twin adapter that you're using.
+    - `target`: a set of key-value pairs that will be passed to the cloud adapter. This is completely free-form, and will potentially be used by the cloud adapter to help with addressing the correct digital twin instance and/or properties for upstream data emissions.
+    - `interval_ms`: the interval (in milliseconds) at which the entity should be queried for changes
+    - `emit_on_change`: a boolean indicating whether data emission should be skipped if the value hasn't changed since the last emission. Set to `true` to enable this behavior.
+    - `conversion`: a conversion that should be applied. Set to `null` if no conversion is needed. Otherwise the conversion is configured with the `mul` and `offset` properties, and the value `y` that is emitted is calculated as `y = mul * x + offset`. Note that conversions are only supported for signal values which can be parsed as `f64`.
+
+This adapter supports [config overrides](../../docs/config-overrides.md). The override filename is `mock_mapping_config.json`, and the default config is located at `res/mock_mapping_config.default.json`.
 
 ## Behavior
 

mapping_clients/in_memory_mock_mapping_client/src/in_memory_mock_mapping_client.rs

@@ -11,7 +11,6 @@ use freyja_common::{config_utils, out_dir};
 use freyja_contracts::mapping_client::*;
 
 const CONFIG_FILE_STEM: &str = "mock_mapping_config";
-const CONFIG_FILE_EXT: &str = "json";
 
 /// Mocks a mapping provider in memory
 pub struct InMemoryMockMappingClient {
@@ -42,7 +41,7 @@ impl MappingClient for InMemoryMockMappingClient {
     fn create_new() -> Result<Self, MappingClientError> {
         let config = config_utils::read_from_files(
             CONFIG_FILE_STEM,
-            CONFIG_FILE_EXT,
+            config_utils::JSON_EXT,
             out_dir!(),
             MappingClientError::io,
             MappingClientError::deserialize,

mapping_clients/mock_mapping_service_client/README.md

@@ -14,16 +14,10 @@ For instructions on other operating systems, see the full documentation [here](h
 
 ## Config
 
-The adapter's default config is located at `res/mock_mapping_client_config.default.json` and will be copied to the build output automatically. This file contains the following properties:
+This adapter supports the following configuration settings:
 
 - `max_retires`: the maximum number of retries permitted when attempting to call the mock service
 - `retry_interval_ms`: the interval between subsequent retry attempts, in milliseconds
 - `mock_mapping_service_url`: the url for the Mock Mapping Service Client
 
-You can override the default values by defining your own `mock_mapping_client_config.json`. The adapter will probe for and unify config in this order, with values near the end of the list taking higher precedence:
-
-- The default config
-- A `mock_mapping_client_config.json` file in the working directory of the executable (for example, the directory you were in when you ran the `cargo run` command)
-- `$FREYJA_HOME/config/mock_mapping_client_config.json`. If you have not set a `$FREYJA_HOME` directory, this defaults to:
-  - Unix: `$HOME/.freyja/config/mock_mapping_client_config.json`
-  - Windows: `%USERPROFILE%\.freyja\config\mock_mapping_client_config.json` (note that Windows support is not guaranteed by Freyja or this adapter)
+This adapter supports [config overrides](../../docs/config-overrides.md). The override filename is `mock_mapping_client_config.json`, and the default config is located at `res/mock_mapping_client_config.default.json`.

mapping_clients/mock_mapping_service_client/src/mock_mapping_service_client.rs

@@ -12,7 +12,6 @@ use freyja_common::{config_utils, out_dir, retry_utils::execute_with_retry};
 use freyja_contracts::mapping_client::*;
 
 const CONFIG_FILE_STEM: &str = "mock_mapping_client_config";
-const CONFIG_FILE_EXT: &str = "json";
 
 /// Mocks a mapping provider in memory
 pub struct MockMappingServiceClient {
@@ -50,7 +49,7 @@ impl MappingClient for MockMappingServiceClient {
     fn create_new() -> Result<Self, MappingClientError> {
         let config = config_utils::read_from_files(
             CONFIG_FILE_STEM,
-            CONFIG_FILE_EXT,
+            config_utils::JSON_EXT,
             out_dir!(),
             MappingClientError::io,
             MappingClientError::deserialize,

mocks/mock_mapping_service/src/main.rs

@@ -27,7 +27,6 @@ use freyja_contracts::mapping_client::{
 };
 
 const CONFIG_FILE_STEM: &str = "mock_mapping_config";
-const CONFIG_FILE_EXT: &str = "json";
 
 struct MappingState {
     count: u8,
@@ -53,7 +52,7 @@ async fn main() {
 
     let config = config_utils::read_from_files(
         CONFIG_FILE_STEM,
-        CONFIG_FILE_EXT,
+        config_utils::JSON_EXT,
         out_dir!(),
         |e| log::error!("{}", e),
         |e| log::error!("{}", e),

provider_proxies/grpc/v1/README.md

@@ -4,14 +4,8 @@ The GRPC Provider Proxy interfaces with providers which support GRPC. It acts as
 
 ## Configuration
 
-This proxy's default config is located at `res/grpc_proxy_config.default.json` and will be copied to the build output automatically. This proxy supports the following configuration settings:
+This proxy supports the following configuration settings:
 
 - `consumer_address`: The address for the proxy's consumer
 
-You can override the default values by defining your own `grpc_proxy_config.json`. The adapter will probe for and unify config in this order, with values near the end of the list taking higher precedence:
-
-- The default config
-- A `grpc_proxy_config.json` file in the working directory of the executable (for example, the directory you were in when you ran the `cargo run` command)
-- `$FREYJA_HOME/config/grpc_proxy_config.json`. If you have not set a `$FREYJA_HOME` directory, this defaults to:
-  - Unix: `$HOME/.freyja/config/grpc_proxy_config.json`
-  - Windows: `%USERPROFILE%\.freyja\config\grpc_proxy_config.json` (note that Windows support is not guaranteed by Freyja or this adapter)
+This adapter supports [config overrides](../../../docs/config-overrides.md). The override filename is `grpc_proxy_config.json`, and the default config is located at `res/grpc_proxy_config.default.json`.

provider_proxies/grpc/v1/src/grpc_provider_proxy.rs

@@ -25,7 +25,6 @@ use freyja_contracts::provider_proxy::{
 };
 
 const CONFIG_FILE_STEM: &str = "grpc_proxy_config";
-const CONFIG_FILE_EXT: &str = "json";
 const SUPPORTED_OPERATIONS: &[OperationKind] = &[OperationKind::Get, OperationKind::Subscribe];
 
 /// Interfaces with providers which support GRPC. Based on the Ibeji mixed sample.
@@ -38,7 +37,7 @@ pub struct GRPCProviderProxy {
     provider_client: DigitalTwinProviderClient<Channel>,
 
     /// Local cache for keeping track of which entities this provider proxy contains
-    entity_operation_map: Arc<Mutex<HashMap<String, OperationKind>>>,
+    entity_operation_map: Mutex<HashMap<String, OperationKind>>,
 
     /// Shared queue for all proxies to push new signal values of entities
     signal_values_queue: Arc<SegQueue<SignalValue>>,
@@ -60,7 +59,7 @@ impl ProviderProxy for GRPCProviderProxy {
     {
         let config = config_utils::read_from_files(
             CONFIG_FILE_STEM,
-            CONFIG_FILE_EXT,
+            config_utils::JSON_EXT,
             out_dir!(),
             ProviderProxyError::io,
             ProviderProxyError::deserialize,
@@ -75,7 +74,7 @@ impl ProviderProxy for GRPCProviderProxy {
         Ok(GRPCProviderProxy {
             config,
             provider_client,
-            entity_operation_map: Arc::new(Mutex::new(HashMap::new())),
+            entity_operation_map: Mutex::new(HashMap::new()),
             signal_values_queue,
         })
         .map(|r| Box::new(r) as _)
@@ -316,7 +315,7 @@ mod grpc_provider_proxy_v1_tests {
                         consumer_address: "[::1]:60010".to_string(),
                     },
                     provider_client: client,
-                    entity_operation_map: Arc::new(Mutex::new(HashMap::new())),
+                    entity_operation_map: Mutex::new(HashMap::new()),
                     signal_values_queue: Arc::new(SegQueue::new()),
                 };
                 assert!(grpc_provider_proxy

provider_proxies/http_mock_provider_proxy/Cargo.toml

@@ -12,6 +12,7 @@ license = "MIT"
 async-trait = { workspace = true }
 axum = { workspace = true }
 crossbeam = { workspace = true }
+freyja-common = { workspace = true }
 freyja-contracts = { workspace = true  }
 log = { workspace = true }
 mock-digital-twin = { workspace = true }
@@ -23,3 +24,6 @@ tokio = { workspace = true }
 tokio-stream = { workspace = true }
 tonic = { workspace = true }
 tower = { workspace = true }
+
+[build-dependencies]
+freyja-build-common = { workspace = true }

provider_proxies/http_mock_provider_proxy/README.md

@@ -0,0 +1,11 @@
+# HTTP Mock Provider Proxy
+
+The HTTP Mock Provider Proxy mocks the behavior of a proxy which communicates with providers via HTTP. This is intended for use with the [Mock Digital Twin](../../mocks/mock_digital_twin/).
+
+## Configuration
+
+This proxy supports the following configuration settings:
+
+- `proxy_callback_address`: The address for the proxy. This is the address that the Mock Digital Twin will use for callbacks.
+
+This adapter supports [config overrides](../../docs/config-overrides.md). The override filename is `http_mock_proxy_config.json`, and the default config is located at `res/http_mock_proxy_config.default.json`.

provider_proxies/http_mock_provider_proxy/build.rs

@@ -2,23 +2,19 @@
 // Licensed under the MIT license.
 // SPDX-License-Identifier: MIT
 
-use std::{env, fs, path::Path};
+use std::env;
 
-const OUT_DIR: &str = "OUT_DIR";
-const RESOURCE_DIR: &str = "res";
-const CONFIG_FILE: &str = "config.json";
+use freyja_build_common::copy_to_build_out_dir;
+
+const RES_DIR_NAME: &str = "res";
+const DEFAULT_CONFIG_FILE: &str = "http_mock_proxy_config.default.json";
 
 fn main() {
     // Current directory of the build script is the package's root directory
     let config_path = env::current_dir()
         .unwrap()
-        .join(RESOURCE_DIR)
-        .join(CONFIG_FILE);
-
-    let target_dir = env::var(OUT_DIR).unwrap();
-    let dest_path = Path::new(&target_dir).join(CONFIG_FILE);
-
-    fs::copy(&config_path, dest_path).unwrap();
+        .join(RES_DIR_NAME)
+        .join(DEFAULT_CONFIG_FILE);
 
-    println!("cargo:rerun-if-changed={}", config_path.to_str().unwrap());
+    copy_to_build_out_dir(config_path, DEFAULT_CONFIG_FILE);
 }

provider_proxies/http_mock_provider_proxy/res/http_mock_proxy_config.default.json

@@ -0,0 +1,3 @@
+{
+    "proxy_callback_address": "127.0.0.1:8801"
+}

provider_proxies/http_mock_provider_proxy/src/config.rs

@@ -4,12 +4,9 @@
 
 use serde::{Deserialize, Serialize};
 
-pub(crate) const CONFIG_FILE: &str = "config.json";
-pub(crate) const CALLBACK_FOR_VALUES_PATH: &str = "/value";
-
-/// Settings for http provider proxy
+/// Config for the http mock provider proxy
 #[derive(Clone, Debug, Serialize, Deserialize)]
-pub(crate) struct Settings {
-    /// the callback server authority for receiving signals from the mock digital twin
-    pub provider_callback_authority: String,
+pub(crate) struct Config {
+    /// The callback address for receiving signals from the mock digital twin
+    pub proxy_callback_address: String,
 }