Merge pull request #3962 from esl/c2s-docs

C2S docs

Update documentation after introducing mongose_c2s.
The migration guide will be updated in a separate PR.

Main changes:

    New routing docs and diagrams, since the old ones were outdated. It was difficult to get any good-looking output from plantuml, so it was replaced with Google docs. The docs were looking bad in night mode, so a separate dark-mode version is used.
    Hook description is updated and unified. There were multiple updates since this doc was out of date.
    Missing configuration options are described.
    Hook API modules moved to src/hooks to make them easier to find and automatically grep. Another reason was naming - we had confusing modules like mongoose_metric_hooks, which contains handlers, not hooks. The rename is the simplest change, and I wanted to avoid code changes in this PR.

doc/configuration/TLS-hardening.md

@@ -17,7 +17,7 @@ None of them is strictly better than the other.
 Below you may find a summary of the differences between them.
 
 * `fast_tls` is faster
-* There are options that OTP TLS (a.k.a `just_tls` in the `ejabberd_c2s` configuration) supports exclusively:
+* There are options that OTP TLS (a.k.a `just_tls` in the C2S listener configuration) supports exclusively:
     * Immediate connection drop when the client certificate is invalid
     * Certificate Revocation Lists
     * More flexible certificate verification options
@@ -49,7 +49,7 @@ The remaining valid values are: `'tlsv1.1'`, `tlsv1`, `sslv3`.
 
 This setting affects the following MongooseIM components:
 
-* Raw XMPP over TCP connections, if `ejabberd_c2s` listener is configured to use `just_tls`
+* Raw XMPP over TCP connections, if a C2S listener is configured to use `just_tls`
 * All outgoing connections (databases, AMQP, SIP etc.)
 * HTTP endpoints
 

doc/configuration/general.md

@@ -170,18 +170,10 @@ If a stanza is addressed to a subdomain of the served domain and this option is
 
 ### `general.routing_modules`
 * **Syntax:** a list of strings representing the routing module names.
-* **Default:** `["mongoose_router_global", "mongoose_router_localdomain", "mongoose_router_external_localnode", "mongoose_router_external", "ejabberd_s2s"]`
+* **Default:** `["mongoose_router_global", "mongoose_router_localdomain", "mongoose_router_external_localnode", "mongoose_router_external", "mongoose_router_dynamic_domains", "ejabberd_s2s"]`
 * **Example:** `routing_modules = ["mongoose_router_global", "mongoose_router_localdomain"]`
 
-Provides an ordered list of modules used for routing messages. If one of the modules accepts packet for processing, the remaining ones are not called.
-
-Allowed module names:
-
-* `mongoose_router_global` - calls the `filter_packet` hook.
-* `mongoose_router_localdomain` - routes packets addressed to a domain supported by the local cluster.
-* `mongoose_router_external_localnode` - delivers packets to an XMPP component connected to the node, which processes the request.
-* `mongoose_router_external` - delivers packets to an XMPP component connected to the local cluster.
-* `ejabberd_s2s` - forwards packets to another XMPP cluster over XMPP Federation.
+Provides an ordered list of modules used for routing messages. All available modules are enabled by default, and you can change their order or disable some of them by providing your own list. See the [Message routing](../../developers-guide/Stanza-routing/#3-message-routing) section of the developer's guide for more information.
 
 ## Miscellaneous
 

doc/configuration/listen.md

@@ -68,7 +68,7 @@ Overrides the default TCP backlog value.
 When set to `true`, [Proxy Protocol](https://www.haproxy.com/blog/haproxy/proxy-protocol/) is enabled and each connecting client has to provide a proxy header. Use only with a proxy (or a load balancer) to allow it to provide the connection details (including the source IP address) of the original client. Versions 1 and 2 of the protocol are supported.
 
 ### `listen.*.hibernate_after`
-* **Syntax:** non-negative integer
+* **Syntax:** non-negative integer or the string `"infinity"`
 * **Default:** `0`
 * **Example:** `hibernate_after = 10`
 
@@ -92,14 +92,6 @@ Maximum allowed incoming stanza size in bytes.
 
 The number of processes accepting new connections on the listening socket.
 
-### `listen.*.max_fsm_queue`
-* **Syntax:** positive integer
-* **Default:** not set - no limit
-* **Example:** `max_fsm_queue = 1000`
-
-Message queue limit to prevent resource exhaustion; overrides the value set in the `general` section.
-This option does **not** work for `s2s` listeners - the `general` value is used for them.
-
 ## Client-to-server (C2S): `[[listen.c2s]]`
 
 Handles XMPP client-to-server (C2S) connections.
@@ -122,16 +114,30 @@ The rule that determines what traffic shaper is used to limit the incoming XMPP
 The rule referenced here needs to be defined in the `access` configuration section.
 The value of the access rule needs to be either the shaper name or the string `"none"`, which means no shaper.
 
+### `listen.c2s.max_connections`
+* **Syntax:** positive integer or the string `"infinity"`
+* **Default:** `"infinity"`
+* **Example:** `max_connections = 10000`
+
+Maximum number of open connections. This is a *soft limit* according to the [Ranch](https://ninenines.eu/docs/en/ranch/2.1/manual/ranch) documentation.
+
+### `listen.c2s.c2s_state_timeout`
+* **Syntax:** non-negative integer or the string `"infinity"`
+* **Default:** `5000`
+* **Example:** `c2s_state_timeout = 10_000`
+
+Timeout value (in milliseconds) used by the C2S state machine when waiting for the connecting client to respond during stream negotiation and SASL authentication. After the timeout the server responds with the `connection-timeout` stream error and closes the connection.
+
 ### `listen.c2s.reuse_port`
 * **Syntax:** boolean
-* **Default:** false
+* **Default:** `false`
 * **Example:** `reuse_port = true`
 
 Enables linux support for `SO_REUSEPORT`, see [https://stackoverflow.com/questions/14388706/how-do-so-reuseaddr-and-so-reuseport-differ] for more details.
 
 ### `listen.c2s.backwards_compatible_session`
 * **Syntax:** boolean
-* **Default:** true
+* **Default:** `true`
 * **Example:** `backwards_compatible_session = false`
 
 Enables backward-compatible session establishement IQs. See https://www.rfc-editor.org/rfc/rfc6121.html#section-1.4:
@@ -284,7 +290,7 @@ The following section configures two C2S listeners.
 * One at port 5222, which accepts a plain TCP connection and allows to use StartTLS for upgrading it to an encrypted one. The files containing the certificate and the DH parameter are also provided.
 * One at port 5223, which accepts only encrypted TLS connections - this is the legacy method as StartTLS is preferred.
 
-Both listeners use ZLIB and the `c2s` and `c2s_shaper` rules for access management and traffic shaping, respectively.
+Both listeners use `c2s` and `c2s_shaper` rules for access management and traffic shaping, respectively.
 
 ## Server-to-server (S2S): `[[listen.s2s]]`
 
@@ -388,6 +394,13 @@ By default, when a component tries to connect and a registration conflict occurs
 It makes implementing the reconnection logic difficult, because the old connection would not allow any other connections.
 By setting this option to `kick_old`, we drop any old connections registered at the same host before accepting new ones.
 
+### `listen.service.max_fsm_queue`
+* **Syntax:** positive integer
+* **Default:** not set - no limit
+* **Example:** `max_fsm_queue = 1000`
+
+Message queue limit to prevent resource exhaustion; overrides the value set in the `general` section.
+
 ### Custom extension to the protocol
 
 In order to register a component for all virtual hosts served by the server (see `hosts` in the `general` section), the component must add the attribute `is_subdomain="true"` to the opening stream element.
@@ -469,7 +482,7 @@ Websocket connections as defined in [RFC 7395](https://tools.ietf.org/html/rfc73
 You can pass the following optional parameters:
 
 #### `listen.http.handlers.mod_websockets.timeout`
-* **Syntax:** positive integer or the string `"infinity"`
+* **Syntax:** non-negative integer or the string `"infinity"`
 * **Default:** `"infinity"`
 * **Example:** `timeout = 60_000`
 
@@ -480,14 +493,14 @@ The time (in milliseconds) after which an inactive user is disconnected.
 * **Default:** not set - pings disabled
 * **Example:** `ping_rate = 10_000`
 
-The time between pings sent by server. By setting this option you enable server-side pinging.
+The time (in milliseconds) between pings sent by server. By setting this option you enable server-side pinging.
 
 #### `listen.http.handlers.mod_websockets.max_stanza_size`
 * **Syntax:** positive integer or the string `"infinity"`
 * **Default:** `"infinity"`
 * **Example:** `max_stanza_size = 10_000`
 
-Maximum allowed incoming stanza size.
+Maximum allowed incoming stanza size in bytes.
 !!! Warning
     This limit is checked **after** the input data parsing, so it does not apply to the input data size itself.
 
@@ -506,6 +519,14 @@ Maximum allowed incoming stanza size.
 This subsection enables external component connections over WebSockets.
 See the [service](#xmpp-components-listenservice) listener section for details.
 
+#### `listen.http.handlers.mod_websockets.c2s_state_timeout`
+
+Same as the [C2S option](#listenc2sc2s_state_timeout)
+
+#### `listen.http.handlers.mod_websockets.backwards_compatible_session`
+
+Same as the [C2S option](#listenc2sbackwards_compatible_session)
+
 ### Handler types: GraphQL API - `mongoose_graphql_handler`
 
 For more information about the API, see the [Admin interface](../graphql-api/Admin-GraphQL.md) and [User interface](../graphql-api/User-GraphQL.md) documentation.
@@ -608,7 +629,7 @@ Number of HTTP connection acceptors.
 * **Default:** `1024`
 * **Example:** `transport.max_connections = "infinity"`
 
-Maximum number of open connections. The default value of 1024 is set by the [Ranch](https://ninenines.eu/docs/en/ranch/1.7/guide/) library.
+Maximum number of open connections. The default value of 1024 is set by the [Ranch](https://ninenines.eu/docs/en/ranch/2.1/guide/) library.
 
 ### TLS (HTTPS) options
 

doc/developers-guide/Hooks-and-handlers.md

@@ -29,7 +29,8 @@ To avoid that coupling and also to enable other ([possibly yet to be written](#s
 mongoose_hooks:offline_message_hook(Acc, From, To, Packet);
 ```
 
-`mongoose_hooks` is a module which serves as an API for calling all hooks in the server.
+`mongoose_hooks` is a module which serves as an API for calling hooks in the server. All such modules are placed in `src/hooks`.
+
 For every hook, there needs to be a function in this module written beforehand which accepts the correct arity of arguments and makes the call to actual low-level hooks mechanism.
 This means that there is some degree of coupling still - but this time between the `ejabberd_sm` module and `mongoose_hooks`, and the latter is always available.
 
@@ -46,10 +47,13 @@ This depends on which handlers are registered to process the event.
 
     This was actually the case before this module was introduced, and hooks' names were just atoms provided as an argument to this low-level API.
     However, we discovered it was causing problems and producing bugs, due to the lack of static code analysis.
-    Now we can have some guarantees thanks to Dialyzer, and each hook invocation has correct number of arguments.
+    Now we can have some guarantees thanks to Dialyzer, and each hook invocation has a correct number of arguments.
     Thanks to this, writing handlers is easier - there is a single source of truth about how a hook is run.
     Remember that a given hook can be invoked from many places in many modules.
 
+    With the new `mongoose_c2s` implementation we introduced a new hook API module, `mongoose_c2s_hooks`.
+    All such API modules are placed in the `src/hooks` directory.
+
 ### Getting results from handlers
 
 Hook handlers are called by "folding".
@@ -73,7 +77,7 @@ The initial value of the accumulator being passed through the sequence of handle
 ### Using accumulators
 
 MongooseIM uses a dedicated data structure to accumulate data related to stanza processing (see ["Accumulators"](accumulators.md)).
-It is instantiated with an incoming stanza, passed along throughout the processing chain, supplied to and returned from certain hook calls, and terminated when stanza is leaving MongooseIM.
+It is instantiated with an incoming stanza, passed along throughout the processing chain, supplied to and returned from certain hook calls, and terminated when the stanza is leaving MongooseIM.
 There are some hooks which don't use this data structure.
 
 If a Mongoose accumulator is passed to a hook, handlers should store their return values in one of 3 ways:
@@ -82,10 +86,10 @@ If a Mongoose accumulator is passed to a hook, handlers should store their retur
 * If the value is to be passed on to be reused within the current processing context, use `mongoose_acc:set(Namespace, Key, Value, Acc)`.
 * If the value should be passed on to the recipient's session, pubsub node etc. use `mongoose_acc:set_permanent(Namespace, Key, Value, Acc)`.
 
-A real life example, then, with regard to `mod_offline` is the `resend_offline_messages_hook` run in `ejabberd_c2s`:
+A real life example, then, with regard to `mod_offline` is the `resend_offline_messages_hook` run in `mod_presence`:
 
 ```erlang
-Acc1 = mongoose_hooks:resend_offline_messages_hook(Acc, StateData#state.jid),
+Acc1 = mongoose_hooks:resend_offline_messages_hook(Acc, Jid),
 Rs = mongoose_acc:get(offline, messages, [], Acc1),
 ```
 
@@ -94,7 +98,7 @@ Rs = mongoose_acc:get(offline, messages, [], Acc1),
 Hooks are meant to decouple modules; in other words, the caller signals that some event took place or that it intends to use a certain feature or a set of features, but how and if those features are implemented is beyond its interest.
 For that reason hooks don't use the "let it crash" approach. Instead, it is rather like "fire-and-forget", more similar in principle to the `Pid ! signal` way.
 
-In practical terms: if a handler throws an error the hook machine logs a message and proceeds to the next handler with an unmodified accumulator.
+In practical terms: if a handler throws an error, the hook machine logs a message and proceeds to the next handler with an unmodified accumulator.
 If there are no handlers registered for a given hook, the call simply has no effect.
 
 ### Sidenote: Code yet to be written
@@ -118,7 +122,7 @@ It is decided when creating a hook and can be checked in the `mongoose_hooks` mo
 
 ## Registering hook handlers
 
-In order to store a packet when `ejabberd_sm` runs `offline_message_hook` the relevant module must register a handler for this hook.
+In order to store a packet when `ejabberd_sm` runs `offline_message_hook`, the relevant module must register a handler for this hook.
 To attain the runtime configurability the module should register the handlers when it's loaded and unregister them when
 it's unloaded.
 That's usually done in, respectively, `start/2` and `stop/1` functions.
@@ -199,12 +203,13 @@ in_subscription(Acc, #{to := ToJID, from := FromJID, type := Type}, _) ->
 
 As seen in this example, a handler receives an accumulator, parameters and extra parameters (in this case - ignored).
 Then it matches to the result of `process_subscription/4` and can return 3 different values:
+
 * `{ok, Acc}` - it allows further processing and does not change the accumulator.
 * `{stop, mongoose_acc:set(hook, result, false, Acc)}` - it stops further processing and returns accumulator with a new value in it.
 * `{stop, Acc}` - it stops further processing and does not change the accumulator.
 
 This is an important feature to note: in some cases our handler returns a tuple  `{stop, Acc}`.
-This skips calling the latter actions in the handler sequence, while the hook call returns the `Acc`.
+This skips calling later actions in the handler sequence, while the hook call returns the `Acc`.
 Further processing is only performed if the first element of return tuple is `ok`.
 
 Watch out! Different handlers may be registered for the same hook - the priority mechanism orders their execution.
@@ -218,7 +223,7 @@ Always ensure what handlers are registered for a given hook (`grep` is your frie
 The following command should give you a list of all the hooks available in MongooseIM:
 
 ```bash
-awk '/\-export\(\[/,/\]\)\./' src/mongoose_hooks.erl | grep -oh "\w*\/" | sed 's/.$//' | sort
+awk '/\-export\(\[/,/\]\)\./' src/hooks/*.erl | grep -oh "\w*/" | sed 's/.$//' | sort
 ```
 It returns:
 ```bash
@@ -230,7 +235,7 @@ adhoc_sm_commands
 xmpp_stanza_dropped
 ```
 
-It just extracts the hooks exported from the `mongoose_hooks` module.
+It just extracts the hooks exported from `mongoose_hooks` and other hook API modules.
 Refer to `grep`/`ack` to find where they're used.
 
 ## Creating your own hooks
@@ -328,7 +333,7 @@ stopping_handler(Acc, #{number := Number}, _) ->
 
 never_run_handler(Acc, #{number := Number}, _) ->
     ?LOG_INFO(#{what => never_run_handler,
-                text => <<"This hook won't run as it's registered with a priority bigger "
+                text => <<"This handler won't run as it's registered with a priority bigger "
                           "than that of stopping_handler/2 is. "
                           "This text should never get printed.">>}),
     {ok, Acc * Number}.
@@ -339,8 +344,8 @@ The module is intended to be used from the shell for educational purposes:
 ```erlang
 (mongooseim@localhost)1> gen_mod:is_loaded(<<"localhost">>, mod_hook_example).
 false
-(mongooseim@localhost)2> gen_mod:start_module(<<"localhost">>, mod_hook_example, [no_opts]).
-{ok,ok}
+(mongooseim@localhost)2> mongoose_modules:ensure_started(<<"localhost">>, mod_hook_example, #{}).
+{started,ok}
 (mongooseim@localhost)3> gen_mod:is_loaded(<<"localhost">>, mod_hook_example).
 true
 (mongooseim@localhost)4> mongoose_logs:set_module_loglevel(mod_hook_example, info).