Add fragmented encoding + redesign fragmented decoding

MIGRATION_GUIDE.md

@@ -1,5 +1,16 @@
 # zcbor v. 0.9.99
 
+* The fragmented payload API has been completely redesigned to accomodate adding the encoding counterpart.
+  The docs have been updated and there's a new section in the README to explain the functionality.
+
+  * You must now define ZCBOR_FRAGMENTS to access the API
+  * `zcbor_*str_decode_fragment()` has been renamed to `zcbor_*str_fragments_start_decode()`
+  * After calling `zcbor_*str_fragments_start_decode()`, you must now retrieve the first fragment manually with `zcbor_str_fragment_decode()`, instead of via an argument.
+  * `zcbor_next_fragment()` and `zcbor_bstr_next_fragment()` have merged and is now called `zcbor_str_fragment_decode()`.
+    It does not take a `prev_fragment` argument, instead, this state is kept internally in the state struct.
+  * `zcbor_bstr_start_decode_fragment()` has been renamed to `zcbor_cbor_bstr_fragments_start_decode()` and does not return a fragment.
+    To retrieve fragments when decoding a CBOR-encoded bstr, use `zcbor_str_fragment_decode()`
+
 
 # zcbor v. 0.9.0
 

README.md

@@ -88,6 +88,31 @@ ZCBOR_STATE_D(decode_state, n, payload, payload_len, elem_count, n_flags);
 ZCBOR_STATE_E(encode_state, n, payload, payload_len, 0);
 ```
 
+Fragmented payloads
+-------------------
+
+zcbor can encode and decode payloads in sections, i.e. the payload can be split into separate buffers/arrays.
+This can be useful e.g. if you send or receive your payload in multiple packets.
+When the current payload section is done, call `zcbor_update_state()` to introduce the next section.
+Note that zcbor does not allow section boundaries to split a zcbor header/value pair.
+This means that the following elements cannot be split between sections:
+
+- Numbers and simple values (integers, floats, bools, undefined, nil)
+- Tags
+- Headers of lists, maps, tstrs, and bstrs
+
+If your payload is split in an unsupported way, you can get around it by making a small section out of the remaining bytes of one section spliced with the start of the next.
+Another option is to leave a little room at the start of each section buffer, and copy the remaining end of one section into the start of the next buffer.
+8 bytes should be enough for this.
+
+Lists and maps can span multiple sections, as long as the individual elements are not split as to break the above rule.
+
+String payloads can be split across multiple payload sections, if `ZCBOR_FRAGMENTS` is enabled, and the `*str_fragments_*()` APIs are used. Note that in the zcbor docs, the term "string fragment" is used for fragmented strings, while the term "payload section" is used for fragmented CBOR payloads, as passed to `zcbor_update_state()`. These do not always line up perfectly, particularly at the start and end of fragmented strings.
+
+CBOR-encoded bstrs can be nested, and there can also be a non-CBOR-encoded innermost string.
+The current innermost string (CBOR-encoded or otherwise) is called the "current string".
+`zcbor_update_state()` modifies all backups so that outer nested CBOR-encoded strings have updated information about the new section.
+
 Configuration
 -------------
 
@@ -103,6 +128,7 @@ Name                      | Description
 `ZCBOR_STOP_ON_ERROR`     | Enable the `stop_on_error` functionality. Note that it also has to be enabled in the state variable (`state->constant_state->stop_on_error`). This makes all zcbor functions abort their execution if called when an error has already happened.
 `ZCBOR_BIG_ENDIAN`        | All decoded values are returned as big-endian. The default is little-endian.
 `ZCBOR_MAP_SMART_SEARCH`  | Applies to decoding of unordered maps. When enabled, a flag is kept for each element in an array, ensuring it is not processed twice. If disabled, a count is kept for map as a whole. Enabling increases code size and memory usage, and requires the state variable to possess the memory necessary for the flags.
+`ZCBOR_FRAGMENTS`         | Enable functions for decoding and encoding byte and text strings in fragments.
 
 
 Python script and module

include/zcbor_common.h

@@ -107,6 +107,18 @@ union {
 	bool payload_moved; /**< Is set to true while the state is stored as a backup
 	                         if @ref zcbor_update_state is called, since that function
 	                         updates the payload_end of all backed-up states. */
+	bool inside_cbor_bstr; /**< True if we are currently inside a CBOR-encoded bstr,
+	                            i.e. that as been started with zcbor_bstr_start_*(), or
+	                            `zcbor_cbor_bstr_fragments_start_*()`. */
+#ifdef ZCBOR_FRAGMENTS
+	bool inside_frag_str; /**< True if we are currently inside a fragmented (non-CBOR-encoded)
+	                           string. This is mutually exclusive with `inside_cbor_bstr`,
+	                           i.e. not set when using `zcbor_cbor_bstr_fragments_start_*()` */
+	ptrdiff_t frag_offset; /**< The offset in the current string at which this payload section starts. Used for non-CBOR-encoded strings. */
+	size_t str_total_len; /**< The total length of the string this fragment is a part of. Used for non-CBOR-encoded strings. */
+	ptrdiff_t frag_offset_cbor; /**< The offset in the current string at which this payload section starts. Used for CBOR-encoded strings. */
+	size_t str_total_len_cbor; /**< The total length of the string this fragment is a part of. Used for CBOR-encoded strings. */
+#endif
 
 /* This is the "decode state", the part of zcbor_state_t that is only used by zcbor_decode.c. */
 struct {
@@ -152,6 +164,7 @@ struct zcbor_state_constant {
 #ifdef ZCBOR_MAP_SMART_SEARCH
 	uint8_t *map_search_elem_state_end; /**< The end of the @ref map_search_elem_state buffer. */
 #endif
+	const uint8_t *curr_payload_section; /**< The currently encoded/decoded payload section. */
 };
 
 #ifdef ZCBOR_CANONICAL
@@ -285,6 +298,9 @@ do { \
 #define ZCBOR_ERR_MAP_FLAGS_NOT_AVAILABLE 20
 #define ZCBOR_ERR_INVALID_VALUE_ENCODING 21 ///! When ZCBOR_CANONICAL is defined, and the incoming data is not encoded with minimal length, or uses indefinite length array.
 #define ZCBOR_ERR_CONSTANT_STATE_MISSING 22
+#define ZCBOR_ERR_INNER_STRING_TOO_LARGE 23 ///! Trying to start a nested string that is too large to fit in the container string.
+#define ZCBOR_ERR_NOT_IN_FRAGMENT 24
+#define ZCBOR_ERR_INSIDE_STRING 25 ///! Currently encoding/decoding a non-CBOR-encoded string, so cannot use most zcbor encoding/decoding functions
 #define ZCBOR_ERR_UNKNOWN 31
 
 /** The largest possible elem_count. */
@@ -392,17 +408,23 @@ static inline void zcbor_error(zcbor_state_t *state, int err)
 	}
 }
 
-/** Whether the current payload is exhausted. */
+/** Whether the current payload (section) is exhausted. */
 static inline bool zcbor_payload_at_end(const zcbor_state_t *state)
 {
 	return (state->payload == state->payload_end);
 }
 
-/** Update the current payload pointer (and payload_end).
+/** Introduce a new payload section.
  *
+ *  Updates the current payload pointer (and payload_end and frag_offset(_cbor)).
  *  For use when the payload is divided into multiple chunks.
  *
- *  This function also updates all backups to the new payload_end.
+ *  This function also updates all backups to the new payload_end,
+ *  and also updates the frag_offset/frag_offset_cbor of all backups.
+ *
+ *  Note that if this is called before the current payload is exhausted, the
+ *  remaining payload will be abandoned.
+ *
  *  This sets a flag so that @ref zcbor_process_backup fails if a backup is
  *  processed with the flag @ref ZCBOR_FLAG_RESTORE, but without the flag
  *  @ref ZCBOR_FLAG_KEEP_PAYLOAD since this would cause an invalid state.
@@ -415,6 +437,17 @@ static inline bool zcbor_payload_at_end(const zcbor_state_t *state)
 void zcbor_update_state(zcbor_state_t *state,
 		const uint8_t *payload, size_t payload_len);
 
+/** Get the the offset into the current string at which the current payload
+ *  pointer (state->payload) belongs. */
+size_t zcbor_current_string_offset(zcbor_state_t *state);
+
+/** Get the remaining number of bytes in the current string, calculated from
+ *  the current payload pointer (state->payload). */
+size_t zcbor_current_string_remainder(zcbor_state_t *state);
+
+/** Can be used on any fragment to tell if it is the final fragment of its string. */
+bool zcbor_is_last_fragment(const struct zcbor_string_fragment *fragment);
+
 /** Check that the provided fragments are complete and in the right order.
  *
  *  If the total length is not known, the total_len can have the value

include/zcbor_decode.h

@@ -371,7 +371,9 @@ bool zcbor_tstr_expect_term(zcbor_state_t *state, char const *str, size_t maxlen
 /** Decode and consume a bstr header.
  *
  * The rest of the string can be decoded as CBOR.
- * A state backup is created to keep track of the element count.
+ * A state backup is created to keep track of the element count and original payload_end.
+ * payload_end is set to the end of the string, so when the payload is exhausted,
+ * the string is considered fully decoded.
  * Call @ref zcbor_bstr_end_decode when done decoding the contents of the bstr.
  *
  * @param[inout] state   The current state of the decoding.
@@ -393,49 +395,46 @@ bool zcbor_bstr_start_decode(zcbor_state_t *state, struct zcbor_string *result);
 bool zcbor_bstr_end_decode(zcbor_state_t *state);
 
 
-/** Supplementary string (bstr/tstr) decoding functions for fragmented payloads: */
+#ifdef ZCBOR_FRAGMENTS
 
-/** Start decoding a bstr/tstr, even if the payload contains only part of it.
- *
- * This must be followed by a call to @ref zcbor_update_state, which can be
- * followed by a call to @ref zcbor_next_fragment. Do not call this function
- * again on subsequent fragments of the same string.
+/** Start decoding a fragmented string. I.e. a string spread over non-consecutive payload sections.
  *
- * This consumes the remaining payload as long as it belongs to the string.
+ * After calling this, you can retrieve a fragment with @ref zcbor_str_fragment_decode,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_decode.
  */
-bool zcbor_bstr_decode_fragment(zcbor_state_t *state, struct zcbor_string_fragment *result);
-bool zcbor_tstr_decode_fragment(zcbor_state_t *state, struct zcbor_string_fragment *result);
+bool zcbor_bstr_fragments_start_decode(zcbor_state_t *state);
+bool zcbor_tstr_fragments_start_decode(zcbor_state_t *state);
 
-/** Extract the next fragment of a string.
+/** Start decoding a fragmented CBOR-encoded bytestring.
  *
- * Use this function to extract all but the first fragment.
- */
-void zcbor_next_fragment(zcbor_state_t *state,
-	struct zcbor_string_fragment *prev_fragment,
-	struct zcbor_string_fragment *result);
-
-/** Decode and consume a bstr header, assuming the payload does not contain the whole bstr.
+ * I.e. a string spread over non-consecutive payload sections.
  *
- * The rest of the string can be decoded as CBOR.
- * A state backup is created to keep track of the element count.
- * Call @ref zcbor_update_state followed by @ref zcbor_bstr_next_fragment when
- * the current payload has been exhausted.
- * Call @ref zcbor_bstr_end_decode when done decoding the contents of the bstr.
+ * This is an alternative to zcbor_*str_fragments_start_decode() to be used if the payload
+ * contains CBOR data that will be decoded directly with other zcbor_*() functions.
+ *
+ * A state backup is created to keep track of the element count and original payload_end.
+ * After calling this, you can decode elements using other zcbor functions,
+ * then update the payload with @ref zcbor_update_state.
+ * Do not use @ref zcbor_str_fragment_decode with this function.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_decode.
+ * When the current payload section contains the end of the string,
+ * payload_end is set to the end of the string, so there is no risk of decoding past the end.
  */
-bool zcbor_bstr_start_decode_fragment(zcbor_state_t *state,
-	struct zcbor_string_fragment *result);
+bool zcbor_cbor_bstr_fragments_start_decode(zcbor_state_t *state);
 
-/** Start decoding the next fragment of a string.
+/** Retrieve a string fragment.
  *
- * Use this function to extract all but the first fragment of a CBOR-encoded
- * bstr.
+ * Consumes bytes from the payload until either the end of the payload or the end of the string.
+ *
+ * Note: Do not use this function with @ref zcbor_cbor_bstr_fragments_start_decode.
  */
-void zcbor_bstr_next_fragment(zcbor_state_t *state,
-	struct zcbor_string_fragment *prev_fragment,
-	struct zcbor_string_fragment *result);
+bool zcbor_str_fragment_decode(zcbor_state_t *state, struct zcbor_string_fragment *fragment);
+
+/** Finish decoding a fragmented string. */
+bool zcbor_str_fragments_end_decode(zcbor_state_t *state);
 
-/** Can be used on any fragment to tell if it is the final fragment of the string. */
-bool zcbor_is_last_fragment(const struct zcbor_string_fragment *fragment);
+#endif /* ZCBOR_FRAGMENTS */
 
 #ifdef __cplusplus
 }

include/zcbor_encode.h

@@ -231,6 +231,46 @@ bool zcbor_bstr_start_encode(zcbor_state_t *state);
  */
 bool zcbor_bstr_end_encode(zcbor_state_t *state, struct zcbor_string *result);
 
+
+#ifdef ZCBOR_FRAGMENTS
+
+/** Start encoding a fragmented string. I.e. a string spread over non-consecutive payload sections.
+ *
+ * After calling this, you can write a fragment with @ref zcbor_str_fragment_encode,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_encode.
+ */
+bool zcbor_bstr_fragments_start_encode(zcbor_state_t *state, size_t total_len);
+bool zcbor_tstr_fragments_start_encode(zcbor_state_t *state, size_t total_len);
+
+/** Start encoding a fragmented CBOR-encoded bytestring.
+ *
+ * I.e. a string spread over non-consecutive payload sections.
+ *
+ * This is an alternative to zcbor_*str_fragments_start_encode() to be used if the payload
+ * contains CBOR data that will be encoded directly with other zcbor_*() functions.
+ *
+ * A state backup is created to keep track of the element count and original payload_end.
+ * After calling this, you can encode elements using other zcbor functions,
+ * then update the payload with @ref zcbor_update_state.
+ * Repeat until the string is fully decoded, then call @ref zcbor_bstr_fragments_end_encode.
+ * When the current payload section contains the end of the string,
+ * payload_end is set to the end of the string, so there is no risk of encoding past the end.
+ */
+bool zcbor_cbor_bstr_fragments_start_encode(zcbor_state_t *state, size_t total_len);
+
+/** Retrieve a string fragment.
+ *
+ * Consume bytes from the payload until either the end of the payload or the end of the string.
+ * Do not use this function with @ref zcbor_cbor_bstr_fragments_start_encode.
+ */
+bool zcbor_str_fragment_encode(zcbor_state_t *state, struct zcbor_string *fragment, size_t *enc_len);
+
+/** Finish encoding a fragmented string. */
+bool zcbor_str_fragments_end_encode(zcbor_state_t *state);
+
+#endif /* ZCBOR_FRAGMENTS */
+
 #ifdef __cplusplus
 }
 #endif