Saving the compiled FSM by serializing it? #297
-
Thank you for this fantastic library. I have a question about storing the compiled FSM and reusing it between invocation. Suppose I have few hundred rules/patterns which doesn't change much over time. But my service is ephemeral because it is hosted in lambda. I want to serialize the compiled instance of FSM and store it in some DB so that on each invocation of the service, it doesn't need to recompile the rules into FSM and it can get same instance by de-serializing it from stored data. Can you provide any direction how can I achieve this? |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 6 replies
-
Good question. First thing, have you measured the latency/compute required to build the FSM? It's possible that it's not even significant. (I genuinely don't know, haven't really done any serious analysis of FSM-building.). If it really is a noticeable problem do please file an issue. Let’s assume that it is significant. Then yes, it would obviously be a good idea to serialize (oops, we say “marshal” in Go) a constructed Quamina instance in a way that allowed fast restore. Which doesn't seem like rocket science. Go has existing serializers for data structures into JSON and CBOR and various other things; once again, I don't know what the efficiency story is. But it does seem like a thing that could be done without too much effort. |
Beta Was this translation helpful? Give feedback.
-
I haven't measured the FSM compilation performance, but the logic is: it'll be always faster to fetch a single binary blob from a very fast key-value store like Dynamo or ScyllaDB or Redis than fetching a hundred rules in raw format from the database. So even if compilation doesn't make much difference but retrieving rules, in most of the cases will be slower than the compiled FSM binary. Anyways, I will try to marshal 😀 it with |
Beta Was this translation helpful? Give feedback.
-
I can't argue with your logic, and my experience running the analogous event-ruler package at AWS agrees with it. I had never previously heard of encoding/gob, I'll be curious to hear about your experience with it. If the thing works well, please consider doing a PR to build this into Quamina. |
Beta Was this translation helpful? Give feedback.
-
@timbray I am trying to serialize the struct, but the biggest issue I face is that all of the fields of structures are unexported. This is the case for majority of structs. This makes all of serialization very difficult. Either the struct fields needs to be exported or a serialize method needs to be implemented in all struct |
Beta Was this translation helpful? Give feedback.
-
Hmm, the key thing you would need to serialize is the coreMatcher type. It references other types: fieldMatcher, valueMatcher, faState, faNext, and smallTable. There are no generics, they're all struct types. What might be a problem is that coreMatcher, fieldMatcher, and valueMatcher use an atomic.Value to support concurrency. I have no idea how that might interact with serialization. |
Beta Was this translation helpful? Give feedback.
-
And to make things worse, I'm currently working intensely on refactoring the Quamina data structures :) So I think you're taking the right approach. Ouch, I forgot to mention something that is highly relevant. Check out https://github.com/timbray/quamina/blob/main/README-pruner.md - @jsmorph has already built a system for remembering which patterns have been added to Quamina. This is for the purpose of supporting deletion, but it seems plausible that those remembered patterns could be exported and used to prepare a new Quamina instance. |
Beta Was this translation helpful? Give feedback.
I can't argue with your logic, and my experience running the analogous event-ruler package at AWS agrees with it.
I had never previously heard of encoding/gob, I'll be curious to hear about your experience with it. If the thing works well, please consider doing a PR to build this into Quamina.