The Redesign of Engula

[huachaohuang]

During the discussion and development of Engula in the past few months, we encountered a few problems:

• Engula positioned itself as a storage engine to build databases at launch. However, we realized that "storage engine" is actually a very confused position. Do I have to use Engula to build a database? Can I use Engula as a key-value store for my applications? We saw that users and contributors have different understanding about what a storage engine is and how it is used. Some people think that it is a key-value service while other people think that it is a library or something. The truth is, even the most well-known storage engines like LevelDB and RocksDB never mention the term "storage engine" in their README or other materials. These projects, including others like Redis, tend to position them as a store or a database instead.
• The design and codebase of Engula is continuously changing. The lack of steady iterations hurt the morale of the team and contributors. This is mostly because Engula tried to do too many things without laying a solid foundation first.
• Engula failed to provide a specific interface and usage for users, which makes the situation even worse.

To address these problems, we are going to make a serious redesign of Engula, in terms of its position and architecture.

First of all, Engula will position itself as a persistent data structure store, used as a database and storage engine. The key features of Engula are as follows:

• Provide a set of persistent data structures to serve different kinds of applications
• Support ACID transactions
• Implement a cloud-native architecture to deliver a cost-effective, highly-scalable, and highly-available service on the cloud

This redesign will be carried out as follows:

• Update the related materials to reflect the new design. The most important one will be the design document.
• Build version 0.3 based on the new design. The release of 0.3 will be served as the milestone of this redesign.

I will create a tracking issue for 0.3 later.

[tisonkun]

Thanks for pushing this redesign!

On the key features that should be what we focus on from now on, I'd like to understand their priorities and steps we achieve them - we cannot achieve all of them at once, though 😆

It writes:

• Provide a set of persistent data structures to serve different kinds of applications
• Support ACID transactions
• Implement a cloud-native architecture to deliver a cost-effective, highly-scalable, and highly-available service on the cloud

Let's review these features button up.

Cloud-native Architecture

For a cloud-native arch, I can see that we're implementing a standalone database now. It can be run standalone which users can play with. However, it's far from a realistic cloud-native arch or we can simply access the database on cloud.

The question is, when we'll start to make progress on this direction? I assume that it's the final feature we focus on, so I don't talk about its implementation but ask for an estimate so that contributors know that we don't focus on this feature yet.

ACID Transactions

I can see there're already several related works:

• Implement the Txn service #379
• txn.proto

However, it works like a single instance transaction which is quite different from distributed transaction.

The question is, when we'll start to define the semantic of distributed transaction over our collection database? I assume that it depends on what data structures we're going to support and realistic works should be mapped into transactions over those data structures.

Data Structures

Thus, we're back to the root features. What data structures do we want to support at the following releases?

Let me try to interpret the current API design.

• All data structures are objects.
• Each object has a unique object_id.
• The object_id is in bytes.
• Currently, we support something like primitives, and its only implementation (data type) is Int64.

Here are some questions that may help on defining the API:

• Is object_id always a String?
• Is object static typing or dynamic typing? So far, it seems dynamic typing.
• What API over the data structure do you want to support? Get/Set is basic but less interesting, and when you want to define API over data structure, the question that whether an object is static typing or dynamic typing is essential.

And finally, apart from a low-level API design, what's your expectation that users use Engula v0.3, v0.4, and v0.5 for?

For example, to cover the common use case of Redis as a cache service, the supported data structure should include simple KV, list, and hash map, all over strings.

The point here is that we cannot achieve all feature at once, so we must focus one thing after another. Thus, contributors can follow the current focused tasks and join force. For example, support more data type on the same expression may be easily scale out or adopting some skill set presented by contributors (e.g., type-exercise-in-rust).

In the contrast, if we talk about multiple things in parallel evenly, but actually omit some of them, those who are attracted by the omitted topics will be upset which should have been avoided. For example, we'd like to build a cloud-native arch, and it's welcome to leave comment as well as point out the risk in the current arch, but we won't spend a lot of time on implementation in the next several releases - it should be clear.

[dragonly]

This is important. A developer appreciates an open-source project by how it solves his/her specific problem in the projects. Without use cases and user stories, the ideas are hard to catch.

We should build things to solve specific problems.

[huachaohuang]

Cloud-native Architecture

The question is, when we'll start to make progress on this direction? I assume that it's the final feature we focus on, so I don't talk about its implementation but ask for an estimate so that contributors know that we don't focus on this feature yet.

It won't be the final feature we focus on. I think we will start working on this in 0.5 or something. We can focus on making Engula work nicely on k8s first. That will be a significant step towards the cloud-native architecture, even if we haven't integrated with cloud vendors.

The current blocker in this direction is the design of Orchestrator, which will be used in different places. I am not that good at k8s and I haven't got time to do a more detailed design for it. So I would like some help in designing the Orchestrator pattern here.

ACID Transactions

The question is, when we'll start to define the semantic of distributed transaction over our collection database? I assume that it depends on what data structures we're going to support and realistic works should be mapped into transactions over those data structures.

I think we will not focus on distributed transaction in the short term. This is because, in our new architecture, I expect that most databases (except for the supersized) can be served by a single cooperator group. In this case, distributed transactions are not required. We can expect that the capability of a cooperator group will be similar to one Aurora database instance, which doesn't do distributed transactions either.

We can start working on distributed transactions when a single cooperator group becomes insufficient.

Data Structures

Here are some questions that may help on defining the API:

• Is object_id always a String?

It is always bytes, or Vec<u8>.

• Is object static typing or dynamic typing? So far, it seems dynamic typing.

I want to support both. Users can create typed collections or generic collections. In a typed collection, all objects have the same type. In a generic collection, the type of individual object is determined at runtime when it is first updated. I think we can focus on generic collections first because they are the general case.

• What API over the data structure do you want to support? Get/Set is basic but less interesting, and when you want to define API over data structure, the question that whether an object is static typing or dynamic typing is essential.

We can start with the built-in types in most programming languages, which are int, float, bytes, strings, set, map, list. We don't need to support them at one time, we can add types and their APIs step by step when someone really wants to use them.

And finally, apart from a low-level API design, what's your expectation that users use Engula v0.3, v0.4, and v0.5 for?

For example, to cover the common use case of Redis as a cache service, the supported data structure should include simple KV, list, and hash map, all over strings.

Yes, I think the early use cases will be similar to Redis. But we are still far from delivering the service we claim in the first few releases. Here are some immature plans I get so far:

• In 0.3, we primarily focus on defining the APIs and providing a workable server.
• In 0.4, we need to do a more detailed design and lay down a basic structure for the internal components like supervisor, transactor, cooperator, stream store, and engine store.
• In 0.5, it would be nice if we can make Engula run with k8s.

As for the standalone server, we will always keep it useful because it is very convenient to use. However, providing a standalone server will not be our core value because there are too many options for that purpose. Our core value is to provide a reliable, cost-effective, and vendor-independent service on the cloud. And the data structure APIs enable us to potentially optimize for diverse applications in the long term.