Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MANTA-5453 Update docs to reflect mantav2 changes #63

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 3 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
-->

<!--
Copyright 2020 Joyent, Inc.
Copyright 2021 Joyent, Inc.
-->

# Manta: a scalable, distributed object store
Expand Down Expand Up @@ -93,11 +93,9 @@ The front door services respond to requests from the internet at large:
service
* [muskie](https://github.com/joyent/manta-muskie): the node.js-based "webapi"
service, this is Manta's "Directory API"
* [buckets-api](https://github.com/joyent/manta-buckets-api): Node.js-based
"buckets-api" service, this is Manta's "Buckets API"

The metadata tiers for the Directory and Buckets APIs store the entire object
namespace (not object data) as well as backend storage system capacity:
The metadata tier APIs store the entire object namespace (not object data) as
well as backend storage system capacity:

* [manatee](https://github.com/joyent/manatee): the "postgres" service, a
high-availability postgres cluster using synchronous replication and automatic
Expand All @@ -108,11 +106,6 @@ namespace (not object data) as well as backend storage system capacity:
* [electric-moray](https://github.com/joyent/electric-moray): Node-based service
that provides the same interface as Moray, but which directs requests to one
or more Moray+Manatee *shards* based on hashing the Moray key.
* [buckets-mdapi](https://github.com/joyent/manta-buckets-mdapi): a Rust-based
API for managing all metadata for the Buckets API
* [buckets-mdplacement](https://github.com/joyent/manta-buckets-mdplacement): a
Rust-based API for handling routing of Buckets API objects to appropriate
nodes in the storage tier.

The storage tier is responsible for actually storing bits on disk:

Expand Down
21 changes: 0 additions & 21 deletions docs/developer-guide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ nitty-gritty content than makes sense for the README.
- [Configuration](#configuration)
- [Configuration Updates](#configuration-updates)
- [Directory API Shard Management](#directory-api-shard-management)
- [Buckets API Shard Management](#buckets-api-shard-management)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand Down Expand Up @@ -405,23 +404,3 @@ find yourself adding additional capacity, deploy the new shard first, coordinate
with all existing shards, then use manta-shardadm to add the shard to list of
shards for the indexing tier.


## Buckets API Shard Management

Buckets API is an experimental feature that serves similar functions as the
Directory API but has a different paradigm for object organization. As opposed
to the hierarchical object support provided by Directory API that comes with
a limit on the maximum number of objects per directory, Buckets API offers a
simpler structure for storing an unlimited number of objects in groups. The
two-level object structure incurs a much smaller overhead in request processing
and is more cost-efficient from a metadata perspective. It is a better option
when the objects in the same bucket are loosely related and do not need a
finer categorization.

Buckets shards are defined and stored in the same way as directory shards
but are separate entities altogether. They are also managed with
`manta-shardadm` which generates the `BUCKETS_MORAY_SHARDS` and
`BUCKETS_HASH_RING_IMAGE` configurations in SAPI metadata.

Note: The "buckets" that Buckets API manages are not to be confused with moray
"buckets" which represent the object namespaces for the data stored in Postgres.
39 changes: 19 additions & 20 deletions docs/mantav2.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
# Mantav2

Starting November 2019, there will be two separate active versions of Manta:
Since November 2019, there have been two separate active versions of Manta:

- "**mantav1**" - a long term support branch of Manta that maintains current
Manta features.
- "**mantav2**" - a new major version of Manta that adds (Buckets API, storage
"rebalancer" service) and removes (jobs, snaplinks, etc.) some major
features, and becomes the focus of future Manta development.
- "**mantav2**" - a new major version of Manta that removes some features (jobs,
snaplinks, MPU, etc.) and makes some major improvements to things like garbage
collection.

At this time, mantav1 is the recommended version for production usage, but
that is expected to change to mantav2 during 2020.
As of 2021H1, mantav2 is the recommended version for new installations.

Originally mantav2 was intended to include a new "Buckets API" but this work was
abandoned in early 2020. There may still be references in documentation and code
to "buckets" but these features should not be deployed or used. Additionally
there was originally mention of online rebalancing of objects. These features
were never completed to the point they could be considered part of the product
and this feature should also be considered abandoned.


## What is mantav2?
Expand All @@ -29,11 +35,6 @@ Significant changes are:
- snaplinks
- metering data under `/<account>/reports/...`
Otherwise the Directory API remains a part of Manta.
- A new "Buckets API" (S3-like) is added. This is the API for which latency
improvements are being made.
- A "rebalancer" system is added for storage tier maintenance.
- The garbage collection (GC) system is improved for larger scale.
- Improved per-account usage data for operators.

The "master" branch of Manta-related git repos is for mantav2. Mantav1
development has moved to "mantav1" branches.
Expand Down Expand Up @@ -78,12 +79,11 @@ The Node.js Manta client is developed in the
- mantav1: Currently on the ["master" branch](https://github.com/joyent/node-manta/tree/master/)
of joyent/node-manta, and published to npm as
["manta"](https://www.npmjs.com/package/manta) -- i.e. `npm install manta`.
- mantav2: Currently on the ["buckets" branch](https://github.com/joyent/node-manta/tree/buckets/)
of joyent/node-manta. It is not yet published to npm.
- mantav2: The mantav1 client should work with mantav2.

*(The intent is to eventually move mantav2 to the "master" branch and publish it
to npm as "mantav2". Mantav1 dev would move to the "mantav1" branch and continue
to publish to npm as "manta".)*
*(The intent is to eventually move a mantav2 version to the "master" branch and
publish it to npm as "mantav2". Mantav1 dev would move to the "mantav1" branch
and continue to publish to npm as "manta".)*


#### Java client
Expand All @@ -93,11 +93,10 @@ The Java Manta client is developed in the

- mantav1: Currently on the ["master" branch](https://github.com/joyent/java-manta/tree/master/)
of joyent/java-manta. Current release versions are 3.x.
- mantav2: Currently on the ["buckets-m1" branch](https://github.com/joyent/java-manta/tree/buckets-m1/)
of joyent/java-manta.
- mantav2: The mantav1 client should work with mantav2.

*(The intent is to eventually move mantav2 to the "master" branch and release it
as a new "4.x" major version. Mantav1 dev would move to the "3.x" branch and
*(The intent is to eventually move a mantav2 to the "master" branch and release
it as a new "4.x" major version. Mantav1 dev would move to the "3.x" branch and
continue to release as 3.x versions.)*


Expand Down
4 changes: 1 addition & 3 deletions docs/operator-guide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,9 @@ essentially:

* A separate filesystem-like namespace *Directory API*, with directories and
objects, accessible over HTTP.
* A *Buckets API* (an experimental feature similar to S3) with objects,
accessible over HTTP.
* *Objects* are arbitrary-size blobs of data
* Users can use standard HTTP `PUT`/`GET`/`DELETE` verbs to create, list, and
remove buckets, directories, and objects.
remove directories, and objects.
* Users can fetch arbitrary ranges of an object, but may not *modify* an object
except by replacing it.

Expand Down
Loading