README.md

Omditor

Status: WIP & Experimental & PoC

Omditor is an incredibly simple markdown editor that uses Omd for the preview and Irmin for the client-server storage. Omditor is offline-first meaning even without a connection you can use the application to edit your markdown and later use Irmin to sync and push your changes (it isn't a PWA so this isn't quite true). Thanks to Irmin's mergeable and branchable structure we get a collaborative, offline-first web application almost for free.

What follows is an explanation of this repository, bare in mind this is all proof-of-concept not some production service. It is a quick experiment and will likely remain that way. This approach is largely based on the very awesome Cuekeeper so be sure to check that out.

How it works?

The client uses Js_of_ocaml and brr + note (my first foray into functional reactive programming). The server uses cohttp and crunches the Javascript to be served. Everything uses irmin. The server holds the main store for the different markdown files (there are only three) and the clients push their changes once they're happy. Until then they are free to locally commit the changes and go and grab a coffee and go outside, knowing their content has been persistently saved to the browser's IndexedDB.

Server

The server has two main roles: to serve the web applications and to provide the HTTP endpoint for the irmin stores to communicate. Before doing that it first sets up a Unix, git, key-value store.

module Store = Irmin_unix.Git.FS.KV (Irmin.Contents.String)
module Sync = Irmin.Sync (Store)
module Http = Irmin_http.Server (Cohttp_lwt_unix.Server) (Store)

Layered over this store is the HTTP server endpoint. To combine this with an existing server I've had to do a bit of Irmin hacking to expose the callback rather than the cohttp server (hence irmin is submoduled). I don't think it is possible to combine two cohttp servers once they are set up (I could be wrong). With the callback in hand, we can set up the whole server callback which includes sending the web app down the line.

let callback repo conn req body =
  let uri = Cohttp.Request.resource req in
  match uri with
  | "" | "/" | "/index.html" ->
      Server.respond_string ~status:`OK ~body:Html.html ()
  | "/static/index.js" ->
      Server.respond_string ~status:`OK
        ~body:(Assets.read "index.js" |> Option.get)
        ()
  | _irmin_path -> Http.callback repo conn req body

Whilst we're at it we can also pre-populate the irmin store with some simple data.

let store () =
  let config = Irmin_git.config ~bare:true repo in
  let* repo = Store.Repo.v config in
  let* t = Store.master repo in
  let* () = Store.set_exn ~info:(info "commit 1") t [ "hello.md" ] "# Hello!" in
  let* () = Store.set_exn ~info:(info "commit 2") t [ "salut.md" ] "# Salut!" in
  let+ () = Store.set_exn ~info:(info "commit 3") t [ "hola.md" ] "# Hola!" in
  repo

And finally we run the server at http://localhost:8080.

let serve repo = Server.create (Server.make ~callback:(callback repo) ())

let main () =
  let* repo = store () in
  serve repo

let () = Lwt_main.run @@ main ()

Client

Overview

The client is quite a bit more complicated for two main reasons:

1. The irmin store is now dealing with a local view of the data and also trying to synchronise and merge updates from the server.
2. The application is reactive so it is built with brr and note.

The idea is fairly simple however. We want to have a local, persistent store where we can make changes, commit them and return to them later. At some point we want to then push these changes up to the server and also sync with whatever is up there.

For the sake of making this PoC projeect simple and as short as possible, we're using Irmin.Contents.String which has the idempotent default merging tactic. A future version would do something cleverer over the Omd abstract syntax tree (I made a start of this on this branch but it needed to extend repr and also we need an Omd.doc -> string function for the client, and it turns out that's pretty challenging)

Store

The first thing we need is a git-store, backed by the browser's IndexedDB. With the right functor magic this is straight-forward enough. I encounter some issues with irmin-indexeddb (see this repo for more details).

module Store =
  Irmin_git.Generic
    (Irmin_indexeddb.Content_store)
    (Irmin_indexeddb.Branch_store)
    (Irmin.Contents.String)
    (Irmin.Path.String_list)
    (Irmin.Branch.String)

Now we need to create the http client to get the remote information. You might be wondering why we need to do this? Irmin (more specifically ocaml-git) doesn't support being a git server just yet so we can't actually pull from it. In the future when this is implemented it will reduce the complexity and amount of data used in this approach by requiring only two stores.

(* No ocaml-git server... so using HTTP remote... *)
module Remote = Irmin_http.Client (Client) (Store)
module Sync = Irmin.Sync (Store)

From here we can then implement a sync function which fetches the main branch from the server for the client and optionally merges this branch into the staging branch. The staging branch is where we do the local work until we are ready to push things to the server. We're using irmin's built-in Irmin.remote_store functionality along with the Http store to make this possible.

let sync ?(merge = true) t =
  let config = Irmin_http.config t.uri in
  let main = t.main in
  Remote.Repo.v config >>= fun repo ->
  Remote.master repo >>= fun remote ->
  Sync.pull_exn main ~depth:1 (Irmin.remote_store (module Remote) remote) `Set
  >>= fun _ ->
  if merge then (
    Brr.Console.log [ Jstr.v "Merging" ];
    Store.merge_into ~info:(info "update staging") ~into:t.staging main >>= function
      | Ok () -> Lwt.return @@ Ok ()
      | Error (`Conflict s) -> 
        (* Of course in practice we'd be more clever here... *)
        Store.Head.get main >>= fun head -> 
        Lwt_result.ok @@ Store.Branch.set (Store.repo t.staging) "staging" head
  )
  else Lwt_result.return ()

As the code makes clear, we're not doing anything particularly smart if we encounter a merge conflict. A common example of when this will occur, is when we have two clients A and B.

1. A makes some changes and hits the commit button so they are now locally committed to A's staging branch.
2. B makes some changes and hits the commit button so they are now locally committed to B's staging branch.
3. A is happy with their changes and hits push to put them on the server.
4. B wants the latest changes so hits sync which fetches the main branch with no problem, but when trying to merge into staging... conflict!

As mentioned, using a cleverer merging strategy (either based on a diffing algorithm or the omd AST) would be a better choice here.

The push and init functions for the store are fairly similar. The init function looks to see if there are fresh commits on the main branch so if staging is behind the idea would be to let the client know (this functionality is not implemented yet).

Application

I won't go into too much detail of the client-side reactive code because I'm not terribly confident with FRP just yet, but the basic idea is that the client has some model of data and you define actions that move the model from one state to another. The "HTML" is built on top of this model (but as a signal) so when something changes the relevant parts are re-rendered.

The model is pretty simple here: the current file, a list of possible files and the text being edited. The actions we can take are: updating the text (the bool is whether we should update the contenteditable div which we only do on a page load or a file change). A local commit, push, sync and file change should all be fairly understandable.

(* Model *)
type t = { file : string; files : string list; editor : Jstr.t }

type action =
  [ `Update of bool * Jstr.t
  | `LocalCommit
  | `Push
  | `Sync
  | `ChangeFile of string ]

If we have a look at one example, the ChangeFile action:

let change_file store s t =
  let open Lwt.Infix in
  let f = { t with file = s } in
  Lwt_js_events.async (fun () ->
      Store.local_get store [ s ] >|= fun l ->
      refresh_send (`Update (true, Jstr.v l)));
  f

You can see we have a "local get" to the store for the file we're after and then asynchronously firing a new update event. The event (and event sender refresh_send) are global and I have a feeling this might be a bit of an anti-pattern... but it works!

One thing we do before changing files, is to commit the content locally so we don't lose the work:

let reducer (store : Store.t) (action : [> action ]) =
  match action with
  | `Update s -> update s
  | `LocalCommit -> commit store
  | `Push -> push store
  | `Sync -> sync store
  (* Changing the file first locally commits it so you don't lose your changes! *)
  | `ChangeFile file -> fun t -> change_file store file (commit store t)
  | `UpdateEditable -> update_editable

I'm not sure if there are race-conditions with Lwt_js_events.async, I haven't experienced them yet but may be worth looking into.

The rest of the code is just for the UI and connecting up the different events, signal and elements.

Conclusion

This is a pretty fun approach to building web applications and feels quite powerful (see some of the extensions below). What's really exciting is the power of OCaml's portable code like irmin which started off life being for MirageOS but this approach to being in control of your implementation unlocks the ability to putting your code in lots of interesting places (laptops, browsers, phones, micro-controllers etc.!).

If you made it this far thanks for reading. I threw this together in just under a week so don't expect production-ready code or anything close to that. I encourage you to have a go with irmin though and see what you can build and let people know about it!

Extensions

• As already mentioned a better merging strategy would unleash the full power of irmin and this approach to building web applications.
• Making the app into a PWA would also make it truly offline-first.
• The UX is not great, the buttons to provide any feedback, there's no way to toggle between your local content and what's on the server, or diff these two. That would be quite a nice experience.
• Could you do peer-to-peer connections rather than relying directly on the server?
• Better UI, writing bindings for CodeMirror shouldn't be too hard and would be better than a contenteditable <div>...
• Full file hierarchy and file creation too, right now you only get the three files at the start.
• An example where the initial git store is pulled from somewhere (e.g. Github) and periodically pushes the changes from the server back to there.
• Unikernel-ise the server because why not.