Rune has no roadmap

[waywardgeek]

Sorry, folks, but it's the truth. There is no internal project at Google supporting Rune. Rune is simply a 20% project for a few Googlers. Rune, like most new programming languages, would require a whole new ecosystem of libraries to be developed before it could be of much use, and the likelihood of that happening is low. Rune was open-sourced so that some of its good ideas could find their way into more mainstream languages, in particular SoA memory layout and detection of timing side channel leakage at compile time.

The mission of proving out Rune's core ideas remains a goal of our 20%-ers. While not exactly a road map, here is the overall plan:

• Win a bunch of memory-intensive benchmarks, proving the value of SoA memory layout.
• Have SoA widely adopted, and maybe devs will pull a few more good ideas from Rune while they're at it.

My hope is that eventually, SoA memory layout will help reduce energy consumption in data centers globally by around 10%, while reducing memory similarly. SoA is hard to integrate into C/C++, but could find its way into Go, Rust, Java, and possibly even Carbon.

Unfortunately, the current compiler lacks many of the optimizations built into the Clang's front-end (the part that runs before LLVM IR generation), such as loop unrolling, making it impossible to do well overall on benchmarks. Also, two major pieces of Rune are a mess: the type infrencing code in src/bind.c, and the LLVM code generator in llvm/genllvm.c. Both problems are much harder than we anticipated, and I'll take the blame for choosing the wrong approach in both cases.

The path forward appears to be:

• Rewrite the type infrencing system to be unification-based.
• Rewrite Rune in Rune.
• Write a new code-generator in Rune that outputs C code rather than LLVM IR, so we can benefit from front-end optimizations built into C compilers.
• Win a bunch of memory-intensive benchmarks.

Rewriting Rune in Rune is needed for several reasons. First, I'm the only contributor with lots of experience writing C code backed by DataDraw generated classes. To make it easier for everyone to contribute, we need to switch to writing in Rune rather than C. Also, there are major improvements needed, and we'd rather do this in Rune. Coding in C is grungy. Coding in Rune is simply more fun. To make rewriting Rune in Rune simpler, without the need for type hints, we need an improved type infrencing algorithm. The current recursive algorithm fails in several ways. For more info, see the header comments in src/bind2.c.

Unfortunately, there is no timeline for these tasks...

Bill

[dibyendumajumdar]

I personally like it is written in C and not Rune.
But yacc and lex put me off ... hand-written parser would be easier to follow.

[0dminnimda]

IMHO, for Rune to have any chance to be adopted means it should support at least c level integration
Once you have it, suddenly a lot of the already writted and battle-tested libraries are at your fingertips, ready to be used

[waywardgeek]

Rune already interfaces to C. See builtin/externC.rn. Could we improve on this? Tuples and structs are meant to interact with C, but we haven't finished LLVM IR annotations to ensure they are identical.

As for yacc/lex, there is some further insanity going on. In the rewrite of Rune in Rune, we're going to switch from Bison/Flex to a custom LALR-1 parser that interprets state tables at runtime rather than spitting out code to do the parsing. I've written a prototype, so I'm confident this won't be too much of a challenge. This will enable domain-specific-languages to be written in Rune as libraries .

[0dminnimda]

+1 on switching to use hand-written parser, it seems to be the way to have less headache and more control (in the early stages it can be simple, you probably don't assume rune to be fed million-line files right now, so no need to spend too much time optimizing it, but rather concentrate on really important things)

Also I recommend you to investigate peg parsers, those parsers are somewhat new, but promissing in their way

---

Thank you for refering to builtin/externC.rn!
While reading rune4python.md I've seen

Currently, you can pass in or receive back small integers (<= u64).

And thought that's it. It immideatly occured to me that just passing ints around is not enough for nice integration, thus I've written about fully supporting c integration. (I think it's beneficiary to extend that section of the documentation)

I will look into the source code for more, this project exites me

[waywardgeek]

Thanks for the PR on C integration. I agree it should be tighter. Passing structs needs to be well supported, and maybe pointers to structs and arrays.

Good C++ integration would be even cooler. All C/C++ code compiles down to functions and globals we can link to easily enough.

I experimented with PEG parsers for my old l42 language, which is now in github.com/waywardgeek/not_rune. The main problem I ran into was that PEG parsers don't tell you if you have any conflicts. Conflicts are resolved by rule ordering. For a never-changing format, that's fine, but Rune will support domain-specific language extensions in libraries. The user needs to know if library A causes parsing conflicts with library B, and import order should not matter. They also need error messages telling them they've introduced conflicts when writing gramar extensions. I also experimented with general parsers, but they were too slow.

Debugging shift-reduce and reduce-reduce conflicts isn't fun, but I've written many recursive-decent parsers by hand and many lex/yacc based parsers. lex/yacc, for folks used to debugging the cryptic conflict messages, is a faster way to write parsers, but that's not why Rune uses it. Compared to the rest of the compiler, a recursive decent parser would be a small effort. It's all about supporting DSLs.

[0dminnimda]

I am not as expirienced in parsers as you are. I am more of an enhusiats in language creation so, thanks for bringing up some of those keywords, it is an interesting research material!

As for dsl extention libraries .. If I understood you correctly, you want the library to be able to extend the language's syntax to accommodate for some domain-specific tasks?

Wouldn't it introduce an ablilty to break an existing code?
For example if your program has a variable called 'x' and library assigns this some funcitonality, then it will break the original program.
One way to solve this is soft keywords - keywords that become special only in some places. Although in my opinion this will make things really messy.

Inconstant syntax is a barrier for any kind of syntax highlighting/autocompletion.
This point is probably not as hot and applicable in the near future. Though it is still something to consider.

What do you think about macro system? From what I've seen rust managed it pretty well and it also doesn't break existing code.

Also what benifits does dsl extentions bring to the table over macro system?
How do you see them being used?

[waywardgeek]

I became interested in DSL extensions from what I found in Ruby and Scheme. Scheme's macro system is powerful enough to implement much of Scheme in Scheme. This is largely due to the lack of syntax in Scheme, and this powerful capability has been lost in modern languages. Is that the sort of macro system you had in mind? I assume you're not talking about C preprocessor macros, which languages since C++ have been trying to leave behind.

As for breaking things, DSL extensions should be keyword protected. For example, we might write:

import webtemplateSyntax
webtemplate myPage(param1, param2, ...) {
    <statements useful specifically for web templates>...
}

The 'webtemplate' keyword, if unique among libraries, gives the syntax designer a blank slate where they can safely design new syntax without breaking existing code. Anything within the block can be completely new syntax, or a mix of existing and new syntax. Regardless, it is not possible for this syntax extension to break existing code that does not use webtemplate as an identifier. Only the webtemplate keyword was added to the set of existing keywords when parsing outside the webtemplate block.

Syntax isolation requires that new keywords used within the new syntax block are not recognized as keywords outside of the new syntax block. This requires some interaction between the state-table interpreter doing the token parsing, and the lexer generating the tokens.

However, syntax extensions do make a nice footgun. As you suggested, it is possible to make 'x' a top level keyword, and add a new statement like:

statement := "x" "=" expression
{
   // Causes all assignments to x to be ignored.
}

Yeah... folks will need to try not do depend on poorly designed syntax extensions that polute the global keyword space.