Justinas Stankevičius

The bizarre world of V

2024-09-25T16:00:00+00:00

V ("The V Programming Language", "vlang") is a programming language that first emerged in 2019. A brainchild of Alexander Medvednikov, it is described as a "[s]imple, fast, safe, compiled language for developing maintainable software"¹, supposedly drawing inspiration from languages like Go, Rust, Swift, C, among others. The vision for V is a language that combines all the good parts of the aforementioned inspirations, while avoiding most of the complexity.

V's tall ambitions and its barebone state upon release resulted in mixed reception in online spaces. This article aims to summarize the situation, covering V's unfulfilled promises, questionable decisions, and its uncertain future.

The article has a loose structure of starting with the more easily falsifiable claims about V, and getting more into subjective criticisms later on.

Disclaimers
Analysis
Outro
- Recommended reading
Footnotes

Disclaimers

Some of you may ask, why write this article, especially at this point in time? The flamewars have mostly ceased, V has a small, passionate community of contributors who have not been detracted from their goals by criticism and keep driving the project forward.

The simple answer is, I usually speak (er, write) about such topics only when I think I have something to say that has not been said before. In this case, I have not seen anyone take this broad of a look at V in a while — one that would look at both the historical context and the current state of V, while at the same time providing the receipts.

V will get discussed outside of its own community again. Someone will ask "what was the V controversy all about anyway?", and the creator will dismiss any arguments on frivolous basis again. I think it's better for me to write down my observations once in a long-form blog post, rather than in various ephemeral discussions.

One thing I must note is, I have no experience developing programming languages. Because of that, this article contains minimal criticisms on hardcore technical / scientific PL development basis, and mostly takes the perspective of a programming language user instead.

I'd also like to clearly state that I have zero intention to cause any harassment to Alex or any other V community member. Let's keep the discussion grounded and professional.

Analysis

Shifting timelines

This section mostly deals with historical claims about V's functionality. Anyone evaluating V for use today should obviously make judgement based on the language's current state. However, I think it is crucial to pay attention to the fact that at V's inception, many claims were made that were even more fantastical than the ones V makes nowadays.

As we will see shortly, many of these claims were not true on V's initial release in 2019. Instead of the functionality reaching parity with the promises made, statements about V's features were slowly watered down over time, until some of them quietly disappeared altogether. V as it exists in 2024 still does not live up to even these (now weaker) promises.

C/C++ translation

The ability of the V compiler to translate arbitrary C and C++ code to V was a feature advertised very early on in V's lifecycle. On February 2019, V's website stated:

V can translate your entire C/C++ project and offer you the safety, simplicity, and up to 200x compilation speed up.

[...]

[The translator] supports the latest standard of notoriously complex C++ and allows full automatic conversion to human readable code.²

This claim was supplemented by links to supposed articles detailing the translation of popular C and C++ software:

Read about translating Doom & Doom 3, LevelDB, SQLite.²

The "Doom & Doom 3" anchor linked to https://vlang.io/doom, which may have existed at some point, but has not been archived by the Wayback Machine. Meanwhile, the LevelDB and SQLite links were placeholders (linking to #).

The promised articles then failed to appear in the next few months:

As of March 3rd, 2019, the hyperlinks become inactive. <a> tags are replaced <u> tags — underlined to look like links. The section describing this feature ends with:

[...] LevelDB, SQLite (coming in March).³
On April 4th, 2019, it states "coming in early April" instead.⁴
On May 6th, 2019, the text becomes simply "coming soon".⁵

To this day, the home page says that the (single) article is coming:

A blog post about translating DOOM will be published. ²⁴

Although in 2022, a tutorials/C2V. Translating simple programs and DOOM./README.md appeared in the GitHub repository.⁴⁶ This is probably the article in question, and Alex just forgot to link it from the homepage.

Now, I'll let one who has never procrastinated on their side project or an article cast the first stone, but let's see what Alex says about the actual functionality elsewhere. In a Hacker News thread in late March 2019, he states:

C++ translation will be done by the time the language is open sourced. I can already compile simpler projects.

I support [translating] STL and even plan to support Boost⁶.

On May 19, 2019, vlang.io states:

[The translator] already supports C and will soon support the latest standard of notoriously complex C++.⁷

The claim of "supporting the latest standard of notoriously complex C++" has been downgraded to "will soon support".

Sometime before June 24, 2019, the "C++ translation", among an assortment of other features, gets a "WIP" (work in progress) label on the website²⁹, seemingly as a reaction to the lukewarm reception in the Hacker News thread about the open-source release on June 22, 2019.⁸ One commenter points out:

Now that (part of?) the code has been released, it seems to be little more than a transpiler from V to C, with allowed inlined C, with most advertised features stubbed out.

There might have been further developments on the translation feature during mid-2019 to end of 2020. I have only so much willingness to scour the archives.

On New Year's Day 2021, V's Twitter account announces:

C2V itself will be open-sourced next week.⁹

More time passes without any official news. Then, on June 22, 2022 — roughly 1.5 years after the "next week" announcement — an "Initial commit" is made to the C2V repository.¹⁰ The v translate compiler command is finally officially released in V 0.3 a week later.¹¹

To summarize C2V, the public was once promised that V "supports the latest standard of notoriously complex C++ [...]". 5 years later, it is nowhere near these capabilities. It seems to be able to translate some C code, although features as simple as reading the command line arguments produce invalid V code⁶². V's roadmap states that full C99 support is slated for version 1.0¹⁸, indicating that the C translation feature is still incomplete (C99 is not really some exotic thing anymore).

C++ translation is nowhere to be seen: C2V's tests directory seemingly does not contain a single C++ test case. The part of the code that would seem to be responsible for C++ translation has not been touched since January of 2023, and the method handling declarations of destructors (an essential feature of C++) is empty.¹²

Indeed, it looks like V can not translate any C++:

$ v version
V 0.4.6 fccd7cd
$ cat trivial.cpp
#include <iostream>
using namespace std;

int main() {
  cout << "Hello from V?" << endl;
  return 0;
}
$ v translate trivial.cpp
C to V translator 0.4.0
  translating /home/justinas/vlang-box/trivial.cpp ... C++ top level
C++ top level
C++ top level
C++ top level
C++ top level
C2V command: '/home/justinas/.vmodules/c2v/c2v' 'trivial.cpp'
C2V failed to translate the C files. Please report it via GitHub.

Native x64 code generation

Producing native code without the use of LLVM or a C compiler as an intermediary was another claim made at the very start of V's public existence.

On February 2019, the V homepage stated:

Does V use LLVM?

No. V compiles directly to machine code. It's one of the main reasons it's so light and fast. Right now only x64 architecture is supported.

V can also emit human readable C, which can then be compiled to run on any platform.²

The parts emphasized (by me) seem to imply that the native code generation backend is the main one, and is one of the primary reasons behind V's claims of fast compilation, while the C generation backend is merely an additional option.

On March 30, 2019, Alex once again reiterates on a Hacker News thread:

Right now [V] can emit x64 machine code¹³

V's initial open-source release on June 22, 2019¹⁴ seemingly did not contain any code implementing a native x64 backend. What looks like a call to the supposed x64 backend is commented out.

That same day, Alex stated in a Hacker News comment:

Work on x64 generation started back in August [...] I haven't touched it in a while, and it simply doesn't compile at the moment.¹⁵

NB: Seeing as this comment was made in July 2019, "August" might be a mistake, unless he meant August 2018 or similar.

As with many other features, "direct machine code generation" gained a "WIP" flag soon after the open-source release.⁸ Once again, I did not bother to go through the entire Git history. However, it seems that the native x64 backend in some state first appeared in a commit made on Nov 22, 2019¹⁶, and the feature is mentioned in the release notes for version 0.1.23¹⁷.

Still to this day, it does not seem that the native backend is in a functional state. For example, it fails as soon as one tries to declare an array:

$ v version
V 0.4.6 4a7c70c
$ cat hello.v
fn main() {
    foo := [1, 2, 3]
    println(foo)
}
$ v -b native hello.v
/home/justinas/vlang-box/compiler/vlib/builtin/builtin.c.v:358:1: warning: globals are not supported yet
  356 | }
  357 |
  358 | __global total_m = i64(0)
      | ~~~~~~~~~~~~~~~~~~~~~~~~~
  359 | // malloc dynamically allocates a `n` bytes block of memory on the heap.
  360 | // malloc returns a `byteptr` pointing to the memory address of the allocated space.
/home/justinas/vlang-box/compiler/vlib/builtin/builtin.c.v:718:1: warning: globals are not supported yet
  716 |
  717 | @[markused]
  718 | __global g_main_argc = int(0)
      | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  719 |
  720 | @[markused]
/home/justinas/vlang-box/compiler/vlib/builtin/builtin.c.v:721:1: warning: globals are not supported yet
  719 |
  720 | @[markused]
  721 | __global g_main_argv = unsafe { nil }
      | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  722 |
  723 | @[if vplayground ?]
/home/justinas/vlang-box/compiler/vlib/builtin/builtin.v:25:1: warning: globals are not supported yet
   23 |
   24 | // will be filled in cgen
   25 | __global as_cast_type_indexes []VCastTypeIndexName
      | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   26 |
   27 | fn __as_cast(obj voidptr, obj_type int, expected_type int) voidptr {
native error: unknown variable `foo`

Or a map:

$ cat map.v
fn main() {
    mut a := map[string]int{}
    a["foo"] = 42
    println(a)
}
$ v -b native map.v |& tail -n 1
native error: expr: unhandled node type: v.ast.MapInit

Or a struct:

$ cat struct.v
struct Foo {
    greeting string
}

fn main() {
    foo := Foo{greeting: "hello"}
    println(foo)
}
$ v -b native struct.v |& grep "error"
native error: unsupported type for mov_reg_to_var ast.TypeInfo(ast.Struct{

You get the idea. "Compilation directly to machine code", touted as the secret of V's fast build times in 2019, may as well not exist even in 2024, let alone back then. It does not seem like there is much effort to change this situation either: the change log for the recent V 0.4.6 release lists 27 merged pull requests in the "C backend" category, whereas the only mention of the native backend is "ci: update native backend ci matrix"⁴¹.

Memory unmanagement

One of the main selling points of V from the very beginning was "innovative memory management".

For context, it is well known that all mainstream paradigms of memory management have certain problems:

Garbage collection usually has a non-negligible performance cost at runtime.
Manual memory management à la C's malloc() and free() is error prone, and in practice results in issues such as memory leaks, double frees, and use-after-free.
The RAII + ownership + borrow checking model of Rust avoids the disadvantages of the two previously mentioned models, but comes with its own cognitive cost of having to explicitly encode information about lifetimes into the program when the compiler is not smart enough to figure it out automatically.

The memory management model of V was envisioned to solve all of these problems at once: there would be no garbage collector, and you wouldn't need to manage memory manually either. Instead, the compiler would take care of freeing memory for you ("like Rust"⁴), but without having to struggle with lifetime specifiers or anything of the sort.

Early V

In early 2019, V's website stated:

Is there garbage collection?

No. V's memory management is similar to Rust but much easier to use. More information about it will be posted in the near future.³

About a month later, this section was expanded:

Is there garbage collection?

No. V manages memory at compilation time (like Rust). Right now only basic cases are handled. For others, manual memory management is required for now. The right approach to solve this will be figured out in the near future.⁴

After V's initial release, the docs were updated to describe the unimpressive status quo:

There's no garbage collection or reference counting. V cleans up what it can during compilation. For example:

[...]

The strings [...] are cleaned up when the function exits.

[...]

For more complex cases manual memory management is required. This will be fixed soon.

V will detect memory leaks at runtime and report them. To clean up, for example, an array, use the free() method[.]¹⁹

I did not go and check whether V 0.0.12 lived up to these promises as documented, because I have no clue how to correctly compile early versions of V. I was, however, able to pull V 0.1.21 (released Sep 30th, 2019) from nixpkgs, and compile a version of the example in the above documentation (after applying some syntax fixes). Valgrind is not happy with it.

$ v version # NB: version in nixpkgs is tagged 0.1.21, V self-identifies as 0.1.20.
V 0.1.20 5ac62bb
$ cat autofree_initial.v
import strings

fn draw_text(s string, x, y int) {
    /* empty */
}

fn draw_scene() {
    name1 := 'Alice'
    name2 := 'Bob'
    draw_text('hello $name1', 10, 10)
    draw_text('hello $name2', 100, 10)
    draw_text(strings.repeat(`X`, 10000), 10, 50)
}

fn main() {
    for i := 0; i < 1000; i++ {
        draw_scene()
    }
}
$ valgrind ./autofree_initial |& grep "definitely lost"
==1107111==    definitely lost: 10,024,000 bytes in 4,000 blocks

Seeing how draw_scene allocates roughly 10⁴ bytes, and it is called 10³ times, Valgrind reporting about 10⁷ bytes is an indication that every single string allocated is leaked. In the intermediate C code (generated via v -o autofree_initial.c autofree_initial.v) it is evident that draw_scene does not contain any calls to free().

At around the same time, V decided to aim for an even stronger guarantee:

(Work in progress) [...] If your V program compiles, it's guaranteed that it's going to be leak free.⁶³

Even Rust, despite having compile-time memory management, does not promise complete freedom from memory leaks, and in its documentation demonstrates a scenario in which memory leaks happen.⁶⁴

V 0.2 and the "zero leaks" demo

It's not a surprise that V pre-0.2 leaked so much memory, because despite the claims in documentation, "autofree" seemingly did not exist at all back then. It was finally officially introduced, gated behind an -autofree flag, in V 0.2 half a year later.²⁰

As an addition, that same day Alex released a video that supposedly demonstrates how autofree prevents leaks²¹. All the demo shows is that the Ved editor, after opening it and scrolling through a large text file once, caps at a certain, lower amount of memory used when compiled with -autofree compared to the same program compiled without autofree (i.e. using the pre-0.2 behavior of "YOLO leak everything"). In no way does it prove (as Alex would later state it does²³) that the program has "zero leaks". For that, at the very least, I would expect the compiled executable to be analysed via Valgrind.

I will add that a video like this is may be a great tool to hype up your creation, but a terrible way to actually prove your feature works, since there are multiple ways in which such demo could be "enhanced" to misrepresent the actual situation. The only proof accepted by the wider community should be instructions that allow one to independently reproduce the results.

But such instructions were not made available, so let's try our best to reproduce by getting V 0.2 and the version of Ved that was available at that time:

$ wget https://github.com/vlang/v/releases/download/0.2/v_linux.zip
$ unzip v_linux.zip
$ mv v v0.2
$ ./v0.2/v version
V 0.2 e4f94b6
$ git clone https://github.com/vlang/ved
$ cd ved
$ git checkout afa14852b78df5234072a3b321e2d1ecb611e120  # last Ved commit before V 0.2 release
$ wget https://raw.githubusercontent.com/azadkuh/sqlite-amalgamation/master/sqlite3.c # get an 8MB file to test with

To test, I will compile Ved, then run it under Valgrind. Once Ved starts, I'll hold the "Page Down" key on my keyboard to scroll to the end of the file, and then promptly close the application so that Valgrind reports the results.

Let's try without any flags first:

$ ../v0.2/v .
Compilation with tcc failed. Retrying with cc ...
$ valgrind ./ved ./sqlite3.c |& grep "definitely lost"
==397075==    definitely lost: 23,386,605 bytes in 3,350,781 blocks

Without autofree, Ved leaks 23 megabytes. What about with autofree?

$ ../v0.2/v -autofree .
Compilation with tcc failed. Retrying with cc ...
$ valgrind ./ved ./sqlite3.c |& grep "definitely lost"
==398821==    definitely lost: 1,584,084 bytes in 1,573,227 blocks

That's much better, Ved only leaks 1.5 megabytes. However, that is not the same as "[a]ll objects are freed during compilation" as the V homepage still states today.²⁴

The V 0.2 release notes²⁰ also promised that autofree would be "enabled by default in 0.3", presumably because it was expected to become good enough by that point. That, of course, did not happen. Instead, before the V 0.2.4 release, garbage collection based on Boehm GC was added²², and remains the default memory management option to this day. In a GitHub discussion about autofree, Alex explains the reasons behind this:

When I started working on V, I was very anti-GC, expecting them to be slow and use a lot more RAM. I integrated a GC just for a test, and was surprised at how well it worked with V [...]

What started as a test and a temporary way to allow developers to write leak free programs, became the stable and well working default option. ²³

I'll give it to him, that is fair reasoning: GC performs "well enough", and for now, does a better job than autofree could.

Status quo

What bothers me the most is how no solid principle behind autofree has ever been explained (at least to my knowledge). V's homepage simply states:

[T]he compiler inserts necessary free calls automatically during compilation.²⁴

This simplistic statement sounds like a thought that any young programmer might have when they encounter manual memory management for the first time: "computers are good at automating stuff, so why can't the compiler insert calls to free() for me?". I don't know enough about PL theory to confidently state whether that is possible in the general case, but people smarter than me seem to think it is not⁷².

When asked about how autofree will function, V's creator sometimes draws parallels to languages such as Rust³ and Lobster²⁵, the latter of which I know little about. There are no explanations on how the ideas from these languages would be implemented, nor how V would avoid limitations of their memory management models.

The GitHub discussion²³ about autofree only gives a couple of notes about techniques that enable autofree to function. One technique mentioned is simple escape analysis: heap-allocated types that do not leave the function they were created in are said to be deallocated at the end of the scope.

Bizarrely, the example that is supposed to demonstrate that autofree handles this case, fails:

$ v version
V 0.4.6 4a7c70c
$ cat autofree.v
struct Foo {
    x int
}

fn foo() {
    foo := &Foo{x: 3} // never escapes
    println(foo)
}

fn main() {
    foo()
}
$ v -autofree autofree.v
$ valgrind ./autofree |& grep "definitely lost"
==146888==    definitely lost: 20 bytes in 2 blocks

Strings, on their own, are covered by escape analysis:

$ cat string.v
import strings

fn produce_str() string {
    foo := strings.repeat(`A`, 10)
    return foo
}

fn main() {
    println(produce_str())
}
$ v -autofree string.v
$ valgrind ./string |& grep "no leaks"
==597912== All heap blocks were freed -- no leaks are possible

But if you put them in a struct, they leak:

$ cat struct.v
import strings

struct Foo {
    x string
}

fn new_foo() Foo {
    return Foo{x: strings.repeat(`A`, 10)}
}

fn main() {
    println(new_foo())
}
$ v -autofree struct.v
$ valgrind ./struct |& grep "definitely lost"
==598343==    definitely lost: 11 bytes in 1 blocks

Not even primitive types are safe: trying to take a pointer to an integer automatically promotes this integer to a heap allocation that is never deallocated⁴⁵.

Escape analysis fails in the other direction too. Trying to return a &string from a function triggers a null pointer dereference, because the string's inner buffer is freed and its address overwritten by 0 before returning:

$ v -autofree dangling.v
$ cat dangling.v
import strings

fn produce_str() &string {
    str1 := strings.repeat(`A`, 10)
    str2 := &str1
    return str2
}

fn main() {
    println(produce_str())
}
$ v -autofree dangling.v
$ ./dangling
7f04dab88a80 : at ???: RUNTIME ERROR: invalid memory access
/tmp/v_1000/dangling.01J1QTVGTZXJMVJVDE3CKG1WAQ.tmp.c:9761: by string_clone
/tmp/v_1000/dangling.01J1QTVGTZXJMVJVDE3CKG1WAQ.tmp.c:11231: by string_str
/tmp/v_1000/dangling.01J1QTVGTZXJMVJVDE3CKG1WAQ.tmp.c:13086: by main__main
/tmp/v_1000/dangling.01J1QTVGTZXJMVJVDE3CKG1WAQ.tmp.c:13131: by main

If we remove the secondary function, V narrowly avoids a double free.

$ cat double.v
import strings

fn main() {
    str1 := strings.repeat(`A`, 10)
    str2 := &str1
    println(str2)
}
$ v -autofree double.v
$ ./double
&AAAAAAAAAA
double string.free() detected

The "double free detected" log message in this case comes from V itself, not a sanitizer or the allocator. V string type's is_lit field acts as something resembling Rust's drop flags⁴⁴ here.

Dangling pointers can also be produced using an array:

$ cat array.v
fn ref_from_array() &int {
    a := [1, 2, 3]
    return &a[0]
}

fn main() {
    println(ref_from_array())
}
$ v -autofree array.v
$ valgrind ./array |& grep -A 4 "Invalid read"
==594613== Invalid read of size 4
==594613==    at 0x634B7B: main__main (in /home/justinas/vlang-testcases/array)
==594613==    by 0x63515C: main (in /home/justinas/vlang-testcases/array)
==594613==  Address 0x4a750f0 is 0 bytes inside a block of size 12 free'd
==594613==    at 0x484988F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)

The only other "technique" described²³ is not worth discussing in detail. It simply states that assigning a string variable to another one (foo := bar) copies the string instead of aliasing. This is true, however not sufficient to avoid memory management issues (as we've demonstrated by using string pointers). Alex also adds that for arrays, bare by-value assignment is forbidden, and the user is always asked to clone() the array explicitly. This is not true (anymore?), arrays are cloned implicitly, just like strings:

$ cat array.v
fn main() {
    foo := [1, 2, 3]
    bar := foo
    println(bar)
}
$ v run array.v
[1, 2, 3]

To summarize, autofree in mid-2024:

Eagerly frees standalone strings and arrays at the end of the block and does not leak them, but this results in dangling pointers and null dereferences.
Does not handle structs at all, and introducing a struct in the mix breaks the small amount of functionality that otherwise works for strings and arrays.
Does not free primitive types that needlessly escape to the heap in the first place.

In 2019 it was stated that there is no garbage collection³ (implying V's "innovative memory management" eliminates the need for GC — it did not say "there is no GC, but we will add it soon and it will be the main technique"). Today, autofree is impotent, only handling a few individual patterns of allocation, and often still getting them wrong. This is an embarrassing state of affairs. V is not a language with "innovative memory management", it is a language using an off-the-shelf garbage collector by default, with an "autofree" option that is not viable in any meaningful way.

To be fair, autofree is at least described on the current website as "still experimental and not production ready yet". However, in the same paragraph, it says that, when used, "[autofree] takes care of most objects (~90-100%)"²⁴. To my knowledge, Alex has never shared a methodology to measure this.

I would test autofree using present-day Ved and the latest V compiler, alas, it immediately crashes if compiled with autofree.

$ pwd
/home/justinas/vlang-testcases/ved
$ git rev-parse HEAD
91d395901829d3c23073ff7d4ae1b137f1a14742
$ v version
V 0.4.6 b6c7b46
$ v -autofree .
$ ./ved
size=gg.Size{
    width: 2560
    height: 1440
}
V panic: as cast: cannot cast `map[string]toml.ast.Value` to `[]toml.ast.Value`
v hash: b6c7b46
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:16660: at _v_panic: Backtrace
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:17160: by __as_cast
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:48683: by toml__Doc_value_
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:48652: by toml__Doc_value
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:49387: by main__Config_init_colors
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:49359: by main__Config_reload_config
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:51422: by main__main
/tmp/v_1000/ved.01J1JK2CEB2HCB4CA18WYEWKQB.tmp.c:55727: by main

Why enabling autofree makes TOML serialization crash remains an open question.

V's website has since pivoted from advertising "innovative memory management"⁶⁵ to "flexible memory management"²⁴, boasting 4 distinct ways of managing memory. Another review of V, linked in "recommended reading" goes into more detail about why swapping between these models is not at all as seamless as the docs suggest.

Redefining terms

More bizzareties lie in the smaller bullet points of V's feature list, where V authors outline several more hefty goals which are not achieved, and at times seemingly not even well understood by the team.

Undefined behavior

V's current homepage proudly proclaims that the language allows "[n]o undefined behavior"²⁴. The claim even has a "new!" tag, indicating that this has recently been achieved, as opposed to when this feature was marked "WIP" back in 2022³⁰ (although the "WIP" caveat remains in the README on GitHub¹).

However, the claim of "no undefined behavior" is far from holding up. In many cases, primitive V code is translated 1-to-1 to syntactically equivalent C code with all the nasty implications of it.

For example, V code that attempts to divide by zero is translated to C code that attempts to divide by zero, a well known case of undefined behavior in C³¹. An issue about this was created in V's issue tracker 3 years ago, and then promptly closed by contributors claiming that "the behavior is the same as Go"³².

Whether the behavior is "the same as Go" is irrelevant here, since V's claim is that it produces no undefined behavior, not that integer division works "exactly as in Go". It is trivial to demonstrate that the behavior is, in fact, not always the same as in Go: in my comment on the issue, I gave an example of V (or rather the C compiler that it delegates to) producing a program that yields 1/0 = 0, which is already different from Go. The behavior of 1/0 = 0 would perhaps be an okay trade-off if it was defined by V and did not trigger undefined behavior in the intermediate C code But it's not, and it does.

The misunderstanding of what undefined behavior means is not limited to external V contributors. When this issue was brought up in a Hacker News thread, Alex reiterated the obviously false claim that division works "just like in Go"³³. The only way that I imagine one could argue that V produces no undefined behavior here is by applying some mental gymnastics à la "division by zero in V is defined to produce undefined behavior in the intermediate C code".

I get it, undefined behavior is not a straightforward concept to grasp. I myself did not understand it well until I read some great articles³⁴⁷³. The problem is claiming that your language has a property X, where X generally has an agreed upon meaning, and then moving the goalposts of what constitutes X.

Static binaries that are not

V's current homepage²⁴ claims that:

[By compiling V code y]ou get a single statically linked binary [...] without any dependencies.

However, this is easily confirmed as false:

$ cat hello.v
fn main() {
    println("hello world")
}
$ v hello.v
$ file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, not stripped
$ ldd hello
        linux-vdso.so.1 (0x00007ffe0bdb1000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0c90200000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0c905f5000)

By default, V produces a binary that is both dynamically linked and relies on the C standard library.

I don't have a bone to pick with the decision to use libc. As Alex rightly points out, for some OSes, libc is the only stable system API³⁵. I do have a small issue with saying "libc is not really a dependency", because a dependency is a dependency.

But my main qualm with the claim is that "static binary" has a defined meaning, and V does not output static binaries by default. In fact, it is hard to find any documentation on how to produce static binaries in V at all. In a GitHub discussion thread, Alex suggests using musl-gcc for this³⁵. Let's ignore the fact that musl-gcc is an external tool which one must acquire separately, and try the suggestion:

$ sudo apt install musl-tools
<...>
$ v -cc musl-gcc hello.v
==================
/usr/bin/ld: pthread_start.c:(.text+0xa9): undefined reference to `__pthread_unregister_cancel'
collect2: error: ld returned 1 exit status
...
==================
(Use `v -cg` to print the entire error message)

builder error:
==================
C error found. It should never happen, when compiling pure V code.
This is a V compiler bug, please report it using `v bug file.v`,
or goto https://github.com/vlang/v/issues/new/choose .
You can also use #help on Discord: https://discord.gg/vlang .

It does not work.

StackOverflow points me to the -freestanding flag³⁶. Let's try that out:

$ cat array.v
fn main() {
    mut foo := [1, 2, 3]
    foo.sort()
    println(foo)
}
$ v -freestanding array.v
$ file ./array
./array: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=19986c1fa887f421581deb1982106b06a5a33ee6, not stripped
$ ./array
V panicsort does not work with -freestanding

The -freestanding flag does make V produce a static binary, but it also has the effect of not linking to libc at all. Combined with the fact that V implements core functionality like sorting an array by delegating to C standard library functions³⁷, this means that a "freestanding" V program can barely do anything non-trivial.

Finally, after filing an issue on V's GitHub I was pointed by a kind stranger to the (obvious in hindsight) solution of using -cflags '-static'⁴². That works:

$ v -cflags '-static' hello.v
$ file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=fa28c8ba332aa9a6f7b5e490e5b8442d6b840ee6, for GNU/Linux 3.2.0, not stripped

Since all this does is request static linkage from the C compiler that V delegates to, I am not sure if there is a way to attempt static linking when using the native code generation backend. But given the state of that backend as a whole, it is currently a pointless exercise.

In short, I think if V promises static binaries, it should either produce them by default, or give an easy, documented way to switch to static linking (e.g. a -static flag in the v command-line tool).

Previously, Alex has stated that "statically linking glibc is not possible"⁴³, which is not technically true as we've just shown (although glibc is definitely not an ideal candidate for static linking³⁸). I am confused why V's homepage promises something that Alex thinks is impossible to achieve.

Other misnomers

Among its other safety features, V claims:

No null (allowed in unsafe code)²⁴

As we have now come to expect, this is not universally true, but at least in this case, the detailed docs spell it out:

Zero-value references, or nil pointers, will NOT be supported in the future, for now data structures such as Linked Lists or Binary Trees that rely on reference fields that can use the value 0, understanding that it is unsafe, and that it can cause a panic.³⁹

Strangely, nil requires unsafe, but the equivalent 0 literal in a pointer context does not. This suggests very primitive reasoning by the compiler: an "evil" keyword is forbidden, but completely equivalent code without the keyword is allowed.

Option types have existed in V since version 0.3¹¹ (before that, they were a weird Option/Result mix). With that in mind, it is weird why it is still allowed to have a 0 valued pointer in safe code instead of forcing users to use ?&T to achieve the goal of having an optional reference. Perhaps it is because a bunch of places in the standard library use 0 pointers and nobody has fixed that yet?

Coincidentally, it was also the version 0.3 change log that announced "[n]ull can be used in unsafe only (for example, for C interop)"¹¹.

I don't think it is honest to advertise "no null" on the front page, and then immediately walk it back in the docs. In addition, dereferencing a null pointer is undefined behavior in C, yet again proving that "no undefined behavior" claims are unsubstantiated.

V also previously claimed "pure functions by default"²⁹, and many people at the time pointed out that allowing I/O in functions is usually a dealbreaker for considering functions "pure".

Today, "purity" as a term is not used, but V's documentation makes the strange claim that "[function] evaluation has no side effects (unless the function uses I/O)"⁴⁰. I am entirely unsure why the authors decided to make this claim, which sounds as empty as "your programs are guaranteed to be bug-free as long as you do not introduce any bugs". In my opinion, this section of the documentation would look a lot more solid if it only contained the core, unambiguous claims: global variables are not allowed by default, and function arguments are immutable by default.

Casting a wide net

Another thing I find strange about the V project is just how much secondary ground it intends to cover, despite the core language being in an unfinished state. I have no problem with community projects like Vinix — I think it is very cool to build a complex software project, such as an operating system, as a proof-of-concept that your creation is not a toy language. What concerns me is bloat in the core v project itself, the current state of these secondary features, and the impact it may have on the ETA of the eventual V 1.0 release.

Backends

For one, take V's several backends. We've already discussed how C code generation is V's main backend, and the direct-to-machine code backend, once claimed to be the main one, is in a state where all it can compile is essentially "hello world".

That did not stop V from gaining additional backends. A prominent one is the JavaScript backend, which comes in four flavours:²⁶

    * `js`                - V outputs JS source code which can be passed to NodeJS to be ran.
    * `js_browser`        - V outputs JS source code ready for the browser.
    * `js_node`           - V outputs JS source code to run with nodejs.
    * `js_freestanding`   - V outputs JS source code with no hard runtime dependency.

The list is majorly confusing, because from the descriptions it is unclear how js and js_node different. It is also unclear what "no hard runtime dependency" means. Perhaps it is supposed to only generate code that fits some ECMAScript standard, and does not assume any browser-ish or Node.js-ish constructs to be present?

Either way, trying to figure out the supposed differences between these backends is moot, since all four produce the same code that hard-depends on a Node.js-like environment.²⁷ That's right, using v -b js_browser does not produce JS code that can run in a browser. The generation of code for Node.js works to some degree, but ends up producing invalid JavaScript syntax when trying to do very simple things such as splitting a string into characters.²⁸

But I'm not worried about syntax. What I think is hard in transpiling something like V into something like JavaScript is semantics. For example, V utilizes a traditional synchronous model with explicit threads and coroutines, while JS usually runs on an implicit, single-threaded event loop.

These two models of concurrency are not directly compatible. So how does V solve this? We can observe that threading in V works normally when using the C backend:

$ cat threads.v
import rand
import rand.seed
import rand.pcg32
import time

fn run (i int) {
    println("hello from thread ${i}")
    mut rng := &rand.PRNG(pcg32.PCG32RNG{})
    rng.seed(seed.time_seed_array(pcg32.seed_len))
    duration := rng.i64_in_range(100_000, 1_000_000) or { panic(err) }
    time.sleep(duration)
    println("goodbye from thread ${i}")
}

fn main() {
    for i := 0; i < 10; i++ {
        spawn run(i)
    }

    // I should build up an array of threads here and use threads.wait(),
    // but that does not compile with the JS backend.
    time.sleep(10_000_000)
}
$ v run threads.v
hello from thread 0
hello from thread 1
hello from thread 2
goodbye from thread 1
hello from thread 3
hello from thread 4
hello from thread 5
hello from thread 6
hello from thread 7
hello from thread 8
goodbye from thread 3
hello from thread 9
goodbye from thread 0
goodbye from thread 6
goodbye from thread 5
goodbye from thread 9
goodbye from thread 2
goodbye from thread 4
goodbye from thread 7
goodbye from thread 8

Threads interleave as expected. But what happens if transpile the same code to JS and run it in Node?

$ v -b js -o threads.js threads.v
$ timeout 5 nodejs threads.js
hello from thread 0
goodbye from thread 0
hello from thread 1
$ timeout 5 nodejs threads.js
hello from thread 0
$ timeout 5 nodejs threads.js
hello from thread 0

It just... hangs? Non-deterministically? There's plenty wrong with the generated code: threads are converted to Promises and time.sleep is implemented as a busy-wait, so the "threads" block the event loop and never interleave. Despite all this, I am still unsure why the program "hangs" and a random thread will seemingly block forever (or for a very long time).

I'll leave it as an exercise to the reader to figure out what V's os.read_file translates to in JS. Hint: it's a no-no in asynchronous code.

What about the Go backend?

$ cat hello.v
fn main() {
    println("hello world")
}
$ v -b go -o hello.go hello.v
using Go WIP backend...
hello.v:2:5: error: unknown function: println
    1 | fn main() {
    2 |     println("hello world")
      |     ~~~~~~~~~~~~~~~~~~~~~~
    3 | }

Let the record show that neither Go nor JS backends are marked as "experimental" in the v build docs, although the native and wasm backends are²⁶. The JavaScript backend is also listed as a "key feature" alongside the C backend in V's GitHub README, implying some degree of importance¹. Again, V 0.4.6 release notes do not seem to mention JavaScript or Go backends at all, suggesting that not much is being done to drive them forward.

The value proposition of having so many different backends is generally unclear to me. Targeting JavaScript because it is the only language that runs natively in the browser (apart from WebAssembly, which is still limited) is somewhat understandable, but compiling to another high-level language such as Go, which V shares a lot of design goals with anyway, seems like a giant timesink. It also raises some of the same questions about semantics: will both of V's threads and coroutines be backed by Go goroutines, or just one of them? How will threads and coroutines differ when targeting Go?

The fact that the compiler does not seem to have an intermediate representation, and every backend has to generate code directly from the AST also means there is a lot of the same work necessary for each backend.

Batteries included (but not yet charged)

Drawing most of its inspiration from Go, V intends to be a small, "simple" language, but at the same time, have a rich standard library. While the breadth of V's stdlib is impressive, there is a lot of work to be done regarding its quality. I only took a cursory look at V's standard library, but it was enough to observe current incoherences.

For one, io.Reader and io.Writer interfaces appeared in V 0.2²⁰, yet to this day the standard library only consumes these interfaces in a few places. Instead, most of the modules require the user to fully buffer the data in memory.

For example, the net.http module⁴⁷ only accepts the request body as an in-memory string, and returns the response body as a single in-memory buffer as well. To stream the response body instead, one must either:

Manually set the on_progress callback and the stop_copying_limit property
Use the strange Downloader interface, which uses the above under the hood.

There does not seem to be an obvious way to stream the request body.

This is a weirdly high-level and inefficient-by-default interface for a language that claims to target, among others, systems engineers, and aims to be "as fast as C"²⁴. The same pattern of "take a buffer with the entire input, return a new allocation with the entire output" is prevalent in other modules such as compress and encoding.

The context module, introduced before V 0.2.4⁴⁸ and modelled after Go's module of the same name, intends to implement request cancellation and deadline functionality. Yet, three years later, the only meaningful usage of the context.Context type outside of tests seems to be in the vweb web framework, where a context.EmptyContext{} is used once⁴⁹. Request cancellation is a problem that is generally applicable to any subroutine that does I/O or any other kind of potentially long-running task, not just V's web framework.

There are a lot of smaller weirdnesses in V's stdlib, but I have no intention to delve deep into code quality in this article. All of these flaws seem fixable without changes to the core language itself.

That said, I'd like to mention one more decision I feel dubious about. Features like threads and channels being first-class citizens in the language have the advantage of creating a very good developer experience when the included tools are sufficient. However, this also means that versions of these primitives created by third parties can not take advantage of the syntax/compiler level support.

This is also one of my gripes with Go, the language I used professionally for the past few years: while the built-in primitives are good enough in many cases, once you hit their limitations and need to reach for a 3rd-party library, you lose the fluent integration with the language. You can't go an alternative thread implementation, nor can you select and <- from a third-party channel. Before the introduction of generics in Go, similar "second-class citizen" status applied to other abstractions outside the standard library, such as data structures.

Where V, in my opinion, takes this to the extreme, is the introduction of language-level SQL syntax that supports its built-in ORM⁵⁰. While SQL is still seen as the king in the database world, it does feel strange that support for this specific technology is embedded so deeply into the V toolchain itself. Given the fact that V currently offers no way for users to introduce similar language syntax extensions by themselves (which can be done in some other languages, e.g. in Rust via macros), this means that interacting with any non-SQL databases will feel massively less nice.

At the very least, it seems possible for 3rd-party SQL database drivers to utilize integration with the sql keyword by implementing the orm.Connection interface, although if the database uses a different dialect, then the support for the dialect needs to either be upstreamed to the orm module, or the database backend needs to re-implement parts of the lower-level orm logic.

V 1.0 and the stability guarantee

The reason why V's extensive scope matters is the implications it has for deliverability of a stable V 1.0 release. V's GitHub repo states:

The V core APIs (primarily the os module) will also have minor changes until they are stabilized in V 1.0. Of course, the APIs will grow after that, but without breaking existing code.

After the 1.0 release V is going to be in the "feature freeze" mode. That means no breaking changes in the language, only bug fixes and performance improvements. Similar to Go.

Will there be V 2.0? Not within a decade after 1.0, perhaps not ever.¹

This is not too dissimilar to 1.0 stability guarantees that languages like Go or Rust provide. V's stability goals are not documented in as much detail, but at the very least we can infer that the syntax of the language itself should not change after 1.0, which seems like an obvious and achievable goal.

Additionally, while V's standard library will continue to evolve, V promises to not make breaking changes to it (e.g. removing any public items or changing function signatures) after the 1.0 release. Given the current state of the stdlib, this will require a huge undertaking of its own. Because no detailed standards for idiomatic V code were set early-on, over time the standard library accumulated modules with very different coding conventions. Before V 1.0 can be released, someone, or a group of people, will need to go through every single element of the ever-growing standard library and ensure that these interfaces fit together well and do not hamper future evolution.

Given how small the core V team is — a quick look at the GitHub stats indicates that only a few developers have been consistently commiting code for the past couple of years and at most a couple dozen are merging PRs — this responsibility will likely fall on Alex himself.

V's roadmap for version 1.0 also mentions the goals to make autofree production-ready and implement full support for the C99 standard in C2V⁵¹. It does not say anything about additional backends, such as direct x86-64 code generation or the JavaScript backend. If it is intended for these to reach complete parity with the C backend, such that users of V can switch between the 2 or 3 first-class backends seamlessly, V 1.0 may take many more years, if not decades, to complete at the current development pace (or, frankly, at any development pace).

The author

This will be the only section where I talk about Alexander, the creator of V, as a person, or rather his online presence. Once again, I'll remind the reader that it is not my intention to bully anyone, and I'll try to only bring up Alex's behavior as it is relevant to his public software projects.

"It's just a bug"

I am not the first to criticise V (and I might not be the last). The criticism is not always taken in stride by Alex or the contributors.

One common answer to commenters pointing out the deficiencies of V is "it's just a bug, we will fix this / we have already fixed this since", or pointing out that the language is at version 0.X, and thus should obviously be seen as pre-production.

I agree that bugs are normal in any software project. However, there must be a line somewhere marking the difference between "generally works, but has some bugs", and "generally does not work". Otherwise, to invoke reductio ad absurdum for one moment, I could claim to have solved the halting problem by writing a program that answers the halting question for one specific application, and then stating that the fact of it not working for any and every program is "just a bug".

I'd like to think that the sections above have pointed out several core areas in V where the problems run deeper than "just having a few bugs here and there". The lack of solid principles while aiming to surpass everything that has come before, the almost complete lack of a plan on how to get there, and at times the V team not fully understanding what they're promising does not inspire confidence.

There are other ways Alex deflects criticism, such as stating that critical blog posts come from "V haters", or that the critics are either creators of other similar programming languages, or shills for these PLs, and thus have a vested interest in scrutinizing V unfairly. To give Alex some credit here, I've seen one or two novelty accounts on Hacker News dedicated solely to critical or outright mean comments regarding V, and I don't think that's a very cool thing to do. As for the second point, I'm aware that at least the creator of Odin did criticize V before the initial release⁶¹, and I won't claim that all of his criticisms were completely valid (although V's closed-source nature and a somewhat ambiguous feature list back then did not make it easy to fully evaluate the language).

"Things change"

As I've covered before, V's memory management claims have changed over time from "at compile time, like Rust" to "mostly at compile time, falling back to RC" to "mostly at compile time, falling back to tracing GC".

bheadmaster: The documentation I've quoted and posted a source to specifically says that all objects are freed either by 1) autofree or 2) reference counting.

amedvednikov: You pointed to an old version of the documentation. RC was changed to tracing GC. Things can change in the design.⁶⁷

My problem with this is that every time without fail new features, whatever state of quality (or existence) they are in, are announced loudly, but when these plans eventually hit a wall, the deprioritization or material downgrades in these features are but a footnote or a silent change to the homepage.

At some point, we have to admit that V is having an identity crisis. V was supposed to be "safe", but does not enforce data race safety, ensure valid memory references, or prevent undefined behavior. It was supposed to be "as fast as C", but the creator does not consider 5% slowdown a meaningful difference, using the strange argument that companies such as Facebook / Meta must not care about such performance delta [on the backend, where V would most likely be used], because they ship heavy JavaScript to their users' browsers⁶⁸.

V is "compiled" and it could be argued that it is "simple", so at least 2 out of its 4 defining qualifiers stand. What is V today, other than a much less mature sparkling Go?

A broader pattern

Before V, there was Volt – Alex's project to create a "fast native desktop client for all major messaging services"⁵². Although the archive of Volt's website is only available starting from 2018, it seems that in 2017, Volt already existed as Eul⁵³.

Volt seems to have followed a similar "overpromise and underdeliver" path as V. A snapshot of the site taken on June 15th, 2018⁵² contains the claim that the 1.0 release for Volt is slated for June 15 (that very same day), and the support for most chat platforms — everything from Telegram to Signal — is supposed to come in June (presumably 2018).

This seemingly did not happen, as commenters on a Hacker News thread in early 2019 point out⁵⁴:

I want to like this, ever since it was called eul. But features are consistently pushed back which suggests that the dev doesnt have a great handle on what is going on, the availability of mac/linux versions seems consistently misleading, it’s not open source, and it uses icons of services not available (e.g. gmail) in a way that seems dishonest. idk.

More than a little disingenuous. While they advertise that the app supports a number of services, when you actually download the app and try it only Slack and Skype are available - the rest are "coming later in February"

Volt's current website⁵⁵ offers a download of version 0.96 for ARM-based Mac OS machines only (other platforms are to be supported "soon"). Icons for messaging services seem to indicate that only Slack, Gmail, Twitter and Discord are supported, with support for 9 more services yet to come. Volt's GitHub repository seems to only be used as a deserted issue tracker and downloads page, despite the 5 year old README stating that "In 2021 [...] the app [will be] open-sourced"⁵⁶.

Despite not taking the messaging world by storm, Volt led to creation of V. Before V had a dedicated website, it had a section on volt.ws⁵⁷. Many of the more impressive claims about V seem to have first been published on that page, including the C/C++ translator "translat[ing] your entire C/C++ codebase", generating x64 code directly, and automatic memory management ("similar to Rust, but [...] much easier")⁵⁸.

If I was any meaner, I would perhaps point and laugh at the tutorial that was supposed to demonstrate using V to develop a 3D shooter game, but only shows code to animate a bouncing square (à la the iconic DVD screensaver)⁵⁸, but at least that tutorial was clearly marked as an "unfinished draft". It's a pity it was never completed, because I would gladly read an article like that, especially targeting people otherwise not experienced in game development.

Before Volt was Volt, back in 2017, there was Gitly, a minimal alternative to the likes of GitHub and GitLab⁵⁹. Gitly might have been Alex's first public project of this scale, and it garnered some pretty positive feedback in the Hacker News thread. The selling points seemed to be similar to ones Volt and V would later tout: simple, fast, lightweight.

However, that version of Gitly seems to have eventually been abandoned, at least according to another Hacker News commenter:

I realized the other day that the V author is the same person that created gitly, which was a really nice looking git forge. I believe the author's stated plan was to open source it, but the website went offline after some time without an open source release ever happening. I hope the author follows through on this one, because both projects look(ed) pretty neat!⁶⁰

To which Alex responded:

It will be back, open source, re-written in V[.]

[...]

I started developing Volt/V in the middle of developing gitly.

That's my biggest drawback. I finish 90% of the project, and jump to a different thing.

I've grown a lot since then, and I'm slowly wrapping up everything.

To be fair, Gitly was in fact rewritten in V, and does exist as an alpha-state open-source project⁶⁶. Nevertheless, the above comment from Alex sheds a light on why his projects consistently fail to get to a production-ready state.

Another commenter on Hacker News claims (I have not verified this independently):

I loved the idea [of V] [...] [b]ut it just didn't seem to go anywhere.

[...]

At one point I even searched in the Discord for messages by Alex that contained "this week", "this month" and "this year", also using "next" instead of "this". It came out to 50+ times that deadlines had been missed (closer to 80+ probably).⁶⁹

He just like me fr

I will not call Alex a fraud or a scammer as many have done before. My most charitable interpretation is that Alex is just like many programmers, including myself. We think of a cool new idea, implement a prototype and it fills us with joy: something that moments ago only existed in our imagination is now a functioning thing. Both "constantly abandoning side projects for new ones" and the inaccuracy of software development estimations are giant memes for a reason.

The author of V is not wrong in his defense that implementation details or even the direction of a software project might change in the course of its development, or that timeline might get adjusted. My problem with that is, again, the lack of transparency in how these changes are made. It is the fact that Alex, to my knowledge, never answers criticisms on this basis in a humble way, admitting that the initial goals were too lofty, deadlines too aggressive, or that the Hacker News commenters doubting that V could do all these things back in 2019 were partially right.

"Failure" does not have to be a failure, and V itself does not have to be the sole product of V's development. Alex has time and time again promised to write blog articles on various aspects of V's development. Had he done that, instead of silently backpedalling on claims about V, we could have had very interesting reads reflecting on the challenges of creating a new programming language. An article covering, say, "how we tried to create autofree and why we're changing course", would in my eyes be much more interesting than the current broken version of autofree itself. Not to mention that it would serve as a much needed piece of communication to V's users and the broader community.

There have also been claims of Alex unjustly banning people from the V community.⁷⁰⁷¹ I can't say I'm particularly interested in looking into each of these cases and evaluating whether the banhammer was deserved. As for my contributions to the V issue tracker, I'd like to think that any outside observer would recognize them as made in good faith, and I hope to avoid the banhammer myself.

Edited on 2024-09-27: About 9 hours after publishing this article and posting it on r/programming, I was "temporarily muted" from V's subreddit. To my knowledge, I have never interacted in the subreddit, nor was I planning on promoting this article in any community spaces of the V project.

Outro

This kind of article is unusual for me and I truly do not know how to end it. Is V all that it claims it is? Certainly not. Is V "unfixable"? Not necessarily, but it needs so much to get "fixed". Not just refocusing in terms of its goals, not just a lot of technical work, but also a major change in the attitude of its creator.

Footnotes

NixOS in the Cloud, step-by-step: part 2

2021-01-11T18:00:00+00:00

In the previous article, we deployed a basic cloud server running NixOS on DigitalOcean and set up a declarative deployment definition for it using Morph.

In this part, we are taking it a step further. We will utilize both Morph and Terraform to spin-up a network of 3 web servers, sitting behind 2 load-balancers, managed by declarative configuration. We will also learn an approach that utilizes Terraform state in Morph network definition, making Morph automatically aware of changes to your infrastructure.

For the uninitiated, Terraform is a popular infrastructure as code tool. It has basically become a de-facto standard across the industry. We will use it to spin up multiple servers on DigitalOcean without the need to point-and-click in the dashboard.

Preparation

Although we are starting from a blank state, I assume you have read the initial article of the series. You will need to utilize the knowledge you've already gained, as well as the resources created in your DigitalOcean account: the NixOS disk image and your uploaded SSH public key.

Let's create a directory and enter it:

$ mkdir part2 && cd part2

nix-shell

Then, let's create a shell.nix file with the tools we will need:

{ pkgs ? import <nixpkgs> { } }:
let
  myTerraform = pkgs.terraform.withPlugins (tp: [ tp.digitalocean ]);
  ter = pkgs.writeShellScriptBin "ter" ''
    terraform $@ && terraform show -json > show.json
  '';
in
pkgs.mkShell {
  buildInputs = with pkgs; [ curl jq morph myTerraform ter ];
}

Notably, besides Terraform, Morph and basic command line utilities, we add ter, a wrapper for the terraform command that will also dump the state from terraform show -json into the current directory. This will come in use soon.

Enter the shell and test that all commands work:

$ nix-shell
$ curl --version | head -n 1
curl 7.72.0 (x86_64-pc-linux-gnu) libcurl/7.72.0 OpenSSL/1.1.1i zlib/1.2.11 libssh2/1.9.0 nghttp2/1.41.0
$ jq --version
jq-1.6
$ morph --version
1.5.0
$ terraform version | head -n 1
Terraform v0.14.3
$ ter version | head -n 1
Terraform v0.14.3

DigitalOcean API token

To use DigitalOcean programmatically, create a personal access token in the DigitalOcean control panel. Keep it in a safe place (e.g. your password manager), as it allows full access to your droplets and other account data.

Upon creation, let's safely set it in our shell session and verify:

$ read -s DIGITALOCEAN_TOKEN
<...enter token here and press Enter...>
$ export DIGITALOCEAN_TOKEN
$ echo DIGITALOCEAN_TOKEN
7fba5cca53d8b6f9565037804a503029451495fac6708392cd6d3646f2e8fe8f

When using read -s, the token will not be echoed (i.e. represented on the screen, not even as asterisks), similar to how entering password for sudo works.

DigitalOcean resource IDs

We need to use two existing resources on our DigitalOcean account: our SSH public key and the NixOS custom image. While we could import them to our Terraform state, or even use a tool like Terraformer to do it for us, let's stick with a manual approach for brevity.

Use curl and jq to query the DigitalOcean API and find out the IDs of the custom image and our SSH key respectively:

$ curl -s -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
    'https://api.digitalocean.com/v2/images?private=true' \
    | jq '.images[] | select(.name == "nixos.qcow2.gz") | .id'
75674995
$ curl -s -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
    'https://api.digitalocean.com/v2/account/keys' \
    | jq '.ssh_keys[0].id'
27010799

The second command assumes you only have one SSH public key set-up in your DigitalOcean account. If that is not the case, you might need to inspect the full response from the API.

Creating our droplets

We are now ready to make a basic resource definition. Let's create a main.tf file:

provider "digitalocean" {}

resource "digitalocean_droplet" "backend" {
  name     = "backend${count.index + 1}"
  region   = "ams3"
  size     = "s-1vcpu-1gb"
  image    = 75674995
  ssh_keys = [27010799]

  count = 2
}

resource "digitalocean_droplet" "loadbalancer" {
  name     = "loadbalancer${count.index + 1}"
  region   = "ams3"
  size     = "s-1vcpu-1gb"
  image    = 75674995
  ssh_keys = [27010799]

  count = 2
}

This file defines a digitalocean resource provider and two digitalocean_droplet resources: both backend and loadbalancer resources will produce 2 servers with dynamically generated names: backend1, backend2, loadbalancer1, and loadbalancer2. Neither DigitalOcean, nor Terraform seem to enforce unique names for droplets, but it is a requirement for what we will want to do later, and a good idea in general.

The definition also utilizes the previously-acquired identifiers to specify that the droplet has to be spawned with our NixOS custom image and include our public SSH key for authentication.

Keys region and size are also required. If you do not live in Europe, you might be better off choosing a region close to you. We will not use any special features of particular regions, but we will use private networking, so the droplets have to all be in the same region. The examples of the possible values for size can be found on a dedicated webpage.

We also need to create a versions.tf file to define the version and full path for the digitalocean provider:

terraform {
  required_providers {
    digitalocean = {
      source  = "digitalocean/digitalocean"
      version = "2.2.0"
    }
  }
}

Let's apply our changes, using the ter wrapper to call Terraform:

$ ter init # only required the first time, or after new modules or providers are added
$ ter apply
<...confirm this with 'yes'...>

In my experience, launching new droplets took about 30 to 40 seconds.

Note: be really careful and always read what ter apply is planning to do. With great power of Terraform also comes great potential for destruction. If, for example, you delete a resource definition from main.tf, or reduce the count of machines, the "obsolete" machines will get destroyed. Terraform will always ask you to confirm the changes, but you have to actually read through them!

By this point, you should have 4 cloud servers launched. We can query the current state via commands like terraform show (or ter show). Feel free to play around and see what information Terraform provides.

$ ter show | head -n 10
# digitalocean_droplet.backend[0]:
resource "digitalocean_droplet" "backend" {
    backups              = false
    created_at           = "2020-12-29T19:48:34Z"
    disk                 = 25
    id                   = "223901656"
    image                = "75674995"
    ipv4_address         = "198.51.100.41"
    ipv4_address_private = "10.110.0.13"
    ipv6                 = false

Terraform allows not only to create new servers, but also to modify the current infrastructure, e.g. increase the amount of instances for a specific resource. Let's add one more backend server:

diff --git a/part2/main.tf b/part2/main.tf
index 3957178..1b9a65d 100644
--- a/part2/main.tf
+++ b/part2/main.tf
@@ -7,7 +7,7 @@ resource "digitalocean_droplet" "backend" {
   image    = 75674995
   ssh_keys = [27010799]

-  count = 2
+  count = 3
 }

Apply the changes once again:

$ ter apply
<...confirm this with 'yes'...>

If you run terraform show again, it should now show 5 machines in total. Terraform can also provide a JSON-based output of your state by passing the -json flag. Let's try that feature out and inspect our current infrastructure:

$ terraform show -json \
    | jq '.values.root_module.resources[] | {name: .values.name, ip: .values.ipv4_address}' 
{
  "name": "backend1",
  "ip": "198.51.100.41"
}
{
  "name": "backend2",
  "ip": "198.51.100.36"
}
{
  "name": "backend3",
  "ip": "203.0.113.74"
}
{
  "name": "loadbalancer1",
  "ip": "198.51.100.42"
}
{
  "name": "loadbalancer2",
  "ip": "203.0.113.122"
}

5 servers at our whim. I would call that a success!

As Morph uses SSH to deploy, it obeys SSH's trust on first use. So, when we deploy our network for the first time, it will ask us to confirm the fingerprint for each machine. For our own convenience, let's pre-fetch the SSH keys for our servers in advance, so we can avoid the manual intervention later and have the deployment proceed automatically.

$ terraform show -json \
    | jq -r '.values.root_module.resources[].values.ipv4_address' show.json \
    | xargs ssh-keyscan >> ~/.ssh/known_hosts

Using Terraform state from Nix

Terraform state is now the source of truth about the servers we have created. We would like to use the information provided by terraform show -json in our network definition and automatically deploy the relevant configuration to all of the services that share a role.

By now, we should have a file called show.json in the current directory. It has been created by the ter wrapper we defined, and updated after each operation, such as ter apply. This file will help us greatly. Nix has a function called builtins.fromJSON, which can deserialize arbitrary JSON data to Nix types. Nix is also a proper functional programming language, which will allow us to do the necessary manipulations with that data.

Put all of this together, and we can read show.json, deserialize it and transform it to a useful data structure which we can use in our network definition. To achieve this, I made a utility file called parsetf.nix:

{ pkgs ? import <nixpkgs> { }
, lib ? pkgs.lib
}:
let
  resourcesInModule = type: module:
    builtins.filter (r: r.type == type) module.resources ++
    lib.flatten (map (resourcesInModule type) (module.child_modules or [ ]));
  resourcesByType = type: resourcesInModule type payload.values.root_module;
  payload = builtins.fromJSON (builtins.readFile ./show.json);
in
{
  inherit resourcesByType;
}

It defines a function named resourcesByType. When this function is called like resourcesByType "digitalocean_droplet", it will return all the droplets we have set-up with Terraform.

It intentionally operates on a single resource type. As time goes on, you might also provision other resources on DigitalOcean, for example, block storage or floating IPs. You will be able to use the information about these resources as well, simply by changing the argument to this function. resourcesByType will even recurse into child modules, which will come in handy once you start using that feature.

Let's see how it works by experimenting in the Nix REPL:

$ nix repl
Welcome to Nix version 2.3.9. Type :? for help.

nix-repl> :l parsetf.nix
Added 1 variables.

nix-repl> droplets = resourcesByType "digitalocean_droplet"

nix-repl> droplets
[ { ... } { ... } { ... } { ... } { ... } ]

nix-repl> (builtins.head droplets).name
"backend"

nix-repl> (builtins.head droplets).values.name
"backend1"

nix-repl> (builtins.head droplets).values.ipv4_address
"198.51.100.41"

First, we loaded everything that parsetf.nix defines into the global namespace. In our case, that's a single function called resourcesByType. When we call it with the proper resource type, we see that it returns a list of five objects, as we currently have 5 servers. Once we get the first element from the list via builtins.head, we can evaluate any of its properties.

Note that .name will always return the name we specified for the Terraform resource, which will be the same for all the instances that this resource defined. Meanwhile .values.name will return the name property defined inside of the resource. This is the property that decides the droplet's actual hostname. As we have made this unique for each droplet, we will be able to use this to uniquely identify them.

Let's continue our REPL session:

nix-repl> balancers = (builtins.filter (d: d.name == "loadbalancer") droplets)

nix-repl> map (b: b.values.ipv4_address) balancers
[ "198.51.100.42" "203.0.113.122" ]

nix-repl> firstBalancer = builtins.head (builtins.filter (d: d.values.name == "loadbalancer1") balancers)

nix-repl> firstBalancer.values.name
"loadbalancer1"

nix-repl> firstBalancer.values.ipv4_address
"198.51.100.42"

As you can see, it is now possible to extract the needed information for both a single machine and a class of machines. All we need is the map and filter operations, familiar from most other programming languages.

Some might ask: why not just read the terraform.tfstate file directly? For one, there will be no such file if using a remote state backend, such as Terraform Cloud, which is a no-brainer when working in a team. The official documentation also says this:

State snapshots are stored in JSON format and new Terraform versions are generally backward compatible with state snapshots produced by earlier versions. However, the state format is subject to change in new Terraform versions, so if you build software that parses or modifies it directly you should expect to perform ongoing maintenence of that software as the state format evolves in new versions.

The output of terraform show -json, meanwhile, has a detailed specification.

Well, what about calling terraform show -json directly from Nix? That is not possible without jumping through some hoops, and that is possibly for the better. As I do not think it would be a good idea to let my expressions run arbitrary commands, I've gone with the wrapper approach. So far it has served me well: you just have to remember to always use ter rather than terraform for operations that might modify the state of your infrastructure.

Marrying Terraform and Morph

Now that we have the ability to use information about resources managed by Terraform in our Nix expressions, let's make a minimal network.nix:

# 1.
let
  resourcesByType = (import ./parsetf.nix { }).resourcesByType;

  droplets = resourcesByType "digitalocean_droplet";
  backends = builtins.filter (d: d.name == "backend") droplets;
  loadbalancers = builtins.filter (d: d.name == "loadbalancer") droplets;

  # 2.
  mkBackend = resource: { modulesPath, lib, name, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];
    deployment.targetHost = resource.values.ipv4_address;
    deployment.targetUser = "root";
    networking.hostName = resource.values.name;
    system.stateVersion = "21.11";
  };

  mkLoadBalancer = resource: { modulesPath, lib, name, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];
    deployment.targetHost = resource.values.ipv4_address;
    deployment.targetUser = "root";
    networking.hostName = resource.values.name;
    system.stateVersion = "21.11";
  };
in
# 3.
{
  network = {
    pkgs = import
      (builtins.fetchGit {
        name = "nixos-21.11-2021-12-19";
        url = "https://github.com/NixOS/nixpkgs";
        ref = "refs/heads/nixos-21.11";
        rev = "e6377ff35544226392b49fa2cf05590f9f0c4b43";
      })
      { };
  };
} // # 5.
# 4.
builtins.listToAttrs (map (r: { name = r.values.name; value = mkBackend r; }) backends) // # 5.
builtins.listToAttrs (map (r: { name = r.values.name; value = mkLoadBalancer r; }) loadbalancers)

This is quite a bit beefier than what we started with in the first article, so let's go over the functionality step-by-step:

(1.) We import parsetf.nix and use the resourcesByType function it defines to get a list of droplets in the Terraform state. We split it into two lists: one for the backends and another for the load balancers.

(2.) Then, we predefine two functions: mkBackend and mkLoadBalancer. They both take a single argument - the terraform resource object - and return yet another function, which generates the NixOS configuration for a concrete machine.

This way, we can generate similar configurations for all of the droplets that share the same role (backend or loadbalancer), while pulling the specifics, like IP address and hostname, from the resource parameter in the outer function.

So far, these two configurations are identical, but we will soon introduce configuration specific to backends and load balancers.

(3.) We introduce an attribute set that defines our network. Just like in part 1, we use the special network attribute to pin the version of nixpkgs that will be used across our machines.

(4.) Then, we use the lists of our resources - backends and loadbalancers. We iterate over the lists using map, and for each resource produce an attribute set of two attributes in the following form:

{
  name = "balancer1";
  value = { modulesPath, lib, name, ... }: { /* configuration options */ };
}

That is, name points to the hostname of the machine, while value points to the function that will generate the config for this specific machine.

We also use builtins.listToAttrs. This function will take this list of attrsets with name-value pairs:

[
  {
    name = "balancer1";
    value = { modulesPath, lib, name, ... }: { /* configuration options */ };
  }
  {
    name = "balancer2";
    value = { modulesPath, lib, name, ... }: { /* configuration options */ };
  }
}

and turn it into a single attrset like

{
  balancer1 = { modulesPath, lib, name, ... }: { /* configuration options */ };
  balancer2 = { modulesPath, lib, name, ... }: { /* configuration options */ };
}

That's exactly the format that Morph expects!

(5.) By this point, we have several attrsets - one with the network key, another with a key for each of our backends and their respective configuration functions, and a similar attrset for load balancers. Using the // operator, we merge these attrsets together.

If we could inspect the final value this expression produces, it would look something like this:

{
  network = {
    pkgs = import
      (builtins.fetchGit {
        name = "nixos-21.11-2021-12-19";
        url = "https://github.com/NixOS/nixpkgs";
        ref = "refs/heads/nixos-21.11";
        rev = "e6377ff35544226392b49fa2cf05590f9f0c4b43";
      })
      { };
  };
  backend1 = { modulesPath, lib, name, ... }: { /* configuration options */ };
  backend2 = { modulesPath, lib, name, ... }: { /* configuration options */ };
  loadbalancer1 = { modulesPath, lib, name, ... }: { /* configuration options */ };
  loadbalancer2 = { modulesPath, lib, name, ... }: { /* configuration options */ };
  loadbalancer3 = { modulesPath, lib, name, ... }: { /* configuration options */ };
}

Now, let's try to deploy the network using Morph:

$ morph deploy network.nix switch
Selected 5/5 hosts (name filter:-0, limits:-0):
          0: backend1 (secrets: 0, health checks: 0)
          1: backend2 (secrets: 0, health checks: 0)
          2: backend3 (secrets: 0, health checks: 0)
          3: loadbalancer1 (secrets: 0, health checks: 0)
          4: loadbalancer2 (secrets: 0, health checks: 0)

<...snip...>

** backend1
updating GRUB 2 menu...
activating the configuration...
setting up /etc...
reloading user units for root...
setting up tmpfiles

Running healthchecks on backend1 (198.51.100.41):
Health checks OK
Done: backend1

<...snip...>

Executing 'switch' on matched hosts:

** loadbalancer2
updating GRUB 2 menu...
activating the configuration...
setting up /etc...
reloading user units for root...
setting up tmpfiles

Running healthchecks on loadbalancer2 (203.0.113.122):
Health checks OK
Done: loadbalancer2

Morph deploys to all five hosts, one by one.

Deploying services to the network

Now that we're done with the Terraform <-> Nix boilerplate, we can start deploying nginx on our droplets. Let's configure nginx on the backends:

diff --git a/part2/network.nix b/part2/network.nix
index 6198eb1..9694bfb 100644
--- a/part2/network.nix
+++ b/part2/network.nix
@@ -13,6 +13,15 @@ let
     deployment.targetUser = "root";
     networking.hostName = resource.values.name;
     system.stateVersion = "21.11";
+
+    networking.firewall.allowedTCPPorts = [ 80 ];
+    services.nginx = {
+      enable = true;
+      virtualHosts.default = {
+        default = true;
+        locations."/".return = "200 \"Hello from ${name} at ${resource.values.ipv4_address}\"";
+      };
+    };
   };

   mkLoadBalancer = resource: { modulesPath, lib, name, ... }: {

Follow this up with a deployment, and check that nginx is working on all of the backends:

$ morph deploy network.nix switch
<...snip...>
$ curl 198.51.100.41
Hello from backend1 at 198.51.100.41
$ curl 198.51.100.36
Hello from backend2 at 198.51.100.36
$ curl 203.0.113.74
Hello from backend3 at 203.0.113.74

Now, let's configure the load balancer machines:

diff --git a/part2/network.nix b/part2/network.nix
index 9694bfb..5fa3925 100644
--- a/part2/network.nix
+++ b/part2/network.nix
@@ -32,8 +32,19 @@ let
     deployment.targetUser = "root";
     networking.hostName = resource.values.name;
     system.stateVersion = "21.11";
-  };

+    networking.firewall.allowedTCPPorts = [ 80 ];
+    services.nginx = {
+      enable = true;
+      upstreams.backend.servers = builtins.listToAttrs
+        (map (r: { name = r.values.ipv4_address_private; value = { }; })
+          backends);
+      virtualHosts.default = {
+        default = true;
+        locations."/".proxyPass = "http://backend";
+      };
+    };
+  };
 in
 {
   network = {

Here, we also configure nginx. However, instead of returning a static message, we define an upstream called backend.

The upstreams.<name>.servers option expects an attribute set where names are the server addresses, and values are options for the specific server (empty in our case). We use the ipv4_address_private attribute from the Terraform resources: by default, DigitalOcean puts all of the droplets created in a single region on their own internal network.

We once again use map to traverse the list of backend resources and then turn it into an attribute set using listToAttrs. At the end, it results in this:

upstreams.backend.servers = {
  "198.51.100.41" = {};
  "198.51.100.36" = {};
  "203.0.113.74" = {};
};

We then also define the default virtual host, which proxies all requests to the upstream. The address of http://backend indicates that nginx will pass the requests to one of the servers defined in the upstream called backend.

Let's deploy and try querying one of the load balancers:

$ morph deploy network.nix switch
<...snip...>
$ curl 198.51.100.42
Hello from backend3 at 203.0.113.74
$ curl 198.51.100.42
Hello from backend2 at 198.51.100.36
$ curl 198.51.100.42
Hello from backend1 at 198.51.100.41
$ curl 198.51.100.42
Hello from backend3 at 203.0.113.74

Our backends take turns serving the requests. Just what we wanted.

Now that the load balancers proxy the requests to backends, there is no need to keep the port 80 on the backends open to the entire world. We can easily change the configuration to only allow connections to this port on the internal network.

diff --git a/part2/network.nix b/part2/network.nix
index 5fa3925..d3f5cca 100644
--- a/part2/network.nix
+++ b/part2/network.nix
@@ -12,11 +12,11 @@ let
     deployment.targetHost = resource.values.ipv4_address;
     deployment.targetUser = "root";
     networking.hostName = resource.values.name;
     system.stateVersion = "21.11";

-    networking.firewall.allowedTCPPorts = [ 80 ];
+    networking.firewall.interfaces.ens4.allowedTCPPorts = [ 80 ];
     services.nginx = {
       enable = true;
       virtualHosts.default = {
         default = true;
         locations."/".return = "200 \"Hello from ${name} at ${resource.values.ipv4_address}\"";

DigitalOcean seems to always have the external network on interface ens3, meanwhile the internal interface is named ens4, but do check on your own just to be sure!

Once again, let's redeploy and see that our cluster still works after the changes:

$ morph deploy network.nix switch
<...snip...>
$ curl 198.51.100.42
Hello from backend2 at 198.51.100.36

By this point, we have a network of NixOS machines, managed by Terraform, with Morph using the Terraform state to deploy the relevant configuration to each machine according to its role.

Finishing touches

To cut down on the noise, we can also move the common properties into a separate NixOS module. Create a file named common.nix:

{ modulesPath, lib, ... }:
{
  imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
    (modulesPath + "/virtualisation/digital-ocean-config.nix")
  ];

  deployment.targetUser = "root";
  system.stateVersion = "21.11";
}

Then, remove these properties from your individual machine definitions, instead, importing the newly created module:

diff --git a/part2/network.nix b/part2/network.nix
index d3f5cca..1dc7801 100644
--- a/part2/network.nix
+++ b/part2/network.nix
@@ -5,14 +5,11 @@ let
   backends = builtins.filter (d: d.name == "backend") droplets;
   loadbalancers = builtins.filter (d: d.name == "loadbalancer") droplets;

-  mkBackend = resource: { modulesPath, lib, name, ... }: {
-    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
-      (modulesPath + "/virtualisation/digital-ocean-config.nix")
-    ];
+  mkBackend = resource: { name, ... }: {
+    imports = [ ./common.nix ];
+
     deployment.targetHost = resource.values.ipv4_address;
-    deployment.targetUser = "root";
     networking.hostName = resource.values.name;
-    system.stateVersion = "21.11";

     networking.firewall.interfaces.ens4.allowedTCPPorts = [ 80 ];
     services.nginx = {
@@ -24,14 +21,11 @@ let
     };
   };

-  mkLoadBalancer = resource: { modulesPath, lib, name, ... }: {
-    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
-      (modulesPath + "/virtualisation/digital-ocean-config.nix")
-    ];
+  mkLoadBalancer = resource: { name, ... }: {
+    imports = [ ./common.nix ];
+
     deployment.targetHost = resource.values.ipv4_address;
-    deployment.targetUser = "root";
     networking.hostName = resource.values.name;
-    system.stateVersion = "21.11";

     networking.firewall.allowedTCPPorts = [ 80 ];
     services.nginx = {

Again, you can find the full code in a GitHub repository.

That's all (for now), folks

Having done all that, we have a network of cloud servers on DigitalOcean, which we can both bootstrap and deploy services on according to the configuration written in Terraform and Nix.

These two posts cover most of the tools and workflows I use to manage my personal infrastructure. While there are individual nuggets of Nix knowledge I have not shared yet, I do not think these warrant a full blog article right now.

One area that would be interesting to see explored is writing Terraform definitions using Nix. As Terraform supports pure JSON as an alternative to its own configuration language, and Nix can both parse and generate JSON, this does not seem impossible. However, I do not immediately see any inherent value or ergonomic improvements in this. Perhaps that shall remain a challenge for another time (or person).

With that said, I hope you learned something. Good luck on your NixOS adventures!

Alternative approaches

There exist alternative approaches of managing your infrastructure and deployments using Nix.

For one, NixOps 1.7 is able to spawn DigitalOcean machines directly. However, as mentioned in the first part, NixOps is a moving target. For version 2.0, the external providers are being moved to their own repositories. There are also significant limitations, e.g.:

Note that we rely on a ssh key resource with the hard-coded name ssh-key. Providing your own key is not supported yet.

Using the DigitalOcean provider with Terraform will give you far more flexibility, allowing you to manage not only your machines, but domain records, attached storage, and other sorts of resources as well.

There is also a dedicated Terraform module for deploying NixOS machines, as described by a tutorial on nix.dev. To be completely honest, I was not aware of this until I had mostly written part 1, and have not tried it for myself. For my needs, a setup where each tool has its own set of responsibilities - Terraform for infrastructure management and Morph or NixOps for configuration management - works well.

NixOS in the Cloud, step-by-step: part 1

2020-12-28T16:30:00+00:00

In the last few months, I migrated both my workstation and my servers (a DigitalOcean VPS and a Raspberry Pi 3) to NixOS. To best summarize the benefits, let's just say that it's like having a "dotfiles" repo, but for your entire system (or multiple!), including custom software, service configuration, drivers, kernel tweaks, etc.

While a similar result could be achieved by more mainstream configuration management tools, such as Puppet or Ansible, they do not integrate deeply into your OS. As such, over a longer period it's likely for your system to accumulate all sorts of manual tweaks done outside of your configuration management framework. The system's configuration is not fully described by your declarative configuration anymore and you will likely not reproduce the system exactly if you have to recreate it from scratch.

NixOS treats the system as mostly immutable and makes it way harder to mess something up: you can't just edit the files under /etc or upgrade globally installed packages by hand. Most meaningful changes you'll only be able to do via edits to configuration.nix and nixos-rebuild switch, which replaces the entire system with a new generation. This also provides an ability to rollback everything easily and effectively, which conventional tools often lack or implement incompletely.

Why this article was made

Despite the situation getting progressively better, while learning Nix and NixOS I still felt like some parts of the stack are under-documented. While there are extensive resources such as the NixOS manual, Nix pills, or nix.dev; what I lacked the most was how-tos or recipes which would show how to connect the tools to achieve your desired goal. Time and time again, I had to resort to reading others' configurations and the source code in the nixpkgs repository.

This series of articles intends to fill this niche somewhat. Most of these methods are the same ones I use to run my own "private cloud". My own infrastructure is quite pedestrian - I only run a few services, including hosting my websites and email - so it's not exactly a guide for "enterprise battle-tested infrastructure". However, once I nailed the process down, I have definitely had a pleasant experience, and my current setup is definitely satisfactory for my needs.

Who this is for

If you - like me a while ago - are getting into Nix / NixOS, but still struggling to put your newly-acquired knowledge to use, you've come to the right place.

I do not intend to teach the core concepts of Nix or the basics of the expression language. The aforementioned resources should get you up and running if you are a complete newbie. Instead, I'll focus on one specific use case - deploying a cloud server running NixOS and managing its configuration from your own workstation.

In this article, we will learn how to:

Generate a custom NixOS image for DigitalOcean
Create a virtual NixOS server on the cloud
Deploy nginx on the server using Morph

Before we begin, let's go over the tools we'll need one by one.

Nix

You should either be running NixOS, or have Nix installed under your preferred Linux distribution.

Again, I assume you have the basic gist of the Nix language. Basically, if you've written at least one Nix derivation, or played around with your configuration.nix, you should be fine.

DigitalOcean

DigitalOcean is a cloud hosting provider. I chose it simply because it is the provider I have been using for the past several years. It works well for me and allows custom images, which is gonna be really important soon.

If you'd like to, you can sign up on DigitalOcean through my referral link and get $100 credit to use freely for 60 days.

Morph

Morph is a tool that allows deploying NixOS machines remotely as easily as you could update your own machine's configuration.

There are multiple competing NixOS deployment tools, including the official NixOps, rival Morph, and a new contender called Colmena. So, why choose Morph specifically?

As Morph's README states, it and similar tools are:

basically fancy wrapper[s] around nix-build, nix copy, nix-env, switch-to-configuration, scp and more

Thus it does not matter that much which tool we use, as all of these tools:

Use the Nix language itself to define your configuration
Use similar Nix-native methods for the deployment process itself
Have a similar layout for the network expressions

Most things you'll learn from this series will be applicable to any of these tools.

As NixOps is currently in the flux somewhere between version 1.7 and 2.0, where 2.0 adds important features, but does not have a stable release yet, it is hard to recommend (although I do use its master branch personally). Colmena might get there soon, as it gains features and polish, but for now, Morph seems like the best alternative.

In particular, Morph has a couple of features I really like. The first is allowing to easily pin a specific version of nixpkgs for your machines. The second is being completely stateless: unlike NixOps, it does not manage a set of deployments in a hidden file somewhere in your home directory. Instead, it does only what you explicitly tell it to do - deploy to machines specified in your .nix file.

Preparation

Enough bikeshedding, let's get down to business. We will need a few things before we are ready.

nix-shell

Let's make a new folder to fit all of our stuff in:

$ mkdir part1 && cd part1

Then, let's define a new shell that will provide the morph package, as well as curl which we will use to query our newly deployed web server. Put this in a new file called shell.nix:

{ pkgs ? import <nixpkgs> { } }:
pkgs.mkShell {
  buildInputs = with pkgs; [ curl morph ];
}

Then enter the shell and you should be able to launch both tools:

$ nix-shell
$ curl --version | head -n 1
curl 7.72.0 (x86_64-pc-linux-gnu) libcurl/7.72.0 OpenSSL/1.1.1i zlib/1.2.11 libssh2/1.9.0 nghttp2/1.41.0
$ morph --version
1.5.0

Great success!

Preparing your DigitalOcean project

Custom image

Besides supporting major distributions out-of-the-box, DigitalOcean also supports custom disk images, which means you can run basically any OS. We will use the DO image generation functionality in nixpkgs to generate one of our own.

Put this in a new file called image.nix:

{ pkgs ? import <nixpkgs> { } }:
let config = {
  imports = [ <nixpkgs/nixos/modules/virtualisation/digital-ocean-image.nix> ];
};
in
(pkgs.nixos config).digitalOceanImage

Then execute:

$ nix-build image.nix

This may take several minutes. Do not worry, we'll only need to do this once. Once the build has finished, you should have a new file under the newly created result/ directory:

$ ls -sh result/*
390M result/nixos.qcow2.gz

A custom image has been built! You can now upload this image to your DigitalOcean account.

SSH key

You should also upload your SSH public key. The custom image we just generated has a hidden superpower: it automatically pulls in the public SSH keys from your DigitalOcean account at creation time. This will let us login via SSH without needing to create a root password.

Create your NixOS droplet!

Finally, we can create a droplet (that's what Digital Ocean calls virtual servers).

A few things to note:

Remember to choose your custom image, called nixos.qcow2.gz
Remember to enable your SSH key under the "Authentication" section. For me, the checkbox was checked by default
I named my droplet nixie. Doing the same will make it easier to follow along
Feel free to choose the smallest ($5/month) flavor. We will not need anything too powerful

You are charged for all of your droplets hourly, even if they are turned off, so remember to destroy anything you do not need anymore in order to not rack up a huge bill.

Verify

Once the droplet has been created, you will see its IP address on the dashboard. Let's try to connect.

$ ssh root@198.51.100.207
The authenticity of host '198.51.100.207 (198.51.100.207)' can't be established.
ED25519 key fingerprint is SHA256:MtP743nmdmL59nJYYeGrvJVw8sNkqpQGvl2yyD5PMOA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '198.51.100.207' (ED25519) to the list of known hosts.

[root@nixie:~]# nixos-version
20.09.2290.647cc06986c (Nightingale)
[root@nixie:~]# exit

Works fine!

Note: this initial NixOS version will depend on your workstation's version of nixpkgs. This is not an ideal situation, but it works fine for provisioning. We will solve this later on.

The custom image we built the droplet from has some important directives in its initial configuration.nix. Let's retrieve this file for later use:

$ scp root@198.51.100.207:/etc/nixos/configuration.nix .
$ chmod 644 configuration.nix  # scp keeps the original 444 permissions, but we will need to edit this file

Managing the droplet with Morph

When using NixOS on your machine, you edit your configuration in /etc/nixos/configuration.nix and activate it via sudo nixos-rebuild switch. However, this "every machine governs itself" model will not scale gracefully as you add remote hosts, especially if the amount of them grows large. Instead, we would like to have all the configuration in one place (a Git repo works well) and be able to push out configuration changes from our workstation to all of these machines using a tool like Morph.

However, this approach leaves /etc/nixos/configuration.nix behind. Your system is now running whatever Morph has built the system from, and only that.

As the custom image we generated contains important stuff to run the system on a DigitalOcean droplets, such as bootloader and hard disk configuration, we will need to keep that when moving to Morph.

Let's check what's inside of the current configuration we've downloaded:

$ cat configuration.nix
{ modulesPath, lib, ... }:
{
  imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
    (modulesPath + "/virtualisation/digital-ocean-config.nix")
  ];
}

Seems a bit short? That's because everything important is hidden away in the digital-ocean-config.nix import. If you are interested what it looks like under the hood, feel free to read the source code in nixpkgs.

For now though, we will rename it to use as a basis for our deployment specification.

$ mv configuration.nix network.nix

Open this file, now called network.nix in your favorite editor. We will do a few small edits to make it look like this instead:

{
  nixie = { modulesPath, lib, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];

    deployment.targetHost = "198.51.100.207";
    deployment.targetUser = "root";
  };
}

Whereas previously we've had the function as the only thing in a file, now the top level value is a set with one key nixie pointing to this function. This is how Morph and similar tools define your servers: one key means one machine to deploy to, and the corresponding value is a function that generates that machine's system configuration, just like the one that might exist in your NixOS workstation's configuration.nix.

Besides that, we added important Morph-specific options: the address of the host to deploy to and the user to connect as.

Let's try to deploy now:

$ morph deploy network.nix switch
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 0)

error: The option `networking.hostName' has conflicting definitions, in `/nix/store/p3bjy5vfknzn1dxq72gb8wwa33bs7hfr-nixos/nixos/modules/virtualisation/digital-ocean-config.nix' and `<unknown-file>'.
<...snip...>

Oops! It seems that the hostname Morph tries to configure automatically conflicts with what is defined in digital-ocean-config.nix. We can fix that by defining it manually in network.nix:

diff --git a/part1/network.nix b/part1/network.nix
index 52e8932..02db4f2 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -1,10 +1,12 @@
 {
-  nixie = { modulesPath, lib, ... }: {
+  nixie = { modulesPath, lib, name, ... }: {
     imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
       (modulesPath + "/virtualisation/digital-ocean-config.nix")
     ];

     deployment.targetHost = "198.51.100.207";
     deployment.targetUser = "root";
+
+    networking.hostName = name;
   };
 }

For those not fluent in patches, this is what our network.nix looks like so far:

{
  nixie = { modulesPath, lib, name, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];

    deployment.targetHost = "198.51.100.207";
    deployment.targetUser = "root";

    networking.hostName = name;
  };
}

The new parameter name that we added to the function definition will be filled out with the same value as the key, in this case, "nixie".

Let's try deploying again:

$ morph deploy network.nix switch

The first deployment will likely take a while. However, on the following deployments, only the parts that changed, e.g. newly added or updated packages, or daemon configuration files, will be rebuilt.

The output of Morph will look like something this:

Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 0)

<...snip...>

Executing 'switch' on matched hosts:

** nixie

<...snip...>

Running healthchecks on nixie (198.51.100.207):
Health checks OK
Done: nixie

Finally some payoff!

Deploying nginx

At last, it's time to deploy something more than just an empty system. We will now make these changes to our configuration:

Enable the nginx HTTP server
Add a default virtual host and a simple "OK" response to HTTP requests
Add a health check that Morph will use to ensure nginx is working properly

diff --git a/part1/network.nix b/part1/network.nix
index 02db4f2..8ab724a 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -8,5 +8,24 @@
     deployment.targetUser = "root";

     networking.hostName = name;
+
+    deployment.healthChecks = {
+      http = [
+        {
+          scheme = "http";
+          port = 80;
+          path = "/";
+          description = "check that nginx is running";
+        }
+      ];
+    };
+
+    services.nginx = {
+      enable = true;
+      virtualHosts.default = {
+        default = true;
+        locations."/".return = "200 \"Hello from Nixie!\"";
+      };
+    };
   };
 }

Time to deploy again!

$ morph deploy network.nix switch
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 1)
<...snip...>
Running healthchecks on nixie (198.51.100.207):
        * check that nginx is running: Failed (Get "http://198.51.100.207:80/": context deadline exceeded (Client.Timeout exceeded while awaiting headers))

Uh oh. Our health check is failing. Can you spot the issue? An experienced NixOS-er will immediately tell what's wrong: by default, NixOS has firewall enabled and we have not specified that we want to open the port 80. Let's fix that:

diff --git a/part1/network.nix b/part1/network.nix
index 8ab724a..ff078d3 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -20,6 +20,8 @@
       ];
     };

+    networking.firewall.allowedTCPPorts = [ 80 ];
+
     services.nginx = {
       enable = true;
       virtualHosts.default = {

And try our deployment one more time:

$ morph deploy network.nix switch
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 1)

<...snip...>

Running healthchecks on nixie (198.51.100.207):
        * check that nginx is running: OK
Health checks OK
Done: nixie

Seems to pass now. Might as well check it via curl, just to be sure:

$ curl http://198.51.100.207
Hello from Nixie!

Yay! We have an nginx instance running on our NixOS server.

Pinning nixpkgs

As mentioned before, currently the server's NixOS version depends on the version of Nixpkgs in your local workstation. That is not always the best idea: for example, you might want to run your workstation on the bleeding edge by using nixpkgs-unstable, but keep your servers on a more stable channel.

Luckily, Morph allows us to change what version of nixpkgs is used to deploy our network:

diff --git a/part1/network.nix b/part1/network.nix
index e89b9b1..3643f5f 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -1,4 +1,15 @@
 {
+  network = {
+    pkgs = import
+      (builtins.fetchGit {
+       name = "nixos-21.11-2021-12-19";
+       url = "https://github.com/NixOS/nixpkgs";
+       ref = "refs/heads/nixos-21.11";
+       rev = "e6377ff35544226392b49fa2cf05590f9f0c4b43";
+      })
+      { };
+  };
+
   nixie = { modulesPath, lib, name, ... }: {

The following expression will check out the commit e6377ff3 from the nixos-21.11 branch and use it as the basis for all of the machines in the network. The name is allowed to be an arbitrary string. I use it to mark the date of the last update of the packages.

Now, let's redeploy and check out what version we are running.

$ morph deploy network.nix switch
<...snip...>
$ morph exec network.nix nixos-version
Selected 1/1 hosts (name filter:-0, limits:-0):
          0: nixie (secrets: 0, health checks: 1)

** nixie
21.11pre-git (Nightingale)

While the default nixos-version utility is not willing to return a commit hash, we do see that we are running from a git checkout, rather than the version we observed before. For more information on various nixpkgs channels, check out this page on the unofficial wiki.

State version

It is also a good idea to explicitly set your system.stateVersion. I will refrain from explaining this option in depth, as an explanation can be found in the wiki. Basically, some stateful services (e.g. Postgres) need manual intervention, such as data migration, when upgrading to a new major version. system.stateVersion prevents breakage that could result from upgrading these packages without the necessary precautions: the affected services will keep using compatible versions, even if we upgrade our NixOS version to a new one.

diff --git a/part1/network.nix b/part1/network.nix
index 3643f5f..04eedaf 100755
--- a/part1/network.nix
+++ b/part1/network.nix
@@ -20,6 +20,8 @@

     networking.hostName = name;

+    system.stateVersion = "21.11"; # Do not change lightly!
+
     deployment.healthChecks = {
       http = [
         {

That's it

Here's the finished network.nix:

{
  network = {
    pkgs = import
      (builtins.fetchGit {
        name = "nixos-21.11-2021-12-19";
        url = "https://github.com/NixOS/nixpkgs";
        ref = "refs/heads/nixos-21.11";
        rev = "e6377ff35544226392b49fa2cf05590f9f0c4b43";
      })
      { };
  };

  nixie = { modulesPath, lib, name, ... }: {
    imports = lib.optional (builtins.pathExists ./do-userdata.nix) ./do-userdata.nix ++ [
      (modulesPath + "/virtualisation/digital-ocean-config.nix")
    ];

    deployment.targetHost = "198.51.100.207";
    deployment.targetUser = "root";

    networking.hostName = name;

    system.stateVersion = "21.11"; # Do not change lightly!

    deployment.healthChecks = {
      http = [
        {
          scheme = "http";
          port = 80;
          path = "/";
          description = "check that nginx is running";
        }
      ];
    };

    networking.firewall.allowedTCPPorts = [ 80 ];

    services.nginx = {
      enable = true;
      virtualHosts.default = {
        default = true;
        locations."/".return = "200 \"Hello from Nixie!\"";
      };
    };
  };
}

Let's recap. Using this process, we:

Created an image that allows us to bootstrap NixOS servers on DigitalOcean, complete with our SSH public key for easy management
Created a NixOS configuration that we can push to remote machines via Morph
Deployed an nginx server and added a health check to ensure it still works after changes to configuration
Pinned this server's packages to a specific commit to ensure stability

The entire source code can be found in a GitHub repository.

What's next?

In the second part of the series, we will learn how to:

Launch several DigitalOcean servers using Terraform
Make Nix aware of Terraform state and the servers it manages
Use that information in our Morph network definition and deploy a setup of several backend servers behind several load balancers.

Producing a podcast the geeky way: Audio

2018-11-11T16:30:00+00:00

A few months ago I started my podcast Corded where I talk to people whose jobs or creations I find interesting. Corded is an ongoing learning experience not only in the sense of producing interesting material, but also the technical intricacies that come with producing an audio/video product.

This article, the first of a three-part series, aims to shed some light on the geeky way I produce my podcast.

Note that I could not emphasize geeky enough: as it often is, when I take up a creative activity, I find the technical side of it just as interesting as the creative side, if not more so. Perhaps this is why when I dabbled in streaming on Twitch, I found that I enjoyed spending hours in OBS trying to make the scene just perfect at least as much as I enjoyed the actual streaming. I will often choose the long route instead of the easy one, just to learn more about the inner workings, rather than make my process as easy as possible. My process will not be the optimal one for everyone or possibly even anyone other than me.

The topic of this first article is audio, the most instrumental part of any podcast.

Hardware

Way before I actually started recording episodes, I became a fan of Bandrew Scott's YouTube channel called Podcastage. Bandrew is constantly putting out in-depth reviews of microphones, audio interfaces and other audio hardware.

It was in part Podcastage that helped me decide on what is so far the only piece of "audio hardware" I use: the Samson Q2U dynamic microphone. Nothing in its price range can beat Q2U's versatility: it can be connected via USB for that no-frills plug-and-play experience, however, an XLR port is also included which makes for a clear upgrade path if one ever decides to upgrade to a standalone audio interface or finds themselves in need of a mixer for live production.

Other than that, I also got a cheapo Neewer boom arm with an included pop filter. It is worthy to note that I had to go to a local store to get a suitable mic clip for the Q2U, as the shock mount with the arm was too large: likely manufactured to accomodate larger condenser microphones.

Remote recording woes

Once I got my gear sorted out, it was time to figure out how best to record both me and the guests. For now, I have opted for remote interviews. This means that I do not have to get more microphones for my guests or even invite them over (possibly meaning a trip from another city just to record for 30 minutes). However, it slightly complicates the process on my side: not only do I need a VoIP solution to communicate, I also need a way to record both my own end and the other end.

For the first issue, I have used both Discord and Zencastr so far. Considering most of my guests do not own a studio-quality microphone in the first place, these applications provide a good-enough quality. I found Zencastr to be less of a hassle as the guests would not need to create a Discord account and configure its audio settings for optimal recording.

Both Zencastr and Discord (using Craig) can record each participant's track separately. However, upon testing Zencastr, I noticed it would often suffer from audio drift issues. For one reason or another, I decided not to try Craig and instead searched for software I could run solely on my own machine and which would let me record both my microphone in and the loopback interface (i.e. 'what you hear') simultaneously. This way, Discord or Zencastr would be used purely as a way to communicate, without using their recording capabilities.

This was not a trivial task, as most digital audio workstations (DAWs) will expect you to use a single audio interface when in my case there were two: the on-board sound card which I would plug my headphones into and the Samson Q2U which acts as a sound card of its own when used via USB.

Finally, I decided on a solution: OBS. Yes, that OBS, mostly used by streamers on Twitch or YouTube. Usually streamers will want the audio on their stream to contain both the loopback interface (e.g. game sounds) and their own microphone. OBS allows to record a setup like that out-of-the-box, and if you tune the settings just right, it will put your audio sources into separate audio tracks in the resulting video file. I would then use Avidemux to extract these separate tracks from the video file. It was the perfect hack, which I gladly used for recording the first three episodes.

Post-production and perfectionism

Of course, just because you have the recording does not mean it is ready for your listeners' ears. Any podcast warrants at least basic post-production: background noise removal, cutting out any mess-ups or awkward silence. Most podcasters will also use an equalizer at least minimally (cutting off anything below 80 or 100 Hz is a usual practice) and apply compression to not have startling changes in audio levels.

Initially I went for the trusty Audacity which I had been using for years whenever I needed to do basic audio edits. For beginner podcasters, using Audacity is a no-brainer. It's free, its noise reduction works like magic, and, once I learned the keybindings, making cuts in the episode felt like swinging a sharp samurai sword at the tracks.

Post-production was (and still is) the most time-consuming part of my process. Even after making the initial cut I would listen to the entire episode 5 to 10 more times, finding something worth fixing every time, be it a sharp breath of the guest, or me going on a tangent that I would find not worthy keeping in.

After all this was done, the episode would be exported to an MP3 file. Many podcasters will tell you that a 64 or 96 kbps mono export is enough. I went a bit overboard and chose 128 kbps as my baseline, with the one exception being episode 2, which included a preview of a track from Kuoga.: that one, I felt, warranted a 256 kbps stereo export.

The move to Linux and a DAW

I have been using Linux as my main OS for about 10 years now (with a 2 year period of being an OS X guy). Windows exists on my hard drive mostly to run a few games that do not perform as well on Linux. So naturally, I wanted to move my podcast production to my preferred OS as well.

As an additional step towards greatness, I decided to look for a proper DAW. Not only is Audacity's multi-track recording limited, it is also a destructive editor. That means that whatever FX or edits you apply to a track in Audacity, it is like painting over something in MS Paint. Sure, you can click "Undo" a few times, but what if you save the project and reopen it? Or if you want to undo an earlier change without affecting the later ones? Want to bypass one of the 10 effects you applied? Tough luck.

Non-destructive editors treat changes more like Adobe Photoshop treats layers. You are free to turn effects on or off or make cuts and then bring the removed parts back without affecting anything else. Besides, everything is applied in real-time as opposed to Audacity that will present you with a progress bar while the waveform changes before your eyes.

In search of a more professional piece of software, I first tried using Ardour, probably the most significant open source DAW. And while I liked its vast recording and routing capabilities, I quickly found that the set of plugins Ardour comes with is very spartan (perhaps on purpose). Scavenging the interwebs for a decent noise removal plugin felt like a backwards step from Audacity, so I quickly turned my attention elsewhere in pursuit of an all-in-one solution.

In the end, I decided on REAPER, a DAW often recommended in the r/podcasting community. The team behind REAPER had just recently started publishing native Linux builds at the time. Despite not being as prevalent as FL Studio or Ableton Live, REAPER is by all means a professional DAW. Just like Ardour, it comes with flexible audio routing, but surpasses it in plugins in terms of both quality and quantity. The bigger userbase also helps: YouTube is full of tutorials for doing all sorts of things using REAPER.

The caveat? REAPER is non-free. However, the WinRAR-like nagware model is a welcome practice. REAPER does not limit functionality even if you use it past the 60-day trial period. Hell, they even offer a set of plugins which you are free to use in any other DAW of your liking. Even if I find the $60 personal/small-business license steep, that is only because I record about one episode per month. Compared to prices of other DAWs of similar level, it is essentially pennies. Pennies that I will pay some time in the future, I promise.

Apart from editing non-destructively, my post-production barely changed with the move from Audacity to REAPER: I still use the same functionality (noise removal, equalization, compression), I still make similar cuts in the episode, and I still export in the same quality.

JACK: the daemon of my nightmares dreams

Another key component to working with audio on Linux is JACK: the "professional sound server daemon". I have been aware of JACK for a while now, perhaps back from all the times I tried to make my PC into a guitar amp using Rakkarack (which, of course, used JACK).

I might have lacked patience back then, because I struggled with just getting JACK up and running a lot. Now, however, I found the experience involved, yet relatively painless. I did need to spend some time learning how to use the audio routing and get JACK to play along nicely with PulseAudio (which is in turn another daemon that most "consumer" Linux software like Firefox or Discord use for audio).

A few evenings learning the ins and outs of JACK yielded this recording setup:

While this may seem convoluted to an untrained eye, it is actually a quite elegant setup:

samson, my microphone (attached via alsa_in), feeds into both REAPER to record and PulseAudio JACK Source for my VoIP solution to transmit.
PulseAudio JACK Sink, i.e. 'what I hear' from the guest via Zencastr via Firefox goes to two destinations as well: REAPER and my headphones.
REAPER outputs to system by default. I do not use monitoring from the DAW when recording.

While this setup is pretty simple, JACK, in conjunction with a DAW like REAPER, provides almost infinite audio routing possibilities. For example, if I ever wanted to do live podcast production, it should be possible to do a mix-minus solely by making the right connections in the software, without any need to shell out money for a hardware mixer.

For more info about using JACK for podcasting check out this video by Dustin Williams, who uses an even geekier JACK setup using jack_mixer, jack_capture, and some helper shell scripts.

The future

As it stands now, I am happy with my Q2U/JACK/REAPER setup for recording and post-production. Aside from getting better at actually using REAPER, the best improvement to make would probably be moving to an in-person recording setup, including a standalone audio interface, a couple more microphones and a dedicated sound-treated space in my home or elsewhere.

However, like I've mentioned before, that change would come with its own set of drawbacks and my low episode output makes it hard to justify the time and money investment it would require.

The next part of the series will focus on the video part of my podcast: the why and the how. If you're able to decipher Lithuanian and like podcasts, consider subscribing to Corded. :)

My reason to be excited for Go 1.4

2014-12-22T09:50:00+00:00

More than a week ago, Go 1.4 was released, bringing new features and bug fixes to the programming language.

Many people consider the support for building shared libraries for Android the most important addition in this version. Others are excited about the new go generate tool. Some are happy that internal packages have landed, as they should help reduce duplication of code.

However, my favorite new feature of Go 1.4 is hidden deep in a section called "Minor changes to the [standard] library" in the release notes. A list item there states:

The testing package has a new facility to provide more control over running a set of tests. If the test code contains a function

func TestMain(m *testing.M)

that function will be called instead of running the tests directly. The M struct contains methods to access and run the tests.

To sum up the benefits of this feature in one sentence, it lets you do global set-up/tear-down for tests.

Why that excites me

Recently, I have been developing a full-blown MVC web application in Go. Having come from Python, where Django wraps you tightly with all sorts of helpers, I initially struggled with several areas of architecture of a web application.

One of those was database testing. Django provides a test runner that creates a fresh database every time the tests are run. This way, you can do away with mocking and run your tests on real data, thus improving their accuracy.

In Go, I don't use any framework for my projects, and even if I did, none seem to have their own sophisticated testing tools. Thus, I had to invent my own way to swap out the database while testing . After thinking the design over for weeks, the final result looked like this:

func TestSomeFeature(t *testing.T) {
    models.TestDBManager.Enter()
    defer models.TestDBManager.Exit()

    // Do the tests
}

Where Enter() is responsible for creating a new Postgres database, running the migrations, swapping out the usual database connection, while Exit() drops the database and restores the original one.

This worked, but was less than ideal. Every single test which utilized the DB had to go through this process and thus as the testing suite grow, the time it took to run tests followed linearly. Even if I wanted to only do this once, there was no single point of entry or exit I could hook on when go test ran.

The introduction of TestMain() made it possible to run these migrations only once. The code now looks like this:

func TestSomeFeature(t *testing.T) {
    defer models.TestDBManager.Reset()

    // Do the tests
}

func TestMain(m *testing.M) {
    models.TestDBManager.Enter()
    // os.Exit() does not respect defer statements
    ret := m.Run()
    models.TestDBManager.Exit()
    os.Exit(ret)
}

While each test must still clean up after itself, that only involves restoring the initial data, which is way faster than doing the schema migrations.

Package	Without TestMain	With TestMain
models	3.017s	0.548s
handlers	9.329s	2.971s
settings	0.008s	0.008s
cli	1.230s	0.528s

The table above illustrates the performance improvements. The time it takes to run tests has reduced significantly, considering the database-heavy parts.

This approach also reduces code duplication by 50%: we now only have one line for database management in each test instead of two.

Less is more

The addition of the TestMain function, as simple change as it is, improved my development process in important ways. This is why Go makes me happy: every 6 months, with a new version of the language, new features come along. Some are big, but most are little. Despite that, they all play an important part in making Go better, without sacrificing core values of the language. I am sure many others will benefit from the improvements brought by Go 1.4.

Alice – Painless Middleware Chaining for Go

2014-05-25T17:00:00+00:00

According to a recent thread on Reddit, many people like it simple when doing web development in Go and use net/http with useful addons (like the Gorilla toolkit) instead of a full-fledged framework.

Such applications often make use of middleware, but wrapping lots of layers of handlers can become messy in the long run:

final := gzipHandler(rateLimitHandler(securityHandler(authHandler(myApp))))

Sure, that works. But then you want to remove a handler, add one or reorder them. Suddenly, you're drowning in parentheses.

Alice (available on GitHub) was created as a solution to simplify chaining while remaining flexible and playing nice with the existing net/http middleware.

It's not a framework, a mux or a toolkit. Its sole functionality is to let you create the same middleware chain like this:

final := alice.New(gzipHandler, ratelimitHandler, 
    securityHandler, authHandler).Then(myApp)

Flaws of other approaches

Many might point out that there are existing solutions for chaining middleware. That's true, but any of the solutions I had looked into had at least one thing that I thought should be done in a differently.

The recent Negroni package allows handlers to be added as middleware like this:

n := negroni.Classic()
// func (n *Negroni) UseHandler(handler http.Handler)
n.UseHandler(myMiddleware)

See the fault here? Traditional net/http middleware wrap the next handler so the wrapper can have a total control. Here, there's simply no handler to pass in. You can't pass in the Negroni instance as it would result in infinite recursion (Negroni calls middleware calls Negroni).

Negroni has its own mechanism for control flow (next() to call the following handlers), but you have to modify your middleware to fully utilize it, which is not ideal.

go-httppipe has the same problem: it's suited for successive handlers, but not wrapper-type middleware.

go-stackbuilder forcibly uses http.ServeMux as the main handler, instead of http.Handler.

func New(mux *http.ServeMux) Builder {
    return Builder{mux}
}

func Build(hs ...interface{}) http.Handler {
    return New(http.DefaultServeMux).Build(hs...)
}

That is not ideal as one might like to bypass the default mux completely, especially when using an alternative one.

Muxchain, like Negroni, doesn't provide a reference to the next handler.

Matt Silverlock's use.go snippet came closest to what I wanted. My only complaint is that the ordering of handlers here is counter-intuitive. Reading the chaining code makes it obvious that

use(myApp, csrf, logging, recovery)

is equivalent to this code:

recovery(logging(csrf(myApp)))

and this request cycle:

recovery -> logging -> csrf -> myApp

So, a reversed order from what you've written in your code.

How Alice is better

Alice adopts Matt's model and fixes the small imperfections. It still requires middleware constructors of form func (http.Handler) http.Handler, but requests now flow exactly the way you order your handlers.

This code:

alice.Chain(recovery, logging, csrf).Then(myApp)

will result in this request cycle:

recovery -> logging -> csrf -> myApp

Here, recovery receives the request and has a reference to logging. It may or may not call logging: it's completely up to recovery, just as without Alice.

One more thing

Deciding on a unified constructor for middleware is the main reason why creating such a convenient API for chaining is even possible.

However, limiting ourselves to one function signature has a drawback. Many middleware have settings one might want (or have) to set. Not only does that not fit into our chosen signature, there is no standard way adopted by developers.

Middleware found in the standard library take the options as additional arguments to the same function:

handler = http.StripPrefix("/old/", handler)

Throttled constructs a rate limiting strategy which has a method to wrap our handler:

th := throttled.RateLimit(throttled.PerMin(30), 
    &throttled.VaryBy{RemoteAddr: true}, 
    store.NewMemStore(1000))
handler := th.Throttle(handler)

And nosurf has additional methods on the handler:

handler := nosurf.New(handler)
handler.ExemptPath("/")
handler.SetFailureHandler(http.NotFoundHandler())

It's nearly impossible for Alice to solve this without resorting to ugly reflection tricks, so it doesn't try to. Instead, when in need of customization, one should create their own constructor.

func myStripPrefix(h http.Handler) http.Handler {
    return http.StripPrefix("/old/", h)
}

This now complies to the constructor interface, so we can plug it into Alice.

alice.New(myStripPrefix).Then(myApp)

Prove me wrong, though

Alice was born in a matter of minutes and based on my own perception of what's right and what's not. Just like I've found other solutions non-ideal, some might find inherent flaws in how Alice works. Although it's unlikely that the very essence of how Alice functions will change, any feedback is welcome.

Check out Alice on GitHub.

Best Practices for Errors in Go

2014-04-26T18:00:00+00:00

Error handling seems to be one of the more controversial areas of Go. Some are pleased with it, while others hate it with passion. Nevertheless, there is a handful of best practices that will make dealing with errors less painful if you're a sceptic and even better if you like it as it is.

Know when to panic

The idiomatic way of reporting errors in Go is having the error as the last return value of a funtion. However, Go also offers an alternative error mechanism called panic that is similar to what is known as exceptions in other programming languages.

Despite not being suitable everywhere, panics are useful in several specific scenarios.

When the user explicitly asks: Okay

In certain cases, the user might want to panic and thus abort the whole application if an important part of it can not be initialized. Go's standard library is ripe with examples of variants of functions that panic instead of returning an error, e.g. regexp.MustCompile. As long as there remains a function that does this in an idiomatic way (regexp.Compile), providing the user with a nifty shortcut is okay.

When setting up: Maybe

A scenario that, in a way, intersects with the previous one, is the setting up phase of an application. In most cases, if preparations fail, there is no point in continuing. One case of this might be a web application failing to bind the required port or being unable to connect to the database server. In that case, there's not much left to do, so panicking is acceptable, even if not explicitly requested by the user.

This behavior can also be observed in the standard library: the net/http muxer will panic if a pattern is invalid in any way.

func (mux *ServeMux) Handle(pattern string, handler Handler) {
    mux.mu.Lock()
    defer mux.mu.Unlock()

    if pattern == "" {
        panic("http: invalid pattern " + pattern)
    }
    if handler == nil {
        panic("http: nil handler")
    }
    if mux.m[pattern].explicit {
        panic("http: multiple registrations for " + pattern)
    }

    // ...

Otherwise: Not really

Although there might be other cases where panic is useful, returned errors remain preferable in most scenarios.

Predefine errors

It's not unusual to see code returning errors like this

func doStuff() error {
    if someCondition {
        return errors.New("no space left on the device")
    } else {
        return errors.New("permission denied")
    }
}

The wrongdoing here is that it's not convenient to check which error has been returned. Comparing strings is error prone: misspelling a string in comparison will result in an error that cannot be caught at compile time, while a cosmetic change of the returned error will break the checking just as much.

Given a small set of errors, the best way to handle this is to predefine each error publicly at the package level.

var ErrNoSpaceLeft = errors.New("no space left on the device")
var ErrPermissionDenied = errors.New("permission denied")

func doStuff() error {
    if someCondition {
        return ErrNoSpaceLeft
    } else {
        return ErrPermissionDenied 
    }
}

Now the previous problems aren't anymore.

if err == ErrNoSpaceLeft {
    // handle this particular error
}

Provide information

Sometimes, an error can happen because of a whole lot of different reasons. Wikipedia lists 41 different HTTP client errors. Let's say we want to treat them as errors in Go (net/http does not). What's more, we'd like to be able to look into the specifics of the error we received and find out whether the error was 404, 410 or 418.

In the last paragraph we developed a pattern for discerning errors from one another, but here it gets a bit messy. Predefining 41 separate errors like this:

var ErrBadRequest = errors.New("HTTP 400: Bad Request")
var ErrUnauthorized = errors.New("HTTP 401: Unauthorized")
// ...

will make our code and documentation messy and our fingers sore.

A custom error type is the best solution to this problem. Go's implicit interfaces make creating one easy: to conform to the error interface, we only need to have an Error() method that returns a string.

type HTTPError struct {
    Code        int
    Description string
}

func (h HTTPError) Error() string {
    return fmt.Sprintf("HTTP %d: %s", h.Code, h.Description)
}

Not only does this act like a regular error, it also contains the status code as an integer. We can then use it as follows:

func request() error {
    return HTTPError{404, "Not Found"}
}

func main() {
    err := request()

    if err != nil {
        // an error occured
        if err.(HTTPError).Code == 404 {
            // handle a "not found" error
        } else {
            // handle a different error
        }
    }

}

The only minor annoyance left here is the need to do a type assertion to a concrete error type. But it's a small price to pay compared to the fragility of parsing the error code from a string.

Provide stack traces

Errors as they exist in Go remain inferior to panics in one important way: they do not provide important information about where in the call stack the error happened.

A solution to this problem has recently been created by the Juju team at Canonical. Their package errgo provides the functionality of wrapping an error into another one that records where the error happened.

Building up on the HTTP error handling example, we'll now put this to use.

package main

import (
    "fmt"

    "github.com/juju/errgo"
)

type HTTPError struct {
    Code        int
    Description string
}

func (h HTTPError) Error() string {
    return fmt.Sprintf("HTTP %d: %s", h.Code, h.Description)
}

func request() error {
    return errgo.Mask(HTTPError{404, "Not Found"})
}

func main() {
    err := request()

    fmt.Println(err.(errgo.Locationer).Location())

    realErr := err.(errgo.Wrapper).Underlying()

    if realErr.(HTTPError).Code == 404 {
        // handle a "not found" error
    } else {
        // handle a different error
    }
}

Our code has changed in several ways. Firstly, we wrap an error into an errgo-provided type. This code now prints the information on where the error happened. On my machine it outputs

/private/tmp/example.go:19

referencing a line in request().

However, our code has become somewhat messier. To get to the real HTTPError we need to do more unwrapping. Sadly, I'm not aware of a real way to make this nicer, so it's all about tradeoffs. If your codebase is small and you can always tell where the error came from, you might not need to use errgo at all.

What about concrete types?

Some might point out that a portion of the type assertions could have been avoided if we returned a concrete type from a function instead of an error interface.

However, doing that in conjunction with the := operator will bring trouble. As a result, the error variable will not be reusable.

func f1() HTTPError { ... }
func f2() OSError { ... }

func main() {
    // err automatically declared as HTTPError
    err := f1()

    // OSError is a completely different type
    // The compiler does not allow this
    err = f2()
}

To avoid this, errors are best returned using the error type. Standard library avoids returning concrete types as well, e.g. os package states:

Often, more information is available within the error. For example, if a call that takes a file name fails, such as Open or Stat, the error will include the failing file name when printed and will be of type *PathError, which may be unpacked for more information.

This concludes my list of best practices for errors in Go. As in other areas, a different mindset has to be adopted when coming into Go from elsewhere. "Different" does not imply "worse" though and deciding on a set of conventions is vital to making Go development even better.

Writing HTTP Middleware in Go

2013-10-19T19:00:00+00:00

In the context of web development, "middleware" usually stands for "a part of an application that wraps the original application, adding additional functionality". It's a concept that usually seems to be somewhat underappreciated, but I think middleware is great.

For one, a good middleware has a single responsibility, is pluggable and self-contained. That means you can plug it in your app at the interface level and have it just work. It doesn't affect your coding style, it isn't a framework, but merely another layer in your request handling cycle. There's no need to rewrite your code: if you decide that you want the middleware, you add it into the equation, if you change your mind, you remove it. That's it.

Looking at Go, HTTP middleware is quite prevalent, even in the standard library. Although it might not be obvious at first, functions in the net/http package, like StripPrefix or TimeoutHandler are exactly what we defined middleware to be: they wrap your handler and take additional steps when dealing with requests or responses.

My recent Go package nosurf is middleware too. I intentionally designed it as one from the very beginning. In most cases you don't need to be aware of things happening at the application layer to do a CSRF check: nosurf, like any proper middleware, stands completely on its own and works with any tools that use the standard net/http interface.

You can also use middleware for:

Mitigating BREACH attack by length hiding
Rate-limiting
Blocking evil bots
Providing debugging information
Adding HSTS, X-Frame-Options headers
Recovering gracefully from panics
...and probably many others

Writing a simple middleware

For the first example, we'll write middleware that only allows users visit our website through a single domain (specified by HTTP in the Host header). A middleware like that could serve to protect the web application from host spoofing attacks.

Constructing the type

For starters, let's define a type for the middleware. We'll call it SingleHost.

type SingleHost struct {
    handler     http.Handler
    allowedHost string
}

It consists of only two fields:

the wrapped Handler we'll call if the request comes with a valid Host.
the allowed host value itself.

As we made the field names lowercase, making them private to our package, we should also make a constructor for our type.

func NewSingleHost(handler http.Handler, allowedHost string) *SingleHost {
    return &SingleHost{handler: handler, allowedHost: allowedHost}
}

Request handling

Now, for the actual logic. To implement http.Handler, our type only needs to have one method:

type Handler interface {
        ServeHTTP(ResponseWriter, *Request)
}

And here it is:

func (s *SingleHost) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    host := r.Host
    if host == s.allowedHost {
        s.handler.ServeHTTP(w, r)
    } else {
        w.WriteHeader(403)
    }
}

ServeHTTP method simply checks the Host header on the request:

if it matches the allowedHost set by the constructor, it calls the wrapped handler's ServeHTTP method, thus passing the responsibility for handling the request.
if it doesn't match, it returns the 403 (Forbidden) status code and the request is dealt with.

The original handler's ServeHTTP is never called in the latter case, so not only does it not get a say in this, it won't even know such a request arrived at all.

Now that we're done coding our middleware, we just need to plug it in. Instead of passing our original Handler directly into the net/http server, we wrap it in the middleware.

singleHosted = NewSingleHost(myHandler, "example.com")
http.ListenAndServe(":8080", singleHosted)

An alternative approach

The middleware we just wrote is really simple: it literally consists of 15 lines of code. For writing such middleware, there exists a method with less boilerplate. Thanks to Go's support of first class functions and closures, and having the neat http.HandlerFunc wrapper, we'll be able to implement this as a simple function, rather than a separate struct type. Here is the function-based version of our middleware in its entirety.

func SingleHost(handler http.Handler, allowedHost string) http.Handler {
    ourFunc := func(w http.ResponseWriter, r *http.Request) {
        host := r.Host
        if host == allowedHost {
            handler.ServeHTTP(w, r)
        } else {
            w.WriteHeader(403)
        }
    }
    return http.HandlerFunc(ourFunc)
}

Here we declare a simple function called SingleHost that takes in a Handler to wrap and the allowed hostname. Inside it, we construct a function analogous to ServeHTTP from the previous version of our middleware. Our inner function is actually a closure, so it can access the variables from the outer function. Finally, HandlerFunc lets us use this function as a http.Handler.

Deciding whether to use a HandlerFunc or to roll out your own http.Handler type is ultimately up to you. While for basic cases a function might be enough, if you find your middleware growing, you might want to consider making your own struct type and separate the logic into several methods.

Meanwhile, the standard library actually uses both ways of building middleware. StripPrefix is a function that returns a HandlerFunc, while TimeoutHandler, although a function too, returns a custom struct type that handles the requests.

A more complex case

Our SingleHost middleware was trivial: we checked one attribute of the request and either passed the request to the original handler, not caring about it anymore, or returned a response ourselves and didn't let the original handler touch it at all. Nevertheless, there are cases where, rather than acting based on what the request is, our middleware has to post-process the response after the original handler has written it, modifying it in some way.

Appending data is easy

If we just want to append some data after the body written by the wrapped handler, all we have to do is call Write() after it finishes:

type AppendMiddleware struct {
    handler http.Handler
}

func (a *AppendMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    a.handler.ServeHTTP(w, r)
    w.Write([]byte("Middleware says hello."))
}

The response body will now consist of whatever the original handler outputted, followed by Middleware says hello..

The problem

Doing other types of response manipulations is a bit harder though. Say, we'd like to prepend data to the response instead of appending it.

If we call Write() before the original handler does, it will lose control over the status code and headers, since the first Write() writes them out immediately.

Modifying the original output in any other way (say, replacing strings in it), changing certain response headers or setting a different status code won't work because of a similar reason: when the wrapped handler returns, those will have already be sent to the client.

To counter this we need a particular kind of ResponseWriter that would work as a buffer, gathering the response and storing it for later use (and modifications). We would then pass this buffering ResponseWriter to the original handler instead of giving it the real RW, thus preventing it from actually sending the response to the user just yet.

Luckily, there's a tool just like that in the Go standard library. ResponseRecorder in the net/http/httptest package does all we need: it saves the response status code, a map of response headers and accumulates the body into a buffer of bytes. Although (like the package name implies) it's intended to be used in tests, it fits our use case as well.

Let's look at an example of middleware that uses ResponseRecorder and modifies everything in a response, just for the sake of completeness.

type ModifierMiddleware struct {
    handler http.Handler
}

func (m *ModifierMiddleware) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    rec := httptest.NewRecorder()
    // passing a ResponseRecorder instead of the original RW
    m.handler.ServeHTTP(rec, r)
    // after this finishes, we have the response recorded
    // and can modify it before copying it to the original RW

    // we copy the original headers first
    for k, v := range rec.Header() {
        w.Header()[k] = v
    }
    // and set an additional one
    w.Header().Set("X-We-Modified-This", "Yup")
    // only then the status code, as this call writes out the headers 
    w.WriteHeader(418)

    // The body hasn't been written (to the real RW) yet,
    // so we can prepend some data.
    data := []byte("Middleware says hello again. ")

    // But the Content-Length might have been set already,
    // we should modify it by adding the length
    // of our own data.
    // Ignoring the error is fine here:
    // if Content-Length is empty or otherwise invalid,
    // Atoi() will return zero,
    // which is just what we'd want in that case.
    clen, _ := strconv.Atoi(r.Header.Get("Content-Length"))
    clen += len(data)
    r.Header.Set("Content-Length", strconv.Itoa(clen))

    // finally, write out our data
    w.Write(data)
    // then write out the original body
    w.Write(rec.Body.Bytes())
}

And here's the response we get by wrapping a handler that would otherwise simply return "Success!" with our middleware.

HTTP/1.1 418 I'm a teapot
X-We-Modified-This: Yup
Content-Type: text/plain; charset=utf-8
Content-Length: 37
Date: Tue, 03 Sep 2013 18:41:39 GMT

Middleware says hello again. Success!

This opens up a whole lot of new possibilities. The wrapped handler is now completely in or control: even after it handling the request, we can manipulate the response in any way we want.

In various cases, your middleware might need to expose certain information to other middleware or your app itself. For example, nosurf needs to give the user a way to access the CSRF token and the reason of failure (if any).

A nice pattern for this is to use a map, usually an unexported one, that maps http.Request pointers to pieces of the data needed, and then expose package (or handler) level functions to access the data.

I used this pattern for nosurf too. Here, I created a global context map. Note that a mutex is also needed, since Go's maps aren't safe for concurrent access by default.

type csrfContext struct {
    token string
    reason error
}

var (
    contextMap = make(map[*http.Request]*csrfContext)
    cmMutex    = new(sync.RWMutex)
)

The data is set by the handler and exposed via exported functions like Token().

func Token(req *http.Request) string {
    cmMutex.RLock()
    defer cmMutex.RUnlock()

    ctx, ok := contextMap[req]
    if !ok {
            return ""
    }

    return ctx.token
}

You can find the whole implementation in the context.go file in nosurf's repository.

While I chose to implement this on my own for nosurf, there exists a handy gorilla/context package that implements a generic map for saving request information. In most cases, it should suffice and protect you from pitfalls of implementing a shared storage on your own. It even has middleware of its own that clears the request data after it's been served.

All in all

The intention of this article was both to draw fellow gophers' attention to middleware as a concept and to demonstrate some of the basic building blocks for writing middleware in Go. Despite being a relatively young language, Go has an amazing standard HTTP interface. It's one of the factors that make coding middleware for Go a painless and even fun process.

Nevertheless, there is still a lack of quality HTTP tools for Go. Most, if not all, of the middleware ideas for Go I mentioned earlier are yet to come to life. Now that you know how to build middleware for Go, why not do it yourself? ;)

P.S. You can find samples for all the middleware written for this post in a GitHub gist.

Embrace Go's HTTP Tools

2013-10-14T20:00:00+00:00

Some time ago I released nosurf, a Go middleware for mitigating Cross-Site Request Forgery attacks. Writing a seemingly simple and small package was enough to fall in love with how Go handles HTTP. Yet, it's up to us to either embrace the standard HTTP facilities or fragmentate, sacrificing composability and modularity.

http.Handler is THE interface

Unified HTTP interfaces for web apps written in certain programming languages, like WSGI for Python and Rack for Ruby are a great idea, but they weren't always there. For instance, Rack only emerged in 2007, when Rails had already been going strong for a while.

Meanwhile in Go, the only interface needed has been in development since 2009, and although it's been through some serious changes since that, by the end of 2011, months before Go 1.0 was released, it had already stabilized.

Of course, I'm talking about the mighty http.Handler.

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

To be able to handle HTTP requests, your type only needs to implement this one method. The method reads the request info from the given *Request and writes a response into the given ResponseWriter. Seems simple enough, right?

Complement, don't replace

Yet, when building abstractions on top of that, some get it wrong. Take for example Mango, described by its author as "a modular web-application framework for Go, inspired by Rack and PEP333".

This is what a Mango application looks like:

func Hello(env mango.Env) (mango.Status, mango.Headers, mango.Body) {
  return 200, mango.Headers{}, mango.Body("Hello World!")
}

Looks simple, concise and very similar to WSGI or Rack, right? Except for one thing. While with dynamic/duck typing, you could have any iterable for a body, here mango.Body is simply a string. Essentially, that takes away the ability to do any sort of streaming responses with Mango. Even if it were to expose a ResponseWriter, anything written to it would clash with the returned values, since they're only returned at the end of the function, after the calls to ResponseWriter have already been made.

That's bad. Whether you need another interface on top of existing net/http is a matter of taste, but even if you do, it should not take functionality away. An interface that is nicer to code with, but takes away important functions is clearly inferior.

The right way

A popular "micro" web framework web.go deals with this in a simple, yet much better way. Its handlers take a pointer to web.Context as an optional first argument.

type Context struct {
    Request *http.Request
    Params  map[string]string
    Server  *Server
    http.ResponseWriter
}
// ...
func hello(ctx *web.Context, val string) string {
    return "hello " + val 
}

web.Context does not take the standard HTTP handler structures away. Instead, the *Request argument is available as a struct member and Context ~~implements the required ResponseWriter methods itself~~ embeds the original ResponseWriter. The string you return from the function (if any) is simply appended to the response.

That is a good design choice and I think it goes well with Go's philosophy. Even though you get a nice higher-level API, you don't have to sacrifice the low-level control over the request handling.

Start now

Go's HTTP library infrastructure, despite growing rapidly, still has some gaps left to fill. But the last thing we need is fragmentation and annoying incompatibilities due to poor design and abstractions that actually take important functionality away. Embracing and supporting the standard Go HTTP facilities is, in my humble opinion, the straightest way to having functional and modular 3rd-party HTTP tools.

Justinas Stankevičius

The bizarre world of V

Table of Contents

Disclaimers

Analysis

Shifting timelines

C/C++ translation

Native x64 code generation

Memory unmanagement

Early V

V 0.2 and the "zero leaks" demo

Status quo

Redefining terms

Undefined behavior

Static binaries that are not

Other misnomers

Casting a wide net

Backends

Batteries included (but not yet charged)

V 1.0 and the stability guarantee

The author

"It's just a bug"

"Things change"

A broader pattern

He just like me fr

Outro

Recommended reading

Footnotes

NixOS in the Cloud, step-by-step: part 2

Preparation

nix-shell

DigitalOcean API token

DigitalOcean resource IDs

Creating our droplets

Using Terraform state from Nix

Marrying Terraform and Morph

Deploying services to the network

Finishing touches

That's all (for now), folks

Alternative approaches

NixOS in the Cloud, step-by-step: part 1

Why this article was made

Who this is for

Nix

DigitalOcean

Morph

Preparation

nix-shell

Preparing your DigitalOcean project

Custom image

SSH key

Create your NixOS droplet!

Verify

Managing the droplet with Morph

Deploying nginx

Pinning nixpkgs

State version

That's it

What's next?

Producing a podcast the geeky way: Audio

Hardware

Remote recording woes

Post-production and perfectionism

The move to Linux and a DAW

JACK: the daemon of my nightmares dreams

The future

My reason to be excited for Go 1.4

Why that excites me

Less is more

Alice – Painless Middleware Chaining for Go

Flaws of other approaches

How Alice is better

One more thing

Prove me wrong, though

Best Practices for Errors in Go

Know when to panic

When the user explicitly asks: Okay

When setting up: Maybe

Otherwise: Not really

Predefine errors