This post dives deep into Alma - A side project which I built in 2022. It was nominated to Side project of the year at React Summit in Amsterdam 2023.

My history with node-based user interfaces goes back to 2017. During my time as a Designer at EA DICE working on the Battlefield-franchise, I did some work within the Frostbite editor - EA's proprietary game engine. It had a visual no-code interface for setting up game logic. It used a node-based UI & I fell in love.

In 2020, I became really interested in WebGL, and wanted to learn everything that was going on behind the scenes of the popular THREE.js JavaScript framework. Around this time, Unity were building a really nice node-based UI for shader development. So I thought...

How can I build a node UI for WebGL shaders?

Understanding WebGL and Shaders

Dan Hollick has made a wonderful introduction to Shaders for the curious mind. I recommend it a lot. But I'll try to do my best to explain these concepts in simple terms.

WebGL

WebGL (Web Graphics Library) is a JavaScript API for rendering high-performance interactive 3D and 2D graphics. It uses your GPU to make very basic math operations super fast in parallel, something the CPU isn't very good at.

Shaders

You may have heard the term before - but what are they? They are small pieces of code which runs once for every pixel on your screen. And it's only job is to determine the final color of each pixel. Sounds simple, right?

But the beautiful thing is that you can get really fancy with the maths and create amazing things.

So... what is Alma exactly?

It is a creative environment where you can use simple nodes & connections to craft stunning visual experiences like the one above - without writing a single line of code.

If you look at this, you might be wondering what exactly you're looking at, and how it relates to WebGL and Shaders.

This is a Node Graph. It is a collection of two distinct pieces.

• Nodes: They can represent anything which compiles to GLSL. Uniforms, Attributes, Variables - but most often Functions.
• Connections: Connections define relationships between Nodes. They are used to "pipe" values from one Node to another.

Alma comes with a wide array of built-in Nodes. Most of them are functions used to compute new values, but some of them are pure primitives like Vectors.

The graph pushes values around between Nodes using connections. But what are these values exactly? This is where the Shader AST comes in.

Shader AST

Alright - this is the bread & butter. The Shader AST is essentially what's doing all the magic in Alma. It's an abstract syntax tree (think a big JSON structure) which compiles to perfectly valid GLSL code, ready to be attached to a WebGL program.

The nice thing with a Shader AST is that we can represent anything using it. Anything like a variable declaration, function call, uniform declaration, and so on.

We construct small objects describing a certain GLSL concept, and push those around the graph. Certain Nodes operate & change values in some way. In the end, the graph is computed into a big syntax tree that we can give to a GLSL compiler to convert it into valid source code.

With that being said - a Node Graph like this:

Produces GLSL code looking like this:

#version 300 es
#ifdef GL_FRAGMENT_PRECISION_HIGH
precision highp int;
precision highp float;
#else
precision mediump int;
precision mediump float;
#endif
#ifndef PI
#define PI (3.141592653589793)
#endif
#ifndef TAU
#define TAU (6.283185307179586)
#endif
#ifndef HALF_PI
#define HALF_PI (1.570796326794896)
#endif

uniform vec2 resolution;
uniform float time;
uniform vec2 mouse;
uniform sampler2D WWaKjnJt;
uniform float WWaKjnJtAspectRatio;
in vec2 v_uv;
layout(location = 0) out vec4 fragColor;
void main() {
  fragColor = vec4(
    (gl_FragCoord.xy / resolution).x,
    (gl_FragCoord.xy / resolution).y,
    sin(time),
    1.0
  );
}

If you look closely, you can see how certain nodes plays a part in the generated output.

With the conceptual approach out of the way, let's get dirty with some architectural talk. How is it built? What's used under the hood?

Picking the right frameworks

@thi.ng/shader-ast

I knew early on that the biggest challenge would be to create a Shader AST with a compiler. I made some efforts to build one myself, but I knew that I was doomed to fail. That's when I found the most beautiful open source project I've ever seen – @thi.ng/umbrella.

Broadly scoped ecosystem & mono-repository of 188 TypeScript projects (and 150 examples) for general purpose, functional, data driven development.

That's how the author, Karsten Schmidt, self describes his enormous project. And that's where I found a few packages called @thi.ng/shader-ast, @thi.ng/shader-ast-glsl & @thi.ng/webgl.

The most beautiful AST implementation I've seen. With excellent TypeScript support as well! This is what Alma runs under the hood for everything WebGL, Shader & GLSL related. And it's bloody fantastic.

MobX

As a node graph propagates a lot of values around, and does a lot of computation - I saw this as a golden opportunity to use MobX for the node graph state management. I've always been a huge fan of its reactivity model, and the author describes the MobX as:

A signal based, battle-tested library that makes state management simple and scalable by transparently applying functional reactive programming.

React Draggable

I settled on react-draggable as the framework of choice when it came to the Nodes themselves. Easy & rigid API for dealing with draggable entities.

The anatomy of a Node

A Node is conceptually a function but with support for multiple return values. If we think about the computational part of a node; it looks just like a function - inputs -> computation -> outputs. An Addition Node may for instance have two inputs and a single output, which assigns the output the sum of the inputs.

A Node consists of Inputs & Outputs. Let's learn more about them.

Inputs

These are responsible for accepting values from incoming connections and feeding them into computation. They are conceptually the same as a function argument.

Inputs have a few properties associated with them:

• Identifier: A unique identifier to keep track of it in our node graph.
• Name: The most obvious, a simple name that describes it.
• Type: A type which describes the value. This will in our case a representation of any GLSL type (float, int, vec2, vec3, etc.).
• Validator: A predicate function which validates an attempted incoming connection. It's there as an extra layer on top of the type-simulation, for extra granularity.
• Default Value: A default value that we always can use as a fallback when a connection disconnects or similar. Must conform to the Input's given type & pass its validator.
• Value: This is the value which will enter the Node. Computation will be done by outputs on this value.

Input values are MobX observables under the hood, which means that we can observe them and run effects when they're mutated.

Outputs

Outputs are used to compute, produce & derive values from the inputs. The following properties are associated with Outputs:

• Identifier: A unique identifier to keep track of it in our node graph.
• Name: A simple name that describes it.
• Type: A type which describes the value. This will in our case a representation of any GLSL type (float, int, vec2, vec3, etc.).
• Value: A MobX computed which tracks & operate on input values.

Output values are MobX computed's, which fits perfectly. I quote the MobX documentation:

Computed values can be used to derive information from other observables. They evaluate lazily, caching their output and only recomputing if one of the underlying observables has changed.

// This function will be used as a MobX computed getter
value: () => {
    // The smoothstep function comes from the @thi.ng/shader-ast package
    return smoothstep<Prim, Prim, Prim>(
        this.resolveValue(this.inputs.edgeA.value),
        this.resolveValue(this.inputs.edgeB.value),
        this.resolveValue(this.inputs.input.value)
    );
}

This means that a Node's output only recomputes if any of its inputs actually mutates. These mutations comes from Connections - let's talk about them.

The anatomy of a Connection

Connections have a sole purpose - propagate values from Outputs to Inputs. That's it. And that's why Alma's connections are MobX reactions. We use the autorun-method which is the simplest kind of MobX reaction. Quoting the MobX docs once again:

The autorun function accepts one function that should run every time anything it observes changes.

Connection reactions look something like this:

this.reactionDisposer = autorun(() => {
    if (this.to.validator(this.from.value)) {
        this.to.setValue(this.from);
    }
});

As you can see, it's very simple. We make sure to validate the incoming value. If it passes, we set the value to the Output. We also keep a reactionDisposer to run later if the connection is disconnected.

The User Interface

I built Alma on React. Shocking. Not really. MobX has nice support for React as well, so that settled it. And as mentioned before, I used React Draggable for the Nodes. I do however get a lot of questions regarding what I'm using to render the connections in Alma. They are pure SVG elements.

An important thing for me was to provide thorough information about the nodes. So I wrote a small description about them all which is shown when a node is selected.

I also created a small toolbar for easy access to common actions & helpers.

Built-in Nodes

Alma has a large extension of built-in nodes.

Arithmetic operations, trigonometry, uniform accessors, vector primitives, math utilities – almost everything you can expect from a GLSL standard library.

Defining your own function node

But my proudest feature in Alma is the ability to "generate" a node from raw GLSL code. If you don't want to bother pulling out tens of nodes to get some basic logic going, you can supply a GLSL function which will be interpreted & turned into a node.

Wrapping it all up

That's pretty much it. Alma is not fancier than that.

Alma was such a wonderful project to work on. It was never intended to blow up like it did. I'm a designer by heart, but an engineer by trial and error. It was purely a journey for me to explore the ins & outs of WebGL, Shader AST's and creative environments as a whole.

What's even more interesting is that I suck at math. Suck. But I'm happy that the math involved in shaders made it at least a bit more fun.

I did eventually rewrite the Node Graph implementation, and built it on top of RxJS instead - but that story is for another time.

Worth knowing is that I started doing a large rewrite of Alma in 2023; something that I unfortunately haven't been able to complete. So if you're interested in diving into the repository on GitHub, make sure to checkout the production branch. Otherwise it won't make much sense to you after having read this article.

Thank you for making it this far. I appreciate it a lot. If you like these kind of posts & want to support me with a few dollars, consider looking into my paid subscriptions.

Until next time.