This post was originally published on my personal website.

Introduction

I decided a few years ago to start working on a development environment that would fullfill a variety of my needs while working on client and personal projects, from sharing a prototype online to publishing generative art on my social networks.

This project became Fragment, which I made publicly available on my Github in September 2022.

Concept

Fragment is built upon the concept of sketches which are Javascript files where you can exports specific functions acting as hooks in a rendering pipeline.

Once Fragment is installed, a sketch file can be created by running the following command:

fragment create

The Command Line Interface (CLI) will ask for the destination folder of your sketch, the name of the files and the starting template of the sketch.

A blank sketch looks like this:

export let props = {};

/**
 * @param {object} params
 * @param {HTMLCanvasElement} params.canvas
 * @param {number} params.width
 * @param {number} params.height
 * @param {number} params.pixelRatio
 */
export let init = ({ canvas, width, height, pixelRatio }) => {};

/**
 * @param {object} params
 * @param {HTMLCanvasElement} params.canvas
 * @param {number} params.width
 * @param {number} params.height
 * @param {number} params.pixelRatio
 * @param {number} params.time
 * @param {number} params.deltaTime
 * @param {number} params.frame
 * @param {number} params.playhead
 * @param {number} params.playcount
 */
export let update = ({ width, height, time, deltaTime }) => {};

/**
 * @param {object} params
 * @param {HTMLCanvasElement} params.canvas
 * @param {number} params.width
 * @param {number} params.height
 * @param {number} params.pixelRatio
 */
export let resize = ({ width, height, pixelRatio }) => {};

export let dispose = ({ width, height, pixelRatio }) => {};

These functions describe the lifecycle of a sketch and are called internally by Fragment in a specific order:

• init() is called once on every sketch load, which happens when Fragment starts and every time the file is saved.
• resize() is called once on every sketch load, and every time the width, height or pixelRatio of the canvas is changed from the interface
• update() is called on every frame. By default Fragment runs a sketch at 60 frames per seconds but this can be changed by exporting a variable from the sketch like this:

// calls update() 24 times per seconds
export let fps = 24; 

// calls update() only once on reload and prop change
export let fps = 0;

• dispose() is called before every reload. This enables custom code to run before a sketch is re-mounted, such as removing existing listeners, destroying textures and so on.

Fragment can work with any library using a <canvas>. Internally, it uses different Renderers, and each of them are running specific code over a canvas node before going into the lifecycle methods of a sketch.

At the time of this writing, Fragment has built-in support for Canvas 2D, THREE.js, p5.js (2D and WebGL modes), and fragment-gl, an internal library which is not documented yet and will be open-source as a separate package in the near future.

To simplify the usage of the most frequent types of sketches, Fragment comes with templates for these libraries.

Fragment also supports custom renderers by using the right hooks and syntax.

Speed

It was obvious from the very start that Fragment needed to feel instant, which was not the case with the initial version. After different attempts and discovering the nightmare resolving modules imports is, I decided to use Vite as the foundation of Fragment. It was, indeed, very fast. I had to dive deep into it to make the environment works the way I intended in the first place: instant reload of sketches, preserving UI states…

I also experimented with Hot Shader Reloading (HSR) with previous front-end tools and ported the code as a Vite plugin, by using a dedicated WebSocket to send the new shader code and replace it into the right WebGL programs.

I'll write a dedicated article about how I implemented this in the coming weeks.

Interactivity

A sketch can also export props which is a single Javascript object that can be edited to create GUIs in the interface. Fragment can build a variety of inputs based on the prop values.

export let props = {
    // TextInput
    text: {
        value: 'hello world',
    },
    // NumberInput with Slider
    radius: {
        value: 12,
        params: {
            min: 0,
            max: 15,
            step: 0.1,
        }
    },
    // VectorInput
    direction: {
        value: [0, 1]
    },
    // ColorInput
    color: {
        value: 'rgb(255, 0, 0),
        displayName: 'backgroundColor'
    },
    // Checkbox
    toggle: {
        value: true,
    },
    // Button
    change: {
        value: () => { console.log('change'); }
    }
};

Fragment also supports Mouse, Keyboard and MIDI inputs out-of-the box, with the ability to configure them on the fly, either from the interface or from code.

import {
    // Mouse triggers
    onMouseDown,
    onMouseUp,
    onMouseMove,
    // Keyboard triggers
    onKeyDown,
    onKeyUp,
    onKeyPress,
    // MIDI triggers
    onNoteOn,
    onNoteOff,
    onControlChange,
} from '@fragment/triggers';

function changeColors() {
    //...
}

export let init = () => {
    onKeyDown('C', changeColors);
}

Triggers are automatically destroyed and re-instantiated between sketch reloads so you don’t need to clean them up manually. Configurations made from the interface is saved in LocalStorage to retrieve them between sketch reloads and work sessions.

Interface

The entire layout of Fragment can be changed on the fly to fit any screen or canvas size.

Exports

Fragment can export images and videos to the disk with simple keyboard shortcuts. It supports PNG, JPEG and WEBP image formats and MP4, GIF, WEBM videos and can also exports frames sequences. The export settings such as quality, format or duration can be changed directly from the interface.

I was a heavy user of canvas-sketch by Matt Deslauriers before creating Fragment and ended up replicating the workflow he implemented on his tool. Many of Fragment's export capabilities are using Matt's tools under the hood.

A sketch can also be built into static files and deployed online.

fragment build sketch.js

Conclusion

Fragment aims to cover many usages in creative coding, from a quick prototype to a full project! I've been using it on all my personal and client projects for the last 3 years now and I'm just excited to push it further. There are still so many ideas left to explore!

It's open-source and built on many open-source projects, such as Vite, Svelte, Polka, clack and many more!

Interested ? Give it a try or subscribe to get updates about the project.

I would love to hear about your thoughts so feel free to reach out if you want to discuss the project.