Learning WebAssembly #10: Image Processing in AssemblyScript

2021-01-30 by Tomas Tulka

WebAssembly is a great fit for image processing. We will manipulate image data with a simple Wasm function written in AssemblyScript and run it in the web browser.

In the previous part of this series, we already learned how to write Wasm modules in AssemblyScript. In this part, we will use this knowledge in a practical scenario: image manipulations with WebAssembly.

We will demonstrate a typical use-case by a simple function for converting an image to grayscale.

Albeit the calculation of the gray color is not very demanding, it clearly demonstrates the real-world usage of WebAssembly on the web: computation-intensive tasks.

You can find the full discussed source code on my GitHub.

Browser Runtime

Unlike our previous experiments with AssemblyScript, this time we will run our Wasm module in a browser. You might first recall how to use a web browser as a Wasm runtime, but the code is quite straight-forward:

WebAssembly
  .instantiateStreaming(fetch('grayscale.wasm'), {})
  .then(({ instance }) => {
    ...
  }

To make the Fetch API work you must serve the web page via HTTP(S).

The second argument of the instantiateStreaming is an object with Wasm imports. All environment imports for Wasm modules, which were compiled from AssemblyScript, are included in an env object:

When running in the browser, we must import an error-callback function abort:

WebAssembly
  .instantiateStreaming(fetch('grayscale.wasm'), {
    env: {
      abort: (_msg, _file, line, column) =>
        console.error(`Error at ${line}:${column}`)
    }
  })

In Node.js those are already provided by the AssemblyScript loader out of the box.

Canvas

To show the image and its transformation in a browser, we will use the HTML canvas element:

<canvas id="canvas" width="500" height="500"></canvas>

And its 2D context:

const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const [width, height] = [canvas.width, canvas.height];

First, we will draw the original image on the canvas:

const img = new Image();
img.src = './my-pretty-picture.png';
img.crossOrigin = 'anonymous';
img.onload = () =>
  ctx.drawImage(img, 0, 0, width, height);

Then, we will get the image data that we will work with:

const imageData = ctx.getImageData(0, 0, width, height);
const data = imageData.data;

The image data is represented as a Uint8ClampedArray a one-dimensional array in the RGBA order, with integer values between 0 and 255 (inclusive).

Memory

Because we will work extensively with memory, we should recall the fundamentals.

We can either create a memory instance from JavaScript and import it into the Wasm module or let the module initialize memory and export its instance into JavaScript.

One benefit of the former approach is the possibility of initializing memory to the needed size.

When exporting memory from Wasm, the initial size is one page (64 KB). To increase its size, we have to call memory.grow() programmatically, which could be impractical, at least in cases where the size is known in advance.

Our image might be bigger than 64 KB, so we had better create a big-enough memory instance:

const arraySize = (width * height * 4) >>> 0;
const nPages = ((arraySize + 0xffff) & ~0xffff) >>> 16;
const memory = new WebAssembly.Memory({ initial: nPages });

Here comes the catch. To be able to import memory into a Wasm module, we have to compile our AssemblyScript code with the --importMemory flag:

$ npm run asbuild:optimized -- --importMemory

This will generate the following line in the compiled Wasm module:

(import "env" "memory" (memory $0 1))

Now, we can import our memory object from JavaScript into Wasm:

WebAssembly
  .instantiateStreaming(fetch('grayscale.wasm'), {
    env:{ memory }
  })

The next step is to initialize the memory with the image data:

const bytes = new Uint8ClampedArray(memory.buffer);

for (let i = 0; i < data.length; i++)
  bytes[i] = data[i];

As you might have noticed, we use the same typed array — Uint8ClampedArray — as image data to format and access memory bytes.

When the memory bytes are filled, we can execute our Wasm function:

instance.exports
  .convertToGrayscale(width, height);

And store the result from memory back into the image:

for (let i = 0; i < data.length; i++)
  data[i] = bytes[i];

ctx.putImageData(imageData, 0, 0);

Image Manipulation

Now, when we have all we need to run an image-manipulation function in a browser, we shall actually write one. As previously mentioned, we will create a function that converts an image to grayscale:

export function convertToGrayscale(
    width: i32, height: i32): void {

  const len = width * height * 4;

  for (let i = 0; i < len; i += 4 /*rgba*/) {
    const r = load<u8>(i);
    const g = load<u8>(i + 1);
    const b = load<u8>(i + 2);

    const gray = u8(
      r * 0.2126 + g * 0.7152 + b * 0.0722);

    store<u8>(i,     gray);
    store<u8>(i + 1, gray);
    store<u8>(i + 2, gray);
  }
}

As the image data contains linear RGBA quadruples of 8-bit signed integers (clamped to the range of values from 0 to 255), we iterate the array of data in 4-incremental steps.

We load the RGB values from memory, compute the gray color, and store the color values back into the memory.

Demo Time

That’s it! Now we have all the know-how needed to develop awesome image processing Wasm functions in AssemblyScript.

Here are some more examples:

• original
• grayscale
• monochrome
• randomized monochrome
• color inversion

You can find the demo source code on my GitHub.