Get Started with Transformers.js in p5.js

How to Integrate Transformers.js with p5.js for Machine Learning in the Browser

This tutorial will guide you through the initial steps of integrating the powerful Transformers.js library into your p5.js sketches. You’ll learn what Transformers.js is, how it relates to Hugging Face, and how to set it up within the p5.js environment to start experimenting with machine learning models directly in your browser.

What You’ll Learn:

• Understanding the role of Hugging Face and Transformers.js in the machine learning landscape.
• Setting up Transformers.js in a p5.js sketch, including handling JavaScript modules and asynchronous operations.
• Performing a basic ‘Hello World’ demonstration using the pipeline API for tasks like sentiment analysis.
• Exploring the potential applications of Transformers.js within creative coding projects.

Prerequisites:

• Basic understanding of JavaScript and p5.js.
• Familiarity with the p5.js web editor.
• A modern web browser.

Step 1: Understanding Transformers.js and Hugging Face

Before diving into the code, it’s essential to understand the ecosystem. Hugging Face is a popular online community and platform, often described as the ‘GitHub for machine learning models.’ It hosts a vast collection of models, datasets, and ‘Spaces’ – live web applications demonstrating model features.

Transformers.js is a JavaScript library that brings the power of Hugging Face’s Transformers models to the browser. Unlike many cloud-based AI services, Transformers.js allows models to run directly on your local machine, ensuring privacy and reducing reliance on external servers. This makes it an excellent choice for creative coding projects where you want to integrate AI capabilities without complex backend setups.

Step 2: Setting Up Your p5.js Sketch for ES6 Modules

Transformers.js utilizes modern JavaScript features, specifically ES6 modules and asynchronous operations (async/await). To use these, your p5.js sketch needs to be configured correctly.

Updating p5.js to Version 2.0:

The p5.js web editor has been updated to support these features. If you haven’t already, ensure you are using p5.js version 2.0 or later. You can typically do this by clicking on the version indicator in the top right corner of the p5.js web editor and selecting a 2.0 version.

Handling Imports:

Traditionally, p5.js sketches load libraries via <script> tags in the index.html file. However, Transformers.js uses the import keyword, which is part of ES6 module syntax. You cannot directly use import in the standard p5.js global mode.

To bridge this gap, we’ll use a function-based import mechanism that’s compatible with the p5.js environment. Instead of:

import { pipeline } from '@xenova/transformers';

We will use:

let pipeline;

async function setup() {
  pipeline = await import(
    'https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/transformers.min.js'
  );
  // ... rest of your setup code
}

Expert Note: The URL points to a specific version of the Transformers.js library hosted on a CDN. Using a specific version (e.g., @2.17.3) ensures consistency. If you omit the version, it will default to the latest.

Step 3: Understanding Async/Await

Loading machine learning models can take time. Transformers.js uses async and await to handle these operations without blocking the main thread.

• async: Declares a function as asynchronous, meaning it can perform operations that take time and will not immediately return a value.
• await: Pauses the execution of an async function until a Promise is resolved (e.g., until a model is fully loaded).

When you call await import(...), the code waits until the library is downloaded and ready before proceeding. This is why the import call must be inside an async function, such as setup() in p5.js.

Step 4: Using the Pipeline API for a “Hello World” Example

The core of Transformers.js is the pipeline API, which simplifies using pre-trained models for various tasks. A pipeline abstracts away the complexities of model loading, preprocessing, inference, and postprocessing.

Sentiment Analysis Example:

Let’s create a simple sentiment analysis example. This task takes a piece of text and determines whether it expresses a positive, negative, or neutral sentiment.

1. Initialize the Pipeline: In your p5.js sketch, within the setup() function (which must be marked as async), initialize the pipeline for sentiment analysis.

let pipeline;
let sentimentResult;

async function setup() {
  // Use the await import to load the library
  const { pipeline } = await import(
    'https://cdn.jsdelivr.net/npm/@xenova/[email protected]/dist/transformers.min.js'
  );

  // Create a pipeline for sentiment analysis
  pipeline = await (await import('@xenova/transformers')).pipeline('sentiment-analysis');

  // Perform sentiment analysis on some text
  sentimentResult = await pipeline('I love using p5.js with AI!');

  console.log(sentimentResult);
  noCanvas(); // No need for a canvas for this text-based example
}

function draw() {
  // Nothing needed in draw for this example
}

Explanation:

• We first import the pipeline function.
• Then, we create an instance of the pipeline, specifying the task as 'sentiment-analysis'. This step downloads and loads the necessary model, which might take a moment the first time it runs.
• Finally, we pass a string of text to the initialized pipeline. The result, an object containing the predicted sentiment (e.g., ‘POSITIVE’ or ‘NEGATIVE’) and a score, is stored in sentimentResult and logged to the console.

Displaying the Result:

To see the result in the browser, you can modify the draw() function or use setup() to display the text:

function setup() {
  // ... (pipeline initialization as above)

  // Perform sentiment analysis
  sentimentResult = await pipeline('I love using p5.js with AI!');

  // Display the result on the webpage
  let resultText = document.createElement('p');
  resultText.textContent = `Sentiment: ${sentimentResult.label} (Score: ${sentimentResult.score.toFixed(2)})`;
  document.body.appendChild(resultText);

  noCanvas();
}

function draw() {
  // No drawing needed
}

Step 5: Exploring Further Possibilities

This sentiment analysis example is just the beginning. Transformers.js supports a wide range of tasks and models, including:

• Text Generation: Creating human-like text.
• Zero-Shot Classification: Classifying text into categories without prior training examples for those specific categories.
• Question Answering: Extracting answers from a given text based on a question.
• Image Classification: Identifying objects or scenes within images.
• Text-to-Image Generation: Creating images from textual descriptions (requires larger models and more resources).

By exploring the Hugging Face model hub and the Transformers.js documentation, you can discover numerous models and tasks to integrate into your p5.js projects, opening up exciting new avenues for creative expression and interaction.

Source: Introduction to Transformers.js (LIVE RECORDING SESSION ARCHIVE) (YouTube)