Run SmolVLM Locally on WebGPU for Radiological Analysis

Introduction

Multimodal vision-language models (VLMs) like SmolVLM are revolutionary for medical imaging, but querying cloud-based APIs with private patient radiographs raises severe privacy and HIPAA compliance concerns. Running models locally in the user's browser using Transformers.js v3 and the WebGPU API eliminates server costs and ensures 100% data privacy.

Architecture Overview

graph TD
    Image(Radiological Scan) --> Canvas[HTML Canvas Resizer]
    Canvas --> WebGPU_Tensors[Float32 Tensor Allocation]
    
    VLM_Loader[Transformers.js Loader] -.-> |Fetch Weight Shards| WebGPU_Device[WebGPU Context GPUBuffer]
    WebGPU_Tensors --> Model_Inference[SmolVLM WebGPU Kernel]
    WebGPU_Device --> Model_Inference
    
    Model_Inference --> Autoregressive_Decoder[WGSL Text Generation]
    Autoregressive_Decoder --> Clean_Report(Local Clinical Findings Report)
        

Prerequisites

• A WebGPU-compatible browser (e.g., Chrome 113+, Edge 113+, Opera, or Safari Technology Preview)
• Basic knowledge of JavaScript and asynchronous operations

Step 1: Initialize WebGPU Context

First, we need to request the WebGPU adapter and verify local hardware acceleration is available in the user's browser.

async function initWebGPU() {
    if (!navigator.gpu) {
        throw new Error("WebGPU is not supported in this browser. Falling back to CPU/WASM.");
    }
    const adapter = await navigator.gpu.requestAdapter();
    if (!adapter) {
        throw new Error("Failed to request GPU adapter.");
    }
    const device = await adapter.requestDevice();
    console.log("WebGPU context successfully initialized on local device:", adapter.name);
    return device;
}

Step 2: Import Transformers.js v3

Import the pipeline modules from the CDN distribution of Transformers.js v3, which introduces native WebGPU execution backends.

import { pipeline, env } from 'https://cdn.jsdelivr.net/npm/@xenova/transformers@3.0.0';

// Enable WebGPU execution globally in Transformers.js
env.backends.onnx.wasm.numThreads = 4;
env.backends.onnx.webgpu = true;

Step 3: Load SmolVLM & Vision Processor

Initialize the multimodal visual question answering pipeline. We target the quantized 4-bit ONNX weights of SmolVLM to keep download sizes lightweight (~1.2GB) and run inference smoothly on consumer GPUs.

async function loadSmolVLM() {
    console.log("Loading SmolVLM weights onto GPU memory...");
    const generator = await pipeline('visual-question-answering', 'Xenova/SmolVLM-Instruct-2B', {
        device: 'webgpu', // Directs execution to the WebGPU device
        dtype: 'fp32' // Or fp16/q4 depending on hardware support
    });
    console.log("SmolVLM is ready for local WebGPU inference.");
    return generator;
}

Step 4: Execute local Radiological Inference

We pass the uploaded image and a clinical prompt to the model to generate a report natively in-browser.

async function analyzeRadiograph(imageElement, prompt) {
    const model = await loadSmolVLM();
    
    const response = await model({
        image: imageElement,
        question: "Assess this chest radiograph. List cardiomediastinal contour, lung expansion, and any consolidation or pleural effusions."
    });
    
    console.log("Analysis Output:", response.answer);
    return response.answer;
}

Conclusion

Deploying SmolVLM locally on WebGPU allows developers to build high-performance, private, zero-latency radiological assistants directly in the browser, opening new avenues for medical imaging tools that run anywhere without expensive cloud GPUs.