Hugging Face + Visual Blocks Custom Components

Visual blocks is an amazing tool from our friends at Google that allows you to easily create and experiment with machine learning pipelines using a visual interface. This repository contains the source code for custom components that allow you to use Hugging Face client and server models in your Visual Blocks pipelines. We've created a few nodes supporting different tasks and models following our Tasks definitions.

Visual Blocks seems to be mostly working in Chrome. If you are having trouble with the interface, try using Chrome, and please submit an issue to the Visual Blocks team.

Important links:

• https://visualblocks.withgoogle.com
• https://github.com/huggingface/visual-blocks-custom-components
• https://www.npmjs.com/package/huggingface-visualblocks-nodes
• https://cdn.jsdelivr.net/npm/huggingface-visualblocks-nodes@latest

Table of Contents

• How to use the custom components
• Nodes and Examples
  • Client Nodes
    • Translation
    • Token Classification
    • Text Classification
    • Object Detection
    • Image Segmentation
    • Image Classification
    • Depth Estimation
    • Background Removal
  • Server Nodes
    • Text Generation and Chat Completion
    • Fill Mask
    • Image Classification
    • Summarization
    • Text Classification
    • Text Generation
    • Text to Image
    • Token Classification
  • Extra Examples
• Local Development

How to use the custom components

To start playing with our custom components you need to Add a custom node to your Visual Blocks project. First you need to start a new project https://visualblocks.withgoogle.com/#/edit/new, then click on the "+" button in the bottom left corner to add a new node.

Then input the pre-bundled code from our npm package. You can do this by pasting the following link into the input field and clicking "Submit":

https://cdn.jsdelivr.net/npm/huggingface-visualblocks-nodes@latest

Then you will be able to see three Hugging Face Collections: Client, Server and Common.

Nodes and Examples

Client Nodes

Client nodes are nodes running tranformers pipelines on the client side using Transformers.js. All Client nodes have WASM and WebGPU (experimental) backend support, and you can find web-compatible models by visiting https://huggingface.co/models?library=transformers.js.

WebGPU support in transformers.js is still experimental and may not work on all devices. Not all models are supported by WebGPU backend yet.

Translation

Translation Node Example

More info:

• https://huggingface.co/tasks/translation
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TranslationPipeline

Token Classification

Token Classification Node Example

More info:

• https://huggingface.co/tasks/token-classification
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TokenClassificationPipeline

Text Classification

Text Classification Node Example

More info:

• https://huggingface.co/tasks/text-classification
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.TextClassificationPipeline

Object Detection

Object Detection Node Example

More info:

• https://huggingface.co/tasks/object-detection
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.ObjectDetectionPipeline

Image Segmentation

Image Segmentation Node Example

More info:

• https://huggingface.co/tasks/image-segmentation
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.ImageSegmentationPipeline

Image Classification

Image Classification Node Example

More info:

• https://huggingface.co/tasks/image-classification
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.ImageClassificationPipeline

Depth Estimation

Depth Estimation Node Example

More info:

• https://huggingface.co/tasks/depth-estimation
• https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.DepthEstimationPipeline

Background Removal

Background Removal Node Example

Server Nodes

Server nodes are nodes running Transformers pipeline tasks using the Hugging Face Serverless API. For a few selected LLM models, it's running using our hosted Text Generation Inference, our fast, optimized inference for LLMs.

You can use the Hugging Face Serverless API for free with limited usage, after which you'll be rate limited. If you need more usage, you can create an account at https://huggingface.co/join and get an API token at https://huggingface.co/settings/tokens or log in using the Hugging Face Login node.

For server nodes you have the option to Login using your Hugging Face account to get more usage and access to private models. Using Hugging Face Hub Login

If successful, you can obtain your Apikey directly from the Hugging Face Hub Login node.

Text Generation and Chat Completion

Chat Template Text Generation Node Example

Chat Completion Node Example

More info:

• https://huggingface.co/tasks/text-generation

Fill Mask

Fill Mask Node Example

More info:

• https://huggingface.co/tasks/fill-mask

Image Classification

Image Classification Node Example

More info:

• https://huggingface.co/tasks/image-classification

Summarization

Summarization Node Example

More info:

• https://huggingface.co/tasks/summarization

Text Classification

Text Classification Node Example

More info:

• https://huggingface.co/tasks/text-classification

Text Generation

Text Generation Node Example

More info:

• https://huggingface.co/tasks/text-generation

Text to Image

Text to Image Node Example

More info:

• https://huggingface.co/tasks/text-to-image

Token Classification

Token Classification Node Example

More info:

• https://huggingface.co/tasks/token-classification

Extra Examples

Background Removal Text to Image

Chat Completion Text to Image Depth

Image Segmentation Webcam Client

Local Development

1. Clone the repository

   git clone https://github.com/huggingface/visual-blocks-custom-components.git
   cd visual-blocks-custom-components
   

2. Install the dependencies

   npm i
   

3. Run the development server

   npm run dev
   

4. Visit Google's staging server

5. Click the + in the bottom left corner to add the custom nodes.

6. Paste in the link to the script (e.g., http://localhost:8080/index.js) and click "Submit".