Skip to main content

ComfyUI Guide

Jarvislabs.ai ComfyUI

ComfyUI is a no-code node-based GUI for stable diffusion. There are nodes for almost everything, in case you don't find the node for your need you can always club together different nodes. Let's say you want to remove the background of an image, you would use rembg custom nodes. Want to mimic a pose from an image? Use ControlNet nodes. There are nodes for any scenario that you can think of.

ComfyUI offers a high level of abstraction that lets you mix and match these nodes. Although this means there is a bit of a learning curve with ComfyUI, once you get past it, ComfyUI is the best tool for stable diffusion.

comfyui_image

Table of Contents​

You can launch ComfyUI on JarvisLabs by clicking the Run on cloud button on the top right.

Click on the API button to access the ComfyUI. comfyui_image

Topics​

In this article, we will see,

  • How to add models to ComfyUI
  • Installing custom nodes manually
  • Adding custom nodes using ComfyUI Manager
  • Setting up ComfyUI API server
  • How to Launch/Modify the parameters used to launch

Adding models to ComfyUI​

To add models to ComfyUI, follow these steps:

  1. Download Models: Choose your preferred source, such as Huggingface or Civitai, which offer numerous stable diffusion checkpoints.

  2. Place Checkpoints: If you're downloading a checkpoint, navigate to the following directory:

    cd ComfyUI/models/checkpoints

  3. Place loRAs: For loRAs, use this directory:

    cd ComfyUI/models/loras

  4. Download Command: Use the wget command to download your model. Replace the URL with your model's URL:

    wget --content-disposition https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0_0.9vae.safetensors?download=true

Note: Ensure each model is placed in its respective directory.

Watch this video tutorial for more details on obtaining download links.

Jarvislabs' ComfyUI instance comes preloaded with over 50 models, allowing you to focus on creativity without the hassle of downloading.

Adding custom nodes manually​

You can also download the custom nodes manually but you need to find the GitHub repo and follow the instructions in their readme file.

Here is an example of how to download a custom node called "ComfyUI_IPAdapter_plus" manually. Follow the below steps:

cd ComfyUI/custom_nodes
git clone https://github.com/cubiq/ComfyUI_IPAdapter_plus.git

If the repo contains a requirements.txt file. Then you need to run the pip command to install the dependencies.

cd  ComfyUI_IPAdapter_plus
pip install -r requirements.txt

If it doesn't contain a requirements.txt file. Make sure to read the readme file and check for manual installation.

ComfyUI manager​

ComfyUI manager is an extension created to manage ComfyUI. It comes with tools to easily add, delete, turn off, or activate different custom nodes in ComfyUI. Additionally, the extension includes a central hub and user-friendly features for accessing a variety of information within ComfyUI.

You can use ComfyUI manager to do the below tasks​

  • Update ComfyUI or any nodes in a few clicks.
  • Install missing custom nodes in your imported workflows.
  • Browse and install custom nodes.
  • Browse and download models.
  • Restart comfyUI in one click
  • Save snapshots of your workflows.
  • Share your workflows on various platforms.

Adding custom nodes using ComfyUI Manager​

Installing custom nodes using comfyUI manager is pretty simple. Click the manager button and click the install custom nodes button. Then just enter the name of the custom node you want to install in the search bar. Click the install button. Once it is done make sure to click the restart button to restart the comfyUI server.

Watch the below video, on how to install custom nodes using the comfyUI manager

Setup ComfyUI API server​

You can set up a ComfyUI server on a ComfyUI instance. To get started you have to install the websocket-client.

pip install websocket-client

Once the websocket-client has been installed - Open the terminal and navigate to the comfyUI example scripts folder or paste the below command to find the basic websocket API scripts.

cd ComfyUI/script_examples

Server address​

The server address has to be changed respective to where the WebUI is hosted. If you are using comfyUI on a local machine then there's no need to change the server address otherwise you need to change the server address to the respective cloud provider url address.

If you want to run the websocket API from the terminal of a JarvisLabs' instance you have to set the server address as below:

server_address = "0.0.0.0:6006"

If you want to access the websocket API of your JarvisLabs' instance from a local machine or another device you can simply copy the "API" link address of the comfyUI instance from the instances page.

Jarvislabs.ai copy api link

Example

server_address = "https://******.notebooksll.jarvislabs.net/"

Saving the workflow in API format​

Change the prompt variable which contains the workflow in api format as a string. You can save your workflow in API format. First, you need to enable the dev setting. Follow the below steps:

Jarvislabs.ai dev settings After you can see the Save the workflow as API button, you can use that to download the indexed workflow.

Replace the indexed workflow with the prompt text variable in the websocket script file to infer your desired workflow.

Running the script​

You can use the below command to use the script:

python websockets_api_example.py

How to Launch/Modify the parameters used to launch​

When the instance is started, we start the main.py in the background, so to kill the process, open a terminal in the jupyterLab notebook.

In the terminal run the below command:

ps aux

Hopefully, you will see the python main.py running in the 6006 port, Copy the kill command below and paste it into the terminal:

kill -9 <PID>

To restart the main.py paste the below command:

python ComfyUI/main.py --port 6006 --listen 0.0.0.0

FAQs​

What is ComfyUI?

ComfyUI makes Stable Diffusion super easy to use, even for people who aren't tech experts! It's like building with blocks: you just drag and drop things to create your artwork, no coding needed. It's got tons of cool features and now works with ControlNets too, so you can make even more amazing stuff!

How to see the comfyUI logs?

Open the jupyterlab notebook and open a new terminal. Check this documentation how to open a new terminal. Run this command to see the comfyUI logs:

tail -f comfy.log
Why am I getting a "Bad Gateway" error after launching a ComfyUI instance?

At Jarvislabs, each ComfyUI instance comes preloaded with over 50 custom nodes by default. The first time you launch the instance, it may take a minute or two to fully load all these custom nodes. During this time, you might encounter a "Bad Gateway" error.

To troubleshoot this:​

  • Check the Logs: Access the instance logs to confirm whether the ComfyUI server is up and running. Use the following command to monitor the logs in real-time.

    tail -f comfy.log

  • Verify the Server Process: Ensure that the ComfyUI server is running in the background by checking for the main.py. You can do this by running:

    ps aux

If you don't see main.py in the list of processes, it's likely that the ComfyUI server has crashed. In this case, you will need to relaunch the instance.

What should I do if the ComfyUI server crashes in the middle of a process?

This can happen due to various reasons. The custom node you're using or you're running out of low storage or etc. But most commonly due to CUDA out of memory. All you need to do is open the terminal and check if the comfyUI server is running using the "ps aux" command; if not, then relaunch the comfyUI server and monitor the CUDA memory while your process is running. Use this command to see the real-time CUDA memory:

watch nvidia-smi

If it's crashed again. Then to solve this issue, choose a GPU with a bigger VRAM.

Why is downloading a model taking so long?

Slow download speeds can occur for several reasons:

  • Host Website Limitations: Download speed also depends on the website you're downloading from. Some websites may have speed limitations or heavy traffic that can slow down your download

  • Recommended Sources: For better performance, it's recommended to download models from reliable sources like Hugging Face or Civitai. However, even on these platforms, downloads may occasionally be slow.

  • Troubleshooting: If your download is particularly slow, try canceling the process and starting it again. This can sometimes resolve temporary speed issues.

Why can't I open the checkpoint folder in JupyterLab?

The checkpoint folder is too large to open directly in JupyterLab, causing its normal functionality to fail but follow the below steps to access the folder:

Use the Terminal: Instead of using the JupyterLab interface, open the terminal within JupyterLab. Using the "cd" command to navigate to the checkpoint folder.

cd /home/ComfyUI/models/checkpoints
Why can't I download models inside the sd1.5 or sdxl folders?

This folders are created on our side to provide default models for every user. Because these folders are managed by us, you won't be able to download additional models directly into them. But you can download the checkpoint models inside the checkpoint folder.

Can I share my workflows with others?

Yes, you can share your workflows with others. Just click the save button on the toolbar, and a JSON file will be saved to your local PC.

What hardware requirements does ComfyUI have?

There are two main ways to run ComfyUI: on your own PC or using an online GPU instance. The requirements vary slightly for each method.

On Your Own PC:​

Hardware​

  • GPU: A decent NVIDIA GPU with at least 8GB of VRAM (e.g., RTX 3060 Ti or higher)
  • CPU: A modern processor (i5 or Ryzen 5 or higher)
  • RAM: 16GB or more

Software​

  • Operating System: Windows 10/11 or Linux
  • Python: Python 3.10 with libraries like PyTorch and CUDA Toolkit pre-installed
  • Command Line: Basic comfort with command prompts

Additional​

  • Download files for the Stable Diffusion model (e.g., 1.5 or 2.1) and ComfyUI itself
  • Sufficient storage space on your PC for models and generated images

Using an Online GPU Instance:​

Hardware​

  • No powerful PC needed; processing happens on cloud servers

Software​

  • Web Browser: Any modern browser like Chrome, Firefox, Edge, etc.
  • Internet Connection: Stable and high-speed for smooth processing and file transfers
  • Cloud Account: Sign up for a service providing access to powerful GPUs for a fee (e.g., Jarvislabs, Google Colab, Paperspace)

Choose the option that best suits your budget, technical skills, and available hardware.

How do I use ControlNets in ComfyUI?

ComfyUI has a dedicated node for ControlNets. You can simply add this node to your workflow and connect it to other nodes to influence the image generation process.