Known errors

Check if this is a known issue before trying your own fix. Entries are keyed by the substring that actually appears in runtime.errorMessage, the build log, or a Python traceback — grep this file for the error you saw.

If you hit something not listed here and figure out a fix, please ask your human to PR it back so future runs benefit.

Build / config errors

These come from the Space build pipeline before the app starts. Read with hf spaces logs <id> --build --tail 500 — find the first error, not the last.

CONFIG_ERROR: torch version in requirements.txt is not compatible with ZeroGPU

Cause: requirements.txt pins torch==X.Y.Z to a version outside the supported set (2.8.0, 2.9.1, 2.10.0, 2.11.0). Fix: Unpin torch (preferred — the runtime preinstalls the latest supported version), or pin to one of the supported values.

Cannot install … because these package versions have conflicting dependencies / ResolutionImpossible

Cause: A dep conflicts with the Gradio SDK pinned by sdk_version: in README. Most commonly pydantic, uvicorn, huggingface_hub, or jinja2 pinned to old values that the SDK no longer accepts. Fix: Unpin the offender. For gradio[mcp] specifically, uvicorn>=0.31.1 and pydantic>=2.11.10 are required.

Build hangs in dependency resolution > 10 min

Cause: pip backtracking through a deep version space. Fix: Pin the conflicting transitive dep. The --build logs will show which one. Bump startup_duration_timeout: 1h in README frontmatter if heavy downloads are expected.

ModuleNotFoundError: No module named 'pkg_resources'

Cause: setuptools 81 dropped pkg_resources; an old package's setup.py imports it. Fix: Bump or unpin the offender. Typical culprits: deepspeed==0.15.x (training-only — usually safe to drop from inference Spaces), openai-whisper==20231117.

400 Bad Request from /api/validate-yaml during create_repo / upload_file

Cause: README frontmatter failed server validation. Most common: short_description over the (undocumented) character cap — target ≤ 60. Fix: Shorten short_description. Long descriptions go in the README body. Also double-check colorFrom/colorTo are one of red|yellow|green|blue|indigo|purple|pink|gray.

403 Forbidden from create_repo for a Gradio / Docker Space

Cause: Gradio and Docker Spaces run on compute and need a paid plan (PRO / Team / Enterprise). The only free exception is ZeroGPU — 2 Spaces for personal accounts in good standing (verified email, account older than 30 days). Fix: On a free account, pass space_hardware="zero-a10g" rather than omitting it — cpu-basic is gated too, so dropping the flavor makes this worse, not better. If ZeroGPU is also rejected, the account is over its 2-Space cap or not in good standing: delete an unused ZeroGPU Space, upgrade to PRO, or ship a Static Space and apply for a community grant. Keep hardware: out of README frontmatter (silently ignored anyway).

403 Forbidden from create_commit(..., create_pr=True)

Cause: Upstream Space has Discussions disabled. Fix: Ask the maintainer to enable Discussions, or push directly if you have write access.

Startup / RUNTIME_ERROR

These come from hf spaces logs <id> --tail 500.

RuntimeError: CUDA has been initialized before importing the spaces package

Cause: Something triggered CUDA init in the main process before import spaces. Usually wrong import order; sometimes a third-party lib eagerly initializing CUDA at import time (e.g. numba.cuda). Fix: Reorder so import spaces is first. For numba-using stacks (NeMo, RAPIDS bits):

import os
os.environ.setdefault("NUMBA_DISABLE_CUDA", "1")
import spaces

RuntimeError: No @spaces.GPU function detected during startup

Cause: The function bound to .click(fn=...) / .submit(...) isn't decorated. Decorating an inner helper doesn't count — the startup scan only walks Gradio's registered handlers. Fix: Decorate the function Gradio binds. If that conflicts with another decorator, wrap explicitly:

@spaces.GPU(duration=60)
def gpu_inner(...): ...
def gradio_handler(...): return gpu_inner(...)

(Or just decorate gradio_handler directly.)

ImportError: cannot import name 'HfFolder' from 'huggingface_hub'

Cause: Old gradio (4.44 and similar) imports HfFolder from huggingface_hub, which was removed in recent hub releases. Fix: Two options.

• Pin huggingface-hub==0.25.0 in requirements.txt (keeps old gradio happy).
• Bump sdk_version in README to 5.x or 6.x (also fixes a lot of other API breaks). If a Gradio custom component locks the major (gradio-image-prompter, gradio_litmodel3d, …), install it with --no-deps so its gradio<5.0 requirement doesn't bind.

ImportError: cannot import name 'is_traceable_wrapper_subclass' from 'torch.utils._python_dispatch'

Cause: A dep with torchaudio<2.1 / torch<2 in its setup.py (e.g. demucs, audiocraft) downgraded torch silently. The build succeeded, the app booted, and import spaces then died on a missing torch symbol. Fix: Install the offender from app.py with --no-deps before import spaces:

import subprocess, sys
subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps",
                "git+https://github.com/facebookresearch/demucs"], check=True)
import spaces

List its actual runtime deps (dora-search einops julius lameenc openunmix pyyaml tqdm for demucs) yourself in requirements.txt.

_pickle.UnpicklingError: Weights only load failed

Cause: torch.load weights-only default flipped to True in torch 2.6. Old checkpoints pickling numpy/object globals fail. Fix: For trusted upstream checkpoints, monkey-patch before the package import:

import torch
_orig = torch.load
torch.load = lambda *a, **k: _orig(*a, **{**k, "weights_only": k.get("weights_only", False)})

Stuck at ZeroGPU init – 10.0% then 60 s timeout

Cause: A library called cuInit in the parent process, poisoning the fork (most often numba.cuda via NeMo). The actual @spaces.GPU body never starts. Fix: os.environ.setdefault("NUMBA_DISABLE_CUDA", "1") as the first line of app.py, before import spaces.

RUNTIME_ERROR right after long APP_STARTING, logs sparse

Cause: Boot exceeded startup_duration_timeout (default 30 min). Big-model loads commonly trigger this. Fix: Bump startup_duration_timeout: 1h in README frontmatter. For Gradio 6 specifically, also set GRADIO_SSR_MODE=false via hf spaces variables add <id> --env GRADIO_SSR_MODE=false to dodge SSR health-check timeouts during slow boot.

RUNNING but the public URL returns 404

Cause: The Space is private. Anonymous Client / browser hits return 404. Fix: Authenticate. gradio_client.Client(space, token=os.environ["HF_TOKEN"]). The kwarg is token=, not hf_token=.

workload was not healthy after 30 min

Cause: Infra-side scheduling or a build that genuinely can't finish in time. Fix: Usually not actionable in code. Bump startup_duration_timeout if heavy downloads are expected; otherwise wait or report.

Exit code 128 / containerd / scheduling failure

Cause: HF infra glitch. Fix: hf spaces restart <id> --factory-reboot. If it persists, retry later or report. Not fixable in code.

Inference-time errors

These appear in hf spaces logs <id> --follow while a request is running.

ZeroGPU illegal duration

Cause: @spaces.GPU(duration=N) is larger than the visitor's tier per-call cap. Fix: Lower N. Tier caps live in the ZeroGPU docs.

ZeroGPU quota exceeded (X requested vs Y left)

Cause: The visitor's remaining quota < requested duration. The comparison is requested vs remaining, not actual vs remaining — a 10-second task left at the default 60 s blocks the user as soon as their remaining drops below 60 s. Fix: Lower duration to the realistic worst case. For input-dependent runtime, use a callable estimator.

RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at .../CUDACachingAllocator.cpp

Cause: Allocator fragmentation under transient memory spikes (high-res pixel-space ops, large attention activations, SR models, video DiTs). Not a clean OOM. Fix: Set expandable segments at the very top of app.py, before any torch import:

import os
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
import spaces
import torch

Usually a single-line fix that replaces lowering resolution or moving to xlarge.

Call hangs forever on first @spaces.GPU entry

Cause: The decorated function returned a CUDA tensor. Unpickling it in the main process triggers torch.cuda._lazy_init(), which ZeroGPU blocks. Fix: Convert to CPU before returning: return tensor.cpu() or .cpu().numpy(). For gr.State, scrub before yielding.

PicklingError at call entry

Cause: An argument crossing the fork boundary contains an unpicklable object — file handle, lock, lambda, closure, or gr.SelectData. Fix: Extract the picklable fields in a thin un-decorated wrapper, pass plain values to the @spaces.GPU function. For gr.SelectData specifically, pull out evt.index[0], evt.index[1] etc. outside the decorator.

RecursionError inside gr.SelectData.__getattr__

Cause: Same as above — gr.SelectData doesn't survive pickle. Fix: Same — extract its fields before crossing the boundary.

CUDA error … flash_fwd_launch_template.h: no kernel image is available for execution on the device (or :188: invalid argument)

Cause: A Flash Attention 3 kernel was loaded — directly via kernels-community/{flash-attn3,vllm-flash-attn3,sgl-flash-attn3}, or indirectly via an old xformers wheel that auto-dispatched to FA3. FA3 has no Blackwell sm_120 build. Fix: Use attn_implementation="sdpa", or "flash_attention_2" with the FA2 wheel from multimodalart/zerogpu-blackwell-wheels. For xformers, the prebuilt wheel from the same dataset auto-dispatches to FA2 — no monkey-patch.

NotImplementedError: sgl_flash_attn3 is only supported on sm80 and above with CUDA >= 12.3

Cause: kernels-community/sgl-flash-attn3 rejects sm_120 at runtime despite the error wording. Same root cause as the FA3 entry above. Fix: Same — SDPA or FA2.

ImportError: cannot import name 'flash_attn_varlen_func' from 'flash_attn' / model insists on attn_implementation="flash_attention_2"

Cause: The model imports flash_attn at module top with no escape hatch, and the runtime doesn't ship it. Fix: Install the prebuilt flash_attn-2.8.3-cp310-cp310-linux_x86_64.whl from multimodalart/zerogpu-blackwell-wheels. Requires python_version: "3.10" in README (wheel is cp310 only). Real flash_attn_2_cuda satisfies xformers' flash_attn_gpu probe too.

For transformers AutoModel-style configs that aren't a hard import, swap attn_implementation="flash_attention_2" → "sdpa". Torch-native, zero deps.

selective_scan_cuda.so undefined symbol / _torchaudio.abi3.so undefined symbol

Cause: A direct-URL prebuilt CUDA wheel pinned to an old torch ABI (cu12torch2.4cxx11abiFALSE, etc.) — won't load on the current runtime. Fix: Drop the URL-pinned wheel from requirements.txt. Use a torch-current wheel from multimodalart/zerogpu-blackwell-wheels, kernels-community, or upstream's release page.

TypeError: 'dict' object is not hashable inside jinja2/utils.py:get

Cause: Old gradio (4.44) + modern starlette / jinja2 cache clash. Fix: Either pin jinja2<3.2 + starlette<0.40 (keeps old gradio), or bump sdk_version past 5.0.

Smoke-test / client errors

Client.__init__() got an unexpected keyword argument 'hf_token'

Cause: Older gradio_client API used hf_token=; current uses token=. Fix: Client(space, token=os.environ["HF_TOKEN"]).

httpx.ReadTimeout on client.predict(...)

Cause: Default timeout too small for the GPU duration. Fix: Client(..., httpx_kwargs={"timeout": 600}). Set this to at least your @spaces.GPU(duration=N) plus 60 s.

404 on what looks like a valid endpoint name

Cause: Streaming endpoint (function uses yield), or a custom gr.Server @app.get/post route that doesn't appear in view_api(). Fix: For streaming, use client.submit(...).result() (or iterate the job). For custom routes, use httpx.post(base_url + "/route", ...) directly — bypass gradio_client.

Result file looks empty / dimensions wrong

Cause: HTTP 200 ≠ correct output. The model snapped to a different size, or the wrong endpoint was hit. Fix: Sniff the returned file's magic bytes (glTF, \x89PNG, RIFF…WEBP, RIFF…WAVE, [4:8]==b"ftyp") and check returned dimensions match what was requested.

Submitting new entries

If you hit an error not in this file and figure out the fix, please ask your human to PR it back to this skill. Format:

• A 1-line heading with the exact error substring an agent would grep for.
• Cause: one sentence on what triggered it.
• Fix: concrete commands or code. If the fix needs more than 5 lines of narrative, point to another reference file (e.g. debugging.md, zerogpu.md) for depth.