Unable to provide valid hw_frames_ctx for CUDA filter chain in JavaCV GPU pipeline

[qiuweichang]

Issue Description

Hello, I'm trying to implement a complete GPU pipeline using JavaCV and encountering an issue with hardware frame context initialization.

Goal

I want to implement a fully GPU-accelerated pipeline for RTSP stream processing:

1. GPU decoding: Using GPU decoders (h264_cuvid, hevc_cuvid)
2. GPU memory retention: Setting PixelFormat = AV_PIX_FMT_CUDA to keep frames in GPU memory
3. Hardware frame upload: Using hwupload_cuda to convert GPU-decoded frames to hardware frames (hw_frames) for CUDA filters
4. Scaling and color conversion: Using scale_cuda CUDA video filter for resizing and RGB conversion
5. GPU encoding: Using NVENC hardware encoder to convert RGB to JPEG

Problem

I'm getting the following error during FFmpegFrameFilter initialization:

Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL!
Query format failed for 'in': Invalid parameter
avfilter_graph_config() error -22

Current Code Approach

java
// GPU decoder initialization
WeakReference weakRecorder = new WeakReference<>(grabber);
grabber = new FFmpegFrameGrabber(config.getRtspUrl());
grabber.setOption("rtsp_transport", "tcp");
grabber.setVideoOption("gpu", "0");
grabber.setVideoCodecName("h264_cuvid");
grabber.setPixelFormat(avutil.AV_PIX_FMT_CUDA);
grabber.setVideoOption("surfaces", "64");
grabber.setVideoOption("extra_hw_frames", "5");

// hwaccel config does not work
#grabber.setVideoOption("hwaccel", "cuda");
#grabber.setVideoOption("hwaccel_device", String.valueOf(gpu));

// GPU filter chain
// String gpuFilter = "scale_cuda=format=rgb24"; // Also reported same error
// String gpuFilter = "vpp_cuda=format=rgb24"; // Also reported same error
String gpuFilter = "hwupload_cuda,scale_cuda=format=rgb24";
filter = new FFmpegFrameFilter(gpuFilter, width, height);
filter.setPixelFormat(avutil.AV_PIX_FMT_CUDA);
filter.start(); // Fails here
avutil.av_log_set_level(avutil.AV_LOG_ERROR);

Error log

[in @ 0x7f484fe39f40] Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL!
[in @ 0x7f484fe39f40] Query format failed for 'in': Invalid parameter
Info: avformat_open_input rejected some options: Option: pixel_format, value: bgr24
Input #0, rtsp, from 'rtsp://192.168.96.136:30242/aisp-data-analysis-center/1916462708309229570':
Metadata:
title : Streamed by ZLMediaKit(git hash:8bf48ed/2024-12-15T11:43:31+08:00,branch:master,build time:2024-12-15T03:44:48)
Duration: N/A, start: 0.023750, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 2560x1440, 25 fps, 25 tbr, 90k tbn
Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s
[h264_cuvid @ 0x7f484fe50780] Invalid pkt_timebase, passing timestamps as-is.
[in @ 0x7f47ec6ac800] Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL!
[in @ 0x7f47ec6ac800] Query format failed for 'in': Invalid parameter
2025-10-15 16:56:30.049 [scheduling-1] INFO c.b.v.c.capture.executor.RtspCaptureExecutor - set h264_cuvid success
2025-10-15 16:56:30.051 [scheduling-1] ERROR c.b.v.c.capture.executor.RtspCaptureExecutor - error:avfilter_graph_config() error -22 (For more details, make sure FFmpegLogCallback.set() has been called.)
org.bytedeco.javacv.FFmpegFrameFilter$Exception: avfilter_graph_config() error -22 (For more details, make sure FFmpegLogCallback.set() has been called.)
at org.bytedeco.javacv.FFmpegFrameFilter.startVideoUnsafe(FFmpegFrameFilter.java:425)
at org.bytedeco.javacv.FFmpegFrameFilter.startUnsafe(FFmpegFrameFilter.java:321)
at org.bytedeco.javacv.FFmpegFrameFilter.start(FFmpegFrameFilter.java:289)

Specific Issue

I learned that. the scale_cuda filter requires hardware frames (hw_frames), but I don't know how to properly provide the CUDA hardware frame context (hw_frames_ctx). The hwupload_cuda filter expects the input to have a valid hardware frame context, which doesn't seem to be automatically set when using GPU decoding with JavaCV.

Questions

1. How can I properly initialize/provide the CUDA hardware frame context (hw_frames_ctx) for the filter chain?
2. Is there a specific sequence or additional configuration needed to make hwupload_cuda work with GPU-decoded frames?
3. Are there alternative approaches to achieve the same goal (full GPU pipeline with color output) while avoiding this hardware frame context issue?

Environment

• JavaCV version: 1.5.12
• GPU: NVIDIA GeForce RTX 2080
• CUDA: 12.2
• OS: Linux
• java：21

gradle：
implementation "org.bytedeco:opencv:4.11.0-1.5.12:linux-x86_64-gpu"
implementation "org.bytedeco:cuda:12.6-9.5-1.5.11:linux-x86_64-redist"
implementation "org.bytedeco:ffmpeg:7.1.1-1.5.12:linux-x86_64-gpl"
implementation "org.bytedeco:openblas:0.3.30-1.5.12:linux-x86_64"
// javacv
implementation "org.bytedeco:javacv:1.5.12"
implementation "org.bytedeco:javacpp:1.5.12:linux-x86_64"
implementation "org.bytedeco:ffmpeg:7.1.1-1.5.12:linux-x86_64"

Any guidance or working examples for implementing a complete GPU pipeline with JavaCV would be greatly appreciated!

[marktech0813]

Hi, Can I handle this issue now?

[marktech0813]

Please check my PR.
thanks.

[qiuweichang]

Thanks for the PR! I'll review and test it as soon as possible.

[qiuweichang]

The problem is solved. Thank you for your handling

[marktech0813]

you're welcome, PR is merged?

[saudet]

@marktech0813 Users can call FFmpegFrameFilter.setPixelFormat(AV_PIX_FMT_NV12) if that's what they need. I don't see how your modifications are useful