nvidia nvmpi encoder for streamFX and obs-studio (e.g. for nvidia jetson. Requires nvmpi enabled ffmpeg / libavcodec)

nvmpi-streamFX-obs

nvidia nvmpi encoder for streamFX and obs-studio (e.g. for nvidia jetson. Requires nvmpi enabled ffmpeg / libavcodec)

Purpose

This is a documentation plus the source files I created / modified in order to build a version of obs-studio that uses the nvidia nvmpi hardware encoder available e.g. on the nvidia jetson series (at least on the jetson series, nvidia nvmpi replaces the well known nvenc hardware encoder). It relies on the integration of nvmpi into ffmpeg and on the obs-StreamFX plugin to make the nvmpi hardware encoder available in obs-studio.

To this end I had to add a handler for nvmpi to StreamFX which I provide here for the benefit of whoever wants to build nvmpi support into obs-StreamFX (it is essentially just a slightly modified copy of StreamFX's nvenc handler).

Limitations

This is not a maintained project, but documents the solution I built for me and am using as of February / March 2021 on nvidia jetson nano 4GB and Ubuntu 20.04 with the kind support of Xaymar. It is based on version 10.0.alpha1 of StreamFX. It can not become part of the StreamFX package as least as long there is no maintainance which I cannot provide due to lack of capacity and resources.

So as development of StreamFX, jetson-ffmpeg and obs-studio continue, you might be required to do modifications on your own. Also the nvmpi-into-ffmpeg integration is an ongoing effort where functional limitations may exist. Status: As of 2021-05-13, the files provided no longer compile with the newer version of StreamFX, see the issue "D_DESC was not declared", so you have to make said adaptions. I currently have no capacity to update them myself.

No jetson specific dependencies have come to my attention, so it would probably work in other environments with nvmpi encoder hardware where an integration into ffmpeg / libavcodec exists.

Documentation

Prerequisite Steps

Using the built instruction provided with the packages (plus the remarks below) you first need to download, build and deploy a working version of

Remarks on building jetson-ffmpeg:

  • I used GCC/G++ version 8 for the compile as the nvidia cuda 10.2 provided with the jetson requires GCC/G++ 8
  • I also found that the compiler flag -fPIC was required in the build configuration statement if on arm64/aarch64 Linux:
./configure --enable-nvmpi --extra-cxxflags=-fPIC --enable-shared
  • In order to avoid conflicts with the ffmpeg standard package that ubuntu provides I left the installation prefix /usr/local/ as the installation default. This will install libavcodec.so etc into /usr/local/lib. However if there are other libav* versions in the system (e.g. from a standard ffmpeg in /usr/lib) it is important to force the correct search order for these libraries e.g. by inserting /usr/local/bin at the beginning of the library search path LD_LIBRARY_PATH.
  • You should test this ffmpeg as described in the building instructions to verify that nvmpi encoders work properly

Remarks on building Browser source for obs-studio

  • Browser source is (of course) not required for hardware encoding in obs, so you can omit it. If you want to use it: the build instructions are not valid for ARM based devices like the jetson. For ARM64 devices like the jetson replace them by the following steps:
  1. The key thing is to download the correct "Chrome embedded Framework" for Linux-arm64 from CEF Builds. Make sure you chose the "Linux-ARM64" tab, and the 'minimal distribution' from the 'stable build' (so do not use the wget with download link provided by obs instructions, because that is for amd64 architecture - not helpful on an arm64 device)
  2. unpack it (using tar -xjf) to a directory of your choosing, e.g. ~/CEF-bin-min
  3. cd CEF-bin-min && mkdir build && cd build
  4. Do this cmake statement (see the explanation in CMakeLists.txt, but you need to add the architecture option):
cmake -G "Unix Makefiles" -DPROJECT_ARCH="arm64" -DCMAKE_BUILD_TYPE=Release
  1. make -j3 cefsimple

This should give you a Chrome embedded Framework ready for integration into obs-studio in ~/CEF-bin-min .

Remarks on building obs-studio with Browser source and StreamFX plugin:

  • I used StreamFX as a frontend-plugin, i.e. put the StreamFX sources into ~/obs-studio/UI/frontend-plugins (assuming that ~/obs-studio holds the obs build environment)
  • StreamFX requires C++17 so GCC/G++ 9 is required. (This poses no problem with cuda 10.2 as on Linux neither obs nor StreamFX compile cuda sources. Obs and StreamFX actually compiled with with GCC/G++ 8, however I experienced the GCC 8 'filesystem' segfault bug)
  • In order to avoid conflicts with the obs-studio standard package that ubuntu provides I chose /usr/local/ as the installation location for the self-built obs by modifying the option -DCMAKE_INSTALL_PREFIX=/usr/local in the cmake statement
  • Also in the cmake statement the option -DCMAKE_CXX_FLAGS="-fPIC" needed to be added (as with jetson-ffmpeg) for correctly linking in arm64/aarch64 Linux and GCC/G++.
  • I have built obs with the browser source which I built myself (see remark above). So the option in cmake statement needed to be -DCEF_ROOT_DIR="../../CEF_binary_min"
  • Obs sometimes complains about missing libraries libobs-frontend-api.so.0 or libobs-opengl.so.0. I cured this by creating links to libobs-frontend-api.so.0.0 resp. libobs-opengl.so.0.0 in /usr/local/bin

At this point the self built obs-studio will not yet offer nvidia nvmpi when checking in settings->output->output mode = advanced, but obs itself should work fine with software encoding, and StreamFX should show up in the obs menu.

Integrating nvmpi encoder into StreamFX and obs-studio

  • The six source files nvmpi* form the nvmpi handler (analogously to StreamFX's nvenc handler). Place them in ~/obs-studio/UI/frontend-plugins/streamfx/source/encoders/handlers beside the nvenc* files.
  • Search the file CMakeLists.txt for 'nvmpi' and transfer the lines over into ~/obs-studio/UI/frontend-plugins/streamfx/CMakeLists.txt, i.e. the build description of the current StreamFX, in an adequate way.
  • In ~/obs-studio/UI/frontend-plugins/streamfx/source/encoders/encoder-ffmpeg.cpp the folowing two lines need to be added directly below the corresponding #includes for nvenc, i.e. below '#include "handlers/nvenc_hevc_handler.hpp"':
#include "handlers/nvmpi_h264_handler.hpp" 
#include "handlers/nvmpi_hevc_handler.hpp"
  • and two lines below should be added in function ffmpeg_manager::ffmpeg_manager() below the line '#ifdef ENABLE_ENCODER_FFMPEG_NVENC':
register_handler("h264_nvmpi", ::std::make_shared<handler::nvmpi_h264_handler>());
register_handler("hevc_nvmpi", ::std::make_shared<handler::nvmpi_hevc_handler>());

Then refresh the build system by re-doing the cmake step for obs-studio, and rebuild obs-studio. If everything went right nvmpi h264 and h265 will now be offered in obs when you choose settings->output->output mode = advanced.

Enjoy!

Footnote: Using obs with xrdp and VirtualGL

Obs-studio requires OpenGL support. If you access your Linux Desktop from remote (e.g. via xrdp or vnc), openGL support will be via software (MESA) or non-existent. I found that using VirtualGL obs-studio runs reasonably well in a remote xrdp session in the same LAN, see xrdp forum.

Similar Resources

GPU PyTorch TOP in TouchDesigner with CUDA-enabled OpenCV

PyTorchTOP This project demonstrates how to use OpenCV with CUDA modules and PyTorch/LibTorch in a TouchDesigner Custom Operator. Building this projec

Jun 15, 2022

TensorFlow implementation of SQN based on RandLA-Net's encoder

TensorFlow implementation of SQN based on RandLA-Net's encoder

SQN_tensorflow This repo is a TensorFlow implementation of Semantic Query Network (SQN). For Pytorch implementation, check our SQN_pytorch repo. Our i

Nov 3, 2022

Ikomia Studio software

Ikomia Studio software

Ikomia Studio Presentation Ikomia Studio is an Open Source desktop application that aims to simplify use, reproducibility and sharing of state of the

Nov 15, 2022

Visual odometry package based on hardware-accelerated NVIDIA Elbrus library with world class quality and performance.

Visual odometry package based on hardware-accelerated NVIDIA Elbrus library with world class quality and performance.

Isaac ROS Visual Odometry This repository provides a ROS2 package that estimates stereo visual inertial odometry using the Isaac Elbrus GPU-accelerate

Nov 26, 2022

The core engine forked from NVidia's Q2RTX. Heavily modified and extended to allow for a nicer experience all-round.

Nail & Crescent - Development Branch Scratchpad - Things to do or not forget: Items are obviously broken. Physics.cpp needs more work, revising. Proba

Nov 22, 2022

NVIDIA Texture Tools samples for compression, image processing, and decompression.

NVTT 3 Samples This repository contains a number of samples showing how to use NVTT 3, a GPU-accelerated texture compression and image processing libr

Nov 16, 2022

Vendor and game agnostic latency reduction middleware. An alternative to NVIDIA Reflex.

LatencyFleX (LFX) Vendor and game agnostic latency reduction middleware. An alternative to NVIDIA Reflex. Why LatencyFleX? There is a phenomenon commo

Nov 29, 2022

TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

TensorRT Open Source Software This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. Included are the sources for Tens

Dec 3, 2022

Forward - A library for high performance deep learning inference on NVIDIA GPUs

 Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

Mar 17, 2021
Comments
  • Request for Ubuntu image with patched OBS pre-installed

    Request for Ubuntu image with patched OBS pre-installed

    Hello,

    I found this project after digging in the OBS documentation for the last couple of days. It seems very promising and I'm trying to set this up for my friend who wants a simple and cheap solution for streaming using OBS.

    I have tried to get this to work based on my very little knowledge of OBS and the Nvidia Jetson, however I don't seem to be able to get a working installation without OBS crashing with a segfault when pressing "start streaming" or "start recording"

    To make this project easier for people to use it would be very helpful to provide an image with OBS preinstalled with support for the NVMPI encoder.

    Atleast some instructions on how to create an image would be nice. For starters, how do I upgrade to Ubuntu 20.04 on my Jetson? The L4T based images provided by Nvidia are Ubuntu 18.04.

  • StreamFX patch failed

    StreamFX patch failed

    $ git apply streamfx.patch --unsafe-paths --directory=../obs-studio/UI/frontend-plugins/streamfx

    error: patch failed: ../obs-studio/UI/frontend-plugins/streamfx/CMakeLists.txt:292

    error: ../obs-studio/UI/frontend-plugins/streamfx/CMakeLists.txt: patch does not apply

    edit: i found what the problem is. the build scripts downloading the wrong version of StreamFX (0.12)

A C++ implementation of Yolov5 to detect mask running in Jetson Xavier nx and Jetson nano.
A C++ implementation of Yolov5 to detect mask running in Jetson Xavier nx and Jetson nano.

yolov5-mask-detect A C++ implementation of Yolov5 to detect mask running in Jetson Xavier nx and Jetson nano.In Jetson Xavier Nx, it can achieve 33 FP

Mar 19, 2022
Face tracking plugin for OBS Studio

OBS Face Tracker Plugin Introduction This plugin provide a filter for face detection and face tracking for mainly a speaking person. This plugin emplo

Nov 27, 2022
Docker files and scripts to setup and run VINS-FUSION-gpu on NVIDIA jetson boards inside a docker container.

jetson_vins_fusion_docker This repository provides Docker files and scripts to easily setup and run VINS-FUSION-gpu on NVIDIA jetson boards inside a d

Oct 12, 2022
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Isaac ROS DNN Inference Overview This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom mode

Dec 5, 2022
GPU Cloth TOP in TouchDesigner using CUDA-enabled NVIDIA Flex
GPU Cloth TOP in TouchDesigner using CUDA-enabled NVIDIA Flex

This project demonstrates how to use NVIDIA FleX for GPU cloth simulation in a TouchDesigner Custom Operator. It also shows how to render dynamic meshes from the texture data using custom PBR GLSL material shaders inside TouchDesigner.

Jul 27, 2022
Openpilot on Jetson Xavier NX
Openpilot on Jetson Xavier NX

Licensing xnxpilot is released under the MIT license. Some parts of the software are released under other licenses as specified. Any user of this soft

Nov 24, 2022
YoloX for a Jetson Nano 4 using ncnn.

YoloX Jetson Nano YoloX with the ncnn framework. Paper: https://arxiv.org/pdf/2107.08430.pdf Special made for a Jetson Nano, see Q-engineering deep le

Nov 11, 2022
With deep learning to neuroscience world with shield for jetson nano - JNEEG (In progress)
With deep learning to neuroscience world with shield for jetson nano - JNEEG (In progress)

With deep learning to neuroscience world with shield for jetson nano - JNEEG (In progress)

Oct 11, 2022
A coverage-guided and memory-detection enabled fuzzer for windows applications.
A coverage-guided and memory-detection enabled fuzzer for windows applications.

WDFuzzer Manual 中文手册见 README_CN.md WDFuzzer:winafl + drmemory WDFuzzer is an A coverage-guided and memory detection abled fuzzer for for windows softw

Nov 15, 2022