This project demonstrate how to infer and track from a 360 videos by using the dewarper plugin.

Deepstream Dewarper App

This project demonstrate how to infer and track from a 360 videos by using the dewarper plugin. Dewarping 360 videos helps to have better inference and tracking accuracy. This can be seen in the image above where the ml model struggles to infer in the original image but does much better in the dewarped surfaces. It also includes a dynamic library libnvdsgst_dewarper.so which has more projection types than the libnvdsgst_dewarper.so file in the DeepStream 5.1

Prequisites:

Please follow instructions in the apps/sample_apps/deepstream-app/README on how to install the prequisites for Deepstream SDK, the DeepStream SDK itself and the apps.

  • You must have the following development packages installed

    GStreamer-1.0 GStreamer-1.0 Base Plugins GStreamer-1.0 gstrtspserver X11 client-side library

  • To install these packages, execute the following command:

    sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev
    libgstrtspserver-1.0-dev libx11-dev

Getting started:

  1. Install Deepstream 5.1 on your platform, verify it is working by running deepstream-app.

  2. Clone the repository preferably in $DEEPSTREAM_DIR/sources/apps/sample_apps.

    $ git clone https://github.com/NVIDIA-AI-IOT/Deepstream-Dewarper-App.git

  3. Replace old dewarper plugin binary with the new binary that includes 15 more projection types. Note: keep the old ones incase you want to revert back and use them

    • Replace the libnvdgst_dewarper.so binary in /opt/nvidia/deepstream/deepstream-5.1/lib/gst-plugins/ with the binary provided in this repo under the plugin_libraries
    • Replace the nvds_dewarper_meta.h file in /opt/nvidia/deepstream/deepstream-5.1/source/includes/
    • note:Jetson library is included here now
  4. Get the Tlt peoplenet model and label file. Download these files under inference_files directory.

  5. Compile the program

    • $ cd deepstream-dewarper-app/
    • $ make
    • $ ./deepstream-dewarper-app [1:file sink|2: fakesink|3:display sink] [1:without tracking| 2: with tracking] [ ] [ ] ... [ ]
    • Single Stream
    • $ ./deepstream-dewarper-app 3 1 file:///home/nvidia/sample_office.mp4 6 one_config_dewarper.txt (to display)
    • // Single Stream for Perspective Projection type (needs config file change)
    • $ ./deepstream-dewarper-app 3 1 file:///home/nvidia/yoga.mp4 0
    • Multi Stream
    • $ ./deepstream-dewarper-app 3 1 file:///home/nvidia/sample_cam6.mp4 6 one_config_dewarper.txt file:///home/nvidia/sample_cam6.mp4 6 one_config_dewarper.txt

The following description focus on the default use-case of detecting people in a cubicle office enviroment but you can use it to test other types of applications that needs the dewarper functionality.

(see Note below). For more information on the general functionality and further examples see the DeepStream Plugin Development Guide.

Dewarping configuration files are provided in dewarper_config_files directory :

  • Example parameters for dewarping fisheye camera/video are given in these config files. You can play with these parameters to get your desired dewarped surface.
  • Details explaining these parameters are given below in this file..

Parameters: uri - represents the input video stream

The dewarping parameters for the given camera can be configured in the config file provided to generate dewarp surfaces input video stream.

Note: gst-nvdewarper plugin uses "VRWorks 360 Video SDK". For further details please refer to https://developer.nvidia.com/vrworks/vrworks-360video/download

For description of general dewarper parameters please visit the DeepStream Plugin Development Guide.



Common Fields/Parameters

Please refer to GST-NVDEWARPER configuration file parameters for details.

  • projection-type - Selects projection type. With the new plugin update (binary files you replaced) 15 more projection types are added. New projection types added are : - 1=PushBroom, 2=VertRadCyl 3= Perspective_Perspective FISH_PERSPECTIVE=4, FISH_FISH=5, FISH_CYL=6, FISH_EQUIRECT=7, FISH_PANINI=8, PERSPECTIVE_EQUIRECT=9, PERSPECTIVE_PANINI=10, EQUIRECT_CYLINDER=11, EQUIRECT_EQUIRECT=12 EQUIRECT_FISHEYE=13, EQUIRECT_PANINI=14, EQUIRECT_PERSPECTIVE=15, EQUIRECT_PUSHBROOM=16, EQUIRECT_STEREOGRAPHIC=17, EQUIRECT_VERTCYLINDER=18
  • top-angle - Top Field of View Angle, in degrees
  • bottom-angle - Bottom Field of View Angle, in degrees
  • pitch - Viewing parameter Pitch, in degrees
  • yaw - Viewing parameter Yaw, in degrees
  • roll - Viewing parameter Roll, in degrees
  • focal length - Focal Lenght of camera lens, in pixels per radian
  • width - dewarped surface width
  • height - dewarped surface height
  • num-batch-buffers - To change the number of surfaces. It should match the number of "surfaces" groups in the configuration file. So if you want two surfaces per buffer you should have "num-batch-buffers"=2 and two surfaces groups ([surface0] and [surface1]). Default value is 4.
Comments
  • Report some issues

    Report some issues

    I've executed this app however I've faced some issues as follows:

    1. there is a wrong file path into deepstream_dewarper_test.c:
    $ git diff
    diff --git a/deepstream_dewarper_test.c b/deepstream_dewarper_test.c
    index 9c65f7a..412b52e 100755
    --- a/deepstream_dewarper_test.c
    +++ b/deepstream_dewarper_test.c
    @@ -738,7 +738,7 @@ main (int argc, char *argv[])
       //  "model-engine-file", "/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/Deepstream-Dewarper-App/inference_files/resnet34_peoplenet_pruned.etlt_b1_gpu0_fp16.engine",
       //  NULL);
       g_object_set (G_OBJECT (nvinfer),
    -    "config-file-path", "config_infer_primary_peoplenet.txt",
    +    "config-file-path", "inference_files/config_infer_primary_peoplenet.txt",
         //"config-file-path", "/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/Deepstream-Dewarper-App/config_infer_primary_peoplenet.txt",
         NULL);
    
    1. Failed to parse UFF model:
    ERROR: [TRT]: Parameter check failed at: ../builder/Network.cpp::addInput::992, condition: isValidDims(dims, hasImplicitBatchDimension())
    ERROR: [TRT]: UFFParser: Failed to parseInput for node input_1
    ERROR: [TRT]: UffParser: Parser error: input_1: Failed to parse node - Invalid Tensor found at node input_1
    parseModel: Failed to parse UFF model
    ERROR: failed to build network since parsing model errors.
    ERROR: Failed to create network using custom network creation function
    ERROR: Failed to get cuda engine from custom library API
    0:00:02.153209672 16358   0x559ea0f410 ERROR                nvinfer gstnvinfer.cpp:613:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1736> [UID = 1]: build engine file failed
    

    Then I fixed inference_files/config_infer_primary_peoplenet.txt as bellow:

    diff --git a/inference_files/config_infer_primary_peoplenet.txt b/inference_files/config_infer_primary_peoplenet.txt
    index e27eea6..bda1ed3 100755
    --- a/inference_files/config_infer_primary_peoplenet.txt
    +++ b/inference_files/config_infer_primary_peoplenet.txt
    @@ -24,18 +24,18 @@
     gpu-id=0
     net-scale-factor=0.0039215697906911373
     tlt-model-key=tlt_encode
    -#tlt-encoded-model=../../models/tlt_pretrained_models/peoplenet/resnet34_peoplenet_pruned.etlt
     tlt-encoded-model=resnet34_peoplenet_pruned.etlt
    -labelfile-path=labels_peoplenet.txt
    -#model-engine-file=../../models/tlt_pretrained_models/peoplenet/resnet34_peoplenet_pruned.etlt_b1_gpu0_fp16.engine
    -model-engine-file=/home/ds_root/peoplenet/resnet34_peoplenet_pruned.etlt_b1_gpu0_fp16.engine
    +labelfile-path=labels.txt
    +model-engine-file=resnet34_peoplenet_pruned.etlt_b1_gpu0_int8.engine
    +
     input-dims=3;544;960;0
     uff-input-blob-name=input_1
     batch-size=1
     process-mode=1
     model-color-format=0
     ## 0=FP32, 1=INT8, 2=FP16 mode
    -network-mode=2
    +network-mode=1
    +
     num-detected-classes=1
     cluster-mode=1
     interval=0
    

    Then I've faced "Segmentation fault" after executing this application.

    1. Segmentation fault:
    $ ./deepstream-dewarper-app 3 1 \
        file:///opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/Deepstream-Dewarper-App/sample_office.mp4 \
        6 \
        one_config_dewarper.txt
    

    LOG:

    Running...
    Decodebin child added: qtdemux0
    Decodebin child added: multiqueue0
    Decodebin child added: aacparse0
    
    (deepstream-dewarper-app:16382): GStreamer-WARNING **: 21:29:17.680: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so': /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block
    
    (deepstream-dewarper-app:16382): GStreamer-WARNING **: 21:29:17.700: Failed to load plugin '/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so': /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block
    Decodebin child added: faad0
    Decodebin child added: h264parse0
    Decodebin child added: capsfilter1
    Decodebin child added: nvv4l2decoder0
    Opening in BLOCKING MODE
    Opening in BLOCKING MODE 
    NvMMLiteOpen : Block : BlockType = 261 
    NVMEDIA: Reading vendor.tegra.display-size : status: 6 
    NvMMLiteBlockCreate : Block : BlockType = 261 
    In cb_newpad
    In cb_newpad
    Segmentation fault (core dumped)
    
    • Device: Jetson Xavier NX
    • Deepstream SDK 5.1

    It would be nice if you have solutions. Thank you.

  • How do I turn off bound boxes?

    How do I turn off bound boxes?

    I run: $ sudo ./deepstream-dewarper-app 1 1 file.mp4 6 one_config_dewarper.txt ./dewarper_config_files/fisheye_to_\ panini_config_dewarper.txt And get: изображение

    I run: $ sudo ./deepstream-dewarper-app 1 2 file.mp4 6 ./dewarper_config_files/fisheye_to_\ panini_config_dewarper.txt And get: изображение

    How to disable the linked boxes to leave just the image conversion? UPD: You need to comment out everything related to the nvosd element.

  • The meaning of the parameters is too difficult to understand

    The meaning of the parameters is too difficult to understand

    This is an interesting project,thank you. But, some of parameters is too difficult to understand,I debug two days, cannot get satisfactory results,I changed pitch, yaw, roll, top-angle ,bottom-angle,width,height,but my 4 surfaces is like the image below: image How can I set the image to 2 rows and 2 columns instead of 4 rows and 1 column I think ,the correct form should be like the image below: image Additionally,I hope we can give a detailed description of the physical meaning of the parameters in the algorithm, preferably with a simple example, such as changing yaw is to change which area of the picture! Thank you very much!

  • Where can I read about different projections?

    Where can I read about different projections?

    Hi here

    I need to use this for a simple case, a camera which is rotated to the right and I want it to correct the perspective distortion.

    I see that there are a lot of projections, but can't seem to find any docs describing them so I can select/understand the best one for my use case.

    Any help would be greatly appreciated.

    Thanks

  • libnvds_dewarper.so fails on using new projections

    libnvds_dewarper.so fails on using new projections

    Trying to run experiment and see that any projection type above 3 is failing to start. The .so library seems to have an outdated check that is preventing it from running.

    Running on x86 and getting this error when replacing the .so file. nvdewarper0: Invalid Projection type (4) selected ** ERROR:nvdewarper.cpp:685:cudaError gst_nvdewarper_do_dewarp(Gstnvdewarper*, NvBufSurface*, NvBufSurface*): assertion failed: (it->projection_type <= NVDS_META_SURFACE_PERSPECTIVE_PERSPECTIVE) Aborted (core dumped)

  • Dewarper for non 360 cameras

    Dewarper for non 360 cameras

    Hi. I want to use your module for flat (non 360) videos to do preprocessing on my frames. The preprocessing includes rotation of frame in 45 degree (or any other degrees) and after that cropping it. So i have one input source and one output surface. My first question:
    Is it possible? which configs should i use for just rotation and crop in the [surfaceX] ?

    My second question: If the answer of first question is yes, can i have 4 input sources(non 360) from 4 cameras and have one surface for each one ? In total 4 surfaces that are rotated and cropped. Thanks in advanced !

  • How connect to CSI camera in deepstream dewarper and capture video to file?

    How connect to CSI camera in deepstream dewarper and capture video to file?

    Hardware Platform Jetson Nano DeepStream Version 5.1 JetPack Version 4.5.1 TensorRT Version 7.1.3

    I install this dewarper , but it is not clear how to connect to the CSI camera, de-warp and save video to a file. How can this be done?

  • lib .so not working on jetson nano

    lib .so not working on jetson nano

    I'm trying to execute dewarper-app on Jetson nano The lib .so in the repository seem not to be as the good format :

    _/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/Deepstream-Dewarper-App/plugin_libraries$ file *
    libnvds_dewarper.so:    ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=a31eec7f81067f554e9e508390007f9d8a582e0a, stripped
    libnvdsgst_dewarper.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=d79269a1c0bcc018cf3cc7e64ca3a18b9c97aa71, not stripped_
    

    Have you got the lib for the jetson nano (arm) ?

Blur is a program made for easily and efficiently adding motion blur to videos through frame blending.
Blur is a program made for easily and efficiently adding motion blur to videos through frame blending.

Blur Blur is a program made for easily and efficiently adding motion blur to videos through frame blending. Features The amount of motion blur is easi

Jun 8, 2022
Example programs for Talking Async videos

Talking Async Example programs for the Talking Async videos, which can be found on YouTube. Episode 1: Why C++20 is the Awesomest Language for Network

Jun 4, 2022
Provides a menu for playing Karaoke videos.
Provides a menu for playing Karaoke videos.

Provides a menu for playing Karaoke videos. Audio input from a microphone is automatically mixed with output from videos. Videos can be paused and scrubbed.

Dec 26, 2021
A UE4 plugin used ffmpeg library to record/publish game viewport output and in-game audio.

Streaming A UE4 plugin used ffmpeg library to record/publish game viewport output and in-game audio. GPL license beacuse used ffmpeg. Structures GameV

Feb 25, 2022
Plugin for VLC that pauses/plays video on mouse click

Pause Click plugin for VLC VLC plugin that allows you to pause/play a video by clicking on the video image. Can be configured to work nicely with doub

Jun 19, 2022
A bittorrent plugin for VLC.

vlc-bittorrent (Bittorrent plugin for VLC) What is this? With vlc-bittorrent, you can open a .torrent file or magnet link with VLC and stream any medi

Jun 15, 2022
Vita Recorder is a plugin that allows to record video clips during your play sessions.

Vita Recorder Vita Recorder is a plugin that allows to record video clips during your play sessions. The code is based off VITA2PC at which has been a

Jun 17, 2022
Free plugin-based video converter
Free plugin-based video converter

SEZE II Смотреть на русском SEZE is a plugin-based video converter for Windows/Linux. You can write your video filter in C++ (see section "How to make

Apr 25, 2022
Sample Project that I'm trying to build on YouTube (Live)

GTK-Project Sample Project that I'm trying to build on YouTube (Live) Streams discussing this: S.No. Date Title Limk 1 30/08/2021 Code Along: Building

Mar 18, 2022
A project for the latest Steam PC (US) release of Resident Evil 4 that patches the game to use high-quality MP4 video.

A project for the latest Steam PC (US) release of Resident Evil 4 that patches the game to use high-quality MP4 video.

Mar 3, 2022
Yakuza Arcade Machine Player - play Virtua Fighter 5: Final Showdown on PC, using Yakuza 6 files.
Yakuza Arcade Machine Player - play Virtua Fighter 5: Final Showdown on PC, using Yakuza 6 files.

Yakuza Arcade Machines Player Yakuza Arcade Machines Player is a launcher that allows you to run Virtua Fighter 5: Final Showdown, standalone and nati

Apr 24, 2022
yangwebrtc is a self-developed rtc architecture supporting Webrtc/Srt/Rtmp, including a variety of video and audio codecs and processing, etc.
yangwebrtc is a self-developed rtc architecture supporting Webrtc/Srt/Rtmp, including a variety of video and audio codecs and processing, etc.

YangWebrtc Overview yangwebrtc是一个自主研发的支持Webrtc/Srt/Rtmp的rtc架构,包含多种视音频编解码和处理等。 支持视频会议、高清录播直播、直播互动等多种视音频应用。 可用于远程教育、远程医疗、指挥调度、安防监控、影视录播、协同办公、直播互动等多种行业应用

Jun 15, 2022
FFmpeg Kit for applications. Supports Android, Flutter, iOS, macOS, React Native and tvOS. Supersedes MobileFFmpeg, flutter_ffmpeg and react-native-ffmpeg.
FFmpeg Kit for applications. Supports Android, Flutter, iOS, macOS, React Native and tvOS. Supersedes MobileFFmpeg, flutter_ffmpeg and react-native-ffmpeg.

FFmpeg Kit for applications. Supports Android, Flutter, iOS, macOS, React Native and tvOS. Supersedes MobileFFmpeg, flutter_ffmpeg and react-native-ffmpeg.

Jun 18, 2022
Kodi is an award-winning free and open source software media player and entertainment hub for digital media
Kodi is an award-winning free and open source software media player and entertainment hub for digital media

website • docs • community • add-ons Welcome to Kodi Home Theater Software! Kodi is an award-winning free and open source software media player and en

Jun 24, 2022
Vireo is a lightweight and versatile video processing library written in C++11

Overview Vireo is a lightweight and versatile video processing library that powers our video transcoding service, deep learning recognition systems an

Jun 15, 2022
Olive is a free non-linear video editor for Windows, macOS, and Linux.
Olive is a free non-linear video editor for Windows, macOS, and Linux.

Olive is a free non-linear video editor for Windows, macOS, and Linux.

Jun 15, 2022
An Open Source PSVita/TV MP4 player with 1080p playback and subtitle support
An Open Source PSVita/TV MP4 player with 1080p playback and subtitle support

Vita-Media-Player An Open Source PSVita/TV MP4 player with 1080p playback and subtitle support 1080i output supported on the PSTV natively and on the

May 28, 2022
Even newer Custom Firmware for Xiaomi Cameras based on Hi3518ev200 Chipset. It includes free RTSP, ONVIF and other improvements based on the work by roleoroleo
Even newer Custom Firmware for Xiaomi Cameras based on Hi3518ev200 Chipset. It includes free RTSP, ONVIF and other improvements based on the work by roleoroleo

Why this Yi-Hack-V5 firmware? The answer is simple: missing updates, RTSP and not based on the latest stock firmware (which features improvements and

Jun 11, 2022
A WFH utility to visually indicate user engagement of audio and video
A WFH utility to visually indicate user engagement of audio and video

DIY: In meeting indicator - WFH Utility The need for in meeting indicator at home So many of you have gotten accustomed to work from home by now. This

Jun 28, 2021