Dataset Synthesizer - NVIDIA Deep learning Dataset Synthesizer (NDDS)

NVIDIA Deep learning Dataset Synthesizer (NDDS)

Overview

NDDS is a UE4 plugin from NVIDIA to empower computer vision researchers to export high-quality synthetic images with metadata. NDDS supports images, segmentation, depth, object pose, bounding box, keypoints, and custom stencils. In addition to the exporter, the plugin includes different components for generating highly randomized images. This randomization includes lighting, objects, camera position, poses, textures, and distractors, as well as camera path following, and so forth. Together, these components allow researchers to easily create randomized scenes for training deep neural networks.

Example of an image generated using NDDS, along with ground truth segmentation, depth, and object poses.
For utilities to help visualize annotation data associated with synthesized images, see the NVIDIA dataset utilities (NVDU) https://github.com/NVIDIA/Dataset_Utilities.

Downloading

This repository uses gitLFS -- DO NOT DOWNLOAD AS .ZIP:

First, install git LFS (large file storage): https://git-lfs.github.com/ , then lfs clone.

For further details, please see https://github.com/NVIDIA/Dataset_Synthesizer/blob/master/Documentation/NDDS.pdf

RELEASE NOTES: 4.22 known issue

If you are using material randomization with more than 10 objects which change materials every frame, you might encounter a hang when stopping the play-in-editor session. The capturing process will still work, but the uniform buffer memory will keep increasing and when user stops the capture session, it takes UE extended time to release the memory. If it takes too long after stopping the play-in-editor session, we recommend to simply shutdown the editor and restart it. The only other workaround is to keep using UE4.21, which requires use of NDDS v1.1.

This problem is specific to UE4 4.22, as it now automatically uses mesh instancing to improve the performance when rendering a large quantity of meshes. Now, every time a new mesh is created or its material is changed, the uniform buffer memory allocation is increased.

This problem affects both DirectX and OpenGL users. Although Vulkan doesn't get affected by this, Vulkan doesn't capture depth and class segmentation.

Motivation

Training and testing deep learning systems is an expensive and involved task due to the need for hand-labeled data. This is problematic when the task demands expert knowledge or not-so-obvious annotations (e.g., 3D bounding box vertices). In order to overcome these limitations we have been exploring the use of simulators for generating labeled data. We have shown in [1,2] that highly randomized synthetic data can be used to train computer vision systems for real-world applications, thus showing successful domain transfer.

Citation

If you use this tool in a research project, please cite as follows:

@misc{to2018ndds,
author = {Thang To and Jonathan Tremblay and Duncan McKay and Yukie Yamaguchi and Kirby Leung and Adrian Balanon and Jia Cheng and William Hodge and Stan Birchfield},
note= {\url{ https://github.com/NVIDIA/Dataset_Synthesizer }},
title = {{NDDS}: {NVIDIA} Deep Learning Dataset Synthesizer},
Year = 2018
}

References

[1] J. Tremblay, T. To, A. Molchanov, S. Tyree, J. Kautz, S. Birchfield. Synthetically Trained Neural Networks for Learning Human-Readable Plans from Real-World Demonstrations. In International Conference on Robotics and Automation (ICRA), 2018.

[2] J. Tremblay, T. To, S. Birchfield. Falling Things: A Synthetic Dataset for 3D Object Detection and Pose Estimation. CVPR Workshop on Real World Challenges and New Benchmarks for Deep Learning in Robotic Vision, 2018.

Comments
  • Importing YCB Objects

    Importing YCB Objects

    Hey,

    in which way did you import the YCB objects into Unreal Engine such that you can use the meshes to render the images? On the YCB Website (http://ycb-benchmarks.s3-website-us-east-1.amazonaws.com/) it is possible to download the processed data or the laserscan. However, they do not provide a .fbx file for the unreal engine.

    How did you create these .fbx files?

    Thank you really much in advance!

    Best regards, Björn

  • annotation error

    annotation error

    Hi after i put cube into example scene, get wrong annotation. why? or how to get right annotation?"projected_cuboid_centroid": [ 256.00009155273438, 287.43289184570312 ], "bounding_box": { "top_left": [ -116833.7734375, -140902.03125 ], "bottom_right": [ 117898.078125, 141403.25 ] }, "cuboid": [ [ 21.611099243164062, -33.871700286865234, -97.358100891113281 ], [ 21.611099243164062, -33.871700286865234, -2.6419000625610352 ], [ 21.611099243164062, 21.59320068359375, -2.6419000625610352 ], [ 21.611099243164062, 21.59320068359375, -97.358100891113281 ], [ -21.611099243164062, -33.871700286865234, -97.358100891113281 ], [ -21.611200332641602, -33.871700286865234, -2.6419000625610352 ], [ -21.611200332641602, 21.59320068359375, -2.6419000625610352 ], [ -21.611099243164062, 21.59320068359375, -97.358100891113281 ] ], "projected_cuboid": [ [ 199.17430114746094, 345.06460571289062 ], [ -1838.110595703125, 3538.15771484375 ], [ -1838.110595703125, -1836.37744140625 ], [ 199.17430114746094, 199.22129821777344 ], [ 312.8258056640625, 345.06460571289062 ], [ 2350.114501953125, 3538.15771484375 ], [ 2350.114501953125, -1836.37744140625 ], [ 312.8258056640625, 199.22129821777344 ] ] as you can see, bounding box value is invalid, projected_cuboid is invalid. projected_cuboid_centroid is not at the center of cube in rgb image.

  • Frame Used for Pose Annotation

    Frame Used for Pose Annotation

    Is the axis in the object model transformed before the ground truth pose is printed in the annotation file? I observed that in some cases when the object model had a coordinate frame with z pointing up, the ground truth pose annotation has a different orientation.

  • TestCapturer not working in 4.22

    TestCapturer not working in 4.22

    Hey folks, so I had been using 4.21, which works well. I wanted to migrate to 4.22 to use ray tracing. That's when I started to encounter problems. So, I recently started from scratch in 4.22, and for whatever reason, the "testscapturer" cannot capture any data. Please help!

  • migrating assets to 4.22

    migrating assets to 4.22

    hey folks, the assets in your 4.22 do not seem to be ready for that version. They are still formatted for 4.21. Any idea about how to fix this? My CAD imports do not have this issue.

  • Unable to find NVSceneCapturerGameMode_BP Game Mode

    Unable to find NVSceneCapturerGameMode_BP Game Mode

    Hi, I am trying to create a simple synthetic cube/cuboid placed in an environment(level) and generate metadata which will give coordinates of the vertices of the cuboid. I am following the steps mentioned in the pdf which can be found on this Ndds.pdf.

    The problem is I'm unable to find the NVSceneCapturerGameMode_BP game logic through the UE4(Unreal Engine) but I can find it locally through the ubuntu file explorer. I've tried both the steps mentioned in last section of page 8 of the pdf but am unable to find it. The same problem with ScreenCapturer_Simple(mentioned on page 10 of pdf) which I can find through ubuntu file explorer but not through UE4.

    Has anyone else faced the same problem? I'm not able to understand why UE4 can't recognise it.

  • Depth image for translucent objects

    Depth image for translucent objects

    Hi there, I am trying to create dataset with translucent object but had no luck. In class segmentation case, I was able to bring translucent objects in image thanks to this step.

    Question is what do i have to do in order to capture depth image of translucent objects in NDDS?

    Here are my results so far. (Cup)

    000016 depth mm 16 000016 depth 000016 is 000016 000016 cs 000016 depth 16 000016 depth cm 8 000016 depth cm 16

  • Fix freezing the UE editor when quitting game mode after capturing (Linux).

    Fix freezing the UE editor when quitting game mode after capturing (Linux).

    Hey there,

    I was using the project on Ubuntu 18.04 with the UnrealEngine 4.21.3 built from source. I could use the project according to the documentation without problems. But when quitting the game mode after recording data, the editor would freeze.

    It was locking in the FNVImageExporter_Thread::Stop on the Thread->Kill(true); statement and therefore probably on HavePendingImageEvent->Wait(); Not resetting the HavePendingImageEvent when stopping the thread fixed the problem.

    I don't know if this was also a problem on Windows or Mac neither if this change introduces a problem on these platforms. Yet, "it worked on my machine".

    Cheers, Stefan

  • Object visibility

    Object visibility

    Hey guys, thanks for making this public. I was wondering about visibility in meta data (for example: 0000.json file). In this video, you guys explained that visibility = 0 is fully visible, and 1 is completely occluded. But in FAT dataset, it is kinda opposite.

    image

    Which one is right? All the data I am exporting have visibility set to 0. How can I change this value?

  • Delay between spawning new actors

    Delay between spawning new actors

    Hey, I want to synthesize datatests with objects lying randomly on the floor. The object shall also lie on top of each other, so i want them to appear at a random position in my TriggerVolume and fall down. When all object lie on the floor, the captured data shall be saved.

    For now, I am using a GroupActorManager for my objects (works well, diffrent objects appear in the defined volume in diffrent rotations) and a SceneCapturer_Simple with a SceneManager_BP to save the data. My problem: a new group of actors spawn in a very short time frame, so the objects can't fall on the floor

    I already tried to vary the Spawn Duration, but that doesn't help.

    Can you please tell me what I'm doing wrong? Do I need another actor or component?

    Thanks in advance :)

  • Broken PDF and Window OS compatibility issue

    Broken PDF and Window OS compatibility issue

    Nice augmentation library! I am trying to utilize the synthesizer to augment my own pose-estimation dataset. I have a few questions about this repository. Firstly, the guide PDF cannot be opened. It seems broken. I want to confirm that it only happens to me or the file is really broken. (https://github.com/NVIDIA/Dataset_Synthesizer/blob/master/Documentation/NDDS.pdf)

    Secondly, Is this library also available on Window OS? Many examples and guidelines I found are all based on Linux OS.

    Waiting for help from any pioneers :)

  • object mask which is not visible

    object mask which is not visible

    Hi everybody. I want to use this tool for some models which use bop formatting. Bop formatting requires object mask and object mask_visib as images As mask, the system displays the siluethe of the object even some parts of it are invisible(behind some other objects) As mask_visib, the system displays the visible object to the camera.

    mask_visib is ok, but how can I get mask (which displays the whole object as a mask even it is behind another object) Thanks thanks in advance

  • Incorrect units in _object_settings.json

    Incorrect units in _object_settings.json

    Hi, I am trying to generate a dataset on a custom 3d object but my object_settings.json keeps giving me the cuboid dimensions in metres instead of centimetres. The object is the correct scale in Unreal Engine when compared to other similar sized objects and is also the correct dimensions in Blender but the cuboid dimensions output are still in the wrong units. Is there anything I can do to resolve this? Or can I just manually change the units in the object_settings file?

    image

  • Ubuntu Version

    Ubuntu Version

    Is there any plan on the roadmap to support a current version of Ubuntu? 16.04 reach EOL almost a year ago and no longer receives security updates so is not a viable option for many.

  • Engine modules are out of date, and cannot be compiled while the engine is ruining. Please build through your IDE.

    Engine modules are out of date, and cannot be compiled while the engine is ruining. Please build through your IDE.

    When I attempt to run NDDS.uproject I get the above error after selecting to rebuild the modules. I have followed the recommend solution from: https://wiki.archlinux.org/title/Unreal_Engine_4#Engine_modules_are_out_of_date,_and_cannot_be_compiled_while_the_engine_is_running._Please_build_through_your_IDE

    However this does not resolve my problem. Any recommendations??

  • cannot get right instance segmentation results and instance seg id while 0">

    cannot get right instance segmentation results and instance seg id while "Time Between Scene Capture" > 0

    i would like to make the objects falling in a box, this process may takes 10 seconds, and then capture the scene with all meta data like the image below. 000000

    My current plan is to set up the spawn time to 10 seconds, and also set the 'Time Between Scene Capture' of the camera to 10 seconds, it works fine, but then I found that all instance id in json file are 0, and I cannot get the instance segmentation map, the data is wrong.

Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models

Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine DSSTNE (pronounced "Destiny") is an open source software library for training and deploying

Sep 26, 2022
Forward - A library for high performance deep learning inference on NVIDIA GPUs
 Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

Mar 17, 2021
A library for high performance deep learning inference on NVIDIA GPUs.
A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

Aug 31, 2022
TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

TensorRT Open Source Software This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. Included are the sources for Tens

Sep 30, 2022
nvidia nvmpi encoder for streamFX and obs-studio (e.g. for nvidia jetson. Requires nvmpi enabled ffmpeg / libavcodec)

nvmpi-streamFX-obs nvidia nvmpi encoder for streamFX and obs-studio (e.g. for nvidia jetson. Requires nvmpi enabled ffmpeg / libavcodec) Purpose This

Jun 25, 2022
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.

iNeural A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms. What is a Neural Network? Work on

Apr 5, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

This is the Vowpal Wabbit fast online learning code. Why Vowpal Wabbit? Vowpal Wabbit is a machine learning system which pushes the frontier of machin

Sep 28, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

Sep 26, 2022
Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

CNTK Chat Windows build status Linux build status The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes

Sep 29, 2022
header only, dependency-free deep learning framework in C++14
header only, dependency-free deep learning framework in C++14

The project may be abandoned since the maintainer(s) are just looking to move on. In the case anyone is interested in continuing the project, let us k

Sep 29, 2022
LibDEEP BSD-3-ClauseLibDEEP - Deep learning library. BSD-3-Clause

LibDEEP LibDEEP is a deep learning library developed in C language for the development of artificial intelligence-based techniques. Please visit our W

Sep 15, 2022
Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Sep 25, 2022
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

Sep 23, 2022
Nimble: Physics Engine for Deep Learning
Nimble: Physics Engine for Deep Learning

Nimble: Physics Engine for Deep Learning

Sep 22, 2022
Deploying Deep Learning Models in C++: BERT Language Model
 Deploying Deep Learning Models in C++: BERT Language Model

This repository show the code to deploy a deep learning model serialized and running in C++ backend.

Mar 24, 2022
TFCC is a C++ deep learning inference framework.

TFCC is a C++ deep learning inference framework.

Aug 11, 2022
AI4Animation: Deep Learning, Character Animation, Control
 AI4Animation: Deep Learning, Character Animation, Control

This project explores the opportunities of deep learning for character animation and control as part of my Ph.D. research at the University of Edinburgh in the School of Informatics, supervised by Taku Komura. Over the last couple years, this project has become a modular and stable framework for data-driven character animation, including data processing, network training and runtime control, developed in Unity3D / Tensorflow / PyTorch.

Sep 26, 2022
KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite English | 简体中文 KSAI Lite是一个轻量级、灵活性强、高性能且易于扩展的深度学习推理框架,底层基于tensorflow lite,定位支持包括移动端、嵌入式以及服务器端在内的多硬件平台。 当前KSAI Lite已经应用在金山office内部业务中,并逐步支持金山

Aug 11, 2022
Deep Learning in C Programming Language. Provides an easy way to create and train ANNs.
Deep Learning in C Programming Language. Provides an easy way to create and train ANNs.

cDNN is a Deep Learning Library written in C Programming Language. cDNN provides functions that can be used to create Artificial Neural Networks (ANN)

Apr 18, 2022