Code accompanying our SIGGRAPH 2021 Technical Communications paper "Transition Motion Tensor: A Data-Driven Approach for Versatile and Controllable Agents in Physically Simulated Environments"

SIGGRAPH ASIA 2021 Technical Communications
Transition Motion Tensor: A Data-Driven Framework for Versatile and Controllable Agents in Physically Simulated Environments

TMT

This paper proposes the Transition Motion Tensor, a data-driven framework that creates physically accurate and novel transitions outside of the motion dataset. It enables the character to grow new motions efficiently and robustly without modifying existing motions. Given several physics-based controllers specializing in different motions, the tensor serves as a guideline for switching between controllers. By querying the tensor for the best transitions, we can create a unified controller capable of producing novel transitions with various behaviors, such as slowing down before higher jumps or jumping immediately for better responsiveness. We apply our framework on both quadrupeds and bipeds, perform quantitative and qualitative evaluations on transition quality, and demonstrate its capability of tackling complex motion planning problems while following user control directives.

Project page: https://inventec-ai-center.github.io/projects/TMT_2021/index.html
Youtube: https://youtu.be/9NzRSZyAOiY
Paper: https://arxiv.org/pdf/2111.15072.pdf

Installation

Linux (recommended: Ubuntu 18.04)

  1. Install dependencies
sudo apt-get update
sudo apt-get install -y vim cmake wget git tar unzip build-essential clang llvm python-dev freeglut3-dev libglew-dev swig python3-pip
  1. Clone the repository
git clone --recursive https://github.com/inventec-ai-center/transition_motion_tensor.git
  1. Install Eigen (version: 3.4.0)
cd ~
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.bz2
tar -xf eigen-3.4.0.tar.bz2
cd eigen-3.4.0
mkdir build_cmake
cd build_cmake
cmake ..
sudo make install
  1. Install Bullet (version: 2.88)
cd ~
wget https://github.com/bulletphysics/bullet3/archive/2.88.zip
unzip 2.88.zip
cd bullet3-2.88/
vim build_cmake_pybullet_double.sh
# replace -DUSE_DOUBLE_PRECISION=ON with -DUSE_DOUBLE_PRECISION=OFF
./build_cmake_pybullet_double.sh
cd build_cmake
sudo make install
  1. Install Open MPI (version: 1.10.7)
cd ~
wget https://download.open-mpi.org/release/open-mpi/v1.10/openmpi-1.10.7.tar.gz
gunzip -c openmpi-1.10.7.tar.gz | tar xf -
cd openmpi-1.10.7
./configure --prefix=/usr/local
sudo make all install -j8
ldconfig
  1. Install Miniconda
cd ~
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod 777 Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
# restart bash to activate conda
conda create --name env_tmt python=3.7
conda activate env_tmt
  1. Install python packages
cd {TMT_REPO}/TMT # {TMT_REPO} is the path where the TMT repository is located
pip install --upgrade pip
pip install -r requirements.txt
  1. Modify the Makefile in TMT/CarlCore/ by specifying the following

    • EIGEN_DIR: Eigen include directory
    • BULLET_INC_DIR: Bullet source directory
    • PYTHON_INC: python include directory
    • PYTHON_LIB: python lib directory
  2. Compile CarlCore

cd {TMT_REPO}/TMT/CarlCore # {TMT_REPO} is the path where the TMT repository is located
make python -j8

How To Use

Running the code

For convenience, we include the pre-trained models under TMT/data/policies and precomputed transition tensors under TMT/data/trans-tensors. To run individual template controllers and unified controllers please follow the instructions below.

  • Template controller
# dog-pace
python TMT_Runner.py --arg_file args/run_dog3d_template_controller_pace_args.txt

# dog-trot
python TMT_Runner.py --arg_file args/run_dog3d_template_controller_trot_args.txt

# dog-canter
python TMT_Runner.py --arg_file args/run_dog3d_template_controller_canter_args.txt

# dog-jump
python TMT_Runner.py --arg_file args/run_dog3d_template_controller_jump_args.txt

# humanoid-walk
python TMT_Runner.py --arg_file args/run_humanoid3d_template_controller_walk_args.txt

# humanoid-jog
python TMT_Runner.py --arg_file args/run_humanoid3d_template_controller_jog_args.txt

# humanoid-run
python TMT_Runner.py --arg_file args/run_humanoid3d_template_controller_run_args.txt
  • Unified controller
# dog (included pace, trot, canter, and jump motions)
python TMT_Runner.py --arg_file args/run_dog3d_unified_controller_args.txt

# humanoid (included walk, jog, and run motions)
python TMT_Runner.py --arg_file args/run_humanoid3d_unified_controller_args.txt

Training Template Controller

  • For dog characters, please follow the instructions from [carl-siggraph2020]
  • For humanoid characters, please follow the instructions from [DeepMimic]

Building Transition Tensor

  1. Collecting transition samples
    • --num_workers : Number of parallel workers
    • --num_trajs: Number of transition samples to collect
    • --input_filename: Input argument file
    • --output_path: Output path to store transition samples
# dog
python TMT_Collect_Transitions.py --num_workers 10 --num_trajs 500000 --input_filename args/record_dog3d_transitions_args.txt --output_path "./output/dog_transitions"

# humanoid
python TMT_Collect_Transitions.py --num_workers 10 --num_trajs 500000 --input_filename args/record_humanoid3d_transitions_args.txt --output_path "./output/humanoid_transitions"
  1. Building transition tensor
    • --input_path: Input path where transition samples are located
    • --alive_thres: Alive rate threshold (default: 0.9)
    • --w_time: Transition duration weight (default: 1.0)
    • --w_stability: Stability weight (default: 0.015)
    • --dt_stability: Stability neighborhood range (default: 0.01)
# dog
python TMT_Build_Transition_Tensor.py --input_path "./output/dog_transitions" --alive_thres 0.9

# humanoid
python TMT_Build_Transition_Tensor.py --input_path "./output/humanoid_transitions" --alive_thres 0.85

Plotting Transition Tensor

We provided a jupyter notebook file TMT/plot_transition_tensor.ipynb to visualize the transition tensor. Visit the file for more information.

Note: You'll need to install python package jupyter before using the jupyter notebook file plot_transition_tensor.ipynb.

Interface

General

  • Right-click and drag will pan the camera
  • Left-click and drag will apply a force on the character at a particular location
  • Scroll wheel will zoom in/out
  • Pressing 'r' will reset the episode
  • Pressing 'l' will reload the argument file and rebuild everything
  • Pressing 'g' will start/stop video recording (required ffmpeg)
  • Pressing space will pause/resume the simulation
  • Pressing '>' will step the simulation one step at a time
  • Pressing 'x' will pelt the character with random boxes

Unified Controller

Dog

  • Pressing 'q' will switch to canter motion
  • Pressing 'e' will switch to jump motion
  • Pressing 'a' or 'd' will switch to pace motion and control the heading
  • Pressing 'w' will switch to trot motion

Humanoid

  • Pressing '1' will switch to walk motion
  • Pressing '2' will switch to jog motion
  • Pressing '3' will switch to run motion

Known Issues and Solutions

Q: Crash with message "util/MathUtil.cpp:175: static tMatrix cMathUtil::RotateMat(const tVector &, double): Assertion std::abs(axis.squaredNorm() - 1) < 0.0001' failed." after throwing random boxes.

A: You can adjust the equation std::abs(axis.squaredNorm() - 1) < 0.0001 in file DeepMimic/DeepMimicCore/util/MathUtil.cpp, from value 0.0001 to 0.1, and re-compile CarlCore.

Q: Getting the error message ImportError: libBulletDynamics.so.2.88: cannot open shared object file: No such file or directory while running the codes.

A: Check libBulletDynamics.so.2.88 is presented under /usr/local/lib/ by running ls -l /usr/local/lib/ | grep libBullet, and then include it in the PATH by running export LD_LIBRARY_PATH=/usr/local/lib/.

Acknowledgement

We thank AI4Animation for providing the quadruped animation system and DeepMimic for providing the motion imitation framework and pre-trained humanoid policies.

Copyright Information

This project is under the terms of the MIT license. It is only for research or education purposes, and not freely available for commercial use or redistribution. This intellectual property belongs to the Inventec Corporation. Licensing is possible if you want to use the code for commercial use. For scientific use, please reference this repository together with the relevant publication.

@article{TMT2021,
	author 	   = {Soeseno, Jonathan Hans and Luo, Ying-Sheng and Chen, Trista Pei-Chun and Chen, Wei-Chao},
	title 	   = {Transition Motion Tensor: A Data-Driven Approach for Versatile and Controllable Agents in Physically Simulated Environments},
	journal    = {SIGGRAPH Asia 2021 Technical Communications},
	year 	   = {2021},
	numpages   = {4},
}
Similar Resources

Repository for the taproot-based rewrite of our 2021 development platform

Repository for the taproot-based rewrite of our 2021 development platform

taproot-mdev2021 This is a blank project fully configured for use of Taproot. It is designed to be a starting point for your own RoboMaster software p

Dec 28, 2022

Repository Containing the Code associated with the Paper: "Learning High-Speed Flight in the Wild"

Repository Containing the Code associated with the Paper:

Learning High-Speed Flight in the Wild This repo contains the code associated with the paper Learning Agile Flight in the Wild. For more information,

Jan 3, 2023

Example code for the research paper "Masked Software Occlusion Culling"; implements an efficient alternative to the hierarchical depth buffer algorithm.

MaskedOcclusionCulling This code accompanies the research paper "Masked Software Occlusion Culling", and implements an efficient alternative to the hi

Dec 22, 2022

Code for the paper Succinct k-mer Set Representations Using Subset Rank Queries on the Spectral Burrows-Wheeler Transform (SBWT)

SBWT This is the code for the paper Succinct k-mer Set Representations Using Subset Rank Queries on the Spectral Burrows-Wheeler Transform (SBWT). The

Jun 18, 2022

Source code from 068A, our 64k demo presented at Syntax 2017

068A source code dump This is a dump of the source code for the engine, graphics tool and player for 068A, our 64k demo released at Syntax 2017. It's

Jan 2, 2023

Source Code and Embedded Design of Our Factory Robot at AUTCup 2015 Competitions

Source Code and Embedded Design of Our Factory Robot at AUTCup 2015 Competitions

AUTCup 2015 @Factory Robot This repository contains the source code and embedded design of our @Factory robot at AUTCup 2015 competitions. The robot w

Oct 20, 2021

Code for our ECE445/ME470 design: Wireless Charging Table with Automatic Alignment

Code for our ECE445/ME470 design: Wireless Charging Table with Automatic Alignment

Qi Wireless Charging Table with Automatic Alignment Code for ECE445/ME470 Senior Design Project SP21 at ZJUI. Team 24: Kaiwen Cao, Tianyi Han, Tingkai

May 1, 2022

DigiMahal is the First Project of Our Team in Sharif University of Technology for Basics of Programming That in this Code we Implemented an Online Shop like DigiKala

DigiMahal is the First Project of Our Team in Sharif University of Technology for Basics of Programming That in this Code we Implemented an Online Shop like DigiKala

DigiMahal is the First Project of Our Team in Sharif University of Technology for Basics of Programming That in this Code we Implemented an Online Shop like DigiKala.

Jul 23, 2022

Proof-of-concept implementation for the paper "Osiris: Automated Discovery of Microarchitectural Side Channels" (USENIX Security'21)

Osiris This repository contains the implementation of the Osiris framework discussed in the research paper "Osiris: Automated Discovery of Microarchit

Nov 11, 2022
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies
Supplementary code for SIGGRAPH 2021 paper: Discovering Diverse Athletic Jumping Strategies

SIGGRAPH 2021: Discovering Diverse Athletic Jumping Strategies project page paper demo video Prerequisites Important Notes We suspect there are bugs i

Dec 6, 2022
Sandbox binary and source code for the Siggraph 2017 paper "Water Wave Packets" by Stefan Jeschke (NVIDIA) and Chris Wojtan (IST Austria)

----------------------------- Manual for wave packet viewer ----------------------------- System requirements: Windows8/8.1/10 with DirectX runtime e

Nov 28, 2022
The artifact associated with our ISSTA 2021 paper "Seed Selection for Successful Fuzzing"

Seed Selection for Successful Fuzzing The artifact associated with our ISSTA 2021 paper "Seed Selection for Successful Fuzzing". While our primary art

Dec 7, 2022
This repo contains source code of our paper presented in IROS2021 "Single-Shot is Enough: Panoramic Infrastructure Based Calibration of Multiple Cameras and 3D LiDARs"
This repo contains source code of our paper presented in IROS2021

Single-Shot is Enough: Panoramic Infrastructure Based Calibration of Multiple Cameras and 3D LiDARs Updates [2021/09/01] first commit, source code of

Dec 19, 2022
Interactive-hex-meshing - Source code for "Interactive All-Hex Meshing via Cuboid Decomposition [SIGGRAPH Asia 2021]".
Interactive-hex-meshing - Source code for

Interactive All-Hex Meshing via Cuboid Decomposition Video demonstration This repository contains an interactive software to the PolyCube-based hex-me

Dec 5, 2022
This is our take on the digitalisation of the board game "b00le0", where you can play versus our AI, or against one of your friends in an online match.
This is our take on the digitalisation of the board game

This is our take on the digitalisation of the board game "b00le0", where you can play versus our AI, or against one of your friends in an online match.

Dec 8, 2022
Enable LoRaWAN communications on your Raspberry Pi Pico or any RP2040 based board. 📡

pico-lorawan Enable LoRaWAN communications on your Raspberry Pi Pico or any RP2040 based board using a Semtech SX1276 radio module. Based on the Semte

Dec 30, 2022
TulipCell is an Excel add-in providing 100+ technical analysis indicators.

Tulip Cell Introduction Tulip Cell is an Excel add-in that provides the technical analysis functions from the Tulip Indicators library. Building Build

Nov 8, 2022
Technical Analysis Indicator Function Library in C

Tulip Indicators Introduction Tulip Indicators is a library of technical analysis functions written in ANSI C. Lots of information is available on the

Jan 7, 2023
A customizable hardware prefetching framework using online reinforcement learning as described in the MICRO 2021 paper by Bera and Kanellopoulos et al.
A customizable hardware prefetching framework using online reinforcement learning as described in the MICRO 2021 paper by Bera and Kanellopoulos et al.

A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning Table of Contents What is Pythia? About the Framework Prerequisites

Jan 1, 2023