MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

layout title nav_order
default
Home
1

MediaPipe


Live ML anywhere

MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

accelerated.png cross_platform.png
End-to-End acceleration: Built-in fast ML inference and processing accelerated even on common hardware Build once, deploy anywhere: Unified solution works across Android, iOS, desktop/cloud, web and IoT
ready_to_use.png open_source.png
Ready-to-use solutions: Cutting-edge ML solutions demonstrating full power of the framework Free and open source: Framework and solutions both under Apache 2.0, fully extensible and customizable

ML solutions in MediaPipe

Face Detection Face Mesh Iris Hands Pose Holistic
face_detection face_mesh iris hand pose hair_segmentation
Hair Segmentation Object Detection Box Tracking Instant Motion Tracking Objectron KNIFT
hair_segmentation object_detection box_tracking instant_motion_tracking objectron knift
Android iOS C++ Python JS Coral
Face Detection
Face Mesh
Iris
Hands
Pose
Holistic
Hair Segmentation
Object Detection
Box Tracking
Instant Motion Tracking
Objectron
KNIFT
AutoFlip
MediaSequence
YouTube 8M

See also MediaPipe Models and Model Cards for ML models released in MediaPipe.

MediaPipe in Python

MediaPipe offers customizable Python solutions as a prebuilt Python package on PyPI, which can be installed simply with pip install mediapipe. It also provides tools for users to build their own solutions. Please see MediaPipe in Python for more info.

MediaPipe on the Web

MediaPipe on the Web is an effort to run the same ML solutions built for mobile and desktop also in web browsers. The official API is under construction, but the core technology has been proven effective. Please see MediaPipe on the Web in Google Developers Blog for details.

You can use the following links to load a demo in the MediaPipe Visualizer, and over there click the "Runner" icon in the top bar like shown below. The demos use your webcam video as input, which is processed all locally in real-time and never leaves your device.

visualizer_runner

Getting started

Learn how to install MediaPipe and build example applications, and start exploring our ready-to-use solutions that you can further extend and customize.

The source code is hosted in the MediaPipe Github repository, and you can run code search using Google Open Source Code Search.

Publications

Videos

Events

Community

Alpha disclaimer

MediaPipe is currently in alpha at v0.7. We may be still making breaking API changes and expect to get to stable APIs by v1.0.

Contributing

We welcome contributions. Please follow these guidelines.

We use GitHub issues for tracking requests and bugs. Please post questions to the MediaPipe Stack Overflow with a mediapipe tag.

Owner
Comments
  • How to render different face effect

    How to render different face effect

    In face effect module I can 3d data as glasses.pbtxt facepaint.pngblob glasses.pngblob.

    I am trying to add few more models to experiment but i couldnt find any documenation or information of pngblob data. It seems like that using glasses.pbtxt 3d model is generated at runtime and glasses.pngblob is getting used as texture. Can you please clear that is it right and how is it happening

    Ques 1- Can you please provide any documentation of pngblob datatype. How can i create new 3d model (pngblob / binarypb ) to render on face. Most common format of 3d model data are OBJ, FBX, etc. Is there any way to convert these format of 3d data to binarypb / pngblob?

    Ques 2- it is mentioned in gl_animation_overlay_calculator.cc that .obj.uuu can be created using the mentioned SimpleObjEncryptor but I couldn't find that. can you please specify where to find that ?

    ANIMATION_ASSET (String, required):
    //     Path of animation file to load and render. Should be generated by
    //     //java/com/google/android/apps/motionstills/SimpleObjEncryptor with
    //     --compressed_mode=true.  See comments and documentation there for more
    //     information on custom .obj.uuu file format.
    
  • Mediapipe CodePens don't run on iOS Safari

    Mediapipe CodePens don't run on iOS Safari

    Hello all,

    I have a project using Mediapipe Hands on iOS and I've been trying to update from the tfjs model to the new Mediapipe api but even when I enable WebGL2, it still fails to work. I've made sure I'm asking permission using navigator.getmedia properly. Wondering if anyone has any ideas on what's going wrong.

    Here's the codepen that I'm testing: https://codepen.io/aionkov/pen/MWjEqWa

    Here's the console:

    [Warning] I1223 11:05:16.032000 1 gl_context_webgl.cc:146] Successfully created a WebGL context with major version 3 and handle 3 (hands_solution_wasm_bin.js, line 9) [Warning] I1223 11:05:16.034000 1 gl_context.cc:340] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)) (hands_solution_wasm_bin.js, line 9) [Warning] W1223 11:05:16.034000 1 gl_context.cc:794] Drishti OpenGL error checking is disabled (hands_solution_wasm_bin.js, line 9) [Warning] E1223 11:05:16.711000 1 calculator_graph.cc:775] INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9) [Warning] Calculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50'] (hands_solution_wasm_bin.js, line 9) [Warning] F1223 11:05:16.712000 1 solutions_wasm.embind.cc:585] Check failed: ::util::OkStatus() == (graph_->WaitUntilIdle()) (OK vs. INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9) [Warning] Calculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50']) (hands_solution_wasm_bin.js, line 9) [Warning] *** Check failure stack trace: *** (hands_solution_wasm_bin.js, line 9) [Warning] undefined (hands_solution_wasm_bin.js, line 9) [Error] Unhandled Promise Rejection: RuntimeError: abort(undefined) at [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:67558 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:67737 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:41049 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:179948 [email protected][wasm code]

    .wasm-function[10471]@[wasm code] .wasm-function[10466]@[wasm code] .wasm-function[10461]@[wasm code] .wasm-function[10458]@[wasm code] .wasm-function[10474]@[wasm code] .wasm-function[515]@[wasm code] .wasm-function[502]@[wasm code]

    [email protected][wasm code] [native code] SolutionWasm$send https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:33:352 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:10:295 https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:11:90 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:22:322 [email protected][native code] (evaluating 'new WebAssembly.RuntimeError(what)') (anonymous function) (hands_solution_wasm_bin.js:9:41099) promiseReactionJob

  • ModuleNotFoundError: No module named 'mediapipe.python._framework_bindings' on Raspberry Pi 3

    ModuleNotFoundError: No module named 'mediapipe.python._framework_bindings' on Raspberry Pi 3

    Hello, I'm having problems using mediapipe on my raspberry pi 3. Using "import mediapipe" gives no error, however using "mp_drawing = mp.solutions.drawing_utils" (for example) gives out the following error message: ModuleNotFoundError: No module named 'mediapipe.python._framework_bindings'

    My installation method was:

    1. sudo apt install ffmpeg python3-opencv python3-pip ;
    2. sudo apt install libxcb-shm0 libcdio-paranoia-dev libsdl2-2.0-0 libxv1 libtheora0 libva-drm2 libva-x11-2 libvdpau1 libharfbuzz0b libbluray2 libatlas-base-dev libhdf5-103 libgtk-3-0 libdc1394-22 libopenexr25 ;
    3. sudo pip3 uninstall mediapipe-rpi3 .

    I'm using a raspberry pi 3b with debian bullseye (32 bit), my python version is 3.9.2 and the opencv version is 4.2.1. P.S. the file name is "maos.py".

    Does anyone know what might be causing this error? (I annexed an image of the error aswell for clarity Error )

  • Accessing landmarks, tracking multiple hands, and enabling depth on desktop

    Accessing landmarks, tracking multiple hands, and enabling depth on desktop

    Hello,

    I found out about Mediapipe after seeing Google's blog post regarding hand tracking. Currently, I am working on using Mediapipe to build a cross platform interface using gestures to control multiple systems. I am using the Desktop CPU example as a base for how to move forward, and I have successfully retrieved the hand landmarks. I just want to ensure that I am retrieving them in the most efficient and proper way.

    The process I use is as follows:

    1. Create a listener of class OutputStreamPoller which listens for the hand_landmarks output stream in the HandLandmark subgraph.
    2. If there is an available packet, load the packet into a variable of class mediapipe::Packet using the .Next() method of the OutputStreamPoller class.
    3. Use the .Get() method of the Packet class and load into another variable called hand_landmarks.
    4. Loop through the variable and retrieve the x, y, and z coordinates and place them into a vector for processing.

    Is this process correct or is there a better way to go about retrieving the coordinates of the hand landmarks?

    I have additional questions, but I am unsure if I should place them in a separate issue. I will ask them here but please let me know if I should open a separate issue.

    1. In the hand tracking examples, only a single hand is to be detected. How would I alter the build such that it can detect multiple hands (specifically 2)?
    2. How would I enable the desktop implementations of hand tracking such that they can capture depth (similar to how the android/ios 3D builds can output z coordinates)?
  • Temperature check on possible memory leak in Holistic JS solution

    Temperature check on possible memory leak in Holistic JS solution

    Hello! I'm using the Holistic web solution in a WebRTC streaming application.

    We've occasionally seen Holistic processing crash in a way that seems indicative of a memory leak: image

    I'm currently unsure about whether the Holistic processing crash is the cause or a symptom of another issue. I've done some memory profiling, but haven't found a reliable way of reproducing it yet.

    Since I don't have too much visibility into the Mediapipe JS internals, I was just hoping to get a temperature check on whether the Mediapipe team thinks:

    1. This issue could be related to the Mediapipe internals / you've seen it before in other contexts
    2. This is definitely not a Mediapipe issue and likely a bug in my application logic

    Basically just trying to determine where to invest additional debugging / investigation efforts.

    FWIW I also came across this thread https://bugs.chromium.org/p/chromium/issues/detail?id=1174675 which indicates there could be some memory leak issues in Chromium that can affect use cases like WebRTC, but the rate of leakage described there seems too slow compared to what I'm perceiving in my application.

    Thanks in advance for your help! Sorry that I'm not able to provide any more details besides that single stack trace. Please let me know if you need any additional information and I'd be happy to circle back with it.

  • Hand tracking landmarks - Z value range

    Hand tracking landmarks - Z value range

    I am failing to find any kind of documentation or example that would explain the exact definition/behavior of the estimated Z coordinates returned by the hand tracking graph.

    We're able to successfully extract the landmark data as X, Y and Z coordinates. The X and Y coordinates are clearly normalized but the Z coordinates appear to take values to which I have no reference (they are not normalized, they are sometimes negative, sometimes positive and don't appear to adhere to any coherent scale. Clear is: They are most likely relative to each other.

    Could somebody shine some light on the estimated Z coordinates - especially the scale they adhere to?

  • base on opencv-4.5.1  , Use  build aar, crash ,dlopen failed: cannot locate symbol

    base on opencv-4.5.1 , Use build aar, crash ,dlopen failed: cannot locate symbol "__subtf3"

    System information (Please provide as much relevant information as possible)

    OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): MediaPipe version: v0.8.6 Bazel version:bazel 4.1.0 Solution (e.g. FaceMesh, Pose, Holistic): HANDTRACK aar Programming Language and version ( e.g. C++, Python, Java): Java Describe the expected behavior:

    My goal was to create an aar to using the handtracking aar on Android Studio with gradle

    When I replace the .aar file in the libs folder (from the face_detection and multi_hand_tracking demo/example projects) with the one that I generated following the steps in https://google.github.io/mediapipe/getting_started/android_archive_library.html

    I get the following crash at System.loadLibrary("mediapipe_jni");

    java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__subtf3" referenced by "/data/app/~~IXWF_H6noRNss6OBiB8kZQ==/com.my.mediapipe.apps.myapplication-Eds2VNkZzhvjhYw-zhjNaw==/lib/arm64/libmediapipe_jni.so"...

    So i Switch to OpenCV 4. but i need opencv-4.5.1
    sed -i -e 's:3.4.3/opencv-3.4.3:4.5.1/opencv-4.5.1:g' WORKSPACE sed -i -e 's:libopencv_java3:libopencv_java4:g' third_party/opencv_android.BUILD

    i use opencv-4.5.1 is different from opencv-4.0.1 in exzample . But it still reports an same error ! nm -D libmediapipe_jni.so |grep subtf3 the symbol is still stay in.

  • How do you use MediaPipe on the web in your own web app?

    How do you use MediaPipe on the web in your own web app?

    I don't need the visualizer I just want to be able to run the MediaPipe Hands with Multi-hand support in my webapp. From my understanding the code is compiled into wasm and then run from a webapp. How would I include the Hands with Multi-hand support app in my on web application?

  • Unable to load the hand detection model

    Unable to load the hand detection model

    I am trying to test the given model in my sample android application. When trying to load the model i face this issue:

    java.lang.IllegalStateException: Internal error: Unexpected failure when preparing tensor allocations: Encountered unresolved custom op: Convolution2DTransposeBias.Node number 165 (Convolution2DTransposeBias) failed to prepare.

    Code: AssetFileDescriptor fileDescriptor = activity.getAssets().openFd("palm_detection.tflite");

  • AttributeError: module 'mediapipe' has no attribute 'solutions'

    AttributeError: module 'mediapipe' has no attribute 'solutions'

    Has anyone had this error when importing the mediapipe library?

    AttributeError: partially initialized module 'mediapipe' has no attribute 'solutions'(most likely due to a circular import)

    import cv2
    import mediapipe as mp
    mp_drawing = mp.solutions.drawing_utils
    mp_face_mesh = mp.solutions.face_mesh
    
  • create face_mesh obj file in python

    create face_mesh obj file in python

    I'm using the face_mesh solution in Python which outputs only the 3D landmarks. I'm looking to turn that into a .obj file but the face_mesh solution API does not provide mesh triangle indices, mesh normals, or mesh UV texture coordinates. Is it possible to do this in Python?

  • Error in box_tracker_calculator.cc

    Error in box_tracker_calculator.cc

    I build an Android AAR for [Box Tracking] and I got this error in box_tracker_calculator.cc

    Box at time 1 already present; ignoring Box at time 2 already present; ignoring Box at time 3 already present; ignoring

    https://github.com/google/mediapipe/blob/cc6a2f7af65977248b2a15f471503da2832f583a/mediapipe/calculators/video/box_tracker_calculator.cc#L381-L388

    And the number keeps increasing.

    I can not understand these codes. Can someone explain this error part code?

  • Running graph directly on OpenGL texture

    Running graph directly on OpenGL texture

    I'm working on a module that wraps Mediapipe and runs an input image(s) using its graphs. The module is used by Android application using JNI and Mediapipe is ran natively using C++.

    Since we're using Android, we use a SurfaceTexture and would, ideally want to use that texture directly as input to the mediapipe graph, instead of copying it.

    I'm looking for a few days now for some practical example or explanation on how to achieve this. At the moment I'm trying to ignore the Android part, as I read that converting the SurfaceTexture to proper OpenGL texture adds another stage to the mix, so I'm running on linux.

    My main issue is all the examples I saw regarding GPU and OpenGL backend use the ImageFrame->GpuBuffer->GlTexture flow where I would like to skip the first two stages and have directly the GlTexture fed into the pipeline.

  • How to compute nose, left head, right head transform matrix

    How to compute nose, left head, right head transform matrix

    Background: Hi, i am glad to know this amazing ai repo. Recently, i use mediapipe as ai engine to inference 3d face landkmark and filament as my 3d rendering engine. I use these two engine to implement ar 3d sticker Question: After i read the blog about MediaPipe 3D Face Transform , i know the pos matrix is used to find the best suitable transform matrix to transform Canonical Face Modelto runtime face model. So is there any possible i can compute the nose, lefthead, righthead transform matrix? I need these trasnform matrix to position different part of 3d sticker, such as a nose decoration.

  • Hardware for handtracking

    Hardware for handtracking

    Hi I am looking for single board computer to run Mediapipe to perform live hand tracking. Some suggestions are raspberry pi 4 or nvidia Jetson Nano. Which of the two can achieve higher fps? Any other board suggestion is welcome Thanks!

  • pose_landmarks emit an empty packet while the corrsponding pose_world_landmarks packet isn't empty

    pose_landmarks emit an empty packet while the corrsponding pose_world_landmarks packet isn't empty

    Please make sure that this is a bug and also refer to the troubleshooting, FAQ documentation before raising any issues.

    System information (Please provide as much relevant information as possible)

    • Have I written custom code (as opposed to using a stock example script provided in MediaPipe): Yes
    • OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Linux archlinux 5.18.16-arch1-1
    • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: -
    • Browser and version (e.g. Google Chrome, Safari) if the issue happens on browser: -
    • Programming Language and version ( e.g. C++, Python, Java): C++
    • MediaPipe version: v0.8.10.2
    • Bazel version (if compiling from source): 5.2.0
    • Solution ( e.g. FaceMesh, Pose, Holistic ): Pose
    • Android Studio, NDK, SDK versions (if issue is related to building in Android environment): -
    • Xcode & Tulsi version (if issue is related to building for iOS): -

    Describe the current behavior: When trying to get output from pose_landmarks stream using OutputStreamPoller#Next, sometimes the output packet is empty while there's a corresponding non-empty output packet in pose_world_landmarks.

    Describe the expected behavior: When the output packet of pose_world_landmarks is not empty, the output packet of pose_landmarks is also not empty.

    Standalone code to reproduce the issue: Diff: https://github.com/homuler/mediapipe/commit/feabd089138188ff4604a925f56f70991fef5736

    Essentially, I modified demo_run_graph_main_gpu.cc to get outputs from pose_landmarks and pose_world_landmarks synchronously.

      ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller pose_landmarks_poller,
                       graph.AddOutputStreamPoller("pose_landmarks", true));
      ASSIGN_OR_RETURN(mediapipe::OutputStreamPoller pose_world_landmarks_poller,
                       graph.AddOutputStreamPoller("pose_world_landmarks", true));
    
      // After starting CalculatorGraph
      while (graph_frames) {
        // ...
    
        mediapipe::Packet pose_landmarks_packet;
        mediapipe::Packet pose_world_landmarks_packet;
        if (!pose_landmarks_poller.Next(&pose_landmarks_packet) || !pose_world_landmarks_poller.Next(&pose_world_landmarks_packet)) {
          break;
        }
        if (pose_landmarks_packet.IsEmpty() && !pose_world_landmarks_packet.IsEmpty()) {
          LOG(WARNING) << "pose_landmarks is empty while pose_world_landmarks exists";
          LOG(WARNING) << "pose_landmarks queue: " << pose_landmarks_poller.QueueSize() << ", pose_world_landmarks queue: " << pose_world_landmarks_poller.QueueSize();
        }
        // ...
      }
    

    Command:

    bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 //mediapipe/examples/desktop/pose_tracking:pose_tracking_gpu
    GLOG_logtostderr=1 ./bazel-bin/mediapipe/examples/desktop/pose_tracking/pose_tracking_gpu --calculator_graph_config_file=mediapipe/graphs/pose_tracking/pose_tracking_gpu.pbtxt
    

    Other info / Complete Logs : I'm not sure if it's a bug, but at least I find it a bit odd. It seems that it occurs when roi and filtered_visibility are calculated almost at the same time.

    I guess the following is occurring.

    Calculation Steps

    NOTE:

    • Step B and Step C are concurrent.
    • Step B is dependent on Step A.
    • Step C is dependent on Step A.
    • Step D is dependent on Step B and Step C.
    • SwitchDemuxCalculator will be triggered whenever its input is updated, but its container node won't be necessarily triggered (it is usually triggered only when all the inputs are given).

    Step A

    1. An input image is added to input_video at X.
    2. poselandmarkgpu__ImagePropertiesCalculator adds its output to image_size.
    3. The new image_size packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator.
    4. PoseLandmarkByRoiGpu adds its output to unfiltered_pose_landmarks and unfiltered_auxiliary_landmarks.
      • In practice, outputs are not added to unfiltered_pose_landmarks and unfiltered_auxiliary_landmarks simultaneously, but the order does not matter here.

    Step B

    1. The new unfiltered_pose_landmarks packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__SwitchDemuxCalculator
    2. poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__SwitchDemuxCalculator triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2.
    3. poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2 adds its output to filtered_visibility.
    4. The new filtered_visibility packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator.
    5. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator relays filtered_visibility to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2.

    Step C

    1. The new unfiltered_auxiliary_landmarks packet triggers poselandmarkgpu__poselandmarkfiltering__LandmarksToDetectionCalculator
    2. poselandmarkgpu__poselandmarkfiltering__LandmarksToDetectionCalculator adds its output to aux_detection.
    3. The new aux_detection packet triggers poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator.
    4. poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator adds its output to roi.
    5. The new roi packet triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator.
    6. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator relays roi to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2.

    Step D

    1. After finishing Step B and Step C, poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 is triggered.
    2. The output of poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 triggers poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator.
    3. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator relays its input to filtered_landmarks (== pose_landmarks).

    The point is that Step B and Step C is concurrent. As long as filtered_visibility is calculated and relayed to LandmarksSmoothingCalculator before roi is calculated (or vice versa), this issue does not occur. However, if filtered_visibility is calculated and roi is calculated before filtered_visibility is relayed to LandmarksSmoothingCalculator, the following occurs.

    Step B and C

    1. poselandmarkgpu__poselandmarkfiltering__switchcontainer_1__VisibilitySmoothingCalculator_2 adds its output to filtered_visibility (Step B # 3).
    2. InputStreamHandler pops the new filtered_visibility packet from the queue and next_timestamp_bound_ for filtered_visibility is updated.
    3. The new filtered_visibility packet is sent to the corresponding input streams.
    4. poselandmarkgpu__poselandmarkfiltering__AlignmentPointsRectsCalculator adds its output to roi (Step C # 4).
    5. InputStreamHandler pops the new roi packet from the queue and next_timestamp_bound_ for roi is updated.
    6. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator is triggered by the new filtered_visibility packet.
    7. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchDemuxCalculator relays each input packet to poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2.
    8. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 is triggered while the roi is empty.
    9. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 does not output, but updates its output timestamp bound.
    10. poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__SwitchMuxCalculator updates the timestamp bound of filtered_landmarks.
    11. If a client is polling the output (using OutputStreamPoller#Next), this change of the timestamp bound is detected and an empty packet is returned to the client.

    If the 5th step occurs after the 7th step finishes, the timestamp bound of roi won't be updated in the 7th step and poselandmarkgpu__poselandmarkfiltering__switchcontainer_2__LandmarksSmoothingCalculator_2 won't be triggered in the 8th step and this issue does not occur.

    This issue also does not occur for world_landmarks (filtered_world_landmarks) because the corresponding SwitchContainer has only one input stream and its container node will be triggered at most once.

    Workaround

    After all, this issue occurs because SwitchDemuxCalculator triggers LandmarksSmoothingCalculator before all the dependent inputs are given. I think we can avoid it by using DefaultInputStreamHandler for the SwitchDemuxCalculator instead of ImmediateInputStreamHandler. When the following patch is applied (though I/F should be improved), this issue seems to be resolved.

    diff --git a/mediapipe/framework/tool/switch_container.proto b/mediapipe/framework/tool/switch_container.proto
    index a9c2d90..f38de33 100644
    --- a/mediapipe/framework/tool/switch_container.proto
    +++ b/mediapipe/framework/tool/switch_container.proto
    @@ -27,4 +27,7 @@ message SwitchContainerOptions {
     
       // Use DefaultInputStreamHandler for muxing & demuxing.
       optional bool synchronize_io = 5;
    +
    +  optional bool synchronize_demux_io = 6;
    +  optional bool synchronize_mux_io = 7;
     }
    diff --git a/mediapipe/framework/tool/switch_demux_calculator.cc b/mediapipe/framework/tool/switch_demux_calculator.cc
    index c4352c8..68b56bc 100644
    --- a/mediapipe/framework/tool/switch_demux_calculator.cc
    +++ b/mediapipe/framework/tool/switch_demux_calculator.cc
    @@ -115,7 +115,7 @@ absl::Status SwitchDemuxCalculator::GetContract(CalculatorContract* cc) {
         }
       }
       auto& options = cc->Options<mediapipe::SwitchContainerOptions>();
    -  if (!options.synchronize_io()) {
    +  if (!options.synchronize_io() && !options.synchronize_demux_io()) {
         cc->SetInputStreamHandler("ImmediateInputStreamHandler");
       }
       cc->SetProcessTimestampBounds(true);
    diff --git a/mediapipe/framework/tool/switch_mux_calculator.cc b/mediapipe/framework/tool/switch_mux_calculator.cc
    index 9982ae4..a44bc00 100644
    --- a/mediapipe/framework/tool/switch_mux_calculator.cc
    +++ b/mediapipe/framework/tool/switch_mux_calculator.cc
    @@ -154,7 +154,7 @@ absl::Status SwitchMuxCalculator::Process(CalculatorContext* cc) {
       // Update the input channel index if specified.
       channel_index_ = tool::GetChannelIndex(*cc, channel_index_);
     
    -  if (options_.synchronize_io()) {
    +  if (options_.synchronize_io() || options_.synchronize_mux_io()) {
         // Start with adding input signals into channel_history_ and packet_history_
         if (cc->Inputs().HasTag("ENABLE") &&
             !cc->Inputs().Tag("ENABLE").IsEmpty()) {
    diff --git a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
    index bb3665f..27d141c 100644
    --- a/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
    +++ b/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt
    @@ -100,6 +100,7 @@ node {
       options: {
         [mediapipe.SwitchContainerOptions.ext] {
           enable: true
    +      synchronize_demux_io: true
           contained_node: {
             calculator: "LandmarksSmoothingCalculator"
             options: {
    
UE4 MediaPipe plugin

UE4 MediaPipe plugin Platforms: Win64 2D features: Face, Iris, Hands, Pose, Holistic 3D features: Face Mesh, World Pose Demo video: https://www.youtub

Aug 9, 2022
Example Qt application that demonstrates how to integrate Mediapipe
Example Qt application that demonstrates how to integrate Mediapipe

Mediapipe integration to Qt application example Example on how to integrate mediapipe as a dynamic library into Qt applicaton on Linux. Resulting appl

Aug 3, 2022
A mediapipe-hand demo infer by ncnn
A mediapipe-hand demo infer by ncnn

The Android demo of Mediapipe Hand infer by ncnn Please enjoy the mediapipe hand demo on ncnn You can try this APK demo https://pan.baidu.com/s/1ArAMH

Aug 4, 2022
The purpose of this project is to apply mediapipe to more AI chips.
The purpose of this project is to apply mediapipe to more AI chips.

1.About This Project Our Official Website: www.houmo.ai Who We Are: We are Houmo - A Great AI Company. We wish to change the world with unlimited comp

Aug 10, 2022
Native runtime package for MediaPipe.NET.

MediaPipe.NET.Runtime Native library package for MediaPipe.NET. This is the first half of the port of MediaPipeUnityPlugin, in order to use MediaPipe

Jul 26, 2022
C++ Live Toolkit are tools subset used to perform on-the-fly compilation and running of cpp code

C++ Live Toolkit CLT (C++ Live Toolkit) is subset of tools that are very light in size, and maintained to help programmers in compiling and executing

Jan 4, 2022
Anomaly Detection on Dynamic (time-evolving) Graphs in Real-time and Streaming manner
Anomaly Detection on Dynamic (time-evolving) Graphs in Real-time and Streaming manner

Anomaly Detection on Dynamic (time-evolving) Graphs in Real-time and Streaming manner. Detecting intrusions (DoS and DDoS attacks), frauds, fake rating anomalies.

Aug 12, 2022
CNStream is a streaming framework for building Cambricon machine learning pipelines
CNStream is a streaming framework for building Cambricon  machine learning pipelines

CNStream is a streaming framework for building Cambricon machine learning pipelines

Aug 4, 2022
Advent of Code 2021 optimized solutions in C++

advent2021-fast These solutions are a work in progress. Advent of Code 2021 optimized C++ solutions. Here are the timings from an example run on an i9

Aug 9, 2022
E-Box solutions Batch 2023
E-Box solutions Batch 2023

EBox Codes for the Assessments in the E-Box platform General Information E-Box is a learning platform helps various students to develop their coding s

Jul 15, 2022
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

ONNX Runtime is a cross-platform inference and training machine-learning accelerator compatible with deep learning frameworks, PyTorch and TensorFlow/Keras, as well as classical machine learning libraries such as scikit-learn, and more.

Aug 17, 2022
A Cross platform implement of Wenet ASR. It's based on ONNXRuntime and Wenet. We provide a set of easier APIs to call wenet models.

RapidASR: a new member of RapidAI family. Our visio is to offer an out-of-box engineering implementation for ASR. A cpp implementation of recognize-on

Aug 11, 2022
Insight Toolkit (ITK) is an open-source, cross-platform toolkit for N-dimensional scientific image processing, segmentation, and registration
 Insight Toolkit (ITK) is an open-source, cross-platform toolkit for N-dimensional scientific image processing, segmentation, and registration

ITK: The Insight Toolkit C++ Python Linux macOS Windows Linux (Code coverage) Links Homepage Download Discussion Software Guide Help Examples Issue tr

Aug 8, 2022
A C++-based, cross platform ray tracing library
A C++-based, cross platform ray tracing library

Visionaray A C++ based, cross platform ray tracing library Getting Visionaray The Visionaray git repository can be cloned using the following commands

Aug 10, 2022
RapidOCR - A cross platform OCR Library based on PaddleOCR & OnnxRuntime
RapidOCR - A cross platform OCR Library based on PaddleOCR & OnnxRuntime

RapidOCR (捷智OCR) 简体中文 | English 目录 RapidOCR (捷智OCR) 简介 近期更新 ?? 2021-12-18 update 2021-11-28 update 2021-11-13 update 2021-10-27 update 2021-09-13 upda

Aug 16, 2022
ClanLib is a cross platform C++ toolkit library.

ClanLib ClanLib is a cross platform toolkit library with a primary focus on game creation. The library is Open Source and free for commercial use, und

Aug 5, 2022
Gesture Recognition Toolkit (GRT) is a cross-platform, open-source, C++ machine learning library designed for real-time gesture recognition.

Gesture Recognition Toolkit (GRT) The Gesture Recognition Toolkit (GRT) is a cross-platform, open-source, C++ machine learning library designed for re

Aug 5, 2022
The Forge Cross-Platform Rendering Framework PC Windows, Linux, Ray Tracing, macOS / iOS, Android, XBOX, PS4, PS5, Switch, Quest 2
The Forge Cross-Platform Rendering Framework PC Windows, Linux, Ray Tracing, macOS / iOS, Android, XBOX, PS4, PS5, Switch, Quest 2

The Forge is a cross-platform rendering framework supporting PC Windows 10 / 7 with DirectX 12 / Vulkan 1.1 with DirectX Ray Tracing API DirectX 11 Fa

Aug 7, 2022
A Cross-Platform(Web, Android, iOS) app to Generate Faces of People (These people don't actually exist) made using Flutter.
A Cross-Platform(Web, Android, iOS) app to Generate Faces of People (These people don't actually exist) made using Flutter.

?? ?? Flutter Random Face Generator A flutter app to generate random faces. The Generated faces do not actually exist in real life (in other words you

Aug 9, 2022