MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

In face effect module I can 3d data as glasses.pbtxt facepaint.pngblob glasses.pngblob.

I am trying to add few more models to experiment but i couldnt find any documenation or information of pngblob data. It seems like that using glasses.pbtxt 3d model is generated at runtime and glasses.pngblob is getting used as texture. Can you please clear that is it right and how is it happening

Ques 1- Can you please provide any documentation of pngblob datatype. How can i create new 3d model (pngblob / binarypb ) to render on face. Most common format of 3d model data are OBJ, FBX, etc. Is there any way to convert these format of 3d data to binarypb / pngblob?

Ques 2- it is mentioned in gl_animation_overlay_calculator.cc that .obj.uuu can be created using the mentioned SimpleObjEncryptor but I couldn't find that. can you please specify where to find that ?

ANIMATION_ASSET (String, required):
//     Path of animation file to load and render. Should be generated by
//     //java/com/google/android/apps/motionstills/SimpleObjEncryptor with
//     --compressed_mode=true.  See comments and documentation there for more
//     information on custom .obj.uuu file format.

Hello all,

I have a project using Mediapipe Hands on iOS and I've been trying to update from the tfjs model to the new Mediapipe api but even when I enable WebGL2, it still fails to work. I've made sure I'm asking permission using navigator.getmedia properly. Wondering if anyone has any ideas on what's going wrong.

Here's the codepen that I'm testing: https://codepen.io/aionkov/pen/MWjEqWa

Here's the console:

[Warning] I1223 11:05:16.032000 1 gl_context_webgl.cc:146] Successfully created a WebGL context with major version 3 and handle 3 (hands_solution_wasm_bin.js, line 9) [Warning] I1223 11:05:16.034000 1 gl_context.cc:340] GL version: 3.0 (OpenGL ES 3.0 (WebGL 2.0)) (hands_solution_wasm_bin.js, line 9) [Warning] W1223 11:05:16.034000 1 gl_context.cc:794] Drishti OpenGL error checking is disabled (hands_solution_wasm_bin.js, line 9) [Warning] E1223 11:05:16.711000 1 calculator_graph.cc:775] INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9) [Warning] Calculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50'] (hands_solution_wasm_bin.js, line 9) [Warning] F1223 11:05:16.712000 1 solutions_wasm.embind.cc:585] Check failed: ::util::OkStatus() == (graph_->WaitUntilIdle()) (OK vs. INTERNAL: CalculatorGraph::Run() failed in Run: (hands_solution_wasm_bin.js, line 9) [Warning] Calculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50 [type.googleapis.com/mediapipe.StatusList='\n\x84\x02\x08\r\x12\xff\x01\x43\x61lculator::Open() for node "handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator" failed: [GL_INVALID_FRAMEBUFFER_OPERATION]: The framebuffer object is not complete.: glCreateShader in third_party/tensorflow/lite/delegates/gpu/gl/gl_shader.cc:50']) (hands_solution_wasm_bin.js, line 9) [Warning] *** Check failure stack trace: *** (hands_solution_wasm_bin.js, line 9) [Warning] undefined (hands_solution_wasm_bin.js, line 9) [Error] Unhandled Promise Rejection: RuntimeError: abort(undefined) at [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:67558 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:67737 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:41049 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands_solution_wasm_bin.js:9:179948 [email protected][wasm code]

.wasm-function[10471]@[wasm code] .wasm-function[10466]@[wasm code] .wasm-function[10461]@[wasm code] .wasm-function[10458]@[wasm code] .wasm-function[10474]@[wasm code] .wasm-function[515]@[wasm code] .wasm-function[502]@[wasm code]

[email protected][wasm code] [native code] SolutionWasm$send https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:33:352 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:10:295 https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:11:90 [email protected]://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/hands.js:22:322 [email protected][native code] (evaluating 'new WebAssembly.RuntimeError(what)') (anonymous function) (hands_solution_wasm_bin.js:9:41099) promiseReactionJob

Hello, I'm having problems using mediapipe on my raspberry pi 3. Using "import mediapipe" gives no error, however using "mp_drawing = mp.solutions.drawing_utils" (for example) gives out the following error message: ModuleNotFoundError: No module named 'mediapipe.python._framework_bindings'

My installation method was:

1. sudo apt install ffmpeg python3-opencv python3-pip ;
2. sudo apt install libxcb-shm0 libcdio-paranoia-dev libsdl2-2.0-0 libxv1 libtheora0 libva-drm2 libva-x11-2 libvdpau1 libharfbuzz0b libbluray2 libatlas-base-dev libhdf5-103 libgtk-3-0 libdc1394-22 libopenexr25 ;
3. sudo pip3 uninstall mediapipe-rpi3 .

I'm using a raspberry pi 3b with debian bullseye (32 bit), my python version is 3.9.2 and the opencv version is 4.2.1. P.S. the file name is "maos.py".

Does anyone know what might be causing this error? (I annexed an image of the error aswell for clarity )

Hello,

I found out about Mediapipe after seeing Google's blog post regarding hand tracking. Currently, I am working on using Mediapipe to build a cross platform interface using gestures to control multiple systems. I am using the Desktop CPU example as a base for how to move forward, and I have successfully retrieved the hand landmarks. I just want to ensure that I am retrieving them in the most efficient and proper way.

The process I use is as follows:

1. Create a listener of class OutputStreamPoller which listens for the hand_landmarks output stream in the HandLandmark subgraph.
2. If there is an available packet, load the packet into a variable of class mediapipe::Packet using the .Next() method of the OutputStreamPoller class.
3. Use the .Get() method of the Packet class and load into another variable called hand_landmarks.
4. Loop through the variable and retrieve the x, y, and z coordinates and place them into a vector for processing.

Is this process correct or is there a better way to go about retrieving the coordinates of the hand landmarks?

I have additional questions, but I am unsure if I should place them in a separate issue. I will ask them here but please let me know if I should open a separate issue.

1. In the hand tracking examples, only a single hand is to be detected. How would I alter the build such that it can detect multiple hands (specifically 2)?
2. How would I enable the desktop implementations of hand tracking such that they can capture depth (similar to how the android/ios 3D builds can output z coordinates)?

Hello! I'm using the Holistic web solution in a WebRTC streaming application.

We've occasionally seen Holistic processing crash in a way that seems indicative of a memory leak:

I'm currently unsure about whether the Holistic processing crash is the cause or a symptom of another issue. I've done some memory profiling, but haven't found a reliable way of reproducing it yet.

Since I don't have too much visibility into the Mediapipe JS internals, I was just hoping to get a temperature check on whether the Mediapipe team thinks:

1. This issue could be related to the Mediapipe internals / you've seen it before in other contexts
2. This is definitely not a Mediapipe issue and likely a bug in my application logic

Basically just trying to determine where to invest additional debugging / investigation efforts.

FWIW I also came across this thread https://bugs.chromium.org/p/chromium/issues/detail?id=1174675 which indicates there could be some memory leak issues in Chromium that can affect use cases like WebRTC, but the rate of leakage described there seems too slow compared to what I'm perceiving in my application.

Thanks in advance for your help! Sorry that I'm not able to provide any more details besides that single stack trace. Please let me know if you need any additional information and I'd be happy to circle back with it.

I am failing to find any kind of documentation or example that would explain the exact definition/behavior of the estimated Z coordinates returned by the hand tracking graph.

We're able to successfully extract the landmark data as X, Y and Z coordinates. The X and Y coordinates are clearly normalized but the Z coordinates appear to take values to which I have no reference (they are not normalized, they are sometimes negative, sometimes positive and don't appear to adhere to any coherent scale. Clear is: They are most likely relative to each other.

Could somebody shine some light on the estimated Z coordinates - especially the scale they adhere to?

System information (Please provide as much relevant information as possible)

OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): MediaPipe version: v0.8.6 Bazel version:bazel 4.1.0 Solution (e.g. FaceMesh, Pose, Holistic): HANDTRACK aar Programming Language and version ( e.g. C++, Python, Java): Java Describe the expected behavior:

My goal was to create an aar to using the handtracking aar on Android Studio with gradle

When I replace the .aar file in the libs folder (from the face_detection and multi_hand_tracking demo/example projects) with the one that I generated following the steps in https://google.github.io/mediapipe/getting_started/android_archive_library.html

I get the following crash at System.loadLibrary("mediapipe_jni");

java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__subtf3" referenced by "/data/app/~~IXWF_H6noRNss6OBiB8kZQ==/com.my.mediapipe.apps.myapplication-Eds2VNkZzhvjhYw-zhjNaw==/lib/arm64/libmediapipe_jni.so"...

So i Switch to OpenCV 4. but i need opencv-4.5.1
sed -i -e 's:3.4.3/opencv-3.4.3:4.5.1/opencv-4.5.1:g' WORKSPACE sed -i -e 's:libopencv_java3:libopencv_java4:g' third_party/opencv_android.BUILD

i use opencv-4.5.1 is different from opencv-4.0.1 in exzample . But it still reports an same error ! nm -D libmediapipe_jni.so |grep subtf3 the symbol is still stay in.

Has anyone had this error when importing the mediapipe library?

AttributeError: partially initialized module 'mediapipe' has no attribute 'solutions'(most likely due to a circular import)

import cv2
import mediapipe as mp
mp_drawing = mp.solutions.drawing_utils
mp_face_mesh = mp.solutions.face_mesh

I don't need the visualizer I just want to be able to run the MediaPipe Hands with Multi-hand support in my webapp. From my understanding the code is compiled into wasm and then run from a webapp. How would I include the Hands with Multi-hand support app in my on web application?

I am trying to test the given model in my sample android application. When trying to load the model i face this issue:

java.lang.IllegalStateException: Internal error: Unexpected failure when preparing tensor allocations: Encountered unresolved custom op: Convolution2DTransposeBias.Node number 165 (Convolution2DTransposeBias) failed to prepare.

Code: AssetFileDescriptor fileDescriptor = activity.getAssets().openFd("palm_detection.tflite");

i 've trained yolo model for object detect, and i want to integrating the model into mediapipe, whether it support or not, if support integrating, how to do, could you give me advice, thanks a lot.

System information:

• MediaPipe v0.8.7.1 :
• Bazel version: 5.2.0
• XCode 14.0.1
• iOS 16 (iPhone 7)

Describe the problem:

I have faced with the problem of building static framework with Mediapipe.

Working solution for dynamic version:

apple_binary(
    name = "face_landmarks_detector_ios",
    deps = [
        ":face_landmarks_detector_ios_lib",
        "@opencv_framework_ios16//:OpencvFramework"
    ],
    data = [
        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
        "//mediapipe/modules/face_landmark:face_landmark.tflite",
    ],
    platform_type = "ios",
    binary_type = "dylib"
)

Not working solution for static framework:

ios_static_framework(
    name = "face_landmarks_detector_ios_static",
    minimum_os_version = "16.0",
    platform_type = "ios",
    families = ["iphone"],
    deps = [
        ":face_landmarks_detector_ios_lib",
        "@opencv_framework_ios16//:OpencvFramework",
    ],
    resources = [
        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
        "//mediapipe/modules/face_landmark:face_landmark.tflite",
    ],
)

And here is this lib to see that is inside 'face_landmarks_detector_ios_lib':

cc_library(
    name = "face_landmarks_detector_ios_lib",
    srcs = [
        "face_landmarks_detector.cc",
    ],
    hdrs = [
        "export.h",
        "face_landmarks_detector.h"
    ],
    copts = ["-std=c++17"],
    deps = [
        "//mediapipe/framework:calculator_framework",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/graphs/face_mesh/subgraphs:face_renderer_gpu",
        "//mediapipe/calculators/core:flow_limiter_calculator",
        ":face_landmarks_detector_front_gpu",
        "//mediapipe/calculators/core:constant_side_packet_calculator",
        ":desktop_live_calculators",
    ] + select({
        "//mediapipe:ios_i386": [],
        "//mediapipe:ios_x86_64": [],
        "//conditions:default": [
            ":mobile_calculators",
            "//mediapipe/framework/formats:landmark_cc_proto",
        ],
    }),
    data = [
        "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
        "//mediapipe/modules/face_landmark:face_landmark.tflite",
    ],
    defines = ["COMPILING_FACE_DETECTOR_API"],
    alwayslink = 1,
)

Problem:

After linking it to the iOS app with XCode I will get issues related to tensor flow models. Especially for:

         "//mediapipe/modules/face_detection:face_detection_short_range.tflite",
        "//mediapipe/modules/face_landmark:face_landmark.tflite",

Somehow ios_static_framework is ignoring this files, rather than dynamic build working good without any issues. Environment and setting of the project is the same for dynamic and static lib.

Here is text from issue when app is running on the device (iPhone 7 iOS 16):

2023-01-08 13:28:55.216549+0200 app-api[13255:7401738] Error NOT_FOUND: ValidatedGraphConfig Initialization failed.
No registered object with name: FaceLandmarksDetectorFrontCpu; Unable to find Calculator "FaceLandmarksDetectorFrontCpu"
No registered object with name: FaceRendererCpu; Unable to find Calculator "FaceRendererCpu"

Questing is: How to link this tflite files to ios_static_framework?

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in Mediapipe): Yes and no (issue exists for both)
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Windows 11
• MediaPipe version: Latest
• Bazel version: N/A
• Solution (e.g. FaceMesh, Pose, Holistic): Hands
• Programming Language and version ( e.g. C++, Python, Java): JavaScript

Describe the expected behavior: MediaPipe inference in JavaScript slows down drastically when there are other 3D applications running in the foreground. Normally the tracking is ~24 FPS, but if I launch a resource-intensive 3D application/game (say Elden Ring), the tracking FPS drops drastically to 3~4 FPS. Switching the foreground application to the MediaPipe JavaScript solution immediately improves the FPS. This is on a i9-9900K + RTX 2080 Ti.

Notably, the Python solution does not have this problem. Tracking FPS remains stable no matter what application is running in the foreground. I am using official demo code for both.

Standalone code you may have used to try to get what you need : I have tried to build the MediaPipe JavaScript solution as an electron app and manually raise its process priority in Task Manager (or through SetPriorityClass Win32 API) to "Realtime". I have also tried to run the electron app with administrator privileges. However, both didn't work.

If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook: The JS hands demo reproduces this.

Other info / Complete Logs : N/A

Please make sure that this is a build/installation issue and also refer to the troubleshooting documentation before raising any issues.

System information (Please provide as much relevant information as possible)

• OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4):
• Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0):
• Programming Language and version ( e.g. C++ 14, Python 3.6, Java ):
• Installed using virtualenv? pip? Conda? (if python):
• MediaPipe version:
• Bazel version:
• XCode and Tulsi versions (if iOS):
• Android SDK and NDK versions (if android):
• Android AAR ( if android):
• OpenCV version (if running on desktop):

Describe the problem:

Provide the exact sequence of commands / steps that you executed before running into the problem:

Complete Logs: Include Complete Log information or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:

1. Added MPPClassificationResult, MPPCategory and their helpers
2. Removed MPPClassifierOptions and helpers since we have decided to in-line them in text classifier options to maintain consistency with Java and Python

I want to detect the occlusion in coming in front of the face region in a video. It will take video frames as input and for each frame it will give an output whether the face the in the current frame is occluded or not.