Initially available under the ML Kit early access program, Pose Detection is now officially part of ML Kit. The library is capable of tracking the human body, including facial landmarks, hands, and feet.
ML Kit Pose Detection API is based on Google's BlazePose pipeline, which combines computer vision and machine learning to infer 33 two-dimensional body landmarks. This represents a significant improvement over the current standard for body pose, which uses the COCO dataset for keypoint detection, according to Google.
The COCO keypoints only localize to the ankle and wrist points, lacking scale and orientation information for hands and feet, which is vital for practical applications like fitness and dance. The inclusion of more keypoints is crucial for the subsequent application of domain-specific pose estimation models, like those for hands, face, or feet.
BlazePose achieves this result by building on top of the previously available BlazeFace and BlazePalm topologies used to create face and hand models. BlazePose achieves real-time performance on mobile phones when using only CPU inference, while using GPU inference makes it also possible to run subsequent ML models for face or hand tracking.
BlazePose includes two different ML models, a fast model and an accurate model. Both models only support the presence of a single person in a frame and work correctly at distances less than 14 feet (4 meters) and when the head is visible.
With the “Fast” mode enabled, you can expect a frame rate of around 30+ FPS on a modern Android device, such as a Pixel 4 and 45+ FPS on a modern iOS device, such as an iPhone X. With the “Accurate” mode enabled, you can expect more stable x,y coordinates on both types of devices, but a slower frame rate overall.
Google is targeting ML Kit Pose Detection to apps helping to stay active home, for example fitness and yoga trackers. Anyway, its applications can go well beyond that and include augmented reality, sign language recognition, full-body gesture control, and more.
Interested developers can head to Google sample apps for Android and iOS to get a taste of this new technology.