At its recent WWDC 2023 developer conference, Apple presented a number of extensions and updates to its machine learning and vision ecosystem, including updates to its Core ML framework, new features for the Create ML modeling tool, and new vision APIs for image segmentation, animal body pose detection, and 3D human body pose.
A first improvement to Core ML, iOS on-device machine learning framework, comes in the form of a faster inference engine, explains Apple engineer Ben Levine. The speedup does not require models to be re-compiled or any associated code changes. Levine did not provide any figures about the speed improvements and said they will be model and hardware dependent.
Core ML also extends its capabilities to support async workflow with the new Async Prediction API. Previously, Core ML supported batch predictions, which are suitable for cases where there is a known amount of work to do and the inputs for all the component batches can be calculated beforehand. Thanks to the new Async Prediction API, you can now dispatch a prediction to a background thread, which means you can run multiple predictions concurrently. In addition, the API supports the possibility of canceling prediction requests, which can further improve responsiveness in certain cases.
For example, Levine demoed an app that colorizes a list of images using Core ML and showed how the new Async Prediction API improved scroll experience by reducing inference time to about the half. Inference concurrency comes with its own caveats, though. Specifically, Levine mentioned the increased memory requirements due to loading multiple inputs and outputs in memory, which must be addressed, e.g., by limiting the number of concurrent operations that the app carries through.
In addition to Core ML inference capabilities, Apple improved model conversion options as well with a new version of its Core ML Tools which help developers optimize the size and performance of their machine learning models. This can be critical as apps starts integrating ever more and bigger ML models, says Apple. The tool now supports several techniques including weight pruning, quantization, and palletization, which can be applied when training a new model or when converting it from Python.
On the front of modeling, Apple extended Create ML by including support for multilingual BERT embeddings and multi-label image classification.
This year, we designed a new embedding model and trained it on billions of labeled text examples. It's a bidirectional encoder representations from transformers model, or BERT for short.
The new BERT model can boost the accuracy of monolingual text classifiers, too, according to Apple.
Finally, the new Augmentation API is designed to help improve model quality when training data is limited. Data augmentation is a technique used to generate new training examples from existing ones using transformations. The Augmentation API will make it possible for developers to build their own augmentation pipelines by combining different transformation steps. Apples says the Augmentation API uses result builders and will results familiar to developers who have used SwiftUI.
As a final note, Apple added new capabilities to its VisionKit framework, including the ability to detect animal poses and track their joints in real-time individually; to lift subjects from images and separate them from their background; and to distinguish multiple individuals in the same image using the new Segmentation API.
If you are interested in delving into all new ML and vision features in the upcoming iOS 17 release, check out the comprehensive WWDC 2023 session recording list.