Apple Core ML: Leveraging the Power of Machine Learning for Mobile

Gone are the days when machine learning (ML) was something that only seasoned developers and software engineers could understand. Modern technologies and frameworks, like iOS machine learning, can now train software to recognize objects, images, speech, gestures, and even emotions — effectively taking mobile apps to another level.

Several frameworks on the market now cut the time needed to train ML models from weeks to hours. Each has its pros and cons. iOS machine learning stands out as one of the best, as this Apple Core ML framework offers a convenient drag-and-drop interface to streamline the training process and harness the power of ML models for embedded apps.

What is Core ML?

So what is Core ML and how do we use it? Core ML is Apple’s foundational machine learning framework for integrating ML models into iOS applications. It’s fully compatible with all Apple products and offers fast performance and easy integration of trained ML models. 

An iPhone app uses Core ML APIs and stored data to analyze information, detect patterns, and make predictions. This means that inferences are made directly on the mobile device. Models are also trained on the user’s device.

A Core ML model is a result of applying an ML algorithm to training data. You can build a model using the Create ML app bundled with Xcode. In this case, no conversion is required, as models trained with Create ML are already in a format suitable for use in an app. There are also Core ML tools and third-party ML libraries available for converting your model into Core ML format.

The framework is built on three main technologies: 

• The CPU (Central Processing Unit): This works with memory-intensive procedures such as natural language processing (NLP).
• The GPU (Graphics Processing Unit): This is good for computation-intensive processes, such as working with images and object detection. 
• The ANE (Apple Neural Engine): The framework was designed to accelerate neural networks. 

One of the good things about Core ML is that it allows you to switch between the CPU, GPU, and ANE on the fly. This optimizes on-device performance and means you don’t have to decide which one to run for each model.

Since its introduction in 2017, Apple Core ML has gone through three iterations as the company  has upgraded the framework and added new tools. It’s currently in its fourth version, known as Core ML 4.0.

Apple Core ML architecture

Core ML supports four training domains that define its architecture: vision, NLP, speech recognition, and sound analysis.

Here’s what that looks like:

• Vision carries out facial recognition, text and object detection, image segmentation, and feature tracking. The vision library uses the iPhone camera as input and does all the heavy lifting from the AVfoundation multimedia framework to the CVPixelBuffer raw image format.
• NLP offers natural language processing and speech recognition with the help of features such as tokenization, language identification, and picking up names from text. 
• Speech recognition and sound analysis track voice patterns and train models to identify different voices.
• There’s also a Gameplay Kit, which develops and organizes your game logic with the help of decision trees.

Core ML models 

The list of Core ML support features and models grows each year. Here are the main categories:

• Image models take data from pictures as input and detect and segment objects.
• Text models process text data, classify and tag words, predict sentiments from sentences, and get trained to answer questions.
• Video models categorize movement and detect poses.
• Sound models classify audio data.
• Tabular models import tabular data, estimate continuous values (regression), and categorize this data.

App integration and conversion from third-party libraries

If you’re using third-party training libraries, you can easily convert them to Core ML with the coremltools Python package.

Every year, the ML team at Apple extends its list of supported libraries. In 2020, they announced support for the most popular neural network libraries, TensorFlow and PyTorch, and turned coremltools into a one-stop shop for converting to Core ML.

That same year, the company introduced a single converter stack called Model Intermediate Language (MIL). MIL was designed to streamline the conversion process and make it easy to add support for new frameworks. It unifies the stack by providing a standard interface for capturing information from different frameworks. 

To integrate an Apple Core ML model into your app, follow these steps:

1. Add a model to your Xcode project.
2. Create the model in code.
3. Get input values to pass to the model.
4. Start using the model to generate predictions.
5. Build and run a Core ML app.

Use cases

Since its introduction, iOS machine learning has enjoyed wide implementation. Here are some successful use cases:

Action classification with Create ML

At WWDC 2020, Apple introduced its Action Classifier model and the Action & Vision app. The app uses Object Detection and Action Classification in Create ML along with the Body Pose Estimation, Trajectory Detection, and Contour Detection features in the Vision framework. The app analyzes your movements during physical activity and helps improve your training performance.

Emotion classification using Core ML and ARKit

The video below shows a developer experimenting with an app powered by Core ML and ARKit, a framework based on facial recognition technology. The model is trained to recognize emotions and showcases the power of the Core ML Vision framework.

Image processing with Style Art

Style Art is a library that processes images using Core ML with a set of pre-trained ML models and converts them to ArtStyle.

Image processing with Image Colorizer

Image Colorizer colorizes grayscale images using the Core ML, Vision, and Core Image frameworks.

Image recognition with SeeFood

SeeFood is an app that uses CoreML to detect various dishes.

Pros and cons of using Core ML  

Apple is constantly upgrading its machine learning technology, and Core ML is getting better every year as it offers users more and more opportunities to implement it in apps.

If you’re still not sure whether it’s for you, check out this list of the main pros and cons of Core ML implementation:

Pros:

• Data privacy ensured by the on-device implementation
• Low latency and near real-time results
• Compatibility with the majority of popular ML models
• Automatic switching between CPU, GPU, and ANE
• Offline availability

Cons:

• Increased app size
• Heavy power consumption