• Machine learning
  • MOBILE

Apple Core ML: Leveraging the Power of Machine Learning for Mobile

Michael Girnyak
10 Sep 2021
5 min
Apple Core ML: Leveraging the Power of Machine Learning for Mobile

Gone are the days when machine learning (ML) was something that only seasoned developers and software engineers could understand. Modern technologies can now train software to recognize objects, images, speech, gestures, and even emotions — effectively taking mobile apps to another level. 

Several frameworks on the market now cut the time needed to train ML models from weeks to hours. Each has its pros and cons. iOS machine learning stands out as one of the best, as Apple’s Core ML framework offers a convenient drag-and-drop interface to streamline the training process and harness the power of ML models for embedded apps.

Want to know more about tech trends?
Sign up to be the first who receive our expert articles

    Success!
    Thank you

    What is Core ML?

    Core ML is Apple’s foundational machine learning framework for integrating ML models into iOS applications. It’s fully compatible with all Apple products and offers fast performance and easy integration of trained ML models. 

    An iPhone app uses Core ML APIs and stored data to analyze information, detect patterns, and make predictions. This means that inferences are made directly on the mobile device. Models are also trained on the user’s device.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 1

    A Core ML model is the result of applying an ML algorithm to training data. You can build a model using the Create ML app bundled with Xcode. In this case, no conversion is required, as models trained with Create ML are already in a format suitable for use in an app. There are also Core ML tools and third-party ML libraries available for converting your model into Core ML format.

    The framework is built on three main technologies: 

    • The CPU (Central Processing Unit): This works with memory-intensive procedures such as natural language processing (NLP).
    • The GPU (Graphics Processing Unit): This is good for computation-intensive processes, such  as working with images and object detection. 
    • The ANE (Apple Neural Engine): The framework was designed to accelerate neural networks. 

    One of the good things about Core ML is that it allows you to switch between the CPU, GPU, and ANE on the fly. This optimizes on-device performance and means you don’t have to decide which one to run for each model.

    Since its introduction in 2017, Core ML has gone through three iterations as Apple has upgraded the framework and added new tools. It’s currently in its fourth version, known as Core ML 4.0.

    Core ML architecture 

    Core ML supports four training domains that define its architecture: vision, NLP, speech recognition, and sound analysis.

    Here’s what that looks like:

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 2
    • Vision carries out facial recognition, text and object detection, image segmentation, and feature tracking. The vision library uses the iPhone camera as input and does all the heavy lifting from the AVfoundation multimedia framework to the CVPixelBuffer raw image format.
    • NLP offers natural language processing and speech recognition with the help of features such as tokenization, language identification, and picking up names from text. 
    • Speech recognition and sound analysis track voice patterns and train models to identify different voices.
    • There’s also a Gameplay Kit, which develops and organizes your game logic with the help of decision trees.

    Core ML models 

    The list of Core ML support features and models grows each year. Here are the main categories:

    • Image models take data from pictures as input and detect and segment objects.
    • Text models process text data, classify and tag words, predict sentiments from sentences, and get trained to answer questions.
    • Video models categorize movement and detect poses.
    • Sound models classify audio data.
    • Tabular models import tabular data, estimate continuous values (regression), and categorize this data.

    App integration and conversion from third-party libraries

    If you’re using third-party training libraries, you can easily convert them to Core ML with the coremltools Python package.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 3

    Every year, the ML team at Apple extends their list of supported libraries. In 2020, they announced support for the most popular neural network libraries, TensorFlow and PyTorch, and turned coremltools into a one-stop shop for converting to Core ML.

    That same year, the company introduced a single converter stack called Model Intermediate Language (MIL). MIL was designed to streamline the conversion process and make it easy to add support for new frameworks. It unifies the stack by providing a standard interface for capturing information from different frameworks. 

    To integrate a Core ML model into your app, follow these steps:

    1. Add a model to your Xcode project.
    2. Create the model in code.
    3. Get input values to pass to the model.
    4. Start using the model to generate predictions.
    5. Build and run a Core ML app.

    Use cases

    Since its introduction, iOS machine learning has enjoyed wide implementation. Here are some successful use cases:

    Action classification with Create ML

    At WWDC 2020, Apple introduced its Action Classifier model and the Action & Vision app. The app uses Object Detection and Action Classification in Create ML along with the Body Pose Estimation, Trajectory Detection, and Contour Detection features in the Vision framework. The app analyzes your movements during physical activity and helps improve your training performance.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 4

    Emotion classification using Core ML and ARKit

    The video below shows a developer experimenting with an app powered by Core ML and ARKit, a framework based on facial recognition technology. The model is trained to recognize emotions and showcases the power of the Core ML Vision framework.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 5

    Image processing with Style Art

    Style Art is a library that processes images using Core ML with a set of pre-trained ML models and converts them to ArtStyle.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 6

    Image processing with Image Colorizer

    Image Colorizer colorizes grayscale images using the Core ML, Vision, and Core Image frameworks.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 7

    Image recognition with SeeFood

    SeeFood is an app that uses CoreML to detect various dishes.

    Apple Core ML: Leveraging the Power of Machine Learning for Mobile - photo 8

    Pros and cons of using Core ML  

    Apple is constantly upgrading its machine learning technology, and Core ML is getting better every year as it offers users more and more opportunities to implement it in apps.

    If you’re still not sure whether it’s for you, check out this list of the main pros and cons of Core ML implementation:

    Pros:

    • Data privacy ensured by on-device implementation
    • Low latency and near real-time results
    • Compatibility with the majority of popular ML models
    • Automatic switching between CPU, GPU, and ANE
    • Offline availability

    Cons:

    • Increased app size
    • Heavy power consumption

    Book a strategy session

    Get actionable insights for your product



      Thank you for reaching out, !

      Make sure to check for details.