Embracing AI: Enhancing mobile apps with Apple Intelligence & Google AI

The rapid advancements in artificial intelligence (AI) are revolutionising the landscape of mobile apps. Recently announced, Apple Intelligence and Google’s AI innovations, including Gemini, are paving the way for more intuitive and context-aware user experiences. As app developers and product owners, understanding and leveraging these capabilities can help position your app to meet evolving user expectations and stay competitive in the market.

Introduction: A Brief History of AI in Mobile Apps

Over the past decade, AI has gradually woven itself into the fabric of mobile technology, transforming how we interact with our devices. Initially, AI’s role in mobile apps was limited to basic functionalities, such as speech and image recognition. Neural networks enabled apps to recognise spoken words and identify objects in images, but these capabilities were often rudimentary and lacked the sophistication we see today.

One of the early milestones in AI integration was the development of recommendation engines by giants like Netflix and Amazon. These systems utilised machine learning algorithms to analyse user behaviour and provide personalised content suggestions, significantly enhancing user engagement and satisfaction. This marked a shift towards more personalised and adaptive user experiences, setting a precedent for the use of AI in consumer applications. Check out, The Science Behind the Netflix Algorithms That Decide What You'll Watch Next, from Wired in 2013.

In addition to personalisation, AI has been used in various other domains within mobile apps over the past decade. Some notable areas include:

• Natural Language Processing (NLP): AI-driven chatbots and virtual assistants like Siri, Google Assistant, and Alexa have become commonplace, providing users with hands-free, conversational interfaces.
  
  

• Predictive Text and Autocorrect: AI has improved the accuracy and efficiency of text input on mobile devices, making typing faster and reducing errors.
  
  

• Augmented Reality (AR): Apps like Pokémon GO and IKEA Place use AI to overlay digital information on the real world, enhancing the user experience with interactive and immersive elements.
  
  

• Health and Fitness: AI-powered apps track user activity, provide personalised workout plans, and monitor health metrics, contributing to a more proactive approach to health management.
  
  

• Security and Fraud Detection: AI algorithms detect suspicious activity and potential fraud in financial transactions, providing an additional layer of security for mobile users.

The use of AI in these areas has not only improved functionality but also set the stage for more advanced integrations. Today, with Apple Intelligence and Google’s Gemini models, we are witnessing a new era where AI capabilities are more deeply embedded into mobile ecosystems, offering unprecedented levels of personalisation, efficiency, and user engagement.

What is Apple Intelligence?

Apple Intelligence is a comprehensive personal intelligence system designed to enhance the functionality and usability of Apple devices. It leverages generative models to process natural language, understand images, and utilise personal context to perform tasks across apps. A key differentiator is Apple’s commitment to privacy, ensuring most processing happens on-device, with external computational power utilised via Apple’s Private Cloud Compute only when necessary.

Image sourced from Apple

Generative Models vs. Diffusion Models: Understanding the Difference

In the realm of artificial intelligence, particularly within the context of Apple Intelligence and Google AI, two significant types of models are frequently mentioned: generative models and diffusion models. Both play crucial roles in enhancing the functionality and user experience of mobile apps, but they operate in fundamentally different ways.

Generative models are a class of AI models designed to generate new data samples from a given distribution. These models learn the underlying patterns and structures in the input data and use this knowledge to create new, similar data. Applications include generating realistic images, coherent text, or even synthetic voices. For example, a generative model might be trained on a dataset of human faces and then used to generate entirely new, realistic-looking faces that do not correspond to any real person.

Diffusion models, on the other hand, are a newer class of generative models that work by iteratively transforming noise into structured data through a diffusion process. They typically involve two key phases: a forward diffusion process that adds noise to data, and a reverse diffusion process that removes noise to generate new data. Key characteristics include noise transformation and stages of refinement, with applications similar to other generative models but particularly noted for generating highly detailed and realistic outputs. For example, a diffusion model might take an initial noisy image and, through a series of steps, gradually transform it into a clear and detailed image of a landscape.

Key Differences

• Generation Process: Generative models directly learn to generate new data from the learned distribution of the training data, while diffusion models involve a two-phase process of adding noise to data and then refining it back to its original form or a new form, often resulting in higher quality and more detailed outputs.
  
  

• Noise Handling: Generative models typically do not explicitly involve noise in their generation process, except for certain types like GANs where noise is used as input to generate data. Diffusion models explicitly use noise as part of their generation mechanism, making them robust in creating high-fidelity outputs from noisy inputs.
  
  

• Application Suitability: Generative models are versatile and can be used for a broad range of applications including text, images, and audio. Diffusion models excel in applications requiring high-quality image generation and have shown superior performance in creating detailed and realistic visuals.Understanding these differences helps developers and product owners choose the right type of model for their specific needs, ensuring optimal performance and user experience in their AI-enhanced mobile apps.

  
  

Apple’s strategy focuses on integrating AI as a feature that enhances the holistic user experience rather than creating standalone AI products. This approach contrasts with other tech giants like Google, which also aim to embed AI deeply within their ecosystems but have different approaches and strengths. A recent article by WIRED, Apple Proved That AI Is a Feature, Not a Product, highlighted Apple’s seamless infusion of AI into existing apps and OS features. 

Google AI: Gemini and Beyond

Google’s AI advancements, particularly with the Gemini models, also represent significant strides in AI integration. Gemini models, including Gemini Nano, Pro, and Ultra, are designed to handle a wide range of tasks and operate efficiently on mobile devices. These models support multimodal capabilities, allowing them to process and understand different types of information such as text, code, audio, image, and video. Google explains some of this in their article, Top 3 Updates for Building with AI on Android at Google I/O ‘24.

Differences Between Gemini Nano, Pro, and Ultra

Gemini Nano

Gemini Nano is designed specifically for mobile devices, optimised to provide AI capabilities on smartphones and tablets. It operates directly on the device, ensuring low latency and real-time processing. One of its key advantages is offline functionality, allowing it to perform AI tasks without needing an active internet connection, which is crucial for applications where connectivity may be limited or unreliable. Additionally, Gemini Nano is optimised for power efficiency to ensure that AI processing does not significantly drain the device’s battery life. While it handles simpler, less computationally intensive AI tasks effectively, it is suitable for everyday applications like voice recognition, basic image processing, and real-time translation. Examples of its applications include voice assistants that need to respond quickly without sending data to the cloud, camera apps that perform real-time enhancements or object recognition, and offline translation apps providing instant translations without internet access.

A video demonstrating smart reply the capabilities of Gemini Nano.

Gemini Pro

Gemini Pro aims to provide more robust AI capabilities with a balance between on-device processing and cloud support. It offers higher processing power compared to Gemini Nano, enabling more complex AI tasks. Gemini Pro combines on-device processing with cloud-based support to handle a broader range of tasks efficiently. This hybrid processing approach ensures that intensive tasks are distributed effectively between the device and the cloud, allowing for better resource management.

With increased functionality, it is suitable for applications requiring moderate computational power and where some tasks can benefit from cloud-based enhancements. For instance, augmented reality (AR) applications that require more detailed scene analysis and object interaction, health monitoring apps that analyse biometric data and provide insights by leveraging both local and cloud processing, and advanced camera features needing more sophisticated image and video processing than what Nano can handle alone are ideal use cases for Gemini Pro.

A video demonstrating advanced multi-modal prompting of a 44 minute movie with Gemini Pro.

Gemini Ultra

Gemini Ultra is designed for the most demanding AI applications, providing top-tier performance and extensive cloud-based capabilities. It offers the highest level of processing power, suitable for extremely complex and resource-intensive AI tasks. Heavily reliant on cloud-based processing, Gemini Ultra can perform tasks that require significant computational resources and large-scale data analysis. Its advanced capabilities support multimodal AI, handling various types of data inputs (text, code, audio, image, video) simultaneously and integrating them seamlessly. It is also highly scalable, making it ideal for applications that need to scale quickly and handle large volumes of data with high throughput and minimal latency.

Examples of applications for Gemini Ultra include sophisticated AI research and development tools that require high computational resources for training and deploying models, enterprise-level applications such as comprehensive business intelligence tools analysing vast amounts of data in real-time, and real-time video analytics for security systems processing and analysing high-definition video feeds continuously.

A video demonstrating how Gemini Ultra can create customised practice questions and explanations to help expand a user’s knowledge of physics.

By understanding the specific capabilities and intended use cases of Gemini Nano, Pro, and Ultra, developers can choose the appropriate model that aligns with their application’s requirements, ensuring optimal performance and user experience.

How It Works

Apple Intelligence

Embedded within Apple’s ecosystem, Apple Intelligence draws from personal context to provide tailored assistance. It utilises large language models and diffusion models to comprehend and generate text and images, employing a semantic index to organise data from various apps for efficient retrieval and action. Privacy is a key focus, with data processed securely on Apple silicon servers using Private Cloud Compute, ensuring user information is not stored externally. The addition of Swift Assist in Xcode 16 further enhances developer productivity by offering cloud-based AI support for coding tasks while maintaining developers’ code privacy.

Google AI (Gemini Models)

Google’s Gemini models offer flexibility with both on-device (Gemini Nano) and cloud-based (Gemini Pro and Ultra) solutions. Gemini Nano is optimised for mobile devices, providing on-device AI capabilities with low latency and offline functionality. For more complex operations, Gemini Pro and Ultra offer advanced performance with cloud-based processing. Google integrates these capabilities into developer tools like Android Studio, enhancing ease of use and productivity for developers. While Google balances on-device capabilities with powerful cloud-based processing, Apple emphasises secure, on-device processing with minimal external data storage.

Comparison

Apple Intelligence is deeply embedded within Apple’s ecosystem, providing tailored assistance based on personal context. It utilises large language models and diffusion models to understand and generate text and images, while employing a semantic index to organise data from various apps for efficient retrieval and action. Privacy is a key focus, with data processed securely on Apple silicon servers using Private Cloud Compute, ensuring user information is not stored externally. The addition of Swift Assist in Xcode 16 further enhances developer productivity by offering cloud-based AI support for coding tasks, while maintaining developers’ code privacy.

In contrast, Google AI, particularly through the Gemini models, offers flexibility with both on-device (Gemini Nano) and cloud-based (Gemini Pro and Ultra) solutions. Gemini Nano is optimised for mobile devices, providing on-device AI capabilities with low latency and offline functionality. For more complex operations, Gemini Pro and Ultra offer advanced performance with cloud-based processing. Google integrates these capabilities into developer tools like Android Studio, enhancing ease of use and productivity for developers. While Google balances on-device capabilities with powerful cloud-based processing, Apple emphasises secure, on-device processing with minimal external data storage.

Recent Examples of Mobile Apps Leveraging AI

AI use in mobile apps is not a new phenomenon; in fact, it has been gradually transforming how we interact with our devices over the past decade. Early implementations of AI in apps focused on basic functionalities like speech and image recognition, which have since evolved into more sophisticated and pervasive applications. Today, AI is seamlessly integrated into many aspects of our digital lives, driving personalisation, enhancing user experiences, and automating tasks. From recommendation engines that suggest music and movies to intelligent assistants that manage our daily schedules, AI has become an integral part of modern mobile applications. Here are some recent examples of mobile apps that effectively leverage AI to provide innovative and user-friendly features.

TikTok

TikTok uses AI extensively to personalise the user experience. Its recommendation engine analyses user behaviour to suggest content that aligns with individual preferences. AI algorithms track what users watch, like, and share to refine content recommendations in real-time. This AI-driven personalisation keeps users engaged by constantly presenting relevant and interesting videos, contributing to TikTok’s rapid growth and high user retention rates. Recently TikTok has even added a virtual assistant to help people create content, Introducing TikTok Creative Assistant: Supercharge content ideation.

A video demo of TikTok's Creative Assistant capability

Google Photos

Google Photos leverages AI for image recognition, organisation, and editing. Its AI-powered search function allows users to find specific photos by typing in keywords like “beach” or “birthday,” Ask Photos: A new way to search your photos with Gemini. Google Photos also offers features like automatic photo enhancements, creating albums, and generating animations from a series of photos. The AI capabilities in Google Photos make it easier for users to manage large photo libraries, find specific images quickly, and enhance their photos with minimal effort, improving the overall user experience.

A video demonstrating Google Photos new image recognition powered by AI

Spotify

Spotify uses AI to power its recommendation algorithms and personalised playlists. Features like Discover Weekly and Daily Mixes are generated using AI that analyses users’ listening habits and preferences. Spotify also uses natural language processing to understand and categorise new music. These AI-driven recommendations help users discover new music that matches their tastes, keeping them engaged and satisfied with the service, and encouraging them to spend more time on the platform. Spotify Debuts a New AI DJ, Right in Your Pocket.

A promotional video demonstrating Spotify's AI DJ feature

Apple Fitness+

Apple Fitness+ uses AI to personalise workout recommendations based on user activity data collected from the Apple Watch, The secret AI integration on your Apple watch you didn't even know about. The app suggests workouts that match users’ fitness levels, goals, and preferences. It also uses AI to provide real-time feedback and coaching during workouts. By providing personalised workout recommendations and real-time feedback, Apple Fitness+ helps users achieve their fitness goals more effectively and maintains their motivation, enhancing the overall value of the app.

Apple’s Fitness+ service recommends similar workouts that you’ve done using AI. Image from Apple.

Replika

Replika is an AI chatbot designed to engage users in meaningful conversations. It uses natural language processing (NLP) to understand and respond to user inputs in a human-like manner. The AI learns from each interaction, becoming more adept at providing personalised and empathetic responses over time. Replika offers users a unique experience by providing a virtual friend that can offer support, conversation, and companionship. Its ability to personalise interactions based on user behaviour and preferences makes it a powerful tool for mental wellness and social connection.

Replika encourages users to converse with an AI chatbot at a deeper level. Image from Replika

Amazon Alexa App

The Amazon Alexa app uses AI to provide a wide range of voice-activated services, How is Alexa using artificial intelligence? From setting reminders and controlling smart home devices to providing personalised news and weather updates, Alexa leverages AI to understand user commands and deliver relevant responses. The integration of AI in the Amazon Alexa app makes it a versatile assistant that can manage various aspects of users’ daily lives, enhancing convenience and accessibility through voice commands.

A video demonstrating Alexa's new AI powered features

These examples demonstrate how AI can be effectively integrated into mobile apps to enhance user experience, provide personalised services, and drive user engagement. By leveraging AI, developers can create more intuitive, responsive, and valuable apps that meet the evolving needs of users.

Leveraging AI in Your Apps

AI is revolutionising how we build and interact with mobile apps, making them smarter, more intuitive, and incredibly efficient. By integrating AI capabilities from Apple Intelligence and Google AI, developers can create apps that stand out in today’s competitive market. Let’s dive into some practical examples of how different types of apps can benefit from these technologies.

Government Agency Apps

Citizen Service Assistance

Imagine a government app where citizens can use natural language commands to get information on services. For example, “Hey Siri, what are the requirements for renewing my driver’s licence?” or “Hey Google, where is the nearest polling station?” This makes accessing important information straightforward and hassle-free.

Personalised Notifications and Reminders

AI can send out personalised notifications about upcoming deadlines, such as tax filing dates, and reminders for appointments at government offices. Picture an app that reminds you about your council tax payment or an upcoming court date, ensuring you never miss a deadline again.

Document Processing and Assistance

Siri and Google AI can assist citizens in filling out forms by providing step-by-step guidance and answering any questions about the required information. Think of an immigration app that walks you through the visa application process, explaining each section of the form and what details are needed.

Travel Apps

Intelligent Itinerary Management

Imagine a travel app where Siri organises your travel details and provides timely updates and reminders. With Gemini’s on-device processing and cloud-based insights, your travel planning becomes a breeze. Your travel app could automatically compile itineraries from your emails and calendar entries, and notify you of flight changes or delays.

On-Screen Awareness

Users can ask Siri or Gemini to pull booking details from emails and messages, adding them directly to their calendars. A travel booking app could scan your emails for confirmations and automatically add those details to your calendar, so you always have your itinerary at hand.

E-commerce Apps

Enhanced Customer Support

Picture an online retail app where Siri and Gemini provide contextual assistance, helping you check order status, track shipments, or initiate returns through voice commands. This means quicker resolutions and a smoother shopping experience.

Personalised Shopping Experiences

Both AI systems can suggest products based on your browsing history, making your shopping experience more personalised. An e-commerce app could recommend products and exclusive deals based on your past purchases and browsing behaviour, keeping you engaged and boosting sales.

Purchase Reminders

Imagine an app that reminds you to purchase or book something based on your previous patterns. For example, an app could remind you to order groceries every week or book parking when you typically need it. This could help you stay on top of your regular tasks without having to remember.

Health and Fitness Apps

Proactive Health Monitoring

Imagine a fitness app that uses Siri and Gemini to remind you about workouts or medication based on your health data and routines. This helps you stay on track with your health goals. Your app could send workout reminders at optimal times, based on your schedule and past activity patterns.

Contextual Insights

This is my personal favourite, one that I’ve already started to leverage with my own health and exercise data. Users can ask Siri or Gemini for health metrics or workout summaries, with data pulled from integrated health apps. A health monitoring app could generate daily summaries of your activity, heart rate, and sleep patterns, providing valuable insights into your overall wellness; like a personal trainer on your device.

By incorporating Apple Intelligence and Google AI into your mobile apps, you can enhance their functionality and user experience, making them smarter, more personalised, and more effective. These examples show just a few ways AI can add real value to your app offerings, ensuring they meet the evolving needs of today’s users.

Next Steps for Mobile App Product Owners

Integrating AI into your mobile apps can seem daunting, but with the right approach, it can significantly enhance your app’s functionality and user experience. Here are some steps to guide you through the process of leveraging the powerful capabilities of Apple Intelligence and Google AI.

1. Understand the Frameworks

Start by familiarising yourself with the key frameworks and tools provided by Apple and Google. For Apple, this includes SiriKit and the App Intents framework, which allow your app to integrate with Siri and utilise Apple Intelligence. On the Google side, explore the Gemini API and AICore, which enable generative AI and on-device processing on Android devices. These resources provide the foundation for integrating AI into your app.

Resources:

Apple: WWDC 2024 Session on App Intents

Apple: WWDC 2024 Session on Integrating Your App with Siri

Google: AI on Google, Android Developers Blog

Additionally, explore the new Assistant Schemas and AICore documentation to define the actions and entities your AI integration will support.

Resource: Apple Documentation on Integrating with Siri and Apple Intelligence

2. Identify Use Cases

Determine how AI can enhance your app’s user experience. Think about scenarios where natural language processing, contextual understanding, personalisation, and image generation could add significant value. Identifying the right use cases is crucial for effective integration. Consider the needs of your users and how AI can help meet those needs more efficiently and intuitively.

3. Plan for Integration

Develop a detailed roadmap for integrating AI features into your app. Start by prioritising functionalities that align with your app’s core offerings and user needs. Ensure your app’s data is well-structured and accessible to facilitate seamless AI integration. This step is vital for setting realistic goals and timelines, as well as for managing resources effectively.

4. Leverage New APIs

Make use of the new APIs and tools provided by Apple and Google to enhance your app’s capabilities. These include APIs for semantic search, image generation, and enhanced user interactions. Leveraging these tools will enable you to implement advanced features that improve the user experience and functionality of your app.

5. Stay Updated

The landscape of AI and mobile app development is rapidly evolving. Keep abreast of the latest updates, best practices, and emerging trends from Apple’s and Google’s developer communities. Participate in forums, attend webinars, and regularly review documentation to stay informed and ensure your app remains competitive and up-to-date with the latest advancements.

By following these steps, you can effectively integrate AI into your mobile apps, leveraging the powerful capabilities of Apple Intelligence and Google AI to create smarter, more intuitive, and highly engaging user experiences.

In Summary

Apple Intelligence and Google AI represent significant advancements in personal intelligence, providing powerful new tools for app developers. While these technologies offer substantial opportunities, it’s essential to remain cautious and realistic about the integration process. Embracing these AI capabilities can position your app for success in this new era of intelligent, context-aware applications, whether you are working within Apple’s ecosystem or Android’s platform.

By leveraging the advancements in AI from both Apple and Google, we as developers have the opportunity to create apps that not only meet user needs but anticipate them, providing a seamless and enriched user experience. As we continue to innovate, let’s utilise these new tools to keep our apps ahead of the curve and delight our users with every interaction.

If you’re interested in learning what ideas might work for your app and what such systems could cost to implement, why not contact us at hello@adapptor.com.au and we’ll see how we can help.