Last week, Apple's WWDC24 finally took place, and iOS 18 will integrate AI capabilities very deeply, with updates mainly including Siri, writing assistant, and image generation:
Siri utilizes Apple Intelligence to achieve new superpowers. With a brand-new design, richer language understanding capabilities, and the ability to input to Siri at any time, communication with Siri is more natural than ever before.
Siri features a new design that is more deeply integrated into the system experience, with an elegant glow surrounding the edges of the screen.
Simply double-tap the bottom of the iPhone or iPad screen to input content to Siri from any point in the system when you don't want to speak out loud.
Leveraging Siri's extensive product knowledge about your device's features and settings. When learning how to do new things on iPhone, iPad, and Mac, you can ask questions, and Siri can quickly provide step-by-step guidance.
Apple Intelligence provides Siri with screen awareness, so it can understand the content on the screen and take action.
Understanding your personal background allows Siri to assist you in a way that is unique to you. Siri can use its knowledge of information on the device to help find the information you need.
Use Siri to execute operations seamlessly within and between applications.
Apple Intelligence supports a new writing tool that can help you find the right words anytime while writing. With enhanced language capabilities, you can summarize an entire lecture in seconds and get a short version of a long group discussion.
You can proofread text, rewrite different versions until the tone and wording are just right, and summarize selected text with just a tap.
Priority notifications are displayed at the top of the cards, so you know at a glance what needs attention. Notifications are grouped so you can browse them more quickly.
Priority messages in Mail elevate time-sensitive messages to the top of the inbox - such as invitations due today or boarding reminders for flights this afternoon. Tap to display a summary of long emails in the Mail app, getting straight to the point. You can also view email summaries directly from the inbox.
Simply tap record in Notes or Phone to record audio and text notes. Apple Intelligence generates text note summaries so you can understand the most important information at a glance.
Use the smart reply feature in Mail to quickly draft email replies that include all the correct details.
Apple Intelligence allows you to express yourself visually in new ways. Create fun, original images and new Genmojis. Use Image Wand to turn sketches into relevant images that complement your notes.
Experience the Image Playground in the app to create fun, original images in just a few seconds. Create new images based on descriptions, suggested concepts, or even people in your photo library.
Try out different concepts in the dedicated Image Playground app and experiment with image styles such as animations, illustrations, and sketches. Create custom images to share with friends in other apps or on social media.
Create all-new Genmojis directly on the keyboard to match any conversation. Provide a description to see a preview and adjust the description until it's perfect.
Image Wand can convert your sketches into relevant images in the Notes app. Draw a circle around your sketch with your finger or Apple Pencil, and Image Wand will analyze the surrounding content to produce complementary visual effects.
Enter a description, and Apple Intelligence will find the most matching photos and videos. Then, it will create a story with unique chapters based on the recognized themes and arrange the photos into a film with its own narrative arc.
Use the cleanup tool in the Photos app to remove distractions from photos. Apple Intelligence can recognize background objects, and with a single tap, you can remove them to take the perfect photo while preserving the original image.
Apple also published an article introducing their LLM deployment plan, which mainly consists of three layers:
On-device LLM inference: Future iOS versions will include a small, low-latency AI model (3 billion parameters) that can understand user commands, the current screen, and perform operations in applications. This model can not only handle simple tasks such as summarization but also support Siri's "AI agent" features, such as processing user commands that require opening and using multiple applications - like "Hey Siri, call an Uber to the nearest Costco." Most importantly, the model runs on Apple Silicon chips (such as M-series chips).
Private cloud computing: The large language model on the device may offload some complex tasks to a more powerful model hosted by Apple's data centers (referred to as "private cloud computing"). These data centers will also run entirely on Apple's M-series chips. The data transmitted will be fully encrypted and protected. The servers are manufactured by Apple itself. In other words, Apple has vertically integrated everything needed to run AI both on the device and within the data centers.
Third-party model inference: Users can also directly use OpenAI's ChatGPT through Siri or certain iOS apps. Note that this is not to replace Siri with ChatGPT - this is a common misunderstanding of the OpenAI collaboration. In fact, ChatGPT is provided as an alternative to the Apple model in specific situations. For example, when a user is about to revise an email, they can choose ChatGPT's response.
Open AI also issued an announcement to introduce their collaboration with Apple:
ChatGPT integration features will be provided for free to iOS users, and paid members can use paid features after logging in.
Even text rewriting uses ChatGPT, and the images seem to use DALL-E.
Siri can also call on ChatGPT's intelligence when needed. User consent is required for the call.
Users can use ChatGPT's features, including images and document understanding, without having to switch between different tools.