Google's new Gemini-powered tools on Pixel 8 Pro failed to impress me
Summarize in Recorder and Smart Replies in Gboard are barely worth the effort of activating AiCore
by Will Sattelberg · Android PoliceReaders like you help support Android Police. When you make a purchase using links on our site, we may earn an affiliate commission. Read More.
Although the December Pixel Feature Drop arrived right around when most users expected it — despite Google's decision to do away with a set schedule for updates — it did come with one big surprise. Gemini, Google's latest large language model, arrived in a few distinct flavors, including one designed specifically to run locally on Android hardware. Dubbed Gemini Nano, it's a tool now available for Pixel 8 Pro users with updated smartphones — though you'll need to jump through a few hoops to use it.
Gemini Nano is designed to be powerful enough to impress on mobile, specifically built for on-device tasks without needing a constant internet connection or costing resources in the background. Although it seemed promising at launch — and, considering how early it is in its lifecycle, absolutely has opportunities to rebound — I see Google's latest LLM as something of a disappointment. Let's take a look at how this tool works on the Pixel 8 Pro, and why it's just not something worth enabling on your device just yet.
Google's Recorder summaries aren't just limited — they're useless
Let's start with the easier examples of the two. We've known AI-created summaries were on their way for Recorder since Google announced the Pixel 8 Pro in October — they were one of the many features that just weren't ready at launch, and with the arrival of Gemini Nano, now we know why. The premise is simple: your phone will take your collection of recordings and deliver a short three-bullet summary detailing what was discussed. Whether you're reading notes to yourself or recording an important conversation, it's easy to see why this sort of thing could help.
On the surface it sounded like another excellent addition to one of Google's best Pixel-exclusive app. As someone who reports on technology news for a living, I've found the recorder app to be a lifesaver. Its automated transcriptions are some of the best in the business — they've saved my hunt for specific pieces of information more times than I can count. it's absolutely an essential tool on my Pixel phones, and the promise of new functionality coming in the form of shortened versions of extended conversations sounded great.
Unfortunately, not only are summaries far more limited than the company initially let on, but any time you manage to generate one, it borders on useless. Based on our own testing across Pixel 8 Pro users at AP, it seems like Recorder is only capable — at least in its current state — of generating summaries for recordings between one and 15 minutes. Maybe I'm an outlier here, but nearly every recording I have saved from various Pixel devices extends well beyond that limit, meaning I can't use it on the vast majority of my library. Longer conversations are where summarized formats would actually help, though, leaving me scratching my head as to why Google would launch a feature in such a limited state.
Maybe it's because summaries are actually terrible. The single recording I have that is of a qualifying length generated a list of bullet points that, charitably, I'd call useless. In one recording, it picked up on actor Brian Cox's name after a scene from Succession was used as an in-context example. Google and Gemini Nano then assumed — incorrectly, I might add — that I'd been working with Brian Cox on some kind of project.
The other two bullet points in this example are technically correct — the project being discussed went well — but they're also meaningless. Where were the actual action points discussed from this call, the things that still needed work? Google's LLM couldn't pick up on them, so instead, it generated what amounts to a virtual pat on the back for a job well done.
As for the rest of my library? The next shortest recording clocks in just shy of 19 minutes, leaving me unable to generate more samples. This limitation alone would be disqualifying in my eyes, but that would require these summaries to seem worthwhile at all. Maybe future upgrades to Recorder and Gemini Nano will turn the tides here, but for now, Pixel 8 owners shouldn't feel like they're missing out on some cool new feature just for buying a smaller phone.
Smart Replies for Gboard just aren't all that smart
Google's other Gemini Nano-powered tool — improved Smart Replies for Gboard — came as a surprise to anyone not paying attention to the AiCore app in November (so, you know, most people). On its face, it's just as limited a feature as Recorder's automatic summaries: you needed to message exclusively on WhatsApp (or a couple of lesser-known apps, Line and KakaoTalk), but the demo Google showed off in its announcement certainly looked promising.
Source: Google
Google's demo.
Sure, it had the potential to make completely wrong suggestions to questions with definitive answers only you could know — what if Fergie had responded to your text? — but at the very least, it looked like it had the opportunity to really wow once it rolled out to users. And keeping it to WhatsApp on devices with Gboard set to US English seems like a great way to get a small selection of testers before a broader launch.
Unfortunately, early real-world tests just aren't much better than Recorder's ability to summarize. Before I really lay into Gemini Nano-powered Smart Replies, let me clarify that there is a small chance the feature isn't actually active for the handful of us at AP who seem to have it. Because suggested text is nothing new to Gboard, it's difficult to ensure this tool has actually rolled out to our Pixel 8 Pro units. I can tell you both myself and Manuel Vonau, though, did everything we could to verify our experience. Not only did the tool continue working after disabling Smart Compose in settings, but it didn't seem to work in other chat apps or after disabling AiCore on-device. In other words, the experience we saw seems, for better or worse, to be the one rolling out to users.
Two examples of Gemini working on two ends of the same conversation.
With that disclosure out of the way, allow me to rain on your parade. The suggested replies are far less wordy than the demo Google showed off, and not in a good way. Delivering quick yes-or-no answers to questions isn't impressive, and when the most contextually appropriate response to any of the questions was simply "pretty good," I'm not sure how else to look at the demos than a false reality of what you should expect. Granted, it's possible Gemini Nano continues to learn as I send suggestions, and Google can certainly make tweaks and improvements on its side, but it's just not the messaging overhaul I was expecting to see from the December Feature Drop.
A rough start for Gemini Nano
Google brought its new on-device LLM to users in such a limited fashion and with so much fanfare that I truly kept my expectations low. This is clearly an early preview of something the company sees as a bright future for Android — it's not being kept as a Pixel-exclusive, so it'll be available on your next Android phone regardless of manufacturer. But even then, I just can't vouch for either of the features launched this month. Both of these tools — and, frankly, Gemini Nano — have a long way to go before I start believing in AI as the true future of mobile platforms.
As they exist right now, they're better seen as slight improvements to existing features. And even then, neither added enough to impress. AiCore's strings leak in November pointed to more abilities in the works, though, and I'm choosing to hold out hope they'll impress more than these first two ever could.
Google Pixel 8 Pro
The Google Pixel 8 Pro is the company's latest flagship, boasting a new Tensor G3 chip, a brighter screen, and a new camera array capable of capturing even more light. As usual, the real power lies in Google's Tensor chip, which offers even more photo enhancement and image editing features.