Vision Pro could gain an Apple Pencil-like controller

Despite Apple emphasizing that the Apple Vision Pro does not require any kind of handheld controller, it is researching how to make an Apple Pencil for it that will have virtual brush tips, seen only by the wearer.

The Apple Pencil lineup has been accused of being confusing for consumers to pick from, but at least the whole range is similar. Apple's latest proposal looks like an oversized, chunky addition - but it's meant to be less a writing implement and more a handheld controller.

"With one illustrative configuration... [the device] is a handheld controller having an elongated marker-shaped housing configured to be grasped within a user's fingers," says the newly-revealed patent application, called "Computer System with Handheld Controllers."

"A handheld controller with a marker-shaped housing may have an elongated housing that spans across the width of a user's hand and that can be held like a pen, pencil, marker, wand, or tool," it continues.

Rather than solely being used for, say, an iPad, Apple suggests that this device could be a controller for everything — a head-mounted device, cellular telephone, tablet computer, laptop computer, wristwatch device, a device with a speaker, or other electronic device..."

Where it is used with head-mounted device, such as the Apple Vision Pro, the patent application proposes that the marker could have a virtual tip.

"The head-mounted device or other device may have a display configured to display virtual content that is overlaid onto real-world content," it says. "[A] handheld controller [could have] a tip portion onto which a computer-generated paint brush head is overlaid."

Apple also suggests a more vague "computer-generated tool head," might be of use.

Internally, this controller could "include an inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc." It could also feature "wireless communications... tracking features such as active or passive visual markers that can be tracked with an optical sensor in an external electronic device," and more.

In typical patent application style, Apple's proposal is as broad-ranging as it conceivably could be, while remaining specific about concerning a "handheld controller [which] may have a housing with an elongated shaft extending between first and second tip portions." But the majority of the example illustrations show a chunky Apple Pencil, complete with a large and physical tip.

The patent application is credited to five inventors, including Yuhao Pan, who previously worked on a proposal to use ultrasonic sensors to authenticate user voices.