Member-only story

Build a Hands-free SwiftUI App Using ARKit and SceneKit

Let’s also use the SwiftUI Charts framework to get a better understanding of the values returned

Mark Lucking

Published in

Better Programming

7 min readFeb 23, 2023

Bon — so my muse is to build some hands-free apps, using FaceID capabilities of high-end devices like the iPhone 14 Pro and the iPad Pro.

I tried to do tasks here and here with some success, but I still felt it was simply too hard to use and unreliable. I took a step back, talked to a colleague, and decided to dig deeper into this. Join me in this paper to find out where I went on my journey.

SwiftUI Charts

The issues were lost in the data — hiding somewhere. I simply had too much data flooding at me at the same time.

As I looked at the problem I realized I could use the new SwiftUI Charts framework to help me understand it. So I set up some code to plot it.

Blended Shape Values Plotted with SwiftUI Charts

You’re looking at the first chart I put together. I have two blended shapes for each eye, the position of which I plot in real time using published variables to display the findings. You can see a clear pattern as I look left or right.

However, there is something wrong with the leftOut column. When I try to stare at the center of the screen, it still registers values; it only drops once I look right.

if (faceAnchor.blendShapes[.eyeLookInLeft]!.doubleValue) > 0.1 {
    looker.eyeLookInLeft = faceAnchor.blendShapes[.eyeLookInLeft]!.doubleValue
}
if (faceAnchor.blendShapes[.eyeLookInRight]!.doubleValue) > 0.1 {
    looker.eyeLookInRight = faceAnchor.blendShapes[.eyeLookInRight]!.doubleValue
}
if (faceAnchor.blendShapes[.eyeLookOutLeft]!.doubleValue) > 0.1 {
    looker.eyeLookOutLeft = faceAnchor.blendShapes[.eyeLookOutLeft]!.doubleValu
}
if (faceAnchor.blendShapes[.eyeLookInLeft]!.doubleValue) > 0.1 {
    looker.eyeLookOutRight = faceAnchor.blendShapes[.eyeLookOutRight]!.doubleValue
}