RealityKit Meets SwiftUI — Build an Augmented Reality Counter iOS App

Add text to an ARView in SwiftUI

AR is in, and my understanding was — not. I had a hard time finding resources on how to add text to an ARView in an iOS application, not to mention manipulating that text with user interaction. So today, we’re taking it back to the basics. We will be building a simple counter app, except in a way we’ve probably never done before. We’re doing it in AR.

If you’d like to jump right into setting up the AR scene, skip ahead to “Setting up ARViewController”, or clone the repo here:

GitHub - martusheff/ar-counter: A counter app in Augmented Reality!

Device Compatibility:

• iPhone 6s and up
• iPad 5th Generation and up
• iPod 7th Generation

ARView Counter App

Here is what will be the result of following today’s tutorial. A counter app being built in SwiftUI with the help of ARKit.

Both the visible button bar (right), and the ARView and confined to their own views, so we will also learn how to pass object states between views, and update the ARView whenever our count increases. Let’s get started.

Above is a demo of our working AR counter.

Setting up Our Project

We will start off by selecting the Augmented Reality App when creating our new project. Hit next.

Then enter a name for the project, and be sure to keep SwiftUI selected. After hitting next, you’ll be prompted to find a location to save your project to. When done, select create, and Xcode will complete setup.

After selecting resume in the simulator pane, the starter code will be populated with presets for adding a box to our AR scene. We can see the box in the simulator above, but because there is no camera available to the simulator, it won’t have the AR experience.

At this point, you can go ahead and run the project (CMD + R) on your iOS device and see this box in the app.

Setting up our Views and Models

With the ARCounter directory selected, hit cmd + option + n to create a new group (folder). Create 2 groups, 1 named View and the other named Model. Within the View group, add a new Swift file with cmd + n .

Select Swift File, and name this file Count . Repeat this process, creating 2 more Swift files named: ARViewContainer , CounterButtonBarView . Move the files into their respective groups. Here’s an image for reference as to what your folder structure should look like.

Setting up Count

In our Count file, we will be creating the Count class. This class contains a published variable named num .

Published is a property wrapper that will update all ObservableObjects that have it as a property whenever it experiences a change in state. More on that later.

Setting up CounterButtonBarView

In our ContentBarButtonView we will want to add the code above. Here, we are setting up our interactive button bar so that way we will be able to update the count that will be displayed in our ARView.

• On line 5 , we are declaring a variable for our instance of Count that is used to track the current count of our application.
• On line 8 , we are embedding the following views in a horizontal stack.
• On line 10 , we create our decrement button. Within this button, we set the action to decrement its views instance of Count by 1. It also prints to console which is helpful for debugging purposes. Within the label section for this button, we are setting an Image using an icon set provided by SFSymbols.
• On line 17 & line 24 , similar button for resetting and incrementing the counter are created.
• From line 31 to line 36 , we are styling the ContentBarButtonView. Feel free to tweak these as you see fit.

Adding our Buttons to the ContentView

We are going to want to update our ContentView to match the above code. The ARViewContainer has been removed from the Content View (and is being moved to the file we created for it earlier.) It will also be removed from the body of the ContentView for the time being.

• On line 6 , we are creating our instance of Count .
• On line 9, we are embedding all further views in a ZStack . This is because we are going to be overlaying the CounterButtonBarView over the AR’s camera view on the Z axis.
• On line 10 , we are embedding the button bar in a VStack .
• On line 11 , we are adding a Spacer() to push down all remaining views to the bottom of the screen.
• On line 12 , we are adding our CounterBarButtonView view and passing in the instance of counter we created for the application on line 6 . The bar button views handling of increment/reset/decrement will update this Count instance.

Now, with your iOS device connected, select it as the device target for build. After running cmd + r, you will see an application with a black screen and button appear. On button tap, your console should be tracking the instance of Count as shown below.

Setting up ARViewController

Now, to setup our AR scene. The above code will handle the creation and updating of the displayed text every time one of the button we created earlier are tapped. We will simplify this by just redrawing the scene on every button tap.

• On line 6, we are setting up a reference that will take in an instance of the Count class we had created earlier. It is set to be an observed object, because it works with the Published property wrapper to update the view every time the published variable num is changed.
• On line 8, we have the constructor for the ARView.
• On line 12, we have the function that will be called to update/redraw the AR view every time the ObservedObject has a Published variable that changes. We call our custom updateCounter() function from here, passing in the existing ARView.
• On line 17, we remove all anchors from the scene, clearing it.
• From line 20 to line 24 we create our MeshResource using .generateText and pass in a series of parameters. First, the text we want to display, which is just going to be the absolute value of the count.num . Second, we pass in our desired extrusion depth to make the text 3d. Last, we set out font with a desired size & font weight.
• From line 26 to line 35 , we check the new value of count.num and assign a color depending on whether it’s positive, negative or zero.
• On lines 37, we create a shader using SimpleMaterial, passing in our color and some other desired presets (feel free to play with these).
• On line 38, we create our Entity using the mesh and shader we had created previously.
• On line 40, we are position out textEntity to be in front of us by changing its position on the Z axis.
• And finally, on lines 41 & 42 , we are adding the entity to the anchor, and the anchor to the scene.

Once complete, the text will have been updated, an action triggered by the change in the published variable num as a result of a user’s tap on button.

Final updates to Content View

Now, in our ContentView we can add our ARViewContainer, and pass in our instance of count.

And that should be it! Run the app and interact with the counter. Below, I have linked the repo again. If you face any troubles, and would like to see my full working code, feel free to clone it & work from it!

Link to Repo: https://github.com/martusheff/ar-counter

Challenges

I find building off of tutorials to be a very good way to learn, so below are some challenges you may consider building upon on this app!

• Add a text field that updates the displayed text.
• Add a color picker to change the color of the text.
• Add an animation to the text every time the counter is updated.