RealityKit & ARKit in SwiftUI — Tap to Swap Item

While developing for the Augmented Reality, we can interact with virtual objects by tapping on them, and today we will add the functionality to tap and swap items. We will include a few items from Apple’s provided USDZ models, and as the user taps on an item, it will be replaced with the next one in the list, cycling through all the items.

Download sample USDZs from the link below:

Quick Look Gallery - Augmented Reality - Apple Developer
Embed Quick Look views in your apps and websites to let users see incredible detailed renderings in 3D or AR.developer.apple.com

Next, we will add them into the project by dragging and dropping into the project

Create a new file named “ItemListEnum” to hold asset names in an enum format.

import Foundation

enum ItemListEnum: String, CaseIterable {
    case chairSwan = "chair_swan"
    case fenderStratocaster = "fender_stratocaster"
    case gramophone = "gramophone"
    case tvRetro = "tv_retro"
}

We’ll utilize a tap gesture to detect user taps on the AR item and swap it with the next item. Since we’re using UIViewRepresentable, the coordinator handles target and actions. Acting as a mediator, the Coordinator in SwiftUI's UIViewRepresentable protocol facilitates communication and event handling between SwiftUI and UIKit views. This seamless integration merges UIKit's imperative nature with SwiftUI's declarative syntax. The coordinator manages the UIKit view's lifecycle, covering instantiation, configuration, and cleanup. Its pivotal role bridges the gap, empowering SwiftUI to access UIKit's rich features while adhering to SwiftUI's modern, declarative paradigm.

Let’s create a new swift file called ARCoordinator.

We’ll utilize the coordinator to implement a tap gesture on the AR item. If the user taps on the item, we’ll replace it with another item.

This will be a final class conforming to the NSObject and ARSessionDelegate.

final class ARCoordinator: NSObject, ARSessionDelegate {

Next, we will add weak references for the ARView and the AnchorEntity so we can pass the reference from UIViewRepresentable.

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?

We’ll add a didTapItem function to the coordinator, which will be attached to the UITapGestureRecognizer action. This function will receive the tap gesture recognizer object, enabling us to detect the tap location and determine if the tap was performed on the virtual item.

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {

Next, we will get the tap location in the ARView

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {
        guard let arView = self.view else { return }
        let userTapLocation = tapGesture.location(in: arView)

We will use view.entity function to find out if entity in the AR scene closest to the specified tap point.

The view.entity returns the closest entity but we will not need it so we can ignore the entity.

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {
        guard let arView = self.view else { return }
        let userTapLocation = tapGesture.location(in: arView)
        
        if let _ = view?.entity(at: userTapLocation) as? ModelEntity {

We’ll need a global counter to cycle through all the assets in the enum, so let’s create a new class. This class will contain two static properties: a counter and the next item.

final class GlobalCounter {
    static var counter = 0
    static var nextItem: Int {
        counter += 1
        return counter % ItemListEnum.allCases.count
    }
}

Back in the ARCoordinator, we will use Bundle.main.path to get the path for the models stored in the bundle. We will use GlobalCounter to get the next item and find that item in the bundle.

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {
        guard let arView = self.view else { return }
        let userTapLocation = tapGesture.location(in: arView)
        
        if let _ = view?.entity(at: userTapLocation) as? ModelEntity {
            let path = Bundle.main.path(
                forResource: ItemListEnum.allCases[GlobalCounter.nextItem].rawValue,
                ofType: "usdz"
            )!

Now, all we gotta do is to load the entity from the url.

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {
        guard let arView = self.view else { return }
        let userTapLocation = tapGesture.location(in: arView)
        
        if let _ = view?.entity(at: userTapLocation) as? ModelEntity {
            let path = Bundle.main.path(
                forResource: ItemListEnum.allCases[GlobalCounter.nextItem].rawValue,
                ofType: "usdz"
            )!
            let url = URL(
                fileURLWithPath: path
            )
            let scene = try! Entity.load(
                contentsOf: url
            )

We gotta make sure to call generateCollisionShapes on scene which creates the shape used to detect collisions between two entities that have collision components.

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {
        guard let arView = self.view else { return }
        let userTapLocation = tapGesture.location(in: arView)
        
        if let _ = view?.entity(at: userTapLocation) as? ModelEntity {
            let path = Bundle.main.path(
                forResource: ItemListEnum.allCases[GlobalCounter.nextItem].rawValue,
                ofType: "usdz"
            )!
            let url = URL(
                fileURLWithPath: path
            )
            let scene = try! Entity.load(
                contentsOf: url
            )
            
            scene.generateCollisionShapes(recursive: true)

Let’s remove all the items from the view and add new entity in

final class ARCoordinator: NSObject, ARSessionDelegate {
    
    weak var view: ARView?
    weak var anchor: AnchorEntity?
    
    @objc func didTapItem(tapGesture: UITapGestureRecognizer) {
        guard let arView = self.view else { return }
        let userTapLocation = tapGesture.location(in: arView)
        
        if let _ = view?.entity(at: userTapLocation) as? ModelEntity {
            let path = Bundle.main.path(
                forResource: ItemListEnum.allCases[GlobalCounter.nextItem].rawValue,
                ofType: "usdz"
            )!
            let url = URL(
                fileURLWithPath: path
            )
            let scene = try! Entity.load(
                contentsOf: url
            )
            
            scene.generateCollisionShapes(recursive: true)
            anchor?.children.removeAll()
            anchor?.addChild(scene)
        }
    }
}

In the ARViewContainer, let’s set the definition for makeCoordinator function, which is an optional function in UIViewRepresentable and return an instance of our ARCoordinator.

struct ARViewContainer: UIViewRepresentable {
    
    func makeCoordinator() -> ARCoordinator {
        ARCoordinator()
    }

Next, we will add gesture recognizer to the ARView.

struct ARViewContainer: UIViewRepresentable {
    
    func makeCoordinator() -> ARCoordinator {
        ARCoordinator()
    }
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        arView.addGestureRecognizer(UITapGestureRecognizer(target: context.coordinator, action: #selector(ARCoordinator.didTapItem(tapGesture:))))

We will set coordinator as the ARView’s session delegate and set anchor and view properties.

struct ARViewContainer: UIViewRepresentable {
    
    func makeCoordinator() -> ARCoordinator {
        ARCoordinator()
    }
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        arView.addGestureRecognizer(UITapGestureRecognizer(target: context.coordinator, action: #selector(ARCoordinator.didTapItem(tapGesture:))))
        
        
        context.coordinator.view = arView
        arView.session.delegate = context.coordinator
        
        let anchor = AnchorEntity(plane: .horizontal)
        
        context.coordinator.anchor = anchor

We will launch the experience with the box and if user taps on the box, we will replace it with the item from our virtual item list.

struct ARViewContainer: UIViewRepresentable {
    
    func makeCoordinator() -> ARCoordinator {
        ARCoordinator()
    }
    
    func makeUIView(context: Context) -> ARView {
        
        let arView = ARView(frame: .zero)
        arView.addGestureRecognizer(UITapGestureRecognizer(target: context.coordinator, action: #selector(ARCoordinator.didTapItem(tapGesture:))))
        
        
        context.coordinator.view = arView
        arView.session.delegate = context.coordinator
        
        let anchor = AnchorEntity(plane: .horizontal)
        
        context.coordinator.anchor = anchor
        
        let box = ModelEntity(mesh: MeshResource.generateBox(size: 0.3), materials: [SimpleMaterial(color: .blue, isMetallic: true)])
        box.generateCollisionShapes(recursive: true)
        
        anchor.addChild(box)
        
        arView.scene.anchors.append(anchor)
        
        return arView
        
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
    
}

Build and run