Spatial Manipulation in Virtual Peripersonal Space: A Study of Motor Strategies

Mixed reality (MR) systems have a rich and extensive history in the context of spatial manipulation [1–3]. The commodification of these systems has helped create a large body of work on spatial interactions, tangible user interfaces, and immersive displays [4–6]. As a result, there is also significant interest in employing spatial user interactions in the engineering domain in the context of VR for training assembly personnel [7,8]. Case in point, human assembly strategies have inspired human–robot interaction research by using peg-in-the-hole micro-assembly as their primary evaluation task [9–12].

At least in principle, the developments in MR systems and interaction techniques [13–15] are generally aligned with the embodied interactions viewpoint, wherein the intention is to incorporate bodily practice into interactions with virtual artifacts such that users perceive the artifact as an extension of themselves; they act through it rather than on it [16,17]. Despite this, spatial manipulation is still, by and large, achieved through arms–length interactions.

History, limitations of technology, and inertia of development have combined to favor interaction in the space farther away from the body in HCI [18] (Fig. 1). We must point out here that precision interfaces are not new as such. They have existed in laparoscopic surgery for some time wherein highly specialized tooling and high-precision motion tracking is used to enable precise manipulation. However, the same cannot be said for spatial user interfaces (SUIs) in general. For example, the distal display monitor and the ubiquity of mouse-based interaction have pushed spatial object manipulation to arms–length interaction [19–21]. Early limitations of stereoscopic displays and the interest in immersive experience created a bias toward deploying display resources to the visual periphery at the expense of the visual fovea [22,23] with which fine motor ability is paired. As a case in point, recent studies in HCI literature [24–26] discuss a decrease in visual perception of a virtual object when placed at distance (beyond arm’s length), as well as, its influence on the manipulation action performed by the user.

We argue that existing design methodologies for spatial interaction techniques need to reconcile with our mental representations of the actions we perform in physical spaces with physical objects — the proximity of the action, the size and shape of the manipulated object, visual feedback, and the corresponding tactile perception should be in synergy. The inspiration for our work comes from some seminal systems such as HoloDesk [27], SpaceTop [28], and MixFab [29] that underscore the importance of spatial interactions in close to the body. While these seminal works show us the power of precise manipulation close to the body, a systematic capturing, characterization, and analysis of actions in this space is missing from the literature. What is more, such a systematic study is critical to tap the potential of precise manipulation of virtual objects in the user’s peripersonal space. Therefore, our goal in this article in to explore the interaction space that merges physical actions (as we perform in daily physical tasks) with virtual interactions in a way that it supports our internal mental representation of the physical world.

Our research draws from Gibson’s seminal adage: “perception is for action’” [30]. The corollary of this concept for interaction design is that action should be designed to match the powers of human perception. While underexplored, HCI literature discuss similar thoughts on co-located action–perception space from the perspective of proximity to the action space [18]. While prior works [27–29,31] have implemented similar co-located action–perception interaction spaces, there is still much to understand about precise manipulations in terms of the types of actions and motor strategies that are active in the peripersonal space. Few works have highlighted spatial interactions from the point of natural user interfaces [32,33]; however, we believe the broader sense of this terminology shifts focus from the fundamental need to understand and advocate the design of interaction spaces that facilitate fine motor control. More importantly, there are also methodological gaps regarding how such interactions should be measured and analyzed. The metrics [34] that typically work for coarse-grained actions such as object docking are likely to miss the fine-grained finger movements, non-Euclidean kinematics of the arms, and concurrent head movements among other things. With this in view, we focus on investigating the following research questions from the point of exploring spatial manipulation in the peripersonal space:

• Q1 

  How do we appropriately measure precise spatial actions performed close to the body in virtual environments ?

• Q2 

  What is the quantitative extent of technological factors that affect the blending of the visuo-motor space close to body for precise motor control ?

• Q3 

  What are the implications of this action–perception blending on design of interaction spaces in the user’s peripersonal space ?

Taking the aforementioned broader questions into consideration, we systematically investigate and present our findings through a set of three controlled lab experiments to understand how spatial interactions designed close to the body (peripersonal space) affect fine motor control and their influence on action-specific perception. In this article, we make three primary contributions: (1) we design and prototype a spatial interface to analyze user behavior for precise spatial task in a real-world scenario having a co-located action–perception space. We use this experiment as a reference to further evaluate user behavior across two experiments having visuo-tactile and visual-only feedback, respectively, for precise virtual tasks; (2) we evaluate user performance through an elaborate version of the classic peg-and-hole bi-manual task across different shapes with the increasing complexity in peg–hole geometry and quantity; (3) we propose a new set of user evaluation metrics in terms of cumulative kinetic energy, energetic transitions for fine and coarse spatial actions. We also evaluate using traditional metrics of the task completion time and peg–hole insertion accuracy to corroborate our findings.

Our analysis highlights the key motor strategies that are followed to perform precise bi-manual tasks in the interaction space close to the body. We quantitatively demonstrate how sensitive the overall user performance is to the blending of the visual and proprioceptive cues in the peripersonal space. The kinetic energy analysis provides fundamental insights into user performance across the three interfaces as it helped capture nuances that are typically lost in arm’s length interactions.

The background of our work stems from the broader areas of action–perception and proprioceptive feedback in spatial interactions focused on precise motor control.

Looking across the mixed-reality continuum [1,2,6], we find that interfaces with head mounted displays, desktop-VR, tablet-AR, and augmented virtuality displays work within medium to large interaction volume for 3D object manipulation tasks [13–15,35]. While there are several works that study SUIs in the peripersonal space, they primarily focus on social behavior of virtual avatar of humans in a VR environment toward enhancing user engagement and immersion [36–39] or social interaction through cross-device interaction [40]. Similarly in ubiquitous computing, the focus is on developing spatially aware systems such as smart display setups [41–48], and input-control mechanisms that control hardware and their software elements using the portable hand-held devices in close proximity to their body [49–51]. However, little is currently known about 3D design and shape manipulation-type interactions in close-to-the-body scenarios where precise and fine motor actions are actually carried out. If anything, these few recent studies show that the role of proximity to action for spatial tasks is quite rich and still largely remains underexplored. Our work seeks to systematically explore this interaction space through three controlled lab experiments by building on the current and growing literature on spatial interactions performed in the peripersonal space.

Another aspect that is of great importance is proprioception. Proprioceptive and kinesthetic control play a key role in spatial manipulation, especially close to the body [52]. The lack of force (kinesthetic control) or tactile (touch) feedback as observed commonly across these systems severely impedes the ability to make fine spatial control; as a result, object manipulation becomes a difficult and high effort task [53]. This is mainly experienced with spatial actions performed close to the body such as MR systems for mid-air pointing and selection actions [54,55]. It has been found that proprioception at an egocentric distance around the periphery of human body helps reduce dependency on the visual feedback for manipulation actions in local and distant mid-air interaction spaces. Recent work by Plaumann et al. discusses a formal study focused on studying the visuo-motor relationship in spatial actions [56]. While the study primarily focuses on macro(coarse), as well as, micro(fine-grain) interactions, the key finding here is that the users experience a visuo-motor mismatch for spatial pointing tasks at distances away from the body, and this discrepancy reduces in pointing actions performed close to the body. On the other hand, Argelaguet and Andujar note visual dependency in virtual environments attributing to distinct motor and visual spaces for spatial interactions [14]. Further, De Boeck et al. [57] also demonstrate that interactions performed proximal to the body improves kinesthetic control by exploiting proprioception. The key finding relevant to our work is that while proprioception enhances manipulative precision close to the body, the same is not true for distal interactions, and these can be further influenced by other sensory perceptions such as visual, audio, tactile, and kinesthetics [58]. To further explore this, we study the differences between users’ motor indexing and movement strategies for peg-and-hole tasks.

Work by Hinckley et al. [59] studied cooperative bi-manual interactions for virtual manipulation and provides a strong evidence for augmenting hand–eye coordination through the use of two hands in conjunction with haptics feedback. In general, two-handed interactions in coordinated tasks have been shown to increase cognitive engagement of the user [60–62] and efficiency of 3D object assembly [63]. Several works explore the advantages of bi-manual spatial interactions [20,64–70] for object selection [71] and manipulation (rotation, translation, scaling) of 3D objects. Alternatively, few recent works [72,73] showcase a hybrid 2D–3D input mode using a tablet surface and a 6DoF controller for bi-manual interactions in a VR setup. Similarly, Brandl et al. [74] explore the combination of two-handed interactions with pen and multi-touch inputs on a surface. Regardless of the wealth of literature, we believe that much is to be discovered regarding bi-manual interactions in the context of proprioception in peripersonal spaces. Hence, we compare bi-manual spatial interactions using our study interface with actual physical interactions. We test this through a MR-based SUI [1,2,6] spatial interaction interface wherein the visual feedback is setup between user and their hands (Fig. 2).

We draw inspiration from prior works [31,27] to explore a co-located visuo-motor space for performing controlled bi-manual experiments in the user’s peripersonal space. For this, we design and prototype one SUI per experiment (Fig. 1).

Below, we discuss some fundamental factors that influenced our design decisions for the exploration of the vision–proprioception spectrum:

Peripersonal space varies with the user’s body structure, further classified based on the body part in action—hands, face, and trunk [18,75,76]. In fact, Galigani et al. [77] make a note that active tool usage in the user’s peripersonal space enhances their proxemic perception, thereby improving spatial manipulation abilities. In our work, we focus on the peripersonal space defined by the upper limbs (Fig. 2(b)), mainly to observe and identify a comfortable interaction distance to perform precise motor tasks in MR systems and also to analyze its relation to the anatomical peripersonal space, specifically the hand, i.e., the perihand space.

There are two key factors for the interaction work space that demand attention. First is the location of the interaction space with respect to the body (specifically the torso). This volume should represent what is currently known as the peripersonal space in the literature. The second factor is the volume of the interaction space, which is defined by the physical limits in sagittal, transversal, and coronal directions [78] from the human body. The manipulation is meant to occur within this volume. We utilize the action field theory [79] to determine the extent of motor control based on the proximity of the object to the user and place where the manipulation action takes place. This theory is of essence from the perspective of interaction amplitude, i.e., coarse and fine motor strategies adopted by the user to manipulate an object in 3D space based on their distance from the object [80].

The average proximity of a user to the manipulated object is somewhere between 1.5 ft and 4 ft, which falls between the proxemic ranges of intimate to peripersonal space. It can be generalized as a space bounded by a user’s arm length [81,18]. However, current MR interaction design methodology integrates actions that are either performed at the boundary of one’s peripersonal space (arm’s length) or the visuo-motor space is disconnected (Fig. 1), which contradicts the need for a co-located action–perception space for precise spatial actions. Prior research [82] also emphasizes on matching the physical and mental representations of the user’s proprioceptive interaction space to performing spatial manipulation tasks.

We build on the aforementioned factors to design our experimental setup as follows:

The experimental hardware setup consists of 10 Optitrack Flex 13 motion capture cameras (field of view: 56 deg; refresh rate: 120 Hz) mounted on a custom gantry built using PVC pipes, measuring 7 ft × 4 ft × 8 ft in volume (Fig. 2(a)). The cameras were synchronized and operated through an Alienware 15R3 laptop computer with an Intel Core i7-7700HQ CPU (2.8GHz), 16GB of GDDR5 RAM, and an NVIDIA GeForce GTX 1060 graphics card having 6 GB video memory, running 64-bit Windows 10 Professional Operating System, which also ran our user evaluation interface. The application was mirrored on a portable monitor.

Our experimental setup (Fig. 2(a)) was designed to facilitate a visually and perceptually coherent interaction space in the user’s peripersonal space to help enable precise spatial interactions in a MR environment. To this effect, we designed the setup to integrate a visuo-motor configuration having the following sequential arrangement: Eyes followed by screen followed by hands, thus maintaining an interaction proximity within the user’s peripersonal space. Our MR-based configuration is specifically an augmented virtuality (AV) interface satisfying four out of six notions for MR, namely, continuum, collaboration, combination, and alignment [6]. The AV interface includes a virtual environment with 3D objects manipulated through physical user input behind the display. The interface was designed with the intention of (a) co-locating the virtual and physical (motor) peripersonal space for high-precision tasks and (b) reducing occlusion caused by placing the user’s hands in front of the screen [23,83,84] as observed for traditional mixed-reality systems.

Inspired by the action-field theory [79], our interaction space is located within the range of 45–60 cm (1.5–2 ft) from the torso (the range signifies anthropometric variations across different users). As for the volume (i.e., the physical limits), we followed an iterative approach starting from a standard table-top dimensions that led to us constructing the motion capture camera mounting frame. The idea was to determine a reasonably small to medium working volume (focused on the upper limbs and the peri-hand space), suitable for precise manipulations while maintaining robust tracking.

Our MR-based user evaluation interface is based on Unity3D in tandem with the optitrack motive API for streaming motion capture data to the former. We tracked user input through reflective markers mounted on custom designed and 3D-printed marker configurations attached to a proxy stem (Fig. 2(c)). The re-designed motion capture setup ensured good tracking coverage and minimized any blind spots due to the compact nature of the interaction space. We were able to track very small objects (0.5–2.5 cm in diameter and up to 4.5 cm in length) with a mean tracking error of 0.035 mm as per the API.

To explore the effect of action–perception blending for performing precise spatial tasks, we conducted the following experiments.

Our aim for the first experiment was to observe, understand, and analyze the motor strategies followed by users to perform precise spatial manipulation tasks in their peripersonal space. The users were asked to manipulate a set of with 3D-printed shapes (Fig. 2(c)) akin to any other object manipulation task in the physical world with a natural and directly perceptible co-located visuo-motor space. This experiment serves as our real-world reference for evaluating the effect of action–perception blending for precise virtual task performed close to the user’s body.

In this experiment, the precise spatial tasks are identical to the ones in experiment 1 with the exception of using the 3D-printed shapes as proxies to manipulate their virtual counterparts using an MR-based spatial interface. The motion capture system (Sec. 3.2) tracks user head movement and the 3D-printed objects, which are reflected in the manipulation of the virtual scene. The visuo-motor configuration (Sec. 3) for the virtual interface is designed with the intent of maintaining an interaction experience similar to the physical world (if the screen was absent). Our interface for experiment 2 serves as a prototypical “representation” of an MR interface facilitating a co-located action–perception space with appropriate visual and tactile feedback.

This experiment is identical to experiment 2 from the point of performing precise virtual tasks using a MR interface. However, the feedback in this interface is purely visual i.e. the users user generic proxies to manipulate the virtual shapes without any tactile feedback.

In this work, we chose the classic peg-in-the-hole assembly task, which is a typical task to study spatial manipulation [85] and psycho-physical evaluation in virtual environments [86,87]. Further, this task involves enough visuo-motor complexity for our specific interest in the user’s ability to match object location and orientation with high precision. In addition, HCI literature has shown studies using peg-and-hole assembly tasks for evaluating spatial user performance in terms of precision and accuracy [59,88,89]. Based on peg-in-the-hole task, the haptic feedback for Real and Representative (Repr.) experiments is facilitated through the interaction between the 3D-printed peg and hole objects during the insertion phase, completing the object assembly.

While peg-and-hole assembly tasks are moderately complex, we added further variables to our experiment design specifically in terms of shape geometry to test the effect of action–perception blending on precise virtual tasks across diverse conditions.

We took cues from day-to-day objects that integrate the broader peg-and-hole insertion action and designed new shapes (Fig. 3) based on (a) number of peg–hole pairs and (b) rotational asymmetry. Keeping the cylinder as our reference (single peg–hole and rotational symmetry), we first designed the shaftkey that had a rotational constraint due to the key-way. Shaftkeys are typically found in electric motor assemblies and mechanical contraptions. Further, the alternate peg–hole pair in 1-peg-1-hole (found in automotive trickle chargers) added positional (two pegs) as well as rotational complexity to the insertion task. Finally, the 3-pin socket was designed for the insertion task using three peg–hole pairs having different orientations. The new peg–hole pairs were also designed to have varying clearance fits: (a) cylinder (peg $∅$⁠: 10 mm; hole $∅$⁠: 10.6 mm, chamfer: 3 × 0.5 mm (tight), 3 × 1 mm (medium), 3 × 1.5 mm (loose)), (b) shaftkey (peg $∅$⁠: 10 mm; hole $∅$⁠: 10.6 mm (tight), 11 mm (medium), 11.4 mm (loose)), (c) 1-peg-1-hole (peg $∅$⁠: 5 mm; hole $∅$⁠: 5.6 mm (tight), 6 mm (medium), 6.4 mm (loose)), and finally (d) 3-Pin Socket (rectangular peg: 3 × 6 mm, 6 × 3 mm, 6 × 3 mm; rectangular hole: 3.4 × 6.4 mm, 6.4 × 3.4 mm, 6.4 × 3.4 mm (tight); 3.5 × 6.5 mm, 6.5 × 3.5 mm, 6.5 × 3.5 mm (medium); 3.6 × 6.6 mm, 6.6 × 3.6 mm, 6.6 × 3.6 mm (loose)).

We implemented head tracking (Fig. 2(b)) for our MR interface that was mapped to the virtual scene camera to help users explore the virtual scene and virtual shapes as they would do in the physical world (in absence of the screen). We iteratively configured the virtual cameras (FoV: 33.4 deg) to appropriately scale the visual rendering of the physical object being tracked, based on its distance from the user. We verified the same experimentally by placing the display mid-way between the user and the physical object. We physically measured the virtual object rendering and found it to be half the size of its physical counter part. We repeated this for multiple distances and the size perception remained consistent based on the law of similar triangles for the view frustum emanating from the camera.

We conduct a set of controlled lab experiments (Sec. 4.1) to test the effect of our proposed experimental setup on user performance for bi-manual peg-and-hole object assembly tasks.

We recruited 39 participants (18–30 years old) who were graduate and undergraduate students from engineering, architecture, and visualization majors. This was a between-subjects experiment where participants were equally distributed across the three experiments (13 participants per group)—Real, Repr., and Generic. To minimize any bias in the experimental data, we also verified and obtained written confirmation through a pretest questionnaire from each participant regarding any injuries on the upper half of their body that may affect user performance.

The motion capture camera system was calibrated every two to three participants (3–4 h) to ensure robust tracking.

We begin by assigning individual tracking to each of the peg-and-hole object pairs. We also wanted to gain a deeper perspective into how user hand kinematics play a role in precise spatial tasks. Therefore, each participant wore tracking markers at their wrist, elbow, and biceps for both arms (Fig. 2(b)). To integrate both object and body tracking, each of the joints were tracked as rigid bodies and care was taken to avoid any tracking confusion by ensuring that no two tracked entities have the same marker configuration. The users also wore a head band, which was tracked to manipulate the application scene. Due to the highly sensitive motion tracking, we applied exponential smoothing to the object and head tracking for a jitter-free interaction experience. After assigning the trackers, participants were given a brief walk-through of the experimental setup. They were allowed to adjust their seating as well as the placement of the display screen as per their comfort. We found that every user placed the screen close to their torso (within their respective peripersonal space).

From the point of standardization, the participants were asked to manipulate the shapes (virtual and physical) through a stem (Fig. 2(c)) on which the 3D-printed peg–hole pairs were attached on one end. In addition for Repr. and Generic experimental groups, participants were also informed of the head tracking and encouraged to use it for exploring the virtual scene and objects during the peg-and-hole assembly task. Following this, the participants practiced with the physical or virtual shapes based on their experimental group to get acquainted with the setup before starting with the user study trials.

Each participant was asked to insert the peg into the hole (Fig. 4) across different geometries and fits (Fig. 3). We varied the study shapes during each study session and trial sequence in increasing order of shape complexity—cylinder, shaftkey, 1-peg-1-hole, and 3-pin socket. The fits were randomized for each shape. To control the trial data for the insertion task only, participants started from a default position (shapes in vertical orientation) after an explicit indication from the study coordinator and the task completion was indicated by the participants to stop recording the trial data.

While there was no time limit for assembling the peg into the hole, participants were able to complete each trial in 5–10 s except for Repr. control group where it took relatively more time to complete the task. We analyze and discuss this user behavior in Sec. 6.

Each user session per experimental group took 45–60 min for completion. For a given experiment, we recorded user data for six trials per shape per fit for every participant, and each participant performed 72 trials across all 12 shapes (four shapes and three fits per shape). For each user trial, we recorded the (a) motion trajectory (position and orientation of object coordinate frame) for peg and hole objects followed by each joint on both arms, and the head (b) task completion time. We also collected self-reported user feedback using the NASA-TLX metrics [90]. To conduct a deeper analysis of user behavior for precise spatial task, we further processed the trajectory data to compute the following user evaluation metrics.

Here, $v^1$ and $v^2$ are two consecutive vectors connecting points p_i−1 and p_i, and p_i and p_i+1; δ is the threshold parameter. We iteratively identified the following values of N and delta for each shape geometry across experiment groups: (a) cylinder and shaftkey (N: 10 (Real, Repr., Generic); δ: 0.01 (Real), 0.05 (Repr.), 0.005 (Generic)), (b) 1-peg-1-hole and 3-Pin Socket (N: 15 (Real, Repr., Generic); δ: 0.05 (Real), 0.0005 (Repr., Generic)). We observe that the parameters are same for all fits of a given shape and consistent for shapes having single peg-and-hole and multiple peg–hole pairs across all experimental groups.

We recorded 936 user trials across the four shapes for a given experimental group and overall 2808 user trials were recorded for all three experiments (Sec. 4.1). In the subsections below, we perform a qualitative evaluation of user performance and behavior for the peg-in-the-hole task and further support our observations with quantitative analysis.

We begin with an in-depth video analysis of user study sessions across the three experiments with the aim of (a) observing and identifying key motor strategy categories for performing the experimental tasks and (b) generating specific hypothesis for quantitative analyses.

• Video Recording and Verification: Before conducting actual user trials, the video camera was arranged (position, orientation, zoom, and focus) in a manner such that it recorded the upper body movements of the participants both in front as well as behind the display screen (Fig. 2(b)). We further reviewed the recordings collected from pilot experiments for video clarity and our ability to identify distinctive user actions at slower (0.25x, 0.5x) playback speeds.

• Coder Recruitment: Each of the three experiments was assigned to one study coordinator involved with the user studies such that the individual analyzing videos for a given experimental group had minimal to no involvement in conducting the experiment. This is to ensure minimize any biases while analyzing user behavior.

• Coding Scheme: The three coders performed a preliminary analysis on 6 of 13 user sessions videos per group and came up with individual coding schemes based on their observations of user behavior for a given experimental group. Following this, the three study coordinators convened to discuss their schemes, and to our surprise, the codes were very similar that made it relatively to finalize on a common scheme to be used for further video analysis across all experimental groups. The codes included but not limited to palmDown, rotatingPeg, seeHole, etc., highlighting the palm facing toward the table while holding the peg–hole object pairs followed by the peg being rotated to align with hole and glancing the hole to ensure proper alignment before insertion, respectively.

• Intercoder Reliability: Once the coders completed analyzing user videos for their respective experiment group, they re-convened to cross-verify each other’s analysis as well as gain any new perspective that might have been overlooked by any of the three coders. Here, each coder took a pass at four videos (about 25% of sample size per group) from another group with the purpose of minimizing any oversight, bias, or unintentional errors in prior analysis. Following this, the coders took a final pass at the videos and further refined their prior analysis of identifying the key motor strategies discussed further.

Drawing from the video analysis, we organized it into three broader categories.

• Bio-Mechanical Stability: One of the key motor strategies followed commonly by users across all experimental groups was to stabilize their upper body, especially the limbs (Fig. 6(a)). A key observation was made by the coders about all users resting their elbows on the table and close to their body before proceeding with performing the peg and hole assembly task. In few cases, the participants stabilized their elbow on the nondominant limb where they held the hole object,*** whereas the dominant limb was suspended in mid-air with a prehensile pose showing the user’s intent to perform an action after ensuring bio-mechanical stability of the other limb. Bio-mechanical stability was observed to be a crucial and one of the first movements executed by all participants to ensure stable grounding of their shoulders and elbows before performing fine wrist and finger level movements to properly align and assemble the peg-and-hole objects.

• Action-Planning: The planning phase often overlapped with the stabilization phase and we identified some key strategies (Fig. 6(b)) observed for users across the three experiments.

  • —

     Grasp: Users typically grasped the 3D-printed peg and hole objects using three to four finger grip which is indicative of precise movements [96,97]. Once stabilized, the grasp allowed the users to make fine motor movements such as rotating (finger level precision) and adjusting the peg to align (wrist level precision) with the hole in most scenarios before insertion. This was observed for all shapes except the cylinder due to rotational asymmetry in their geometry.

  • —

     Peek: While grasp helped align the shapes, users were often observed to glance into the peg and hole after alignment for a follow-up confirmation of their prior alignment motor strategy. In this scenario, the users turned the peg and hole face towards them as well as aligned their head for an appropriate and non-skewed visual perception. Following this, the users initiated the insertion action phase which came with its own challenges as the geometric complexity increased with the study shapes.

• Action Refinement: Unlike cylinder that had rotational symmetry, the remaining three shapes were found to be challenging by the users to successfully insert the peg into the hole in their first attempt. Rotational asymmetry often resulted in the peg–hole pairs intersection due to improper assembly, which forced the users to retract the shapes. Further, the participants re-strategized their actions by adjusting the grasp (rotational alignment) and re-confirming it with by a brief peeking until the two objects were fully inserted. This was a crucial phase as the participants refined their fine motor actions only and no adjustments were made to coarse actions that were performed for bio-mechanical stability. For the Repr. experimental group, participants were found to struggle relatively more than the other two groups, and this was due to a visuo-tactile disparity where the participants often found their proprioceptive perception not matching the visual feedback, thereby leading to relatively larger task completion times and user frustration.

We further confirmed this our observation analyses by reviewing the velocity profiles (Fig. 7) and found maximum activity occurring at the object level. Based on this, we analyze user performance for the manipulations performed at peg level in the subsections below.

We found the cumulative K.E., change detection, and completion time to be nonnormally distributed using Kolmogorov–Smirnov test. Further, we analyzed the performance metrics for a given shape geometry across the three control groups using a nonparametric Kruskal–Wallis test. On the basis of our observational analyses, we make the following hypotheses:

• H1

   Representative (Fig. 1) should enable similar motor strategies (cumulative K.E and change detection) with respect to Real as they both afford kinesthetic feedback. Hence, there should be no statistical significance.

• H2

   Generic (Fig. 1) should exhibit higher consumption of energy and motor strategy (cumulative K.E. and change detection) with respect to Real due to the absence of kinesthetic feedback in the former. Here, we should observe statistical significance.

• H3

   Representative should enable same performance (completion time and docking accuracy) with respect to Real. We should not observe statistical significance.

• H4

   Generic should have less accuracy and take relatively more time to complete the docking with respect to Real. This should lead to statistical significance for both completion time and accuracy.

• H5

   Generic should exhibit higher energy consumption for both view manipulation (cumulative K.E. for the head) and peg with respect to Real. We should observe statistical significance for number of change points detected and cumulative kinetic energy for both the head and the peg.

We observe an overall statistical significance (Fig. 8) for each shape across all control groups (cylinder: p = 6.4 × 10⁻⁹; shaftkey: p = 4.91 × 10⁻⁴; 1-peg-1-hole: p = 6.12 × 10⁻⁶; 3-pin: p = 3.53 × 10⁻⁶). On further post hoc analysis using Tukey’s multiple comparison test, we found the mean kinetic energies to be similar for Real and Generic across shaftkey and 1-peg-1-hole having p values as 0.8 and 0.2, respectively. This was also observed during the video analysis as users in Repr. group struggled to assemble these shapes due to vision–perception disparity. The mean energy consumption was found to be consistently higher for Generic across all shapes except the 3-pin socket. This is attributed to the user familiarity with the three pin sockets as a daily object and their ability to assemble it solely based on proprioception (overcoming the disparity in Repr.). The aforementioned analyses confirm our hypothesis H1 only for the shaftkey and 1-peg-1-hole shapes and is rejected for the remaining shapes. However, our hypothesis H2 holds through for all shapes with higher mean cumulative K.E. for Generic than Real. The anomalous user behavior for the 3-pin socket is further corroborated in NASA-TLX [98] with consistently higher mental (13.9), physical (13.5), temporal effort (14.2), and frustration (13.1) as well as, lower overall performance (7.8) from user-reported scores.

We observed a change detection trend similar to the cumulative kinetic energy (Sec. 6.3) across the single (cylinder and shaftkey) and multiple (1-peg-1-hole and 3-pin socket) peg–hole pairs (Fig. 9). For cylinder and shaftkey, the mean energetic transitions were found to be higher for Real and Repr. than Generic, which shows that tactile perception led to additional effort from the users as we observed in action–refinement motor strategy followed to ensure proper assembly of the peg–hole object pairs. This confirms the hypothesis H2 but with higher mean transitions for Real (cylinder: p = 9.56 × 10⁻¹⁰; shaftkey: p = 9.75 × 10⁻¹⁰), also, rejects H1 with higher mean transition points for Repr. with respect to real due to the vision–proprioception disparity (cylinder: p = 9.56 × 10⁻¹⁰; shaftkey: p = 9.56 × 10⁻¹⁰; 1-peg-1-hole: p = 9.56 × 10⁻¹⁰; 3-pin: p = 9.56 × 10⁻¹⁰). In case of 1-peg-1-hole and 3-pin socket, Real had the lowest transitions, which shows that participants re-strategized their docking at a higher frequency for virtual manipulation. Thus, rejecting the hypothesis H1 (1-peg-1-hole: p = 9.56 × 10⁻¹⁰; 3-pin: p = 9.56 × 10⁻¹⁰), but confirming hypothesis H2.

Similar to change detection, we observe similar completion time trends for single and multiple peg–hole pairs (Fig. 10). For cylinder and shaftkey, the mean task completion time increases from Real to Generic, thus confirming the hypothesis H4 (cylinder: p = 9.56 × 10⁻¹⁰; shaftkey: p = 9.75 × 10⁻¹⁰). We also observe a higher mean task completion time for Repr. with respect to Real, rejecting H3. In case of 1-peg-1-hole and 3-pin socket, the mean task completion time for Repr. (σ : 11.71 and 28.2 s, respectively) is higher than Real (5.15 and 4.26 s) and Generic (σ : 11.51 and 11.06 s), which is surprising and contradictory to H3. This is also observed for NASA-TLX scores on a 21-point scale across 1-peg-1-hole and3-pin socket for Repr. having higher mental (10.3 and 13.9, respectively), physical (10.2 and 13.5, respectively), temporal (11 and 14.3, respectively), and overall effort (10.7 and 7.8, respectively) along with lower performance (12.2 and 14.8, respectively) and higher frustration (10.8 and 13.1, respectively) scores. We also observed the same during the studies as the participant struggle increased with shape difficulty. The mean completion time for Generic was observed to be higher than Real (cylinder: p = 9.56 × 10⁻¹⁰; shaftkey: p = 9.75 × 10⁻¹⁰; 1-peg-1-hole: p = 9.56 × 10⁻¹⁰; 3-pin: p = 2.23 × 10⁻⁸). Thus, confirming the hypothesis H4.

While each participant was allowed to assemble the peg and hole, the accuracy should have been same for all shapes across control groups with kinesthetic feedback and different for Generic. As expected, we observed similar mean accuracy values for Real and Repr. confirming hypothesis H3 (p = 0.08) and higher mean accuracy for Generic confirming hypothesis H4 (p < 0.05). This is an artifact of the Real and Repr. setup itself, that the error is simply the inaccuracy error of tracking. While for Generic, the error is purely caused by visual cues.

What really surprised us (as evident from our initial hypotheses) is the significant time disparity between Repr. and Generic for 1-peg-1-hole and 3-pin socket cases (Sec. 6.5). This is even after considering the fact that the mean docking accuracy was found lower for Generic. There are three potential factors for this that we considered, namely: (1) size disparity (difference between perceived dimensions of parts visually and through touch), (2) tracking disparity (angular and positional errors caused during multi-object and arm tracking), and (3) visual-tactile preference. Here, the third factor is particularly interesting.

Since our visual feedback was in a completely virtual space (rather than a typical mixed-reality setup), there was always a possibility that the sizes of the objects rendered could be different from those being held and manipulated. This could lead to a perceptual mismatch between what the users were seeing and what they were feeling in their hands.

In general, all vision-based systems have tracking errors especially when multiple objects are being tracked (in our case: two hand-held objects, six joints on the arms, and the head). In addition, in our case, noise was inevitably introduced due to reflection from the participants’ apparel and hair.

Given the choice between visual and haptic perception, humans tend to use the former for coarse manipulations, but rely on kinesthetic perception for precise docking tasks [10]. This can be a potential cause for the additional time that we observe in Repr. users.

Each of these factors contribute to the quality of the blend between the action space (shoulder–elbow–wrist–hand–object) and the perception space (visual and proprioceptive). We made significant efforts to resolve the size disparity out the very outset of our experimental setup design (Sec. 4). Therefore, it is likely that the increased effort in Repr. users is a combination of the tracking disparity and the visual-tactile preference as also shown by a consistently wider spread and higher mean scores in the NASA-TLX. The tracking error for the Real and Repr. amounts to 3–5 deg and are identical because in both cases, the user is physically inserting a peg in a hole. In contrast, the main cue for the user to asses success of insertion was based on collision in the Generic interface, thereby accounting for higher errors (5 − −20 deg). Therefore, it is reasonable to assume that tracking disparities may have minor influences over user performance in the Representative control group.

We believe that the visual-tactile preference may have played a bigger role. To verify this quantitatively, we conducted an analysis of user’s head movement by computing the cumulative K.E. consumed with each control group (Fig. 11). We assumed a spherical representation of the head to compute the cumulative K.E. [99]). We observe that Real, Repr., and Generic are all statistically different from each other with mean cumulative K.E. ordered as Real > Generic > Repr. for shapes except the 3-pin socket where it is Real > Repr. > Generic. In effect, users moved very little while using the Repr. interface compared to Real and Generic. This suggests that proprioception kicked in as a preferred means for assessing the success of insertion closer to the end of the task. This was also observed in our video analysis wherein users relied on visual feedback until the initial shape alignment and relied more on their tactile perception of the peg–hole contact to insert. This is a critical factor that needs a detailed study in its own right.

Incidentally, this also may connect with the third potential cause, that is, the preference for perceptual modality as observed for higher energy consumed for the head for the Real group, which is purely physical interaction. Despite lower head movement, the completion times were higher for the virtual interfaces. From the point of motor strategies, we observed that most users performed the docking task by first stabilizing the nondominant arm (which controlled the hole), followed by grasping the peg and hole stems through a two to three finger precision grip with their palms facing down towards the table. Further, the users adjusted the hole into their field of view and then manipulated the peg to complete the docking.

Our study revealed some important design and technological considerations for future interfaces that seek to enable high-precision tasks. Therefore, we draw a deeper understanding of our research questions (Sec. 1.2) by highlighting following three important considerations.

While hand-held controllers are fairly common and robust, other technological aspects need careful consideration. This is especially true since current AR/VR/XR headsets are primarily designed for interaction at a distance. Despite prior implementations of MR interfaces in the peripersonal space (Sec. 1.1), a major challenge that we faced was the lack of guidelines for implementing a close-range motion capture system to facilitate a co-located vision-proprioception space for precise spatial interactions. By using our setup, we encountered a tracking error of 3–5 deg with our interfaces for Real and Repr. experimental groups (Sec. 6.6), which affected user performance for the latter in terms of higher energetic transitions (Fig. 9) and resulting in higher task completion time (Fig. 10). Despite this, our setup performed relatively better than existing commercially available tracking devices. This draws a direct understanding of the quantitative extent to which technological factors affect precise control in spatial interactions (Q2), which until now has largely remained unexplored. Therefore, despite the available resources that allow tracking of user movement and spatial actions, there is not any hardware setup that solely focuses on tracking fine motor movements. Hence, our quantitative and qualitative assessments highlight the critical importance of precise tracking for an uninhibited and intuitive spatial manipulation experience for interaction close to the body. We see tremendous opportunity for improvement in commodity sensors to enable robust tracking at sub-millimeter scale.

While our behind-the-display interaction approach did not support a see-through display, it facilitated a co-located visuo-motor space for most users combining the user’s proprioceptive bi-manually coordinated actions. However, the most crucial challenge that we faced was for the Repr. interface, wherein the participants repeatedly encountered vision–proprioception disparity despite visual and tactile perception being provided for the peg-and-hole assembly task. The unexpected increase in completion time for the Repr. interface (Fig. 10) and higher energetic transitions (Fig. 9) is a critical example that points toward greater care in blending the visual and proprioceptive spaces. The action-specific visual perception [100,101] of manipulating such small objects will need co-locating the human motor zone with small-sized ultra-high resolution see-through displays. There is a promising research potential in leveraging 4K visual displays that mobile devices already possess for configuring AR and VR systems. Second, our study (Sec. 6.6) also suggests that instead of a physically realistic force and tactile feedback, what is really needed is a consistently mapped combination of visual and tactile cues that support each other seamlessly and get out of the user’s way when needed. In summary, it is important that design of future interaction spaces reflect and facilitate the innate blending of action-proprioception space (Q3) for performing high-precision interactions in intimate spaces where the sense of touch overshadows visual feedback in many cases [102].

High-precision manipulation tasks often require a higher-degree of hand–eye coordination whose effect can be seen in the motion trajectory of the action performed. From a kinematics standpoint, the movement and therefore manipulative ability enabled by the shoulder-arm-wrist complex is primarily responsible for ballistic actions [103] in the outer regions of the peripersonal space leading up to the extra-personal space (beyond arm’s length). This is the space in which current spatial interactions are typically designed and studied. In contrast, finger-level manipulations can afford to perform highly precise activities (such as hand writing and use sharp and fine tools) because of the highly redundant degrees-of-freedom [104]. A direct consequence of these kinematic differences is that the types of metrics (completion time and accuracy) typically used in spatial manipulation literature are not nearly enough to capture fine-grained spatial manipulations. We propose that body kinematics plays a major role in characterizing task precision as opposed to Euclidean metrics such as path efficiency and deviation proposed in related areas such as minimally invasive surgery [91–93]. In our prior attempts, none of these metrics were able to clearly elucidate the precise tasks studied in this article, i.e., they evaluated poorer user performance for Real experimental group followed by Repr. and finally, Generic that is counterintuitive to our assumption of Real being the gold standard for the remaining two experiment groups. To that effect, our proposed kinetic energy metric was able to clearly show differences across the three interfaces studied in this article as well facilitate with logical insights that could be corroborated with the motor strategies extracted from the video analysis (Sec. 6.1). In this article, we build upon existing knowledge on measuring precise actions and take initiative in proposing new metrics that focus on “how the action is performed” than focusing on the end outcomes of a given precise action. We believe that this will encourage future research on exploring metrics that appropriately precise spatial actions in the user’s peripersonal space (Q1). In fact, there is a rich set of research questions yet to be asked regarding the synchronization of head, arm, and finger level movements in precise spatial tasks.

The goal of this article was to study precision in spatial manipulation in the peripersonal space and understand motor strategies in this space. To this end, we developed three variants (real, representative, and generic) of a tablet-based virtual manipulation system with camera-based motion tracking of hand-held objects. The data collected through the peg–hole task using these variants shed light on the common motion strategies used while performing precise manipulation—bio-mechanical stability, object grasp, exploring the scene and objects through brief glances, and refining actions to ensure proper and intersection free alignment of peg–hole pairs. Moreover, we find that while the gross strategies remained the same across the three variants, the main change was observed in terms of finer finger-level manipulation, the cognitive load, the time taken, and the accuracy. The main underlying principle at play here is the quality of blending between the visual and tactile feedback in precise tasks and need a deeper investigation. Another critical issue that was revealed in our study was the need for defining metrics that align with the unique way in which our hands and fingers work. Here, our energy-based metrics seem to capture important information regarding the task. However, further research should be done to investigate other metrics, perhaps with the help of extremely precise tracking of hands, fingers, objects, and head. Overall, not only do our findings align with the current psychology and motor behavior of spatial manipulation, but they also pave way for research in the design, development, and evaluation of future spatial interfaces for fine-grained tasks, especially for 3D modeling and design applications. We believe that the learning outcomes of our study will motivate the development of better technology, principled approaches for interaction design, as well as new avenues for exploring high-precision spatial manipulation.

This material is based upon work supported by the National Science Foundation under Grant #2008800 and partially supported by AggieE_Challenge program, Texas A&M University Triads for Transformation (T3) program, and Orthopaedic Research and Education Foundation (Award #0014566A). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.