\section{Manipulating Objects with the Hands in AR}
\label{augmented_reality}

\AR devices generate and integrate virtual content into the user's perception of their real environment (\RE), creating the illusion of the \emph{presence} of the virtual \cite{azuma1997survey,skarbez2021revisiting}.
Among the different types of devices, \AR headsets leave the hands free to interact with virtual objects.
This promises natural and intuitive interactions similar to those with everyday real objects \cite{billinghurst2021grand,hertel2021taxonomy}.

\subsection{What is Augmented Reality?}
\label{what_is_ar}

The first \AR headset was invented by \textcite{sutherland1968headmounted}: With the technology available at the time, it was already capable of displaying virtual objects at a fixed point in space in real time, giving the user the illusion that the content was present in the room.
Fixed to the ceiling, the headset displayed a stereoscopic (one image per eye) perspective projection of the virtual content on a transparent screen, taking into account the user's position, and thus already following our interaction loop presented in \figref[introduction]{interaction-loop}.

\subsubsection{A Definition of AR}
\label{ar_definition}

The first formal definition of \AR was proposed by \textcite{azuma1997survey}: (1) combine real and virtual, (2) be interactive in real time, and (3) register real and virtual\footnotemark.
Each of these characteristics is essential: the real-virtual combination distinguishes \AR from \VR, a movie with integrated digital content is not interactive and a 2D overlay like an image filter is not registered.
There are also two key aspects of this definition: it does not focus on technology or method, but on the user's perspective of the system experience, and it does not specify a particular human sense, \ie it can be auditory \cite{yang2022audio}, haptic \cite{bhatia2024augmenting}, or even olfactory \cite{brooks2021stereosmell} or gustatory \cite{brooks2023taste}.
Yet, most research has focused on visual augmentation, and the term \AR (without a prefix) is almost always understood as visual \AR.

\footnotetext{This third characteristic has been slightly adapted to use the version of \textcite{marchand2016pose}, the original definition was: \enquote{registered in \ThreeD}.}

\begin{subfigs}{ar_applications}{Examples of \AR applications. }[][
  \item Visuo-haptic surgery training with cutting into virtual soft tisues \cite{harders2009calibration}.
  \item Interactive guide in document verification tasks by comparing with virtual references \cite{hartl2013mobile}.
  \item SpaceTop is transparent \AR desktop computer featuring direct hand manipulation of \ThreeD content \cite{lee2013spacetop}.
  \item Inner Garden is a spatial \AR zen garden made of real sand visually augmented to create a mini world that can be reshaped by hand \cite{roo2017inner}.
  ]
  \subfigsheight{41mm}
  \subfig{harders2009calibration}
  \subfig{hartl2013mobile}
  \subfig{lee2013spacetop}
  \subfig{roo2017inner}
\end{subfigs}

\subsubsection{Applications of AR}
\label{ar_applications}

Advances in technology, research, and development have enabled many uses of \AR, including medical, educational, industrial, navigation, collaboration, and entertainment applications \cite{dey2018systematic}.
For example, \AR can provide surgical training simulations in safe conditions \cite{harders2009calibration} (\figref{harders2009calibration}), or improve student learning of complex concepts and phenomena such as optics or chemistry \cite{bousquet2024reconfigurable}.
It can also guide workers in complex tasks, such as assembly, maintenance or verification \cite{hartl2013mobile} (\figref{hartl2013mobile}), reinvent the way we interact with desktop computers \cite{lee2013spacetop} (\figref{lee2013spacetop}), or can create complete new forms of gaming or tourism experiences \cite{roo2017inner} (\figref{roo2017inner}).
Most of (visual) \AR/\VR experiences can now be implemented with commercially available hardware and software solutions, especially for tracking, rendering and display.
However, the user experience in \AR is still highly dependent on the display used.

\subsubsection{AR Displays}
\label{ar_displays}

To experience a virtual content combined and registered with the \RE, an output device that display the \VE to the user is necessary.
There is a large variety of \AR displays with different methods of combining the real and virtual content, and different locations on the \RE or the user \cite[p.126]{billinghurst2015survey}.

In \emph{\VST-\AR}, the virtual images are superimposed to images of the \RE captured by a camera \cite{marchand2016pose}, and the combined real-virtual image is displayed on a screen to the user, as illustrated in \figref{itoh2022indistinguishable_vst}. An example is shown in \figref{hartl2013mobile}.
This augmented view through the camera has the advantage of a complete control on the real-virtual combination such as mutual occlusion between real and virtual objects \cite{macedo2023occlusion}, coherent lighting and no delay between the real and virtual images \cite{kruijff2010perceptual}.
But, due to the camera and the screen, the user's view is degraded with a lower resolution, frame rate, field of view, and an overall visual latency compared to proprioception \cite{kruijff2010perceptual}.

An \emph{\OST-\AR} display directly combines the virtual images with the real world view using a transparent optical system \cite{itoh2022indistinguishable} to be like augmented glasses, as illustrated in \figref{itoh2022indistinguishable_ost}. An example is shown in \figref{lee2013spacetop}.
These displays feature a direct, preserved view of the \RE at the cost of more difficult registration (spatial misalignment or temporal latency between the real and virtual content) \cite{grubert2018survey} and mutual real-virtual occlusion \cite{macedo2023occlusion}.

Finally, \emph{projection-based \AR} overlays the virtual images on the real world using a projector, as illustrated in \figref{roo2017one_2}, \eg \figref{roo2017inner}.
It doesn't require the user to wear the display, but requires a real surface to project the virtual on, and is vulnerable to shadows created by the user or the real objects \cite[p.137]{billinghurst2015survey}.

\begin{subfigs}{ar_displays}{Simplified operating diagram of \AR display methods. }[][
  \item \VST-\AR \cite{itoh2022indistinguishable}.
  \item \OST-\AR \cite{itoh2022indistinguishable}.
  \item Spatial \AR \cite{roo2017one}.
  ]
  \subfigsheight{44mm}
  \subfig{itoh2022indistinguishable_vst}
  \subfig{itoh2022indistinguishable_ost}
  \subfig{roo2017one_2}
\end{subfigs}

Regardless the \AR display, it can be placed at different locations \cite{bimber2005spatial}, as shown in \figref{roo2017one_1}.
\emph{Spatial \AR} is usually projection-based displays placed at fixed location (\figref{roo2017inner}), but it can also be \OST or \VST \emph{fixed windows} (\figref{lee2013spacetop}).
Alternatively, \AR displays can be \emph{hand-held}, like a \VST smartphone (\figref{hartl2013mobile}), or body-attached, like a micro-projector used as a flashlight \cite[p.141]{billinghurst2015survey}.
Finally, \AR displays can be head-worn like \VR \emph{headsets} or glasses, providing a portable experience.

\fig[0.75]{roo2017one_1}{Locations of \AR displays from eye-worn to spatially projected. Adapted by \textcite{roo2017one} from \textcite{bimber2005spatial}.}

\subsubsection{Presence and Embodiment in AR}
\label{ar_presence_embodiment}

Presence and embodiment are two key concepts that characterize the user experience in \AR and \VR.
While there is a large literature on these topics in \VR, they are less defined and studied for \AR \cite{genay2022being,tran2024survey}.
These concepts will be useful for the design, evaluation, and discussion of our contributions:
In particular, we will investigate the effect of the visual feedback of the virtual hand when touching haptic texture augmentation (\chapref{xr_perception}) and manipulating virtual objects (\chapref{visual_hand}), and explore the plausibility of co-localized visuo-haptic texture augmentations (\chapref{vhar_textures}).

\paragraph{Presence}
\label{ar_presence}

\AR and \VR are both essentially illusions.
The virtual content does not actually exist physically, but is only digitally simulated and rendered to the user's senses through display devices.
Such experience of disbelief suspension in \VR is what is called \emph{presence}, and it can be decomposed into two dimensions: place illusion and plausibility \cite{slater2009place,slater2022separate}.
\emph{Place illusion} is the sense of the user of \enquote{being there} in the \VE (\figref{presence-vr}).
It emerges from the real time rendering of the \VE from the user's perspective: to be able to move around inside the \VE and look from different point of views.
\emph{Plausibility} is the illusion that the virtual events are really happening, even if the user knows that they are not real.
It doesn't mean that the virtual events are realistic, \ie that reproduce the real world with high fidelity \cite{skarbez2017survey}, but that they are believable and coherent with the user's expectations.
In the same way, a film can be plausible even if it is not realistic, such as a cartoon or a science-fiction movie.

For \AR, \textcite{slater2022separate} proposed to invert place illusion to what we can call \enquote{object illusion}, \ie the sense of the virtual object to \enquote{feels here} in the \RE (\figref{presence-ar}).
As with \VR, virtual objects must be able to be seen from different angles by moving the head, but also, this is more difficult, appear to be coherent enough with the \RE \cite{skarbez2021revisiting}, \eg occlude or be occluded by real objects \cite{macedo2023occlusion}, cast shadows or reflect lights.
The plausibility can be applied to \AR as is, but the virtual objects must additionally have knowledge of the \RE and react accordingly to it to be, again, perceived as coherently behaving with the real world \cite{skarbez2021revisiting}.

\begin{subfigs}{presence}{
    The sense of immersion in virtual and \AEs. Adapted from \textcite{stevens2002putting}.
  }[][
  \item Place illusion is the sense of the user of \enquote{being there} in the \VE.
  \item Objet illusion is the sense of the virtual object to \enquote{feels here} in the \RE.
  ]
  \subfigsheight{35mm}
  \subfig{presence-vr}
  \subfig{presence-ar}
\end{subfigs}

\paragraph{Embodiment}
\label{ar_embodiment}

The \emph{sense of embodiment} is the \enquote{subjective experience of using and having a body} \cite{blanke2009fullbody}, \ie the feeling that a body is our own.
In everyday life, we are used to being, seeing and controlling our own body, but it is possible to embody a virtual body as an avatar while in \AR \cite{genay2022being} or \VR \cite{guy2023sense}.
This illusion arises when the visual, proprioceptive and (if any) haptic sensations of the virtual body are coherent \cite{kilteni2012sense}.
It can be decomposed into three subcomponents: \emph{Agency}, which is the feeling of controlling the body; \emph{Ownership}, which is the feeling that \enquote{the body is the source of the experienced sensations}; and \emph{Self-Location}, which is the feeling \enquote{spatial experience of being inside [the] body} \cite{kilteni2012sense}.
In \AR, it could take the form of body accessorization, \eg wearing virtual clothes or make-up in overlay, of partial avatarization, \eg using a virtual prothesis, or a full avatarization \cite{genay2022being}.

\subsection{Direct Hand Manipulation in AR}
\label{ar_interaction}

A user in \AR must be able to interact with the virtual content to fulfil the second point of \textcite{azuma1997survey}'s definition (\secref{ar_definition}) and complete our proposed visuo-haptic interaction loop (\figref[introduction]{interaction-loop}).
In all examples of \AR applications shown in \secref{ar_applications}, the user interacts with the \VE using their hands, either directly or through a physical interface.

\subsubsection{User Inputs and Interaction Techniques}
\label{interaction_techniques}

For a user to interact with a computer system (desktop, mobile, \AR, etc.), they first perceive the state of the system and then acts upon it through an input device \cite[p.145]{laviolajr20173d}.
Such input devices form an input \emph{\UI} that captures and translates user's actions to the computer.
Similarly, an output \UI render and display the state of the system to the user (such as an \AR/\VR display, \secref{ar_displays}, or an haptic actuator, \secref{wearable_haptic_devices}).

Inputs \UI can be either an \emph{active sensing}, \ie a held or worn device, such as a mouse, a touch screen, or a hand-held controller, or a \emph{passive sensing}, that does not require a contact, such as eye trackers, voice recognition, or hand tracking \cite[p.294]{laviolajr20173d}.
The captured information from the sensors is then translated into actions within the computer system by an \emph{interaction technique}.
For example, a cursor on a screen can be moved using either with a mouse or with the arrow keys on a keyboard, or a two-finger swipe on a touchscreen can be used to scroll or zoom an image.
Choosing useful and efficient \UIs and interaction techniques is crucial for the user experience and the tasks that can be performed within the system.

\subsubsection{Tasks with Virtual Environments}
\label{ve_tasks}

\textcite{laviolajr20173d} (p.385) classify interaction techniques into three categories based on the tasks they enable users to perform: manipulation, navigation, and system control.
\textcite{hertel2021taxonomy} proposed a similar taxonomy of interaction techniques specifically for \AR headsets.

The \emph{manipulation tasks} are the most fundamental tasks in \AR and \VR systems, and the building blocks for more complex interactions.
\emph{Selection} is the identification or acquisition of a specific virtual object, \eg pointing at a target as in \figref{grubert2015multifi}, touching a button with a finger, or grasping an object with a hand.
\emph{Positioning} and \emph{rotation} of a selected object are the change of its position and orientation in \ThreeD space respectively.
It is also common to \emph{resize} a virtual object to change its size (\figref{piumsomboon2013userdefined_2}).
These three operations are geometric (rigid) manipulations of the object: they do not change its shape.

The \emph{navigation tasks} are the movements of the user within the \VE.
Travel is the control of the position and orientation of the viewpoint in the \VE, \eg physical walking, velocity control, or teleportation.
Wayfinding is the cognitive planning of the movement, such as path finding or route following (\figref{grubert2017pervasive}).

The \emph{system control tasks} are changes to the system state through commands or menus such as creating, deleting, or modifying virtual objects, \eg as in \figref{roo2017onea}. It is also the input of text, numbers, or symbols.

\comans{JG}{In, Figure 2.24 I suggest removing d. or presenting it as separate figure as it shows no interaction technique (The caption is “Interaction techniques in AR” but a visualization of a spatial registration technique).}{It has been removed and replaced by an example of resizing a virtual object.}
In this thesis we focus on manipulation tasks of virtual content directly with the hands, more specifically on touching visuo-haptic textures with a finger (\partref{perception}) and positioning and rotating virtual objects pushed and grasp by the hand (\partref{manipulation}).

\begin{subfigs}{interaction-techniques}{Interaction techniques in \AR. }[][
  \item Spatial selection of virtual item of an extended display using a hand-held smartphone \cite{grubert2015multifi}.
  \item Resizing a virtual object with a bimanual gesture \cite{piumsomboon2013userdefined}.
  \item Displaying as an overlay registered on the \RE the route to follow \cite{grubert2017pervasive}.
  \item Virtual drawing on a real object with a hand-held pen \cite{roo2017onea}.
  ]
  \subfigsheight{35.5mm}
  \subfigbox{grubert2015multifi}
  \subfigbox{piumsomboon2013userdefined_2}
  \subfigbox{grubert2017pervasive}
  \subfigbox{roo2017onea}
\end{subfigs}

\subsubsection{The Gap between Real and Virtual}
\label{real_virtual_gap}

In \AR and \VR, the state of the system is displayed to the user as a \ThreeD spatial \VE.
With an \AR headset, the \VE can be experienced at a 1:1 scale and as an integral part of the \RE.
The rendering gap between the real and virtual elements, as described on our interaction loop in \figref[introduction]{interaction-loop}, is thus experienced as narrow or even not consciously perceived by the user.
This manifests as a sense of presence of the virtual, as described in \secref{ar_presence}.

As the gap between real and virtual rendering is reduced, one could expect a similar and seamless interaction with the \VE as with a \RE, which \textcite{jacob2008realitybased} called \emph{reality based interactions}.
As of today, an \AR system tracks itself with the user in \ThreeD, using tracking sensors and pose estimation algorithms \cite{marchand2016pose}.
It enables the \VE to be registered with the \RE and the user simply moves to navigate within the virtual content.
However, direct hand manipulation of virtual content is a challenge that requires specific interaction techniques \cite{billinghurst2021grand}.
It is often achieved using two interaction techniques: \emph{tangible objects} and \emph{virtual hands} \cite[p.165]{billinghurst2015survey}.

\subsubsection{Manipulating with Tangibles}
\label{ar_tangibles}

As \AR integrates visual virtual content into \RE perception, it can involve real surrounding objects as \UI: to either visually augment them (\figref{roo2017inner}), or to use them as physical proxies to support interaction with virtual objects \cite{ishii1997tangible}.
Each virtual object is coupled to a real object and physically manipulated through it, providing a direct, efficient and seamless interaction with both the real and virtual content \cite{billinghurst2005designing}.
The real objects are called \emph{tangible} in this usage context.

Methods have been developed to automatically pair and adapt the virtual objects for rendering with available tangibles of similar shape and size \cite{hettiarachchi2016annexing,jain2023ubitouch} (\figref{jain2023ubitouch}).
The issue with these \emph{space-multiplexed} interfaces is the large number and variety of tangibles required.
An alternative is to use a single \emph{universal} tangible object like a hand-held controller, such as a cube \cite{issartel2016tangible} or a sphere \cite{englmeier2020tangible}.
These \emph{time-multiplexed} interfaces require interaction techniques that allow the user to pair the tangible with any virtual object, \eg by placing the tangible into the virtual object and pressing the fingers \cite{issartel2016tangible} (\figref{issartel2016tangible}), similar to a real grasp (\secref{grasp_types}).

Still, the virtual visual rendering and the real haptic sensations can be incoherent.
In \VR, some discrepancy between the real and virtual objects is acceptable because the real object is not visible to the user \cite{detinguy2019how,detinguy2019universal}.
In \AR, however, the real object may be partially or fully visible, and the user can see that their hand is not touching the real and virtual objects at the same time.
This is particularly true in \OST-\AR, where the virtual objects are inherently slightly transparent allowing the paired real objects to be seen through them \cite{macedo2023occlusion}.
In a pick-and-place task with real objects in \OST-\AR, a difference in size \cite{kahl2021investigation} (\figref{kahl2021investigation}) and shape \cite{kahl2023using} (\figref{kahl2023using_1}) of the virtual objects does not affect user performance or presence, and that small variations (\percent{\sim 10} for size) were not even noticed by the users.
This suggests the feasibility of using simplified real objects in \AR whose spatial properties (\secref{object_properties}) abstract those of the virtual objects.
Similarly, in \secref{tactile_rendering} we described how a material property (\secref{object_properties}) of a touched real object can be modified using wearable haptic devices \cite{detinguy2018enhancing,salazar2020altering}: it could be used to render coherent visuo-haptic material perceptions directly touched with the hand in \AR.

\begin{subfigs}{ar_tangibles}{Manipulating virtual objects through real objects. }[][
  \item Ubi-Touch paired the movements and screw interaction of a virtual drill with a real vaporizer held by the user \cite{jain2023ubitouch}.
  \item A real cube that can be moved into the \VE and used to grasp and manipulate virtual objects \cite{issartel2016tangible}.
  \item Size and
  \item shape difference between a real object and a virtual one is acceptable for manipulation in \AR \cite{kahl2021investigation,kahl2023using}.
  ]
  \subfigsheight{37.5mm}
  \subfig{jain2023ubitouch}
  \subfig{issartel2016tangible}
  \subfig{kahl2021investigation}
  \subfig{kahl2023using_1}
\end{subfigs}

\subsubsection{Manipulating with Virtual Hands}
\label{ar_virtual_hands}

Initially tracked by active sensing devices such as gloves or controllers, it is now possible to track hands in real time using passive sensing (\secref{interaction_techniques}) and computer vision algorithms natively integrated into \AR/\VR headsets \cite{tong2023survey}.
Our hands allow us to manipulate real everyday objects (\secref{grasp_types}), hence virtual hand interaction techniques seem to be the most natural way to manipulate virtual objects \cite[p.400]{laviolajr20173d}.

The user's hand being tracked is reconstructed as a \emph{virtual hand} model in the \VE \cite[p.405]{laviolajr20173d}.
The simplest models represent the hand as a rigid \ThreeD object that follows the movements of the real hand with 6 \DoF (position and orientation in space) \cite{talvas2012novel}.
An alternative is to model only the fingertips (\figref{lee2007handy}) or the whole hand (\figref{hilliges2012holodesk_1}) as points.
The most common technique is to reconstruct all the phalanges of the hand in an articulated kinematic model (\secref{hand_anatomy}) \cite{borst2006spring}.

The contacts between the virtual hand model and the virtual objects are then simulated using heuristic or physics-based techniques \cite[p.405]{laviolajr20173d}.
Heuristic techniques use rules to determine the selection, manipulation and release of a virtual object (\figref{piumsomboon2013userdefined_1}).
However, they produce unrealistic behaviour and are limited to the cases predicted by the rules.
Physics-based techniques simulate forces at the points of contact between the virtual hand and the virtual object.
In particular, \textcite{borst2006spring} proposed an articulated kinematic model in which each phalanx is a rigid body simulated with the god-object method \cite{zilles1995constraintbased}:
The virtual phalanx follows the movements of the real phalanx, but remains constrained to the surface of the virtual objects during contact.
The forces acting on the object are calculated as a function of the distance between the real and virtual hands (\figref{borst2006spring}).
More advanced techniques simulate the friction phenomena \cite{talvas2013godfinger} and finger deformations \cite{talvas2015aggregate}, allowing highly accurate and realistic interactions, but which can be difficult to compute in real time.

\begin{subfigs}{virtual-hand}{Manipulating virtual objects with virtual hands. }[][
  \item A fingertip tracking that allows to select a virtual object by opening the hand \cite{lee2007handy}.
  \item Physics-based hand-object manipulation with a virtual hand made of numerous many small rigid-body spheres \cite{hilliges2012holodesk}.
  \item Grasping a through gestures when the fingers are detected as opposing on the virtual object \cite{piumsomboon2013userdefined}.
  \item A kinematic hand model with rigid-body phalanges (in beige) that follows the real tracked hand (in green) but kept physically constrained to the virtual object. Applied forces are shown as red arrows \cite{borst2006spring}.
  ]
  \subfigsheight{37mm}
  \subfigbox{lee2007handy}
  \subfigbox{hilliges2012holodesk_1}
  \subfigbox{piumsomboon2013userdefined_1}
  \subfigbox{borst2006spring}
\end{subfigs}

However, the lack of physical constraints on the user's hand movements makes manipulation actions tiring \cite{hincapie-ramos2014consumed}.
While the user's fingers traverse the virtual object, a physics-based virtual hand remains in contact with the object, a discrepancy that may degrade the user's performance in \VR \cite{prachyabrued2012virtual}.
Finally, in the absence of haptic feedback on each finger, it is difficult to estimate the contact and forces exerted by the fingers on the object during grasping and manipulation \cite{maisto2017evaluation,meli2018combining}.
While a visual feedback of the virtual hand in \VR can compensate for these issues \cite{prachyabrued2014visual}, the visual and haptic feedback of the virtual hand, or their combination, in \AR needs to be investigated as well.

\subsection{Visual Feedback of Virtual Hands in AR}
\label{ar_visual_hands}

When interacting with a physics-based virtual hand method (\secref{ar_virtual_hands}) in \VR, the visual feedback of the virtual hand has an influence on perception, interaction performance, and preference of users \cite{prachyabrued2014visual,argelaguet2016role,grubert2018effects,schwind2018touch}.
In a pick-and-place manipulation task in \VR, \textcite{prachyabrued2014visual} and \textcite{canales2019virtual} found that the visual hand feedback whose motion was constrained to the surface of the virtual objects similar as to \textcite{borst2006spring} (\enquote{Outer Hand} in \figref{prachyabrued2014visual}) performed the worst, while the visual hand feedback following the tracked human hand (thus penetrating the virtual objects, \enquote{Inner Hand} in \figref{prachyabrued2014visual}) performed the best, though it was rather disliked.
\textcite{prachyabrued2014visual} also found that the best compromise was a double feedback, showing both the virtual hand and the tracked hand (\enquote{2-Hand} in \figref{prachyabrued2014visual}).
While a realistic rendering of the human hand increased the sense of ownership \cite{lin2016need}, a skeleton-like rendering provided a stronger sense of agency \cite{argelaguet2016role} (\secref{ar_embodiment}), and a minimalist fingertip rendering reduced typing errors \cite{grubert2018effects}.
A visual hand feedback while in \VE also seems to affect how one grasps an object \cite{blaga2020too}, or how real bumps and holes are perceived \cite{schwind2018touch}.

\fig{prachyabrued2014visual}{Visual hand feedback affect user experience in \VR \cite{prachyabrued2014visual}.}

Conversely, a user sees their own hands in \AR, and the mutual occlusion between the hands and the virtual objects is a common issue (\secref{ar_displays}), \ie hiding the virtual object when the real hand is in front of it, and hiding the real hand when it is behind the virtual object (\figref{hilliges2012holodesk_2}).
While in \VST-\AR, this could be solved as a masking problem by combining the real and virtual images \cite{battisti2018seamless}, \eg in \figref{suzuki2014grasping}, in \OST-\AR, this is much more difficult because the \VE is displayed as a transparent 2D image on top of the \ThreeD \RE, which cannot be easily masked \cite{macedo2023occlusion}.

Since the \VE is intangible, adding a visual feedback of the virtual hand in \AR that is physically constrained to the virtual objects would achieve a similar result to the double-hand feedback of \textcite{prachyabrued2014visual}.
A virtual object overlaying a real object in \OST-\AR can vary in size and shape without degrading user experience or manipulation performance \cite{kahl2021investigation,kahl2023using}.
This suggests that a visual hand feedback superimposed on the real hand as a partial avatarization (\secref{ar_embodiment}) might be helpful without impairing the user.

Few works have compared different visual feedback of the virtual hand in \AR or with wearable haptic feedback.
Rendering the real hand as a semi-transparent hand in \VST-\AR is perceived as less natural but seems to be preferred to a mutual visual occlusion for interaction with real and virtual objects \cite{buchmann2005interaction,piumsomboon2014graspshell}.
Similarly, \textcite{blaga2017usability} evaluated direct hand manipulation in \VST-\AR with a skeleton-like rendering \vs no visual hand feedback: while user performance did not improve, participants felt more confident with the virtual hand (\figref{blaga2017usability}).
In a collaborative task in \OST-\AR \vs \VR headsets, \textcite{yoon2020evaluating} showed that a realistic human hand rendering was the most preferred over a low-polygon hand and a skeleton-like hand for the remote partner.
\textcite{genay2021virtual} found that the sense of embodiment with robotic hands overlay in \OST-\AR was stronger when the environment contained both real and virtual objects (\figref{genay2021virtual}).
Finally, \textcite{maisto2017evaluation} and \textcite{meli2018combining} compared the visual and haptic feedback of the hand in \VST-\AR, as detailed in the next section (\secref{vhar_rings}).
Taken together, these results suggest that a visual augmentation of the hand in \AR could improve usability and performance in direct hand manipulation tasks, but the best rendering has yet to be determined.

\begin{subfigs}{visual-hands}{Visual feedback of the virtual hand in \AR. }[][
  \item Grasping a virtual object in \OST-\AR with no visual hand feedback \cite{hilliges2012holodesk}.
  \item Simulated mutual-occlusion between the hand grasping and the virtual object in \VST-\AR \cite{suzuki2014grasping}.
  \item Grasping a real object with a semi-transparent hand in \VST-\AR \cite{buchmann2005interaction}.
  \item Skeleton rendering overlaying the real hand in \VST-\AR \cite{blaga2017usability}.
  \item Robotic rendering overlaying the real hands in \OST-\AR \cite{genay2021virtual}.
  ]
  \subfigsheight{29.5mm}
  \subfig{hilliges2012holodesk_2}
  \subfig{suzuki2014grasping}
  \subfig{buchmann2005interaction}
  \subfig{blaga2017usability}
  \subfig{genay2021virtual}
\end{subfigs}

\subsection{Conclusion}
\label{ar_conclusion}

\AR systems integrate virtual content into the user's perception as if it were part of the \RE.
\AR headsets now enable real-time pose estimation of the head and hands, and high-quality display of virtual content, while being portable and mobile.
They create \AEs that users can explore with a strong sense of the presence of the virtual content.
However, without direct and seamless interaction with the virtual objects using the hands, the coherence of the \AE experience is compromised.
In particular, when manipulating virtual objects in \OST-\AR, there is a lack of mutual occlusion and interaction cues between the hands and the virtual content, which could be mitigated by a visual augmentation of the hand.
A common alternative approach is to use real objects as proxies for interaction with virtual objects, but this raises concerns about their coherence with visual augmentations.
In this context, the use of wearable haptic systems worn on the hand seems to be a promising solution both for improving direct hand manipulation of virtual objects and for coherent visuo-haptic augmentation of touched real objects.
