\chapter{Conclusion}
\mainlabel{conclusion}

\chaptertoc

We conclude this thesis manuscript by summarizing our contributions and the main findings of our research, and by presenting future work and perspectives for each of our research axes.

\section{Summary}

In this manuscript, we showed how \OST-\AR headsets and wearable haptics can improve direct hand interaction with virtual and augmented objects. % by augmenting the perception of the real and manipulation of the virtual.
Wearable haptics can provide rich tactile feedback on virtual objects and augment the perception of real objects, both directly touched by the hand, while preserving freedom of movement and interaction with the \RE.
However, their integration with \AR is still in its infancy and presents many design, technical and human challenges.
We have structured this thesis around two research axes: \textbf{(I) modifying the visuo-haptic texture perception of real surfaces} and \textbf{(II) improving the manipulation of virtual objects}.

\noindentskip In \partref{perception}, we focused on the perception of wearable virtual textures that augment real surfaces.
Texture is a fundamental property of an object, perceived equally by sight and touch.
It is also one of the most studied haptic augmentations, but has not yet been integrated into \AR or \VR.
We \textbf{(1) proposed a wearable visuo-haptic texture augmentation system}, \textbf{(2)} evaluated how the perception of haptic texture augmentations is \textbf{affected by the visual feedback of the virtual hand} and the environment (real, augmented, or virtual), and \textbf{(3)} investigated the \textbf{perception of co-localized visuo-haptic texture augmentations}.

In \chapref{vhar_system}, we presented a system for \textbf{augmenting any real surface} with virtual \textbf{roughness textures with visuo-haptic feedback} using an \OST-\AR headset and a wearable vibrotactile device worn on the middle phalanx of the finger.
It allows \textbf{free visual and touch exploration} of the textures as if they were real, allowing the user to view them from different angles and touch them with the bare finger without constraints on hand movement.
The user studies in the next two chapters were based on this system.

In \chapref{xr_perception}, we explored how the perception of wearable haptic augmented textures is affected by the visual feedback of the virtual hand and the environment, whether it is real, augmented or virtual.
We augmented the perceived roughness of the real surface with virtual vibrotactile patterned textures and rendered the visual conditions by switching the \OST-\AR headset to a \VR-only view.
We then conducted a psychophysical user study with 20 participants to evaluate the perceived roughness augmentation in these three visual conditions.
The textures were perceived as \textbf{rougher when touched with the real hand alone compared to a virtual hand} in either \AR or \VR, possibly due to the \textbf{perceived latency} between finger movements and different visual, haptic, and proprioceptive feedbacks.

In \chapref{vhar_textures}, we investigated the perception of co-localized visual and wearable haptic texture augmentations on real surfaces.
We transposed the \textbf{data-driven visuo-haptic textures} from the \HaTT database to the system presented in \chapref{vhar_system} and conducted a user study with 20 participants to rate the coherence, realism, and perceived roughness of the combination of nine visuo-haptic texture pairs.
Participants integrated roughness sensations from both visual and haptic modalities well, with \textbf{haptics dominating perception}, and consistently identified and matched \textbf{clusters of visual and haptic textures with similar perceived roughness}.

\noindentskip In \partref{manipulation}, we focused on improving the manipulation of virtual objects directly with the hand using an \OST-\AR headset.
Our approach was to design visual augmentations of the hand and delocalized haptic feedback, based on the literature, and evaluate them in user studies.
We first considered \textbf{(1) the visual augmentation of the hand} and then the \textbf{(2)} combination of different \textbf{visuo-haptic feedback of the hand when manipulating virtual objects}.

In \chapref{visual_hand}, we investigated the visual feedback of the virtual hand as an augmentation of the real hand.
Seen as an \textbf{overlay on the user's hand}, it provides feedback on hand tracking and interaction with virtual objects.
We compared the six commonly used visual hand augmentations in the \AR literature in a user study with 24 participants, where we evaluated their effect on user performance and experience in two representative manipulation tasks.
The results showed that a visual hand augmentation improved user performance, perceived effectiveness and confidence, with a \textbf{skeleton-like rendering being the most performant and effective}.
This rendering provided a detailed view of the tracked phalanges while being thin enough not to hide the real hand.

In \chapref{visuo_haptic_hand}, we then investigated visuo-haptic feedback to direct hand manipulation with virtual objects using wearable vibrotactile haptics.
In a user study with a similar design and 20 participants, we compared two vibrotactile contact techniques, provided at \textbf{four different delocalized positions on the user's hand}, and combined with the two most representative visual hand augmentations from the previous chapter.
The results showed that providing vibrotactile feedback \textbf{improved the perceived effectiveness, realism, and usefulness when provided close to the fingertips} and that the visual hand augmentation complemented the haptic contact feedback in providing a continuous feedback on hand tracking.

\section{Future Work}

The wearable visuo-haptic feedback we presented for augmenting the perception of real objects (\partref{perception}) and the manipulation of virtual objects (\partref{manipulation}) have some limitations and open questions.
In this section we present some future work for each chapter that could address these issues.

\subsection*{Augmenting the Visuo-haptic Texture Perception of Real Surfaces}

\paragraph{Augmented Object Properties.}

We focused on visuo-haptic augmentation of roughness using vibrotactile feedback, because it is one of the most salient properties of surfaces (\secref[related_work]{object_properties}), one of the most studied in haptic perception (\secref[related_work]{texture_rendering}), and equally perceived by sight and touch (\secref[related_work]{visual_haptic_influence}).
However, many other wearable augmentation of object properties should be considered, such as hardness, friction, temperature, or local deformations.
Such integration of haptic augmentation of a real surface has been addressed with the hand-held devices of \citeauthor{culbertson2017ungrounded} \cite{culbertson2017importance,culbertson2017ungrounded}, but will be more challenging with wearable haptic devices.
In addition, combination with pseudo-haptic rendering techniques \cite{ujitoko2021survey} should be systematically investigated to expand the range of possible wearable haptic augmentations.

\paragraph{Fully Integrated Tracking.}

In our system, we registered the \RE and the \VE using fiducial markers and a webcam external to the \AR headset.
This only allowed us to estimate poses of the index finger and the surface to be augmented with the haptic texture, but it was reliable and accurate enough for our needs.
In fact, preliminary tests we conducted showed that the built-in tracking capabilities of the HoloLens~2 were not able to track hands wearing a vibrotactile voice-coil device.
A more robust hand pose estimation system would support wearing haptic devices on the hand as well as holding real objects.

The spatial registration error \cite{grubert2018survey} and the temporal latency \cite{diluca2019perceptual} between the \RE and \VE should also be reduced to be imperceptible.
The effect of these spatial and temporal errors on the perception and manipulation of the virtual object should be systematically investigated.
Prediction of hand movements should also be considered to overcome such issues \cite{klein2020predicting,gamage2021predictable}.

A complementary solution would be to embed tracking sensors in the wearable haptic devices, such as an inertial measurement unit (IMU) or cameras \cite{preechayasomboon2021haplets}.
This would allow a complete portable and wearable visuo-haptic system to be used in practical applications.

\subsection*{Perception of Haptic Texture Augmentation in Augmented and Virtual Reality}

\paragraph{Visual Representation of the Virtual Texture.}

In our user study, we assessed the effect of touching a vibrotactile texture augmentation with a real hand or a virtual hand, in \AR or \VR.
To control for the visual feedback, we decided not to display the virtual texture so that participants only saw and touched a uniform white real surface.
The visual information of a texture is as important as the haptic sensations for the perception of roughness, and the interaction between the two to form the overall texture perception is complex \cite{bergmanntiest2007haptic,yanagisawa2015effects,vardar2019fingertip}.
In particular, it remains to be investigated how the vibrotactile patterned textures we employed can be represented visually in a convincing way, as the visuo-haptic coupling of such virtual patterned textures is not trivial \cite{unger2011roughness}. % even with real textures \cite{klatzky2003feeling}.

\paragraph{Broader Visuo-Haptic Conditions.}

Our study was conducted with an \OST-\AR headset, but the results may be different with a \VST-\AR headset, where the \RE is seen through cameras and screens (\secref[related_work]{ar_displays}), and the perceived simultaneity between visual and haptic stimuli, real or virtual, may be different \cite{knorlein2009influence}.
The effect of perceived visuo-haptic simultaneity on the augmented haptic perception and its magnitude should also be systematically investigated, for example by inducing different delays between visual and haptic feedback \cite{diluca2011effects}.

We also focused on the perception of roughness augmentation using wearable vibrotactile haptics and a square wave signal to simulate a patterned texture.
Our objective was not to accurately reproduce real textures, but to induce different perceived roughness on the same real surface with well controlled parameters.
However, more accurate models for simulating interaction with virtual textures should be applied to wearable haptic augmentations \cite{unger2011roughness}.

Another limitation that may have affected the perception of the haptic texture augmentations is the lack of compensation for the frequency response of the actuator and amplifier \cite{asano2012vibrotactile,culbertson2014modeling,friesen2024perceived}.
The dynamic response of the finger should also be considered, and may vary between individuals \cite{delhaye2012textureinduced}.

\subsection*{Perception of Visual and Haptic Texture Augmentations in Augmented Reality}

\paragraph{Applicability of the Method.}

As in the previous chapter, our aim was not to accurately reproduce real textures, but to alter the perception of a real surface being touched with simultaneous visual and haptic texture augmentations.
However, the results also have some limitations, as they addressed a small set of visuo-haptic textures that augmented the perception of smooth and white real surfaces.
Visuo-haptic texture augmentation might be difficult on surfaces that already have strong visual or haptic patterns \cite{asano2012vibrotactile}, or on objects with complex shapes.
A real surface could be indeed augmented not only to add visuo-haptic textures, but also to amplify, diminish, mask, or replace the existing real texture.
In addition, the visual textures used were simple color images not intended for use in an \ThreeD \VE, and enhancing their visual quality could improve the perception of visuo-haptic texture augmentation.
It would also be interesting to replicate the experiment in more controlled visuo-haptic environments, in \VR or with world-grounded haptic devices.
This would enable to better understand how the rendering quality, spatial registration and latency of virtual textures can affect their perception.
Finally, the role of visuo-haptic texture augmentation should also be evaluated in more complex tasks, such as object recognition and assembly, or in more concrete use cases, such as displaying and touching a museum object or a 3D printed object before it is manufactured.

\paragraph{Specificities of Direct Touch.}

The haptic textures used were recordings and models of the vibrations of a hand-held probe sliding over real surfaces \cite{culbertson2014one}.
We generated the vibrotactile textures from velocity magnitude of the finger, but the perceived roughness of real textures also depends on other factors such as the contact force, angle, posture or surface of the contact \cite{schafer2017transfer}.
The respective importance of these factors on the haptic texture perception is not yet fully understood \cite{richardson2022learning}.
It would be interesting to determine the importance of these factors on the perceived realism of virtual vibrotactile textures in the context of bare finger touch.
Finger based captures of real textures should also be considered \cite{balasubramanian2024sens3}.
Finally, the virtual texture models should also be adaptable to individual sensitivities \cite{malvezzi2021design,young2020compensating}.

\subsection*{Visual Augmentation of the Hand for Manipulating virtual objects in AR}

\paragraph{Other AR Displays.}

The visual hand augmentations we evaluated were displayed on the Microsoft HoloLens~2, which is a common \OST-\AR headset \cite{hertel2021taxonomy}.
We purposely chose this type of display, because in \OST-\AR the lack of mutual occlusion between the hand and the virtual object is the most challenging to solve \cite{macedo2023occlusion}.
We therefore hypothesized that a visual hand augmentation would be more beneficial to users with this type of display.
However, the user's visual perception and experience are different with other types of displays, such as \VST-\AR, where the \RE view is seen through cameras and screens (\secref[related_work]{ar_displays}).
In particular, the mutual occlusion problem and the latency of hand pose estimation  could be overcome with a \VST-\AR headset.
In this case, the occlusion rendering could be the most natural, realistic and effective augmentation.
Yet, a visual hand augmentation could still be beneficial to users by providing depth cues and feedback on hand tracking, and should be evaluated as such.

\paragraph{More Practical Usages.}

We conducted the user study with two manipulation tasks that involved placing a virtual cube in a target volume, either by pushing it on a table or by grasping and lifting it.
These tasks are indeed fundamental building blocks for more complex manipulation tasks \cite[p.390]{laviolajr20173d} such as stacking or assembling, which should be investigated as well.
They can indeed require users to perform more complex finger movements and interactions with the virtual object.
Depending on the task, the importance of position, orientation and depth information of the hand and the object may vary and affect the choice of visual hand augmentation.
More practical applications should also be considered, such as medical, educational or industrial scenarios, which may have different needs and constraints (\eg, the most natural visual hand augmentation for a medical application, or the easiest to understand and use for an educational context).

Similarly, a broader experimental study might shed light on the role of gender and age, as our subject pool was not sufficiently diverse in this regard.
Finally, all visual hand augmentations received low and high rank rates from different participants, suggesting that users should be able to choose and personalize some aspects of the visual hand augmentation according to their preferences or needs, and this should also be evaluated.

\subsection*{Visuo-Haptic Augmentation of Hand Manipulation With virtual objects in AR}

\paragraph{More Diverse Haptic Stimuli.}

The haptic feedback we considered was limited to vibrotactile feedback using \ERM motors.
The simpler contact vibration technique (Impact technique) was sufficient to confirm contact with the cube.
However, more diverse vibrotactile stimuli may be required for more complex interactions, such as rendering hardness (\secref[related_work]{hardness_rendering}), textures (\secref[related_work]{texture_rendering}), friction \cite{konyo2008alternative,jeon2011extensions,salazar2020altering}, or edges and shape of virtual objects.
This will require considering a wider ranger of haptic actuators and sensations (\secref[related_work]{wearable_haptic_devices}), such as pressure or stretching of the skin.
More importantly, the best compromise between well-rounded haptic feedback and wearability of the system with respect to \AR constraints should be analyzed (\secref[related_work]{vhar_haptics}).

\paragraph{Personalized Haptics.}

Some users found the vibration feedback too strong, suggesting that adapting and personalizing the haptic feedback to one's preference should be investigated \cite{malvezzi2021design,umair2021exploring}.
In addition, although it was perceived as more effective and realistic when provided close to the point of contact, other positionings, such as the wrist, may be preferred and still be sufficient for a given task.
The interactions in our user study were also restricted to the thumb and index fingertips, with haptic feedback provided only for these contact points, as these are the most commonly used parts of the hand for manipulation tasks.
It remains to be explored how to support rendering for different and larger areas of the hand, and how to position a delocalized feedback for points other than the fingertips could be challenging.

\section{Perspectives}

Our goal was to improve direct hand interaction with virtual objects using wearable haptic devices and an \OST-\AR headset.
We aimed to provide more plausible and coherent perception and more natural and effective manipulation of the visuo-haptic augmentations.
Our contributions have enabled progress towards a seamless integration of the virtual into the real world.
They also allow us to outline longer-term research perspectives.

\subsection*{Towards Universal Wearable Haptic Augmentation}

We saw how complex the sense of touch is (\secref[related_work]{haptic_hand}).
Multiple sensory receptors all over the skin allow us to perceive different properties of objects, such as their texture, temperature, weight or shape.
Particularly concentrated in the hands, cutaneous sensory feedback, together with the muscles, is crucial for grasping and manipulating objects.
In this manuscript, we showed how wearable haptic devices can provide virtual tactile sensations to support direct hand interaction with an \OST-\AR headset.
We investigated both the visuo-haptic perception of texture augmenting real surfaces (\partref{perception}) and the manipulation of virtual objects with visuo-haptic feedback of hand contact with virtual objects (\partref{manipulation}).

However, unlike the visual sense, which can be fully immersed in the virtual using an \AR/\VR headset, there is no universal wearable haptic device that can reproduce all the haptic properties perceived by the hand (\secref[related_work]{wearable_haptics}).
Thus, the haptic renderings and augmentations we studied were limited to specific properties of roughness (\chapref{vhar_system}) and contact (\chapref{visuo_haptic_hand}) using vibrotactile feedback.
A systematic and comparative study of existing wearable haptic devices and renderings should therefore be carried out to assess their ability to reproduce the various haptic properties \cite{culbertson2017importance,friesen2024perceived}.
More importantly, the visuo-haptic coupling of virtual and augmented objects should be studied systematically, as we did for textures in \AR (\chapref{vhar_textures}) or as done in \VR \cite{choi2021augmenting,gunther2022smooth}.
Attention should also be paid to the perceptual differences of wearable haptics in \AR \vs \VR (\chapref{xr_perception}).
This would allow to assess the relative importance of visual and haptic feedback in the perception of object properties, and how visual feedback may support or compensate for limitations in wearable haptic feedback \cite{kim2020defining}.

One of the main findings of studies on the haptic perception of real objects is the importance of certain perceived properties over others in discriminating between objects \cite{hollins1993perceptual,baumgartner2013visual,vardar2019fingertip}.
It would therefore be interesting to determine which wearable haptic augmentations are most important for the perception and manipulation of virtual and augmented objects with the hand in \AR and \VR.
Similar user studies could then be conducted, to reproduce as many haptic properties as possible in virtual object discrimination tasks.
These results would enable the design of more universal wearable haptic devices that provide rich haptic feedback that best meets users' needs for interaction in \AR and \VR.

% systematic exploration of the parameter space of the haptic rendering to determine the most important parameters their influence on the perception
% measure the difference in sensitivity to the haptic feedback and how much it affects the perception of the object properties

\subsection*{Responsive Visuo-Haptic Augmented Reality}

We reviewed the diversity of \AR and \VR reality displays and their respective characteristics in rendering (\secref[related_work]{ar_displays}) and the manipulation of virtual content with the hand (\chapref{visual_hand}).
The diversity of wearable haptic devices and the different sensations they can provide is even more important (\secref[related_work]{wearable_haptics}) and an active research topic \cite{pacchierotti2017wearable}.
Coupling wearable haptics with \AR headsets also requires the haptic actuator to be placed on the body other than at the hand contact points (\secref[related_work]{vhar_haptics}).
In particular, in this thesis we investigated the perception of haptic texture augmentation using a vibrotactile device on the median phalanx (\chapref{vhar_system}) and also compared different positions of the haptics on the hand for manipulating virtual objects (\chapref{visuo_haptic_hand}).

Haptic feedback should be provided close to the point of contact of the hand with the virtual, to enhance the realism of texture augmentation (\chapref{vhar_textures}) and to render contact with virtual objects (\chapref{visuo_haptic_hand}), \eg rendering fingertip contact with a haptic ring worn on the middle or proximal phalanx.
However, the task at hand, the user's sensitivity and preferences, the limitations of the tracking system, or the ergonomics of the haptic device may require the use of other form factors and positions, such as the wrist or arm.
Similarly, collaborative and transitional experiences between \AR and \VR are becoming more common, and would involve different setups and modalities \cite{roo2017onea}.
Novel \AR/\VR displays are already capable of transitioning from \AE to \VE \cite{feld2024simple}, and haptic feedback should also adapt to these transitions (\chapref{xr_perception}).

Therefore, a visuo-haptic augmented reality system should be able to adapt to any \AR/\VR display, any wearable haptic device worn anywhere on the body, and support personalization of haptic feedback.
In other words, the visuo-haptic rendering system should be designed to be responsive to the context of use.
This would require the development and validation of methods to automatically calibrate the haptic feedback to the user's perception of what it was designed to represent.
Methods should also be developed to allow the user to easily adjust the haptic feedback to their preferences at runtime, but without exposing all the design parameters \cite{kim2020defining}.

Finally, more practical use cases and applications of visuo-haptic \AR should be explored and evaluated.
For example, capturing visuo-haptic perceptions of objects and sharing them in a visioconference with different \AR or \VR setups per participant, or in a medical teleconsultation.
It could also be projecting and touching a visuo-haptic sample in a real wall for interior design, then switching to \VR to see and touch the complete final result, or manipulating a museum object with a real proxy object in visuo-haptic \AR, or even as it was in the past in its original state in \VR.
%Another example could be a medical teleconsultation, where the doctor could palpate a distant patient with haptic augmentation, but in \VR.

% design, implement and validate procedures to automatically calibrate the haptic feedback to the user's perception in accordance to what it has been designed to represent
% + let user free to easily adjust (eg can't let adjust whole spectrum of vibrotactile, reduce to two or three dimensions with sliders using MDS)

%- Visio en réalité mixte : ar avec avatars distants, vr pour se retrouver dans l'espace de l'autre ou un espace distant, et besoin de se faire toucher des objets à distance
%- Ou bien en cours, voir l'échantillon à toucher dans lenv de travail ou en contexte en passant en VR
%- Ex : médecin palpation, design d'un objet, rénovation d'un logement (AR en contexte courant, VR pour voir et toucher une fois terminé)