erwan/phd-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP vhar_textures

Browse Source
This commit is contained in:
Erwan Normand
2024-09-29 23:13:19 +02:00
parent fcbda4a3e7
commit 3a8f68c094
13 changed files with 313 additions and 368 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
4-conclusion/conclusion.tex
View File

@@ -5,28 +5,32 @@
\section{Summary}
% we presented our research on direct hand interaction with real and virtual everyday objects, visually and haptically augmented using immersive \AR and wearable haptic devices.
In this thesis, entitled \enquote{\textbf{\ThesisTitle}}, we presented our research towards a coherent, natural and seamless visuo-haptic augmented reality that enables perception and interaction with everyday objects directly with the hand.
Worn on the hand, wearable haptics can provide a rich tactile feedback on \VOs and augment the perception of real objects, while preserving the freedom of movement and interaction with the \RE.
In this thesis, entitled \enquote{\textbf{\ThesisTitle}}, we showed how wearable haptics, worn on the outside of the hand, can improve direct hand interaction in immersive \AR by augmenting the perception and manipulation of the virtual.
Wearable haptics can provide a rich tactile feedback on \VOs and augment the perception of real objects, both directly touched with the hand, while preserving the freedom of movement and interaction with the \RE.
However, their integration with \AR is still in its infancy, and presents many design, technical and human challenges.
We focused our research on two: \textbf{(I) modifying the texture perception of tangible surfaces}, and \textbf{(II) improving the manipulation of \VOs}.
We structured our research on two axes: \textbf{(I) modifying the texture perception of tangible surfaces}, and \textbf{(II) improving the manipulation of \VOs}.
\noindentskip In \partref{perception} we focused on modifying the perception of wearable virtual visuo-haptic textures that augments tangible surfaces.
Texture is a fundamental property of an object, perceived equally by sight and touch.
It is also one of the most known haptic augmentation, but it had not yet been integrated with \AR or \VR.
%However, haptic texture augmentation had not yet been integrated with \AR or \VR.
%We designed wearable visuo-haptic texture augmentations and evaluated how the degree of virtuality and the rendering of the visuals influenced the perception of the haptic textures.
We (1) proposed a \textbf{wearable visuo-haptic texture augmentation system}, (2) evaluated how the perception of haptic textures is \textbf{affected by the visual virtuality of the hand} and the environment (real, augmented, or virtual), and (3) investigated the \textbf{perception of co-localized visuo-haptic texture augmentations}.
We \textbf{(1)} proposed a \textbf{wearable visuo-haptic texture augmentation system}, \textbf{(2)} evaluated how the perception of haptic textures is \textbf{affected by the visual virtuality of the hand} and the environment (real, augmented, or virtual), and \textbf{(3)} investigated the \textbf{perception of co-localized visuo-haptic texture augmentations}.
\noindentskip In \chapref{vhar_system},
\noindentskip In \chapref{xr_perception}, we explored how the perception of wearable haptic augmented textures is affected by the visual virtuality of the hand and the environment, being either real, augmented or virtual.
We augmented the perceived roughness of the tangible surface by rendering virtual vibrotactile patterned textures on the voice-coil, and rendered the visual conditions with an immersive \OST-\AR headset that could be switched to a \VR only view.
\noindentskip In \chapref{xr_perception} we explored how the perception of wearable haptic augmented textures is affected by the visual virtuality of the hand and the environment, whether it is real, augmented or virtual.
We augmented the perceived roughness of the tangible surface with virtual vibrotactile patterned textures, and rendered the visual conditions by switching the \OST-\AR headset to a \VR-only view.
We then conducted a psychophysical user study with 20 participants and extensive questionnaires to evaluate the perceived roughness augmentation in these three visual conditions.
The textures were perceived as \textbf{rougher when touched with the real hand alone} compared to a virtual hand in either \AR or \VR, possibly due to the \textbf{perceived latency} between finger movements and different visual, haptic, and proprioceptive feedbacks.
The textures were perceived as \textbf{rougher when touched with the real hand alone compared to a virtual hand} in either \AR or \VR, possibly due to the \textbf{perceived latency} between finger movements and different visual, haptic, and proprioceptive feedbacks.
\noindentskip In \chapref{vhar_textures},
\noindentskip In \chapref{vhar_textures}, we investigated the perception of co-localized visual and wearable haptic texture augmentations on tangible surfaces.
We transposed the \textbf{data-driven visuo-haptic textures} from the \HaTT database to the system presented in \chapref{vhar_system} and conducted a user study with 20 participants to rate the coherence, realism, and perceived roughness of nine visuo-haptic texture pairs.
Participants integrated roughness sensations from both visual and haptic modalities well, with \textbf{haptics predominating the perception}, and consistently identified and matched \textbf{clusters of visual and haptic textures with similar perceived roughness}.
\noindentskip In \partref{manipulation} we focused on improving the manipulation of \VOs directly with the hand in immersive \OST-\AR.
Our approach was to design, based on the literature, and evaluate in user studies the effect of visual rendering of the hand and the delocalized haptic rendering.
We first considered (1) \textbf{the visual rendering as hand augmentation} and then the (2) combination of different visuo-haptic \textbf{rendering of the hand manipulation with \VOs}.
Our approach was to design visual renderings of the hand and delocalized haptic rendering, based on the literature, and to evaluate them in user studies.
We first considered \textbf{(1) the visual rendering as hand augmentation} and then the \textbf{(2)} combination of different visuo-haptic \textbf{rendering of the hand manipulation with \VOs}.
\noindentskip In \chapref{visual_hand}, we investigated the visual rendering as hand augmentation.
Seen as an \textbf{overlay on the user's hand}, such visual hand rendering provide feedback on the hand tracking and the interaction with \VOs.
@@ -65,18 +69,34 @@ This would allow a complete portable and wearable visuo-haptic system to be used
\paragraph{Visual Representation of the Virtual Texture}
The main limitation of our study is the absence of a visual representation of the virtual texture.
The main limitation of this user study was the absence of a visual representation of the virtual texture.
This is indeed a source of information as important as haptic sensations for the perception of both real textures \cite{baumgartner2013visual,bergmanntiest2007haptic,vardar2019fingertip} and virtual textures \cite{degraen2019enhancing,gunther2022smooth}, and their interaction in the overall perception is complex.
Specifically, it remains to be investigated how to visually represent vibrotactile textures in an immersive \AR or \VR context, as the visuo-haptic coupling of such grating textures is not trivial \cite{unger2011roughness} even with real textures \cite{klatzky2003feeling}.
Specifically, it remains to be investigated how to visually represent the vibrotactile patterned textures used in an immersive \AR or \VR context, as the visuo-haptic coupling of such patterned textures is not trivial \cite{unger2011roughness} even with real textures \cite{klatzky2003feeling}.
\paragraph{Broader Visuo-Haptic Conditions}
Also, our study was conducted with an \OST-\AR headset, but the results may be different with a \VST-\AR headset.
Finally, we focused on the perception of roughness sensations using wearable haptics in \AR \vs \VR using a square wave vibrotactile signal, but different haptic texture rendering methods should be considered.
More generally, many other haptic feedbacks could be investigated in \AR \vs \VR using the same system and methodology, such as stiffness, friction, local deformations, or temperature.
Our study was conducted with an \OST-\AR headset, but the results may be different with a \VST-\AR headset, where the \RE is seen through cameras and screens, and the perceived simultaneity between visual and haptic stimuli, real or virtual, is different.
We also focused on the perception of roughness augmentation using wearable vibrotactile haptics and a square wave signal to simulate a patterned texture: Our objective was not to accurately reproduce real textures, but to induce various roughness perception on the same tangible surface with a well controlled parameters.
Yet, more accurate models to simulate interaction with virtual textures should be transposed to wearable haptic augmentations, such as in \textcite{unger2011roughness}.
Another limitation that may have affected the perception of the haptic texture augmentations is the lack of compensation for the frequency response of the actuator and amplifier \cite{asano2012vibrotactile,culbertson2014modeling,friesen2024perceived}.
The dynamic response of the finger should also be considered.
\subsection*{Perception of Visual and Haptic Texture Augmentations in Augmented Reality}
\paragraph{Adapt to the Specificities of Direct Touch}
As with the previous chapter, our objective was not to accurately reproduce real textures, but to alter the perception of simultaneous visual and haptic roughness augmentation of a tangible surface directly touched by the finger in \AR.
Yet, the haptic textures used models from the vibrations of a hand-held probe sliding over real surfaces captured.
We generated the vibrotactile textures based only on the finger speed, but the perceived roughness of real textures depends on other factors, such as the force of contact, the angle, the posture or the surface of the contact \cite{schafer2017transfer}, but their respective importance in the perception is not yet fully understood \cite{richardson2022learning}.
%Comparison from
These results have of course some limitations as they addressed a small set of visuo-haptic textures augmenting the perception of smooth white tangible surfaces.
Indeed, the increase in visuo-haptic texture perception may be limited on surfaces that already have strong visual or haptic patterns \cite{asano2012vibrotactile}, or on objects with complex shapes.
In addition, the haptic textures used were modelled from the vibrations of a probe sliding over the captured surfaces.
Finally, the visual textures used were also simple color captures not meant to be used in an immersive \VE.
However,
In addition of these limitations, both visual and haptic texture models should be improved by integrating the rendering of spatially localized breaks, edges or patterns, like real textures \cite{richardson2022learning}, and by being adaptable to individual sensitivities, as personalized haptics is a promising approach \cite{malvezzi2021design,young2020compensating}.
\subsection*{Visual Rendering of the Hand for Manipulating \VOs in AR}
\paragraph{Other AR Displays}
@@ -84,7 +104,7 @@ More generally, many other haptic feedbacks could be investigated in \AR \vs \VR
The visual hand renderings we evaluated were displayed on the Microsoft HoloLens~2, which is a common \OST-\AR headset \cite{hertel2021taxonomy}.
We purposely chose this type of display as it is with \OST-\AR that the lack of mutual occlusion between the hand and the \VO is the most challenging to solve \cite{macedo2023occlusion}.
We thus hypothesized that a visual hand rendering would be more beneficial to users with this type of display.
However, the user's visual perception and experience is different with other types of displays, such as \VST-\AR, where the \RE view is seen through a screen (\secref[related_work]{ar_displays}).
However, the user's visual perception and experience is different with other types of displays, such as \VST-\AR, where the \RE view is seen through cameras and screens (\secref[related_work]{ar_displays}).
While the mutual occlusion problem and the hand tracking latency can be overcome with \VST-\AR, the visual hand rendering could still be beneficial to users as it provides depth cues and feedback on the hand tracking, and should be evaluated as such.
\paragraph{More Ecological Conditions}