WIP xr-perception

1-background/introduction/introduction.tex

@@ -126,12 +126,12 @@ Because the visuo-haptic \VE is displayed in real time, colocalized and aligned
 
 In this context, we identify two main research challenges that we address in this thesis:
 \begin{enumerate*}[label=(\Roman*)]
-\item providing plausible and coherent visuo-haptic augmentations, and
-\item enabling effective manipulation of the augmented environment.
+\item \textbf{providing plausible and coherent visuo-haptic augmentations}, and
+\item \textbf{enabling effective manipulation of the augmented environment}.
 \end{enumerate*}
 Each of these challenges also raises numerous design, technical and human issues specific to each of the two types of feedback, wearable haptics and immersive \AR, as well as multimodal rendering and user experience issues in integrating these two sensorimotor feedbacks into a coherent and seamless visuo-haptic \AE.
 
-\subsectionstarbookmark{Provide Plausible and Coherent Visuo-Haptic Augmentations}
+\subsectionstarbookmark{Challenge I: Provide Plausible and Coherent Visuo-Haptic Augmentations}
 
 Many haptic devices have been designed and evaluated specifically for use in \VR, providing realistic and varied kinesthetic and tactile feedback to \VOs.
 Although closely related, (visual) \AR and \VR have key differences in their respective renderings that can affect user perception.
@@ -148,7 +148,7 @@ So far, \AR can only add visual and haptic sensations to the user's overall perc
 These added virtual sensations can therefore be perceived as out of sync or even inconsistent with the sensations of the \RE, for example with a lower rendering quality, a temporal latency, a spatial shift, or a combination of these.
 It is therefore unclear to what extent the real and virtual visuo-haptic sensations will be perceived as realistic or plausible, and to what extent they will conflict or complement each other in the perception of the \AE.
 
-\subsectionstarbookmark{Enable Effective Manipulation of the Augmented Environment}
+\subsectionstarbookmark{Challenge II: Enable Effective Manipulation of the Augmented Environment}
 
 Touching, grasping and manipulating \VOs are fundamental interactions for \AR \cite{kim2018revisiting}, \VR \cite{bergstrom2021how} and \VEs in general \cite{laviolajr20173d}.
 As the hand is not occupied or covered with a haptic device to not impair interaction with the \RE, as described in the previous section, one can expect a seamless and direct manipulation of the hand with the virtual content as if it were real.
@@ -177,8 +177,8 @@ Our approach is to
 
 We consider two main axes of research, each addressing one of the research challenges identified above:
 \begin{enumerate*}[label=(\Roman*)]
-\item modifying the perception of tangible surfaces using visuo-haptic texture augmentations, and
-\item improving the manipulation of virtual objects using visuo-haptic augmentations of the hand-object interaction.
+\item \textbf{modifying the texture perception of tangible surfaces}, and % with visuo-haptic texture augmentations, and
+\item \textbf{improving the manipulation of virtual objects}.% with visuo-haptic augmentations of the hand-object interaction.
 \end{enumerate*}
 Our contributions in these two axes are summarized in \figref{contributions}.
 
@@ -188,33 +188,33 @@ Our contributions in these two axes are summarized in \figref{contributions}.
   The second axis focuses on \textbf{(II)} improving the manipulation of \VOs with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.
 ]
 
-\subsectionstarbookmark{Axis I: Modifying the Perception of Tangible Surfaces with Visuo-Haptic Texture Augmentations}
+\subsectionstarbookmark{Axis I: Modifying the Texture Perception of Tangible Surfaces}
 
 Wearable haptic devices have proven to be effective in modifying the perception of a touched tangible surface, without modifying the tangible, nor covering the fingertip, forming a haptic \AE \cite{bau2012revel,detinguy2018enhancing,salazar2020altering}.
 %It is achieved by placing the haptic actuator close to the fingertip, to let it free to touch the surface, and rendering tactile stimuli timely synchronised with the finger movement.
 %It enables rich haptic feedback as the combination of kinesthetic sensation from the tangible and cutaneous sensation from the actuator.
 However, wearable haptic \AR have been little explored with visual \AR, as well as the visuo-haptic augmentation of textures.
 Texture is indeed one of the main tactile sensation of a surface material \cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by both sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) rendering \cite{unger2011roughness,culbertson2014modeling,strohmeier2017generating}.
-For this first axis of research, we propose to design and evaluate the perception of virtual visuo-haptic textures augmenting tangible surfaces. %, using an immersive \AR headset and a wearable vibrotactile device.
+For this first axis of research, we propose to \textbf{design and evaluate the perception of virtual visuo-haptic textures augmenting tangible surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
 To this end, we (1) design a system for rendering virtual visuo-haptic texture augmentations, to (2) evaluate how the perception of these textures is affected by the visual virtuality of the hand and the environment (\AR \vs \VR), and (3) investigate the perception of co-localized visuo-haptic texture augmentations in \AR.
 
 First, an effective approach to rendering haptic textures is to generate a vibrotactile signal that represents the finger-texture interaction \cite{culbertson2014modeling,asano2015vibrotactile}.
 Yet, to achieve the natural interaction with the hand and a coherent visuo-haptic feedback, it requires a real time rendering of the textures, no constraints on the hand movements, and a good synchronization between the visual and haptic feedback.
-Thus, our first objective is to design an immersive, real time system that allows free exploration with the bare hand of visuo-haptic texture augmentations on tangible surfaces.
+Thus, our first objective is to \textbf{design an immersive, real time system that allows free exploration with the bare hand of visuo-haptic texture augmentations on tangible surfaces}.
 
-Second, many works have investigated the haptic rendering of virtual textures, but few have integrated them with immersive \VEs or have considered the influence of the visual rendering on their perception.
+Second, many works have investigated the haptic augmentation of textures, but none have integrated them with \AR and \VR, or have considered the influence of the visual rendering on their perception.
 Still, it is known that the visual feedback can alter the perception of real and virtual haptic sensations \cite{schwind2018touch,choi2021augmenting} but also that the force feedback perception of grounded haptic devices is not the same in \AR and \VR \cite{diluca2011effects,gaffary2017ar}.
-Hence, our second objective is to understand how the perception of haptic texture augmentation differs depending on the degree of visual virtuality of the hand and the environment.
+Hence, our second objective is to \textbf{evaluate how the perception of haptic texture augmentation is affected by the visual virtuality of the hand and the environment}.
 
 Finally, some visuo-haptic texture databases have been modeled from real texture captures \cite{culbertson2014penn,balasubramanian2024sens3}, to be rendered as virtual textures with graspable haptics that are perceived as similar to real textures \cite{culbertson2015should,friesen2024perceived}.
 However, the rendering of these textures in an immersive and natural visuo-haptic \AR using wearable haptics remains to be investigated.
-Our third objective is to evaluate the perception of simultaneous and co-localized visuo-haptic texture augmentation of tangible surfaces in \AR, directly touched by the hand, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
+Our third objective is to \textbf{evaluate the perception of simultaneous and co-localized visuo-haptic texture augmentation of tangible surfaces in \AR}, directly touched by the hand, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
 
-\subsectionstarbookmark{Axis II: Improving Virtual Object Manipulation with Visuo-Haptic Augmentations of the Hand}
+\subsectionstarbookmark{Axis II: Improving the Manipulation of Virtual Objects}
 
 In immersive and wearable visuo-haptic \AR, the hand is free to touch and interact seamlessly with real, augmented, and virtual objects, and one can expect natural and direct contact and manipulation of \VOs with the bare hand.
 However, the intangibility of the visual \VE, the display limitations of current visual \OST-\AR systems and the inherent spatial and temporal discrepancies between the user's hand actions and the visual feedback in the \VE can make the interaction with \VOs particularly challenging.
-%However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the real environment, the visual feedback, and the haptic feedback, can make the interaction with \VOs with bare hands particularly challenging.
+%However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the \RE, the visual feedback, and the haptic feedback, can make the interaction with \VOs with bare hands particularly challenging.
 Two particular sensory feedbacks are known to improve such direct \VO manipulation, but they have not been properly investigated in immersive \AR: visual rendering of the hand \cite{piumsomboon2014graspshell,prachyabrued2014visual} and delocalized haptic rendering \cite{lopes2018adding,teng2021touch}.
 For this second axis of research, we propose to design and evaluate \textbf{the role of visuo-haptic augmentations of the hand as interaction feedback with \VOs in immersive \OST-\AR}.
 We consider the effect on the user performance an experience of (1) the visual rendering as hand augmentation and (2) combination of different visuo-haptic rendering of the hand manipulation with \VOs
@@ -227,7 +227,7 @@ Thus, our fourth objective is to \textbf{investigate the visual rendering as han
 Second, as described above, wearable haptics for visual \AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE.
 Previous works have shown that wearable haptics that provide feedback on the hand manipulation with \VOs in \AR can significantly improve the user performance and experience \cite{maisto2017evaluation,meli2018combining}.
 However, it is unclear which positioning of the actuator is the most beneficial nor how a haptic augmentation of the hand compares or complements with a visual augmentation of the hand.
-Our last objective is to \textbf{investigate the visuo-haptic rendering of the hand manipulation} with \VOs in \OST-\AR using wearable vibrotactile haptic.
+Our last objective is to \textbf{investigate the visuo-haptic rendering of hand manipulation with \VOs} in \OST-\AR using wearable vibrotactile haptic.
 
 \section{Thesis Overview}
 \label{thesis_overview}
@@ -237,35 +237,35 @@ In \textbf{\partref{background}}, we describe the context and background of our
 
 In \textbf{\chapref{related_work}}, we then review previous work on the perception and manipulation with virtual and augmented objects, directly with the hand, using either wearable haptics, \AR, or their combination.
 First, we overview how the hand perceives and manipulate real everyday objects.
-Second, we present wearable haptics and haptic augmentations of roughness and hardness of real objects.
+Second, we present wearable haptics and haptic augmentations of texture and hardness of real objects.
 Third, we introduce \AR, and how \VOs can be manipulated directly with the hand.
 Finally, we describe how multimodal visual and haptic feedback have been combined in \AR to enhance perception and interaction with the hand.
 
 We then address each of our two research axes in a dedicated part.
 
 \noindentskip
-In \textbf{\partref{perception}}, we describe our contributions to the first axis of research, augmenting the visuo-haptic texture perception of tangible surfaces.
-We evaluate how the visual rendering of the hand (real or virtual), the environment (\AR or \VR) and the textures (displayed or hidden) affect the roughness perception of virtual vibrotactile textures rendered on real surfaces and touched directly with the index finger.
+In \textbf{\partref{perception}}, we describe our contributions to the first axis of research: modifying the visuo-haptic texture perception of tangible surfaces.
+We evaluate how the visual rendering of the hand (real or virtual), the environment (\AR or \VR) and the textures (coherent, different or not shown) affect the perception of virtual vibrotactile textures rendered on real surfaces and touched directly with the index finger.
 
 In \textbf{\chapref{vhar_system}}, we detail a system for rendering visuo-haptic virtual textures that augment tangible surfaces using an immersive \AR/\VR headset and a wearable vibrotactile device.
 The haptic textures are rendered as a real-time vibrotactile signal representing a grating texture, and is provided to the middle phalanx of the index finger touching the texture using a voice-coil actuator.
-The tracking of the real hand and environment is done using marker-based technique, and the visual rendering of their virtual counterparts is done using the immersive \OST \AR headset Microsoft HoloLens~2.
+The tracking of the real hand and environment is done using marker-based technique, and the visual rendering of their virtual counterparts is done using the immersive \OST-\AR headset Microsoft HoloLens~2.
 
-In \textbf{\chapref{xr_perception}}, we investigate, in a user study, how different the perception of virtual haptic textures is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
-We use psychophysical methods to measure the user roughness perception of the virtual textures, and extensive questionnaires to understand how this perception is affected by the visual rendering of the hand and the environment.
+In \textbf{\chapref{xr_perception}}, we investigate, in a user study, how different the perception of haptic texture augmentations is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
+We use psychophysical methods to measure the user perception, and extensive questionnaires to understand how this perception is affected by the visual rendering of the hand and the environment.
 
 In \textbf{\chapref{ar_textures}}, we evaluate the perception of visuo-haptic texture augmentations, touched directly with one's own hand in \AR.
 The virtual textures are paired visual and tactile models of real surfaces \cite{culbertson2014one} that we render as visual and haptic overlays on the touched augmented surfaces.
 Our objective is to assess the perceived realism, coherence and roughness of the combination of nine representative visuo-haptic texture pairs.
 
 \noindentskip
-In \textbf{\partref{manipulation}}, we describe our contributions to the second axis of research: improving the manipulation of \VOs using visuo-haptic augmentations of the hand as interaction feedback with \VOs in immersive \OST-\AR.
+In \textbf{\partref{manipulation}}, we describe our contributions to the second axis of research: improving the free and direct hand manipulation of \VOs using visuo-haptic augmentations of the hand as interaction feedback with \VOs in immersive \OST-\AR.
 
-In \textbf{\chapref{visual_hand}}, we investigate in a user study the effect of six visual renderings as hand augmentations for the direct manipulation of \VOs, as a set of the most popular hand renderings in the \AR literature.
-Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate the user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a \VO directly with the hand.
+In \textbf{\chapref{visual_hand}}, we investigate in a user study of six visual renderings as hand augmentations, as a set of the most popular hand renderings in the \AR literature.
+Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate their effect on the user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a \VO directly with the hand.
 
-In \textbf{\chapref{visuo_haptic_hand}}, we evaluate in a user study two vibrotactile contact techniques, provided at four different positionings on the user's hand, as haptic rendering of the hand manipulation with \VOs.
-They are compared to the two most representative visual hand renderings from the previous chapter, and the user performance and experience are evaluated within the same \OST-\AR setup and manipulation tasks.
+In \textbf{\chapref{visuo_haptic_hand}}, we evaluate in a user study the visuo-haptic rendering of manual object manipulation with two vibrotactile contact techniques, provided at four different positionings on the user's hand, as haptic rendering of the hand manipulation with \VOs.
+They are compared to the two most representative visual hand renderings from the previous chapter, resulting in sixteen visuo-haptic hand renderings that are evaluated within the same experimental setup and design.
 
 \noindentskip
 In \textbf{\chapref{conclusion}}, we conclude this thesis and discuss short-term future work and long-term perspectives for each of our contributions and research axes.

1-background/related-work/2-wearable-haptics.tex

@@ -135,7 +135,7 @@ Several types of vibrotactile actuators are used in haptics, with different trad
 
 An \ERM is a direct current (DC) motor that rotates an off-center mass when a voltage or current is applied (\figref{precisionmicrodrives_erm}). \ERMs are easy to control, inexpensive and can be encapsulated in a few millimeters cylinder or coin form factor. However, they have only one \DoF because both the frequency and amplitude of the vibration are coupled to the speed of the rotation, \eg low (high) frequencies output at low (high) amplitudes, as shown on \figref{precisionmicrodrives_erm_performances}.
 
-\begin{subfigs}{erm}{Diagram and performance of \ERMs. }[][
+\begin{subfigs}{erm}{Diagram and performance of an \ERM. }[][
   \item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives~\footnotemark.
   \item Amplitude and frequency output of an \ERM as a function of the input voltage.
   ]
@@ -357,9 +357,9 @@ We describe them in the \secref{vhar_haptics}.
 Haptic systems aim to provide virtual interactions and sensations similar to those with real objects.
 The complexity of the haptic sense has led to the design of numerous haptic devices and renderings.
 While many haptic devices can be worn on the hand, only a few can be considered wearable as they are compact and portable, but they are limited to cutaneous feedback.
-If the haptic rendering is timely associated with the user's touch actions on a real object, the perceived haptic properties of the object can be modified.
-Several rendering methods have been developed to modify the perceived roughness and hardness, mostly using vibrotactile feedback and, to a lesser extent, pressure feedback.
-However, not all of these haptic augmentations have been already transposed to wearable haptics.
+If the haptic rendering of the device is timely associated with the user's touch actions on a real object, the perceived haptic properties of the object can be modified.
+Several haptic augmentation methods have been developed to modify the perceived roughness and hardness, mostly using vibrotactile feedback and, to a lesser extent, pressure feedback.
+However, not all of these haptic augmentations have been already transposed to wearable haptics, and use of wearable haptic augmentations have not been yet studied in the context of \AR.
 
 %, unlike most previous actuators that are designed specifically for fingertips and would require mechanical adaptation to be placed on other parts of the hand.
 % thanks to the vibration propagation and the sensory capabilities distributed throughout the skin, they can be placed without adaption and on any part of the hand

1-background/related-work/3-augmented-reality.tex

@@ -6,7 +6,7 @@ Immersive systems such as headsets leave the hands free to interact with \VOs, p
 
 %\begin{subfigs}{sutherland1968headmounted}{Photos of the first \AR system \cite{sutherland1968headmounted}. }[
 %        \item The \AR headset.
-%        \item Wireframe \ThreeD \VOs were displayed registered in the real environment (as if there were part of it).
+%        \item Wireframe \ThreeD \VOs were displayed registered in the \RE (as if there were part of it).
 %    ]
 %    \subfigsheight{45mm}
 %    \subfig{sutherland1970computer3}
@@ -237,7 +237,7 @@ Our hands allow us to manipulate real everyday objects with both strength and pr
 Initially tracked by active sensing devices such as gloves or controllers, it is now possible to track hands in real time using cameras and computer vision algorithms natively integrated into \AR/\VR headsets \cite{tong2023survey}.
 
 The user's hand is therefore tracked and reconstructed as a \emph{virtual hand} model in the \VE  \cite{billinghurst2015survey,laviolajr20173d}.
-The simplest models represent the hand as a rigid 3D object that follows the movements of the real hand with \qty{6}{\DoF} (position and orientation in space) \cite{talvas2012novel}.
+The simplest models represent the hand as a rigid \ThreeD object that follows the movements of the real hand with \qty{6}{\DoF} (position and orientation in space) \cite{talvas2012novel}.
 An alternative is to model only the fingertips (\figref{lee2007handy}) or the whole hand (\figref{hilliges2012holodesk_1}) as points.
 The most common technique is to reconstruct all the phalanges of the hand in an articulated kinematic model (\secref{hand_anatomy}) \cite{borst2006spring}.
 
@@ -246,7 +246,7 @@ Heuristic techniques use rules to determine the selection, manipulation and rele
 However, they produce unrealistic behaviour and are limited to the cases predicted by the rules.
 Physics-based techniques simulate forces at the points of contact between the virtual hand and the \VO.
 In particular, \textcite{borst2006spring} have proposed an articulated kinematic model in which each phalanx is a rigid body simulated with the god-object method \cite{zilles1995constraintbased}:
-The virtual phalanx follows the movements of the real phalanx, but remains constrained to the surface of the virtual objects during contact.
+The virtual phalanx follows the movements of the real phalanx, but remains constrained to the surface of the \VOs during contact.
 The forces acting on the object are calculated as a function of the distance between the real and virtual hands (\figref{borst2006spring}).
 More advanced techniques simulate the friction phenomena \cite{talvas2013godfinger} and finger deformations \cite{talvas2015aggregate}, allowing highly accurate and realistic interactions, but which can be difficult to compute in real time.
 
@@ -264,7 +264,7 @@ More advanced techniques simulate the friction phenomena \cite{talvas2013godfing
 \end{subfigs}
 
 However, the lack of physical constraints on the user's hand movements makes manipulation actions tiring \cite{hincapie-ramos2014consumed}.
-While the user's fingers traverse the virtual object, a physics-based virtual hand remains in contact with the object, a discrepancy that may degrade the user's performance in \VR \cite{prachyabrued2012virtual}.
+While the user's fingers traverse the \VO, a physics-based virtual hand remains in contact with the object, a discrepancy that may degrade the user's performance in \VR \cite{prachyabrued2012virtual}.
 Finally, in the absence of haptic feedback on each finger, it is difficult to estimate the contact and forces exerted by the fingers on the object during grasping and manipulation \cite{maisto2017evaluation,meli2018combining}.
 While a visual rendering of the virtual hand in \VR can compensate for these issues \cite{prachyabrued2014visual}, the visual and haptic rendering of the virtual hand, or their combination, in \AR is under-researched.
 
@@ -303,7 +303,7 @@ In a collaborative task in immersive \OST-\AR \vs \VR, \textcite{yoon2020evaluat
 Finally, \textcite{maisto2017evaluation} and \textcite{meli2018combining} compared the visual and haptic rendering of the hand in \VST-\AR, as detailed in the next section (\secref{vhar_rings}).
 Taken together, these results suggest that a visual rendering of the hand in \AR could improve usability and performance in direct hand manipulation tasks, but the best rendering has yet to be determined.
 %\cite{chan2010touching} : cues for touching (selection) \VOs.
-%\textcite{saito2021contact} found that masking the real hand with a textured 3D opaque virtual hand did not improve performance in a reach-to-grasp task but displaying the points of contact on the \VO did.
+%\textcite{saito2021contact} found that masking the real hand with a textured \ThreeD opaque virtual hand did not improve performance in a reach-to-grasp task but displaying the points of contact on the \VO did.
 %To the best of our knowledge, evaluating the role of a visual rendering of the hand displayed \enquote{and seen} directly above real tracked hands in immersive OST-AR has not been explored, particularly in the context of \VO manipulation.
 
 \begin{subfigs}{visual-hands}{Visual hand renderings in \AR. }[][

1-background/related-work/4-visuo-haptic-ar.tex

@@ -97,7 +97,7 @@ For example, in a fixed \VST-\AR screen (\secref{ar_displays}), by visually defo
 %In particular, in \AR and \VR, the perception of a haptic rendering or augmentation can be influenced by the visual rendering of the \VO.
 
 \subsubsection{Perception of Visuo-Haptic Rendering in AR and \VR}
-\label{AR_vs_VR}
+\label{ar_vr_haptic}
 
 Some studies have investigated the visuo-haptic perception of \VOs rendered with force-feedback and vibrotactile feedback in \AR and \VR.