WIP conclusion

1-background/introduction/introduction.tex

@@ -89,16 +89,16 @@ In particular, it has been implemented by enhancing the haptic perception of tan
 \figref{salazar2020altering} shows an example of modifying the perceived stiffness of a tangible object in \VR using simultaneous pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum3}).
 \figref{bau2012revel} shows another example of visuo-haptic \AR rendering of virtual texture when running the finger on a tangible surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum3}).
 
-Current visual \AR systems often lack haptic feedback, creating a deceptive and incomplete user experience when reaching the \VE with the hand.
+Current \AR systems often lack haptic feedback, creating a deceptive and incomplete user experience when reaching the \VE with the hand.
 All visual \VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties congruently and interact with them with confidence and efficiency.
 It is therefore necessary to provide haptic feedback that is consistent with the visual \AE and ensures the best possible user experience.
 The integration of wearable haptics with \AR seems to be one of the most promising solutions, but it remains challenging due to their many respective characteristics and the additional constraints of combining them.
 
 \begin{subfigs}{visuo-haptic-environments}{Visuo-haptic environments with different degrees of reality-virtuality. }[][
-  \item Visual \AR environment with a real, tangible haptic object used as a proxy to manipulate a \VO \cite{kahl2023using}.
-  \item Visual \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a \VO \cite{meli2018combining}.
+  \item \AR environment with a real, tangible haptic object used as a proxy to manipulate a \VO \cite{kahl2023using}.
+  \item \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a \VO \cite{meli2018combining}.
   \item A tangible object seen in a visual \VR environment whose haptic perception of stiffness is augmented with the hRing haptic device \cite{salazar2020altering}.
-  \item Visuo-haptic rendering of texture on a touched tangible object with a visual \AR display and haptic electrovibration feedback \cite{bau2012revel}.
+  \item Visuo-haptic rendering of texture on a touched tangible object with a \AR display and haptic electrovibration feedback \cite{bau2012revel}.
   ]
   \subfigsheight{31mm}
   \subfig{kahl2023using}
@@ -121,7 +121,7 @@ Because the visuo-haptic \VE is displayed in real time, colocalized and aligned
 
 \fig{interaction-loop}{The interaction loop between a user and a visuo-haptic augmented environment.}[
   One interact with the visual (in blue) and haptic (in red) \VE through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with \VOs.
-  The \VE is rendered back to the user co-localized with the real one (in gray) using a visual \AR headset and a wearable haptic device.
+  The \VE is rendered back to the user co-localized with the real one (in gray) using a \AR headset and a wearable haptic device.
 ]
 
 In this context, we identify two main research challenges that we address in this thesis:
@@ -140,7 +140,7 @@ First, the user's hand and \RE are visible in \AR, unlike \VR where there is tot
 As such, in \VR, visual sensations are particularly dominant in perception, and conflicts with haptic sensations are also specifically created to influence the user's perception, for example to create pseudo-haptic \cite{ujitoko2021survey} or haptic retargeting \cite{azmandian2016haptic} effects.
 Moreover, many wearable haptic devices take the form of controllers, gloves or exoskeletons, all of which cover the fingertips and are therefore not suitable for \AR.
 The user's hand must be indeed free to touch and interact with the \RE while wearing a wearable haptic device.
-It is possible instead to place the haptic actuator close to the point of contact with the \RE, as described above to implement haptic \AR, \eg providing haptic feedback on the nail \cite{ando2007fingernailmounted,teng2021touch}, another phalanx \cite{asano2015vibrotactile,salazar2020altering} or the wrist \cite{pezent2022design,sarac2022perceived} for rendering fingertip contacts with virtual content.
+It is possible instead to place the haptic actuator close to the point of contact with the \RE, as described above to implement haptic augmentations, \eg providing haptic feedback on the nail \cite{ando2007fingernailmounted,teng2021touch}, another phalanx \cite{asano2015vibrotactile,salazar2020altering} or the wrist \cite{pezent2022design,sarac2022perceived} for rendering fingertip contacts with virtual content.
 Therefore, when touching a virtual or augmented object, the real and virtual visual sensations are seen as co-localized, but the virtual haptic feedback is not.
 It remains to be investigated how such potential discrepancies affect the overall perception to design visuo-haptic renderings adapted to \AR.
 
@@ -155,7 +155,7 @@ As the hand is not occupied or covered with a haptic device to not impair intera
 Thus, augmenting a tangible object has the advantage of physically constraining the hand, allowing for easy and natural interaction, but manipulating a purely \VO with the bare hand can be challenging without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
 
 In addition, current \AR systems have visual rendering limitations that also affect interaction with \VOs. %, due to depth underestimation, a lack of mutual occlusions,  and hand tracking latency.
-Visual \AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
+\AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
 But the depth perception of the \VOs is often underestimated \cite{peillard2019studying,adams2022depth}, and there is often a lack of mutual occlusion between the hand and a \VO, \ie that the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
 Finally, as illustrated in \figref{interaction-loop}, interacting with a \VO is an illusion, because in fact the real hand is controlling in real time a virtual hand, like an avatar, whose contacts with \VOs are then simulated in the \VE.
 Therefore, there is inevitably a latency delay between the real hand's movements and the \VO's return movements, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the touched \VO \cite{prachyabrued2014visual}.
@@ -193,22 +193,23 @@ Our contributions in these two axes are summarized in \figref{contributions}.
 Wearable haptic devices have proven to be effective in modifying the perception of a touched tangible surface, without modifying the tangible, nor covering the fingertip, forming a haptic \AE \cite{bau2012revel,detinguy2018enhancing,salazar2020altering}.
 %It is achieved by placing the haptic actuator close to the fingertip, to let it free to touch the surface, and rendering tactile stimuli timely synchronised with the finger movement.
 %It enables rich haptic feedback as the combination of kinesthetic sensation from the tangible and cutaneous sensation from the actuator.
-However, wearable haptic \AR have been little explored with visual \AR, as well as the visuo-haptic augmentation of textures.
+However, wearable haptic augmentations have been little explored with \AR, as well as the visuo-haptic augmentation of textures.
 Texture is indeed one of the main tactile sensation of a surface material \cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by both sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) rendering \cite{unger2011roughness,culbertson2014modeling,strohmeier2017generating}.
-For this first axis of research, we propose to \textbf{design and evaluate the perception of virtual visuo-haptic textures augmenting tangible surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
-To this end, we (1) design a system for rendering virtual visuo-haptic texture augmentations, to (2) evaluate how the perception of these textures is affected by the visual virtuality of the hand and the environment (\AR \vs \VR), and (3) investigate the perception of co-localized visuo-haptic texture augmentations in \AR.
+%With a better understanding of how visual factors influence the perception of haptically augmented tangible objects, the many wearable haptic systems that already exist but have not yet been fully explored with \AR can be better applied and new visuo-haptic renderings adapted to \AR can be designed.
+For this first axis of research, we propose to design and evaluate the \textbf{perception of wearable virtual visuo-haptic textures augmenting tangible surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
+To this end, we (1) design a system for rendering wearable visuo-haptic texture augmentations, to (2) evaluate how the perception of haptic texture augmentations is affected by the visual virtuality of the hand and the environment (real, augmented, or virtual), and (3) investigate the perception of co-localized visuo-haptic texture augmentations.
 
 First, an effective approach to rendering haptic textures is to generate a vibrotactile signal that represents the finger-texture interaction \cite{culbertson2014modeling,asano2015vibrotactile}.
 Yet, to achieve the natural interaction with the hand and a coherent visuo-haptic feedback, it requires a real time rendering of the textures, no constraints on the hand movements, and a good synchronization between the visual and haptic feedback.
-Thus, our first objective is to \textbf{design an immersive, real time system that allows free exploration with the bare hand of visuo-haptic texture augmentations on tangible surfaces}.
+Thus, our first objective is to \textbf{design an immersive, real time system} that allows free exploration with the bare hand of \textbf{wearable visuo-haptic texture augmentations} on tangible surfaces.
 
 Second, many works have investigated the haptic augmentation of textures, but none have integrated them with \AR and \VR, or have considered the influence of the visual rendering on their perception.
 Still, it is known that the visual feedback can alter the perception of real and virtual haptic sensations \cite{schwind2018touch,choi2021augmenting} but also that the force feedback perception of grounded haptic devices is not the same in \AR and \VR \cite{diluca2011effects,gaffary2017ar}.
-Hence, our second objective is to \textbf{evaluate how the perception of haptic texture augmentation is affected by the visual virtuality of the hand and the environment}.
+Hence, our second objective is to \textbf{evaluate how the perception of wearable haptic texture augmentation is affected by the visual virtuality of the hand and the environment} (real, augmented, or virtual).
 
 Finally, some visuo-haptic texture databases have been modeled from real texture captures \cite{culbertson2014penn,balasubramanian2024sens3}, to be rendered as virtual textures with graspable haptics that are perceived as similar to real textures \cite{culbertson2015should,friesen2024perceived}.
 However, the rendering of these textures in an immersive and natural visuo-haptic \AR using wearable haptics remains to be investigated.
-Our third objective is to \textbf{evaluate the perception of simultaneous and co-localized visuo-haptic texture augmentation of tangible surfaces in \AR}, directly touched by the hand, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
+Our third objective is to \textbf{evaluate the perception of simultaneous and co-localized wearable visuo-haptic texture augmentations} of tangible surfaces in \AR, directly touched by the hand, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
 
 \subsectionstarbookmark{Axis II: Improving the Manipulation of Virtual Objects}
 
@@ -216,7 +217,7 @@ In immersive and wearable visuo-haptic \AR, the hand is free to touch and intera
 However, the intangibility of the visual \VE, the display limitations of current visual \OST-\AR systems and the inherent spatial and temporal discrepancies between the user's hand actions and the visual feedback in the \VE can make the interaction with \VOs particularly challenging.
 %However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the \RE, the visual feedback, and the haptic feedback, can make the interaction with \VOs with bare hands particularly challenging.
 Two particular sensory feedbacks are known to improve such direct \VO manipulation, but they have not been properly investigated in immersive \AR: visual rendering of the hand \cite{piumsomboon2014graspshell,prachyabrued2014visual} and delocalized haptic rendering \cite{lopes2018adding,teng2021touch}.
-For this second axis of research, we propose to design and evaluate \textbf{the role of visuo-haptic augmentations of the hand as interaction feedback with \VOs in immersive \OST-\AR}.
+For this second axis of research, we propose to design and evaluate \textbf{visuo-haptic augmentations of the hand as interaction feedback with \VOs} in immersive \OST-\AR.
 We consider the effect on the user performance an experience of (1) the visual rendering as hand augmentation and (2) combination of different visuo-haptic rendering of the hand manipulation with \VOs
 
 First, the visual rendering of the virtual hand is a key element for interacting and manipulating \VOs in \VR \cite{prachyabrued2014visual,grubert2018effects}.
@@ -224,7 +225,7 @@ A few works have also investigated the visual rendering of the virtual hand in \
 But \OST-\AR has significant perceptual differences from \VR due to the visibility of the real hand and environment, and these visual hand augmentations have not been evaluated in the context of \VO manipulation with the bare hand.
 Thus, our fourth objective is to \textbf{investigate the visual rendering as hand augmentation} for direct manipulation of \VOs in \OST-\AR.
 
-Second, as described above, wearable haptics for visual \AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE.
+Second, as described above, wearable haptics for \AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE.
 Previous works have shown that wearable haptics that provide feedback on the hand manipulation with \VOs in \AR can significantly improve the user performance and experience \cite{maisto2017evaluation,meli2018combining}.
 However, it is unclear which positioning of the actuator is the most beneficial nor how a haptic augmentation of the hand compares or complements with a visual augmentation of the hand.
 Our last objective is to \textbf{investigate the visuo-haptic rendering of hand manipulation with \VOs} in \OST-\AR using wearable vibrotactile haptic.
@@ -253,7 +254,7 @@ The haptic textures are rendered as a real-time vibrotactile signal representing
 The tracking of the real hand and environment is done using marker-based technique, and the visual rendering of their virtual counterparts is done using the immersive \OST-\AR headset Microsoft HoloLens~2.
 
 In \textbf{\chapref{xr_perception}}, we investigate, in a user study, how different the perception of haptic texture augmentations is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
-We use psychophysical methods to measure the user perception, and extensive questionnaires to understand how this perception is affected by the visual rendering of the hand and the environment.
+We use psychophysical methods to measure the user perception, and extensive questionnaires to understand how this perception is affected by the visual virtuality of the hand and the environment.
 
 In \textbf{\chapref{vhar_textures}}, we evaluate the perception of visuo-haptic texture augmentations, touched directly with one's own hand in \AR.
 The virtual textures are paired visual and tactile models of real surfaces \cite{culbertson2014one} that we render as visual and haptic overlays on the touched augmented surfaces.

4-conclusion/conclusion.tex

@@ -3,21 +3,28 @@
 
 \section{Summary}
 
-In this thesis, entitled \enquote{\textbf{\ThesisTitle}}, we presented our research on direct hand interaction with real and virtual everyday objects, visually and haptically augmented using immersive \AR and wearable haptic devices.
-The first axis of our research was the \textbf{}.
-The second axis was the \textbf{}.
+% we presented our research on direct hand interaction with real and virtual everyday objects, visually and haptically augmented using immersive \AR and wearable haptic devices.
+In this thesis, entitled \enquote{\textbf{\ThesisTitle}}, we presented our research towards a coherent, natural and seamless  visuo-haptic augmented reality that enables perception and interaction with everyday objects directly with the hand.
+Worn on the hand, wearable haptics can provide a rich tactile feedback on \VOs and augment the perception of real objects, while preserving the freedom of movement and interaction with the \RE.
+However, their integration with \AR is still in its infancy, and presents many design, technical and human challenges.
+We focused our research on two: \textbf{(I) modifying the texture perception of tangible surfaces}, and \textbf{(II) improving the manipulation of \VOs}.
 
-\noindentskip In \partref{perception},
+\noindentskip In \partref{perception} we focused on modifying the perception of wearable virtual visuo-haptic textures that augments tangible surfaces.
+%We designed wearable visuo-haptic texture augmentations and evaluated how the degree of virtuality and the rendering of the visuals influenced the perception of the haptic textures.
+We (1) proposed a \textbf{wearable visuo-haptic texture augmentation system}, (2) evaluated how the perception of haptic textures is \textbf{affected by the visual virtuality of the hand} and the environment (real, augmented, or virtual), and (3) investigated the \textbf{perception of co-localized visuo-haptic texture augmentations}.
 
 \noindentskip In \chapref{vhar_system},
 
-\noindentskip In \chapref{xr_perception},
+\noindentskip In \chapref{xr_perception}, we explored how the perception of wearable haptic augmented textures is affected by the visual virtuality of the hand and the environment, being either real, augmented or virtual.
+We augmented the perceived roughness of the tangible surface by rendering virtual vibrotactile patterned textures on the voice-coil, and rendered the visual conditions with an immersive \OST-\AR headset that could be switched to a \VR only view.
+We then conducted a psychophysical user study with 20 participants and extensive questionnaires to evaluate the perceived roughness augmentation in these three visual conditions.
+The textures were perceived as \textbf{rougher when touched with the real hand alone} compared to a virtual hand in either \AR or \VR, possibly due to the \textbf{perceived latency} between finger movements and different visual, haptic, and proprioceptive feedbacks.
 
 \noindentskip In \chapref{vhar_textures},
 
-\noindentskip In \partref{manipulation}, we addressed the challenge of improving the manipulation of \VOs directly with the hand in immersive \OST-\AR.
-Our approach was to design, based on the literature, and evaluate in user studies the effect of visual rendering of the hand and the delocalized haptic rendering
-We first focused on (1) \textbf{the visual rendering as hand augmentation} and then on the (2) \textbf{combination of different visuo-haptic rendering of the hand manipulation with \VOs}.
+\noindentskip In \partref{manipulation} we focused on improving the manipulation of \VOs directly with the hand in immersive \OST-\AR.
+Our approach was to design, based on the literature, and evaluate in user studies the effect of visual rendering of the hand and the delocalized haptic rendering.
+We first considered (1) \textbf{the visual rendering as hand augmentation} and then the (2) combination of different visuo-haptic \textbf{rendering of the hand manipulation with \VOs}.
 
 \noindentskip In \chapref{visual_hand}, we investigated the visual rendering as hand augmentation.
 Seen as an \textbf{overlay on the user's hand}, such visual hand rendering provide feedback on the hand tracking and the interaction with \VOs.
@@ -27,7 +34,7 @@ This rendering provided a detailed view of the tracked phalanges while being thi
 
 \noindentskip In \chapref{visuo_haptic_hand}, we then investigated the visuo-haptic rendering as feedback of the direct hand manipulation with \VOs using wearable vibrotactile haptics.
 In a user study with a similar design and 20 participants, we compared two vibrotactile contact techniques, provided at \textbf{four different delocalized positions on the user's hand}, and combined with the two most representative visual hand renderings from the previous chapter.
-The results showed that providing vibrotactile feedback \textbf{improved the perceived effectiveness, realism, and usefulness when it is provided close to the fingertips}, and that the visual hand rendering complemented the haptic hand rendering well in giving a continuous feedback on the hand tracking.
+The results showed that providing vibrotactile feedback \textbf{improved the perceived effectiveness, realism, and usefulness when it was provided close to the fingertips}, and that the visual hand rendering complemented the haptic hand rendering well in giving a continuous feedback on the hand tracking.
 
 \section{Future Work}
 
@@ -68,7 +75,7 @@ More generally, many other haptic feedbacks could be investigated in \AR \vs \VR
 
 \subsection*{Perception of Visual and Haptic Texture Augmentations in Augmented Reality}
 
-\subsection*{Visual Rendering of the Hand for Manipulating Virtual Objects in AR}
+\subsection*{Visual Rendering of the Hand for Manipulating \VOs in AR}
 
 \paragraph{Other AR Displays}
 
@@ -85,7 +92,7 @@ While these tasks are fundamental building blocks for more complex manipulation
 Similarly, a broader experimental study might shed light on the role of gender and age, as our subject pool was not sufficiently diverse in this regard.
 Finally, all visual hand renderings received low and high rank rates from different participants, suggesting that users should be able to choose and personalize some aspects of the visual hand rendering according to their preferences or needs, and this should be also be evaluated.
 
-\subsection*{Visuo-Haptic Rendering of Hand Manipulation With Virtual Objects in AR}
+\subsection*{Visuo-Haptic Rendering of Hand Manipulation With \VOs in AR}
 
 \paragraph{Richer Haptic Feedback}
 

references.bib

@@ -1174,6 +1174,17 @@
   doi = {10/gs6x4v}
 }
 
+@article{friston2014measuring,
+  title = {Measuring {{Latency}} in {{Virtual Environments}}},
+  author = {Friston, Sebastian and Steed, Anthony},
+  date = {2014},
+  journaltitle = {IEEE Trans. Vis. Comput. Graph.},
+  volume = {20},
+  number = {4},
+  pages = {616--625},
+  doi = {10/f6f4ct}
+}
+
 @article{frohner2019can,
   title = {Can {{Wearable Haptic Devices Foster}} the {{Embodiment}} of {{Virtual Limbs}}?},
   author = {Frohner, Jakob and Salvietti, Gionata and Beckerle, Philipp and Prattichizzo, Domenico},