Corrections intro

1-background/introduction/introduction.tex

@@ -4,8 +4,8 @@
 \chaptertoc
 
 %This PhD manuscript shows how wearable haptics, worn on the outside of the hand, can improve direct hand interaction in immersive \AR by augmenting the perception of the virtual content and its manipulation.
-This PhD manuscript shows how immersive \AR, which integrates visual virtual content into the real world perception, and wearable haptics, worn on the outside of the hand, can improve the free and direct interaction of the hand with virtual objects.
-Our goal is to achieve a more plausible and coherent perception, as well as a more natural and effective manipulation of the visuo-haptic augmentations. %interaction of the hand with the virtual content.%, moving towards a seamless integration of the virtual into the real world.
+This thesis shows how immersive \AR, which integrates visual virtual content into the real world perception, and wearable haptics, worn on the outside of the hand, can improve the free and direct interaction of the hand with virtual and augmented objects.
+Our goal is to enable users to perceive and interact with wearable visuo-haptic augmentations in a more realistically and effectively, as if they were real. %interaction of the hand with the virtual content.%, moving towards a seamless integration of the virtual into the real world.
 %We are particularly interested in enabling direct contact of virtual and augmented objects with the bare hand.
 %The aim of this thesis is to understand how immersive visual and wearable haptic augmentations complement each other in the context of direct hand perception and manipulation with virtual and augmented objects.
 
@@ -15,10 +15,10 @@ Our goal is to achieve a more plausible and coherent perception, as well as a mo
 \subsectionstarbookmark{Hand Interaction with Everyday Objects}
 
 In daily life, \textbf{we simultaneously look at, touch and manipulate the everyday objects} around us without even thinking about it.
-Many of these object properties can be perceived in a complementary way through all our sensory modalities, such as their shape, size or material \cite{baumgartner2013visual}.
-Vision often precedes touch, enabling us to anticipate the tactile sensations we will feel when touching the object \cite{yanagisawa2015effects}, \eg hardness or texture, and even to predict properties that we cannot see, \eg weight or temperature.
+Many of these object properties can be perceived in a complementary way through all our sensory modalities, such as their shape or material \cite{baumgartner2013visual}.
+Vision often precedes touch, enabling us to anticipate the tactile sensations we will feel when touching the object \cite{yanagisawa2015effects}, \eg hardness or texture, and even to anticipate properties that we cannot see, \eg weight or temperature.
 Information from different sensory sources can be complementary, redundant or contradictory \cite{ernst2004merging}.
-This is why we sometimes want to touch an object to check one of its properties that we have seen and to compare or confront our visual and tactile sensations.
+%This is why we sometimes want to touch an object to check one of its properties that we have seen and to compare or confront our visual and tactile sensations.
 We then \textbf{instinctively construct a unified perception of the properties of the object} we are exploring and manipulating from our sensory modalities, as well as from the movement of our hand and fingers on the object \cite{ernst2002humans}.
 
 The sense of touch also allows us to perceive our environment, but also to represent ourselves in space and interact with the surrounding objects.
@@ -78,12 +78,6 @@ It is technically and conceptually closely related to \emph{\VR}, which complete
 It describes the degree of virtuality of the environment along an axis, with one end being \RE and the other end being pure \VE, \ie indistinguishable from the real world (as in \emph{The Matrix} movies).
 Between these two extremes lies \MR, which includes \AR and \VR as different levels of mixing real and virtual environments \cite{skarbez2021revisiting}.\footnote{This is the original and classic definition of \MR, but there is still debate about how to define and characterize \AR and \MR experiences \cite{speicher2019what,skarbez2021revisiting}.}
 
-We call \AR/\VR \emph{systems} the computational set of hardware (input devices, sensors and displays) and software (tracking, simulation and rendering) that allows the user to interact with the \VE. % by implementing the interaction loop we proposed in \figref{interaction-loop}.
-\AR and \VR systems can address any of the human senses, but most focus on visual augmentation \cite[p.144]{billinghurst2015survey} and \cite{kim2018revisiting}.
-Many visual displays have been explored, from projection systems to hand-held displays.
-\textbf{\AR headsets are the most promising displays as they are portable and provide the user with an immersive augmented environment} \cite{hertel2021taxonomy}.
-%but the most \textbf{promising devices are \AR headsets}, which are \textbf{portable displays worn directly on the head}, providing the user with an \textbf{immersive visual augmented environment}.
-
 \begin{subfigs}{rv-continuums}{Reality-virtuality continuums. }[][
   \item For the visual sense, as originally proposed by and adapted from \textcite{milgram1994taxonomy}.
   \item Extension to include the haptic sense on a second, orthogonal axis, proposed by and adapted from \textcite{jeon2009haptic}.
@@ -93,20 +87,27 @@ Many visual displays have been explored, from projection systems to hand-held di
 \end{subfigs}
 
 %Concepts of virtuality and augmentation can also be applied for sensory modalities other than vision.
-\textcite{jeon2009haptic} proposed to describe visuo-haptic \AR/\VR systems with two orthogonal reality-virtuality continuums, one for vision and one for touch, as shown in \figref{visuo-haptic-rv-continuum5}.
+\textcite{jeon2009haptic} proposed to describe visuo-haptic \AR/\VR with two orthogonal reality-virtuality continuums, one for vision and one for touch, as shown in \figref{visuo-haptic-rv-continuum5}.
 The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of virtuality for each visual or haptic feedback: real, augmented and virtual.
 For example, (visual) \AR using a real object as a proxy to manipulate a virtual object is considered \emph{haptic reality} (\eg \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum5}), whereas a device that provides synthetic haptic feedback when touching a virtual object is considered \emph{haptic virtuality} (\eg \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum5}).
-\textbf{A \emph{haptic augmentation} is then the combination of real and virtual haptic stimuli} \cite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum5}).
-In particular, it has been implemented by augmenting the haptic perception of real objects by providing timely virtual tactile stimuli using wearable haptics:
-\figref{salazar2020altering} shows an example of modifying the perceived stiffness of a real object in \VR using simultaneous pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum5}).
-\figref{bau2012revel} shows another example of visuo-haptic augmentation of virtual texture when running the finger over a real surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum5}).
+\textbf{A \emph{haptic augmentation} is then the combination of real and virtual haptic stimuli}, such that the virtual haptic sensations modify the perception of the real object \cite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum5}).
+%In particular, it has been implemented by augmenting the haptic perception of real objects by providing timely virtual tactile stimuli using wearable haptics:
+\figref{salazar2020altering} shows an example of modifying the perceived stiffness of a real object in \VR using simultaneous wearable pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum5}).
+\figref{bau2012revel} shows another example of visuo-haptic augmentation of virtual texture using reverse electrovibration when running the finger over a real surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum5}).
 
-The illusion of \enquote{being there} when in \VR or of the virtual content to \enquote{feels here} when in \AR \cite{slater2022separate,skarbez2021revisiting} is called \emph{presence}.
+In this thesis we call \AR/\VR \emph{systems} the computational set of hardware (input devices, sensors, displays and haptic devices) and software (tracking, simulation and rendering) that allows the user to interact with the \VE. % by implementing the interaction loop we proposed in \figref{interaction-loop}.
+Many \AR displays have been explored, from projection systems to hand-held displays.
+\textbf{\AR headsets are the most promising display technology as they are portable and provide the user with an immersive augmented environment} \cite{hertel2021taxonomy}.
+Commercial headsets also have integrated real-time self-location and mapping of the \RE and hand tracking of the user.
+While \AR and \VR systems can address any of the human senses, most focus only on visual augmentation \cite[p.144]{billinghurst2015survey} and \cite{kim2018revisiting}.
+%but the most \textbf{promising devices are \AR headsets}, which are \textbf{portable displays worn directly on the head}, providing the user with an \textbf{immersive visual augmented environment}.
+
+\emph{Presence} is the illusion of \enquote{being there} when in \VR, or the illusion of the virtual content to \enquote{feel here} when in \AR \cite{slater2022separate,skarbez2021revisiting}.
 One of the most important aspects of this illusion is the \emph{plausibility}, \ie the illusion that the virtual events are really happening. %, even if the user knows that they are not real.
-However, when an \AR/\VR system lacks haptic feedback, it creates a deceptive and incomplete user experience when the hand reaches the virtual content.
-All visual virtual objects are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties and interact with them with confidence and efficiency.
-It is also necessary to provide a haptic feedback that is coherent with the visual virtual objects and ensures the best possible user experience, as we argue in the next section.
-The \textbf{integration of wearable haptics with \AR} appears to be one of the most promising solutions, but it \textbf{remains challenging due to their many respective characteristics and the additional constraints of combining them}.
+However, when an \AR/\VR system lacks haptic feedback, it may create a deceptive and incomplete user experience when the hand reaches the virtual content.
+All (visual) virtual objects are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties and interact with them with confidence and efficiency.
+It is also necessary to provide a haptic feedback that is coherent with the virtual objects and ensures the best possible user experience, as we argue in the next section.
+The \textbf{integration of wearable haptics with \AR appears to be one of the most promising solutions}, but it remains challenging due to their respective limitations and the additional constraints of combining them, as we will overview in the next section.
 
 \begin{subfigs}{visuo-haptic-environments}{Visuo-haptic environments with varying degrees of reality-virtuality. }[][
   \item \AR environment with a real haptic object used as a proxy to manipulate a virtual object \cite{kahl2023using}.
@@ -135,25 +136,25 @@ Because the visuo-haptic \VE is displayed in real time and aligned with the \RE,
 \fig{interaction-loop}{The interaction loop between a user and a visuo-haptic augmented environment as proposed in this thesis.}[
   A user interacts with the visual (in blue) and haptic (in red) \VEs through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with virtual objects.
   The visual and haptic \VEs are rendered back using an immersive \AR headset and wearable haptics, and are perceived by the user to be registered and co-localized with the \RE (in gray).
-  \protect\footnotemark
+  %\protect\footnotemark
 ]
 
 In this context, we focus on two main research challenges:
 \textbf{(I) providing plausible and coherent visuo-haptic augmentations}, and
 \textbf{(II) enabling effective manipulation of the augmented environment}.
-Each of these challenges also raises numerous design, technical and human issues specific to wearable haptics and immersive \AR.
+Each of these challenges also raises numerous design, technical, perceptual and user experience issues specific to wearable haptics and immersive \AR.
 %, as well as virtual rendering and user experience issues.% in integrating these two sensorimotor feedbacks into a coherent and seamless visuo-haptic augmented environment.
 
-\footnotetext{%
-  The icons are \href{https://creativecommons.org/licenses/by/3.0/}{CC BY} licensed:
-  \enquote{\href{https://thenounproject.com/icon/finger-pointing-4230346/}{finger pointing}} by \href{https://thenounproject.com/creator/leremy/}{Gan Khoon Lay},
-  \enquote{\href{https://thenounproject.com/icon/hololens-1499195/}{HoloLens}} by \href{https://thenounproject.com/creator/daniel2021/}{Daniel Falk}, and
-  \enquote{\href{https://thenounproject.com/icon/vibration-6478365/}{vibrations}} by \href{https://thenounproject.com/creator/iconbunny/}{Iconbunny}.
-}
+%\footnotetext{%
+%  The icons are \href{https://creativecommons.org/licenses/by/3.0/}{CC BY} licensed:
+%  \enquote{\href{https://thenounproject.com/icon/finger-pointing-4230346/}{finger pointing}} by \href{https://thenounproject.com/creator/leremy/}{Gan Khoon Lay},
+%  \enquote{\href{https://thenounproject.com/icon/hololens-1499195/}{HoloLens}} by \href{https://thenounproject.com/creator/daniel2021/}{Daniel Falk}, and
+%  \enquote{\href{https://thenounproject.com/icon/vibration-6478365/}{vibrations}} by \href{https://thenounproject.com/creator/iconbunny/}{Iconbunny}.
+%}
 
 \subsectionstarbookmark{Challenge I: Providing Plausible and Coherent Visuo-Haptic Augmentations}
 
-\textbf{Many haptic devices have been designed and evaluated specifically for use in \VR}, providing varied kinesthetic and tactile feedback to virtual objects, and adding realism when interacting with them \cite{culbertson2018haptics}.
+\textbf{Many haptic devices have been designed and evaluated specifically for use in \VR}, providing the user with rich kinesthetic and tactile feedback on virtual objects, increasing the realism and effectiveness of interaction with them \cite{culbertson2018haptics}.
 Although closely related, \AR and \VR have key differences in their respective renderings that can affect user perception.
 
 %As such, in \VR, visual sensations are particularly dominant in perception, and conflicts with haptic sensations are also specifically created to influence the user's perception, for example to create pseudo-haptic \cite{ujitoko2021survey} or haptic retargeting \cite{azmandian2016haptic} effects.
@@ -166,7 +167,7 @@ It remains to be investigated how such potential discrepancies affect the overal
 
 %So far, most of the \AR studies and applications only add visual and haptic sensations to the user's overall perception of the environment, but conversely it is more difficult to remove sensations.
 %Visual and haptic augmentations of the \RE add sensations to the user's overall perception.
-The \textbf{added visual and haptic virtual sensations may also be perceived as incoherent} with the sensations of \RE, for example with a lower rendering quality, a temporal latency, a spatial shift, or a combination of these.
+The \textbf{added visual and haptic virtual sensations may also be perceived as incoherent} with the sensations of the real objects, for example with a lower rendering quality, a temporal latency, a spatial shift, or a combination of these.
 Moreover, in \AR the user can still see the real world environment, including their hands, augmented real objects and worn haptic devices, unlike \VR where there is total control over the visual rendering. % of the hand and \VE.
 It is therefore unclear to what extent the real and virtual visuo-haptic sensations will be perceived as a whole, and to what extent they will conflict or complement each other. % in the perception of the augmented environment.
 With a better understanding of \textbf{how visual factors can influence the perception of haptic augmentations}, the many wearable haptic systems that already exist but have not yet been fully explored with \AR can be better applied, and new visuo-haptic augmentations adapted to \AR can be designed.
@@ -181,9 +182,9 @@ However, \textbf{manipulating a purely virtual object with the bare hand can be
 In addition, current \AR systems have visual rendering limitations that also affect interaction with virtual objects. %, due to depth underestimation, a lack of mutual occlusions,  and hand tracking latency.
 \AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
 However, the depth perception of virtual objects is often underestimated \cite{peillard2019studying,adams2022depth}.
-There is also often \textbf{a lack of mutual occlusion between the hand and a virtual object}, \ie that the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
-Finally, as illustrated in \figref{interaction-loop}, interaction with a virtual object is an illusion, because the real hand controls in real time a virtual hand, like an avatar, whose contacts with virtual objects are then simulated in the \VE.
-Therefore, there is inevitably a latency between the movements of the real hand and the return movements of the virtual object, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the virtual object touched \cite{prachyabrued2014visual}.
+There is also often \textbf{a lack of mutual occlusion between the hand and a virtual object}, that is the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
+Finally, as illustrated in our interaction loop \figref{interaction-loop}, interaction with a virtual object is an illusion, because the real hand controls in real time a virtual hand, like an avatar, whose contacts with virtual objects are then simulated in the \VE.
+Therefore, there is inevitably a latency between the movements of the real hand and the feedback movements of the virtual object, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the virtual object touched \cite{prachyabrued2014visual}.
 These three rendering limitations make it \textbf{difficult to perceive the position of the fingers relative to the object} before touching or grasping it, but also to estimate the force required to grasp the virtual object and move it to a desired location.
 
 Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a virtual object with the hand.
@@ -193,7 +194,7 @@ Yet, it is unclear which type of visual and haptic feedback, or their combinatio
 \label{contributions}
 
 %The aim of this thesis is to understand how immersive visual and wearable haptic augmentations complement each other in the context of direct hand perception and manipulation with virtual and augmented objects.
-As described in the Research Challenges section above, providing a coherent and effective visuo-haptic augmented environment to a user is complex and raises many issues.
+As described in the research challenges section, providing a coherent and effective visuo-haptic augmented environment to a user is complex and raises many issues.
 Our approach is to:
 \begin{enumerate*}[label=(\arabic*)]
 \item design immersive and wearable visuo-haptic renderings that augment both the objects being interacted with and the hand interacting with them, and
@@ -205,10 +206,10 @@ We consider two main axes of research, each addressing one of the research chall
 \item \textbf{augmenting the texture perception of real surfaces}, and % with visuo-haptic texture augmentations, and
 \item \textbf{improving the manipulation of virtual objects}.% with visuo-haptic augmentations of the hand-object interaction.
 \end{enumerate*}
-Our contributions in these two axes are summarized in \figref{contributions}.
+Our contributions are summarized in \figref{contributions}.
 
 \fig[0.95]{contributions}{Summary of our contributions through the simplified interaction loop.}[
-  The contributions are represented in dark grey boxes, the research axes in dark green circles and the research objectives in light green circles.
+  The contributions are represented in dark grey boxes, the research axes in green circles. % and the research objectives in light green circles.
   The first axis is \textbf{(I)} the design and evaluation of the perception of visuo-haptic texture augmentations of real surfaces, directly touched by the hand.
   The second axis focuses on \textbf{(II)} improving the manipulation of virtual objects with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.
 ]
@@ -220,7 +221,7 @@ Wearable haptic devices have proven effective in modifying the perception of a t
 %It enables rich haptic feedback as the combination of kinesthetic sensation from the real and cutaneous sensation from the actuator.
 However, wearable haptic augmentation with \AR has been little explored, as well as the visuo-haptic augmentation of texture.
 Texture is indeed one of the most fundamental perceived properties of a surface material \cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) augmentation \cite{unger2011roughness,culbertson2014modeling,asano2015vibrotactile,strohmeier2017generating,friesen2024perceived}.
-Coherently substitute the visuo-haptic texture of a surface directly touched by a finger is an important step towards a \AR capable of visually and haptically augmenting the \RE of a user in a plausible way.
+%Coherently substitute the visuo-haptic texture of a surface directly touched by a finger is an important step towards a \AR capable of visually and haptically augmenting the \RE of a user in a plausible way.
 For this first axis of research, we propose to design and evaluate the \textbf{perception of wearable virtual visuo-haptic textures augmenting real surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
 To this end, we (1) design a system for rendering wearable visuo-haptic texture augmentations, to (2) evaluate how the perception of haptic texture augmentations is affected by visual feedback of the virtual hand and the environment, and (3) investigate the perception of co-localized visuo-haptic texture augmentations.
 
@@ -233,7 +234,7 @@ Second, many works have investigated the haptic augmentations of texture, but no
 Still, it is known that visual feedback can alter the perception of real and virtual haptic sensations \cite{schwind2018touch,choi2021augmenting} but also that the force feedback perception of grounded haptic devices is not the same in \AR and \VR \cite{diluca2011effects,gaffary2017ar}.
 Hence, our second objective is to \textbf{evaluate how the perception of wearable haptic texture augmentation is affected by the visual feedback of the virtual hand and the environment} (real, augmented or virtual).
 
-Finally, some visuo-haptic texture databases have been created from real texture captures \cite{culbertson2014penn,balasubramanian2024sens3} to be rendered as virtual textures with hand-held haptic devices that are perceived as similar to real textures \cite{culbertson2015should,friesen2024perceived}.
+Finally, visuo-haptic texture databases have been created from real texture captures \cite{culbertson2014penn,balasubramanian2024sens3} to be rendered as virtual textures with hand-held haptic devices that are perceived as similar to real textures \cite{culbertson2015should,friesen2024perceived}.
 However, the rendering of these textures in an immersive and natural visuo-haptic \AR using wearable haptics remains to be investigated.
 Our third objective is to \textbf{evaluate the perception of simultaneous and co-localized wearable visuo-haptic texture augmentations} of real surfaces in \AR. %, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
 

1-background/related-work/4-visuo-haptic-ar.tex

@@ -74,7 +74,7 @@ The overall perception can then be modified by changing one of the sensory modal
 \textcite{yanagisawa2015effects} altered the perceived roughness, stiffness, and friction of real tactile materials touched by the finger by superimposing different real visual textures using a half-mirror.
 In a similar setup, but in immersive \VST-\AR, \textcite{kitahara2010sensory} overlaid visual textures on real textured surfaces touched through a glove: many visual textures were found to match the real haptic textures.
 \textcite{degraen2019enhancing} and \textcite{gunther2022smooth} also combined multiple virtual objects in \VR with \ThreeD-printed hair structures or with everyday real surfaces, respectively.
-They found that the visual perception of roughness and hardness influenced the haptic perception, and that only a few real, tangible objects seemed to be sufficient to match all the visual virtual objects (\figref{gunther2022smooth}).
+They found that the visual perception of roughness and hardness influenced the haptic perception, and that only a few real objects seemed to be sufficient to match all the visual virtual objects (\figref{gunther2022smooth}).
 %Taken together, these studies suggest that a set of haptic textures, real or virtual, can be perceived as coherent with a larger set of visual virtual textures.
 
 \fig{gunther2022smooth}{In a passive touch context in \VR, only a smooth and a rough real surfaces were found to match all the visual virtual objects \cite{gunther2022smooth}.}