diff --git a/1-introduction/introduction/introduction.tex b/1-introduction/introduction/introduction.tex index 386863d..1da023a 100644 --- a/1-introduction/introduction/introduction.tex +++ b/1-introduction/introduction/introduction.tex @@ -120,23 +120,23 @@ The most mature devices are \HMDs, which are portable headsets worn directly on % In particular, \textcite{jeon2009haptic} proposed extending the \RV continuum to include haptic feedback by decoupling into two orthogonal haptic and visual axes (see \figref{visuo-haptic-rv-continuum3}). % -The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of \RV for each visual or haptic axis (real, augmented, virtual). +The combination of the two axes defines 9 types of \vh environments, with 3 possible levels of \RV for each \v or \h axis: real, augmented and virtual. % -For example, a \vAE that uses a tangible (touchable) object as a proxy to manipulate virtual content is considered a \glsdisp{hRE}{\acrlong{hRE}} (\acrshort{hRE}; see \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum3}), whereas a device that provides synthetic haptic feedback when touching a \VO is considered a \glsdisp{hVE}{\acrlong{hVE}} (\acrshort{hVE}; see \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum3}). +For example, a \v-\AE that uses a tangible (touchable) object as a proxy to manipulate virtual content is considered a \h-\RE (\eg \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum3}), whereas a device that provides synthetic haptic feedback when touching a \VO is considered a \h-\VE (\eg \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum3}). % -\Gls{hAR} is then the combination of real and virtual haptic stimuli~\autocite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum3}). +Haptic \AR is then the combination of real and virtual haptic stimuli~\autocite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum3}). % In particular, it has been implemented by enhancing the haptic perception of tangible objects by providing timely tactile stimuli using wearable haptics. % \figref{salazar2020altering} shows an example of modifying the perceived stiffness of a tangible object in \VR using simultaneous pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum3}). % -\figref{bau2012revel} shows another example of visuo-haptic \AR rendering of texture when running the finger on a tangible surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum3}). +\figref{bau2012revel} shows another example of \vh-\AR rendering of texture when running the finger on a tangible surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum3}). -Current (visual) \AR systems often lack haptic feedback, creating a deceptive and incomplete user experience when reaching the \VE with the hand. +Current \v-\AR systems often lack haptic feedback, creating a deceptive and incomplete user experience when reaching the \VE with the hand. % -All visual \VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties congruently and interact with them with confidence and efficiency. +All \v-\VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties congruently and interact with them with confidence and efficiency. % -It is therefore necessary to provide haptic feedback that is consistent with the \vAE and ensures the best possible user experience. +It is therefore necessary to provide haptic feedback that is consistent with the \v-\AE and ensures the best possible user experience. % The integration of wearable haptics with \AR seems to be one of the most promising solutions, but it remains challenging due to their many respective characteristics and the additional constraints of combining them. @@ -145,8 +145,8 @@ The integration of wearable haptics with \AR seems to be one of the most promisi }[ \item Visual \AR environment with a real, tangible haptic object used as a proxy to manipulate a \VO~\autocite{kahl2023using}. \item Visual \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a \VO~\autocite{meli2018combining}. - \item A tangible object seen in a visual \VR environment whose haptic perception of stiffness is augmented with the hRing haptic device~\autocite{salazar2020altering}. - \item Visuo-haptic rendering of texture on a touched tangible object with a visual \AR display and haptic electrovibration feedback~\autocite{bau2012revel}. + \item A tangible object seen in a \v-\VR environment whose haptic perception of stiffness is augmented with the hRing haptic device~\autocite{salazar2020altering}. + \item Visuo-haptic rendering of texture on a touched tangible object with a \v-\AR display and haptic electrovibration feedback~\autocite{bau2012revel}. ] \subfigsheight{31mm} \subfig{kahl2023using} @@ -158,25 +158,25 @@ The integration of wearable haptics with \AR seems to be one of the most promisi \section{Research Challenges of Wearable Visuo-Haptic Augmented Reality} -The integration of wearable haptics with \AR to create a \vhAE is complex and presents many perceptual and interaction challenges, \ie sensing the \AE and acting effectively upon it. +The integration of wearable haptics with \AR to create a \vh \AE is complex and presents many perceptual and interaction challenges, \ie sensing the \AE and acting effectively upon it. % We are particularly interested in enabling direct contact and manipulation of virtual and augmented objects with the bare hand. % -Our goal is to enable congruent, intuitive and seamless perception of and interaction with the \vhAE. +Our goal is to enable congruent, intuitive and seamless perception of and interaction with the \vh-\AE. -The experience of such a \vhAE relies on an interaction loop with the user, as illustrated in \figref{interaction-loop}. +The experience of such a \vh-\AE relies on an interaction loop with the user, as illustrated in \figref{interaction-loop}. % The \RE and the user's hand are tracked in real time by sensors and reconstructed in visual and haptic VEs. % The interactions between the virtual hand and objects are then simulated and rendered as visual and haptic feedback to the user using an \AR headset and a wearable haptic device. % -Because the visuo-haptic \VE is displayed in real time, colocalized and aligned with the real one, the user is given the illustion of directly perceiving and interacting with the virtual content as if it were part of the \RE. +Because the \vh-\VE is displayed in real time, colocalized and aligned with the real one, the user is given the illustion of directly perceiving and interacting with the virtual content as if it were part of the \RE. \fig{interaction-loop}{% The interaction loop between a user and a visuo-haptic augmented environment. % }[% One interact with the visual (in blue) and haptic (in red) virtual environment through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with \VOs. % - The virtual environment is rendered back to the user colocalized with the real one (in gray) using a visual \AR headset and a wearable haptic device. % + The virtual environment is rendered back to the user colocalized with the real one (in gray) using a \v-\AR headset and a wearable haptic device. % ] %This to ensure the best possible user experience, taking into account the current capabilities and limitations of wearable haptics and augmented reality technologies. @@ -188,7 +188,7 @@ In this context, we identify two main research challenges that we address in thi \item enabling effective interaction with the augmented environment. \end{enumerate*} % -Each of these challenges also raises numerous design, technical and human issues specific to each of the two types of feedback, wearable haptics and immersive \AR, as well as multimodal rendering and user experience issues in integrating these two sensorimotor feedbacks into a coherent and seamless \vhAE. +Each of these challenges also raises numerous design, technical and human issues specific to each of the two types of feedback, wearable haptics and immersive \AR, as well as multimodal rendering and user experience issues in integrating these two sensorimotor feedbacks into a coherent and seamless \vh-\AE. %These challenges are illustrated in the visuo-haptic interaction loop in \figref{interaction-loop}. @@ -208,7 +208,7 @@ Moreover, many wearable haptic devices take the form of controllers, gloves or e % The user's hand must be indeed free to touch and interact with the \RE. % -It is possible instead to place the haptic actuator close to the point of contact with the \RE, as described above to implement \hAR, \eg providing haptic feedback on another phalanx~\autocite{asano2015vibrotactile,salazar2020altering} or the wrist~\autocite{sarac2022perceived} for rendering fingertip contacts with virtual content. +It is possible instead to place the haptic actuator close to the point of contact with the \RE, as described above to implement \h-\AR, \eg providing haptic feedback on another phalanx~\autocite{asano2015vibrotactile,salazar2020altering} or the wrist~\autocite{sarac2022perceived} for rendering fingertip contacts with virtual content. % Therefore, when touching a virtual or augmented object, the real and virtual visual sensations are seen as colocalised, but the virtual haptic feedback is not. % @@ -255,7 +255,7 @@ Yet, it is unclear which type of visual and haptic feedback is the best suited t The aim of this thesis is to understand how immersive visual and wearable haptic feedback compare and complement each other in the context of direct hand perception and interaction with augmented and \VOs. % -As described in the Research Challenges section above, providing a convincing, consistent and effective \vhAE to a user is complex and raises many issues. +As described in the Research Challenges section above, providing a convincing, consistent and effective \vh-\AE to a user is complex and raises many issues. % Our approach is to % @@ -289,13 +289,13 @@ Our contributions in these two axes are summarized in \figref{contributions}. % Very short abstract of contrib 2 -Wearable haptic devices have proven to be effective in modifying the perception of a touched tangible surface, without modifying the tangible, nor covering the fingertip, forming a \hAE~\autocite{bau2012revel,detinguy2018enhancing,salazar2020altering}. +Wearable haptic devices have proven to be effective in modifying the perception of a touched tangible surface, without modifying the tangible, nor covering the fingertip, forming a \h-\AE~\autocite{bau2012revel,detinguy2018enhancing,salazar2020altering}. % %It is achieved by placing the haptic actuator close to the fingertip, to let it free to touch the surface, and rendering tactile stimuli timely synchronised with the finger movement. % %It enables rich haptic feedback as the combination of kinesthetic sensation from the tangible and cutaneous sensation from the actuator. % -However, wearable \hAR have been little explored with visual \AR, as well as the visuo-haptic augmentation of textures. +However, wearable \h-\AR have been little explored with \v-\AR, as well as the visuo-haptic augmentation of textures. % Texture is indeed one of the main tactile sensation of a surface material~\cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by both sight and touch~\cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) rendering~\cite{unger2011roughness,culbertson2014modeling,strohmeier2017generating}. % @@ -317,19 +317,19 @@ Hence, our second objective is to understand how the perception of haptic textur Finally, some visuo-haptic texture databases have been modelled from real texture captures~\autocite{culbertson2014penn,balasubramanian2024sens3}, to be rendered as virtual textures with graspable haptics that are perceived as similar to real textures~\autocite{culbertson2015should,friesen2024perceived}. % -However, the rendering of these textures in an immersive and natural visuo-haptic \AR using wearable haptics remains to be investigated. +However, the rendering of these textures in an immersive and natural \vh-\AR using wearable haptics remains to be investigated. % Our third objective is to evaluate the perception of simultaneous and co-localized visuo-haptic texture augmentation of tangible surfaces in \AR, directly touched by the hand, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture. \subsectionstarbookmark{Improving Virtual Object Interaction with Visuo-Haptic Augmentations of the Hand} -In immersive and wearable visuo-haptic \AR, the hand is free to touch and interact seamlessly with real, augmented, and virtual objects, and one can expect natural and direct contact and manipulation of \VOs with the bare hand. +In immersive and wearable \vh-\AR, the hand is free to touch and interact seamlessly with real, augmented, and virtual objects, and one can expect natural and direct contact and manipulation of \VOs with the bare hand. % -However, the intangibility of the vVE, the many display limitations of current visual \AR systems and wearable haptic devices, and the potential discrepancies between these two types of feedback can make the interaction with \VOs particularly challenging. +However, the intangibility of the vVE, the many display limitations of current \v-\AR systems and wearable haptic devices, and the potential discrepancies between these two types of feedback can make the interaction with \VOs particularly challenging. %However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the real environment, the visual feedback, and the haptic feedback, can make the interaction with \VOs with bare hands particularly challenging. % -Still two types of sensory feedback are known to improve such direct \VO interaction, but they have not been studied in combination in immersive \vAE: visual rendering of the hand~\autocite{piumsomboon2014graspshell,prachyabrued2014visual} and hand-object interaction rendering with wearable haptics~\autocite{lopes2018adding,teng2021touch}. +Still two types of sensory feedback are known to improve such direct \VO interaction, but they have not been studied in combination in immersive \v-\AE: visual rendering of the hand~\autocite{piumsomboon2014graspshell,prachyabrued2014visual} and hand-object interaction rendering with wearable haptics~\autocite{lopes2018adding,teng2021touch}. % For this second axis of research, we propose to design and evaluate the role of visuo-haptic augmentations of the hand as interaction feedback with \VOs. % @@ -339,11 +339,11 @@ First, the visual rendering of the virtual hand is a key element for interacting % A few works have also investigated the visual rendering of the virtual hand in \AR, from simulating mutual occlusions between the hand and \VOs~\autocite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay~\autocite{blaga2017usability,yoon2020evaluating}, augmenting the real hand. % -But visual \AR has significant perceptual differences from \VR due to the visibility of the real hand and environment, and these visual hand augmentations have not been evaluated in the context of \VO manipulation. +But \v-\AR has significant perceptual differences from \VR due to the visibility of the real hand and environment, and these visual hand augmentations have not been evaluated in the context of \VO manipulation. % Thus, our fourth objective is to evaluate and compare the effect of different visual hand augmentations on direct manipulation of \VOs in \AR. -Finally, as described above, wearable haptics for visual \AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE. +Finally, as described above, wearable haptics for \v-\AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE. % Previous works have shown that wearable haptics that provide feedback on the hand interaction with \VOs in \AR can significantly improve the user performance and experience~\autocite{maisto2017evaluation,meli2018combining}. % diff --git a/config/acronyms.tex b/config/acronyms.tex index 6a491eb..e6c66ac 100644 --- a/config/acronyms.tex +++ b/config/acronyms.tex @@ -33,25 +33,20 @@ } \let\AE\undefined +\let\v\undefined + \acronym{AE}{augmented environment} \acronym{AR}{augmented reality} -\acronym{hAE}{haptic augmented environment} -\acronym{hAR}{haptic augmented reality} +\acronym{h}{haptic} \acronym{HMD}{head-mounted display} -\acronym{hRE}{haptic real environment} -\acronym{hVE}{haptic virtual environment} \acronym{MR}{mixed reality} \acronym{OST}{optical see-through} \acronym{RE}{real environment} \acronym{RV}{reality-virtuality} -\acronym{vAE}{visual augmented environment} +\acronym{v}{visual} \acronym{VCA}{voice-coil actuator} \acronym{VE}{virtual environment} -\acronym{vhAE}{visuo-haptic augmented environment} -\acronym{vhRE}{visuo-haptic real environment} -\acronym{vhVE}{visuo-haptic virtual environment} +\acronym{vh}{visuo-haptic} \acronym{VO}{virtual object} \acronym{VR}{virtual reality} -\acronym{vRE}{visual real environment} \acronym{VST}{visual see-through} -\acronym{vVE}{visual virtual environment}