erwan/phd-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove \VO and \AE acronym

Browse Source
This commit is contained in:
Erwan Normand
2024-10-18 14:49:07 +02:00
parent 625892eb9c
commit bd4a436f18
17 changed files with 164 additions and 168 deletions
Download Patch File Download Diff File Expand all files Collapse all files

72
1-background/introduction/introduction.tex
View File

@@ -95,7 +95,7 @@ Many visual displays have been explored, from projection systems to hand-held di
%Concepts of virtuality and augmentation can also be applied for sensory modalities other than vision.
\textcite{jeon2009haptic} proposed to describe visuo-haptic \AR/\VR systems with two orthogonal reality-virtuality continuums, one for vision and one for touch, as shown in \figref{visuo-haptic-rv-continuum5}.
The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of virtuality for each visual or haptic feedback: real, augmented and virtual.
For example, (visual) \AR using a real object as a proxy to manipulate a \VO is considered \emph{haptic reality} (\eg \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum5}), whereas a device that provides synthetic haptic feedback when touching a \VO is considered \emph{haptic virtuality} (\eg \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum5}).
For example, (visual) \AR using a real object as a proxy to manipulate a virtual object is considered \emph{haptic reality} (\eg \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum5}), whereas a device that provides synthetic haptic feedback when touching a virtual object is considered \emph{haptic virtuality} (\eg \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum5}).
\textbf{A \emph{haptic augmentation} is then the combination of real and virtual haptic stimuli} \cite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum5}).
In particular, it has been implemented by augmenting the haptic perception of real objects by providing timely virtual tactile stimuli using wearable haptics:
\figref{salazar2020altering} shows an example of modifying the perceived stiffness of a real object in \VR using simultaneous pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum5}).
@@ -104,13 +104,13 @@ In particular, it has been implemented by augmenting the haptic perception of re
The illusion of \enquote{being there} when in \VR or of the virtual content to \enquote{feels here} when in \AR \cite{slater2022separate,skarbez2021revisiting} is called \emph{presence}.
One of the most important aspects of this illusion is the \emph{plausibility}, \ie the illusion that the virtual events are really happening. %, even if the user knows that they are not real.
However, when an \AR/\VR system lacks haptic feedback, it creates a deceptive and incomplete user experience when the hand reaches the virtual content.
All visual \VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties and interact with them with confidence and efficiency.
It is also necessary to provide a haptic feedback that is coherent with the visual \VOs and ensures the best possible user experience, as we argue in the next section.
All visual virtual objects are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties and interact with them with confidence and efficiency.
It is also necessary to provide a haptic feedback that is coherent with the visual virtual objects and ensures the best possible user experience, as we argue in the next section.
The \textbf{integration of wearable haptics with \AR} appears to be one of the most promising solutions, but it \textbf{remains challenging due to their many respective characteristics and the additional constraints of combining them}.
\begin{subfigs}{visuo-haptic-environments}{Visuo-haptic environments with varying degrees of reality-virtuality. }[][
\item \AR environment with a real haptic object used as a proxy to manipulate a \VO \cite{kahl2023using}.
\item \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a \VO \cite{meli2018combining}.
\item \AR environment with a real haptic object used as a proxy to manipulate a virtual object \cite{kahl2023using}.
\item \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a virtual object \cite{meli2018combining}.
\item A real object seen in a visual \VR environment whose haptic perception of stiffness is augmented with the hRing haptic device \cite{detinguy2018enhancing,salazar2020altering}.
\item Visuo-haptic texture augmentation of a real object to be touched using a handheld \AR display and haptic electrovibration feedback \cite{bau2012revel}.
]
@@ -125,7 +125,7 @@ The \textbf{integration of wearable haptics with \AR} appears to be one of the m
\label{research_challenges}
The integration of wearable haptics with \AR to create a visuo-haptic augmented environment is complex and presents many perceptual and interaction challenges.
% \ie sensing the \AE and acting effectively upon it.
% \ie sensing the augmented environment and acting effectively upon it.
In this thesis, we propose to \textbf{represent the user's experience with such a visuo-haptic augmented environment as an interaction loop}, shown in \figref{interaction-loop}.
It is based on the interaction loops of users with \ThreeD systems \cite[p.84]{laviolajr20173d}.
The \RE and the user's hand are tracked in real time by sensors and reconstructed in visual and haptic \VEs.
@@ -133,7 +133,7 @@ The interactions between the virtual hand and objects are then simulated, and re
Because the visuo-haptic \VE is displayed in real time and aligned with the \RE, the user is given the illusion of directly perceiving and interacting with the virtual content as if it were part of the \RE.
\fig{interaction-loop}{The interaction loop between a user and a visuo-haptic augmented environment as proposed in this thesis.}[
A user interacts with the visual (in blue) and haptic (in red) \VEs through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with \VOs.
A user interacts with the visual (in blue) and haptic (in red) \VEs through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with virtual objects.
The visual and haptic \VEs are rendered back using an immersive \AR headset and wearable haptics, and are perceived by the user to be registered and co-localized with the \RE (in gray).
\protect\footnotemark
]
@@ -142,7 +142,7 @@ In this context, we focus on two main research challenges:
\textbf{(I) providing plausible and coherent visuo-haptic augmentations}, and
\textbf{(II) enabling effective manipulation of the augmented environment}.
Each of these challenges also raises numerous design, technical and human issues specific to wearable haptics and immersive \AR.
%, as well as virtual rendering and user experience issues.% in integrating these two sensorimotor feedbacks into a coherent and seamless visuo-haptic \AE.
%, as well as virtual rendering and user experience issues.% in integrating these two sensorimotor feedbacks into a coherent and seamless visuo-haptic augmented environment.
\footnotetext{%
The icons are \href{https://creativecommons.org/licenses/by/3.0/}{CC BY} licensed:
@@ -153,7 +153,7 @@ Each of these challenges also raises numerous design, technical and human issues
\subsectionstarbookmark{Challenge I: Providing Plausible and Coherent Visuo-Haptic Augmentations}
\textbf{Many haptic devices have been designed and evaluated specifically for use in \VR}, providing varied kinesthetic and tactile feedback to \VOs, and adding realism when interacting with them \cite{culbertson2018haptics}.
\textbf{Many haptic devices have been designed and evaluated specifically for use in \VR}, providing varied kinesthetic and tactile feedback to virtual objects, and adding realism when interacting with them \cite{culbertson2018haptics}.
Although closely related, \AR and \VR have key differences in their respective renderings that can affect user perception.
%As such, in \VR, visual sensations are particularly dominant in perception, and conflicts with haptic sensations are also specifically created to influence the user's perception, for example to create pseudo-haptic \cite{ujitoko2021survey} or haptic retargeting \cite{azmandian2016haptic} effects.
@@ -168,26 +168,26 @@ It remains to be investigated how such potential discrepancies affect the overal
%Visual and haptic augmentations of the \RE add sensations to the user's overall perception.
The \textbf{added visual and haptic virtual sensations may also be perceived as incoherent} with the sensations of \RE, for example with a lower rendering quality, a temporal latency, a spatial shift, or a combination of these.
Moreover, in \AR the user can still see the real world environment, including their hands, augmented real objects and worn haptic devices, unlike \VR where there is total control over the visual rendering. % of the hand and \VE.
It is therefore unclear to what extent the real and virtual visuo-haptic sensations will be perceived as a whole, and to what extent they will conflict or complement each other. % in the perception of the \AE.
It is therefore unclear to what extent the real and virtual visuo-haptic sensations will be perceived as a whole, and to what extent they will conflict or complement each other. % in the perception of the augmented environment.
With a better understanding of \textbf{how visual factors can influence the perception of haptic augmentations}, the many wearable haptic systems that already exist but have not yet been fully explored with \AR can be better applied, and new visuo-haptic augmentations adapted to \AR can be designed.
\subsectionstarbookmark{Challenge II: Enabling Effective Manipulation of the Augmented Environment}
Touching, \textbf{grasping and manipulating \VOs are fundamental interactions for \AR} \cite{kim2018revisiting}, \VR \cite{bergstrom2021how} and \VEs in general \cite[p.385]{laviolajr20173d}.
Touching, \textbf{grasping and manipulating virtual objects are fundamental interactions for \AR} \cite{kim2018revisiting}, \VR \cite{bergstrom2021how} and \VEs in general \cite[p.385]{laviolajr20173d}.
Since the hand is not occupied or covered with a haptic device so as to not impair interaction with the \RE, as described in the previous section, one can expect a seamless and direct manipulation of the hand with the virtual content as if it were real.
When augmenting a real object, the user's hand is physically constrained by the object, allowing for easy and natural interaction.
However, \textbf{manipulating a purely \VO with the bare hand can be challenging} without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
However, \textbf{manipulating a purely virtual object with the bare hand can be challenging} without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
In addition, current \AR systems have visual rendering limitations that also affect interaction with \VOs. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency.
In addition, current \AR systems have visual rendering limitations that also affect interaction with virtual objects. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency.
\AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
However, the depth perception of \VOs is often underestimated \cite{peillard2019studying,adams2022depth}.
There is also often \textbf{a lack of mutual occlusion between the hand and a \VO}, \ie that the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
Finally, as illustrated in \figref{interaction-loop}, interaction with a \VO is an illusion, because the real hand controls in real time a virtual hand, like an avatar, whose contacts with \VOs are then simulated in the \VE.
Therefore, there is inevitably a latency between the movements of the real hand and the return movements of the \VO, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the \VO touched \cite{prachyabrued2014visual}.
These three rendering limitations make it \textbf{difficult to perceive the position of the fingers relative to the object} before touching or grasping it, but also to estimate the force required to grasp the \VO and move it to a desired location.
However, the depth perception of virtual objects is often underestimated \cite{peillard2019studying,adams2022depth}.
There is also often \textbf{a lack of mutual occlusion between the hand and a virtual object}, \ie that the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
Finally, as illustrated in \figref{interaction-loop}, interaction with a virtual object is an illusion, because the real hand controls in real time a virtual hand, like an avatar, whose contacts with virtual objects are then simulated in the \VE.
Therefore, there is inevitably a latency between the movements of the real hand and the return movements of the virtual object, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the virtual object touched \cite{prachyabrued2014visual}.
These three rendering limitations make it \textbf{difficult to perceive the position of the fingers relative to the object} before touching or grasping it, but also to estimate the force required to grasp the virtual object and move it to a desired location.
Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a \VO with the hand.
Yet, it is unclear which type of visual and haptic feedback, or their combination, is best suited to guide the manipulation of a \VO. %, and whether one or the other of a combination of the two is most beneficial for users.
Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a virtual object with the hand.
Yet, it is unclear which type of visual and haptic feedback, or their combination, is best suited to guide the manipulation of a virtual object. %, and whether one or the other of a combination of the two is most beneficial for users.
\section{Approach and Contributions}
\label{contributions}
@@ -210,7 +210,7 @@ Our contributions in these two axes are summarized in \figref{contributions}.
\fig[0.95]{contributions}{Summary of our contributions through the simplified interaction loop.}[
The contributions are represented in dark grey boxes, the research axes in dark green circles and the research objectives in light green circles.
The first axis is \textbf{(I)} the design and evaluation of the perception of visuo-haptic texture augmentations of real surfaces, directly touched by the hand.
The second axis focuses on \textbf{(II)} improving the manipulation of \VOs with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.
The second axis focuses on \textbf{(II)} improving the manipulation of virtual objects with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.
]
\subsectionstarbookmark{Axis I: Augmenting the Texture Perception of Real Surfaces}
@@ -240,23 +240,23 @@ Our third objective is to \textbf{evaluate the perception of simultaneous and co
\subsectionstarbookmark{Axis II: Improving the Manipulation of Virtual Objects}
In immersive and wearable visuo-haptic \AR, the hand is free to touch and interact seamlessly with real, augmented and virtual objects.
Hence, a user can expect natural and direct contact and manipulation of \VOs with the bare hand.
However, the intangibility of the visual \VE, the display limitations of current visual \OST-\AR systems and the inherent spatial and temporal discrepancies between the user's hand actions and the visual feedback in the \VE can make interaction with \VOs particularly challenging.
%However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the \RE, the visual feedback, and the haptic feedback, can make the interaction with \VOs with bare hands particularly challenging.
Two particular sensory feedbacks are known to improve such direct \VO manipulation, but have not been properly investigated in immersive \AR: visual feedback of the virtual hand \cite{piumsomboon2014graspshell,prachyabrued2014visual} and delocalized haptic feedback \cite{lopes2018adding,teng2021touch}.
For this second axis of research, we propose to design and evaluate \textbf{visuo-haptic augmentations of the hand as interaction feedback with \VOs} in immersive \OST-\AR.
We consider the effect on user performance and experience of (1) the visual feedback of the virtual hand as augmentation of the real hand and (2) different delocalized haptic feedback of \VO manipulation with the hand in combination with visual hand augmentations.
Hence, a user can expect natural and direct contact and manipulation of virtual objects with the bare hand.
However, the intangibility of the visual \VE, the display limitations of current visual \OST-\AR systems and the inherent spatial and temporal discrepancies between the user's hand actions and the visual feedback in the \VE can make interaction with virtual objects particularly challenging.
%However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the \RE, the visual feedback, and the haptic feedback, can make the interaction with virtual objects with bare hands particularly challenging.
Two particular sensory feedbacks are known to improve such direct virtual object manipulation, but have not been properly investigated in immersive \AR: visual feedback of the virtual hand \cite{piumsomboon2014graspshell,prachyabrued2014visual} and delocalized haptic feedback \cite{lopes2018adding,teng2021touch}.
For this second axis of research, we propose to design and evaluate \textbf{visuo-haptic augmentations of the hand as interaction feedback with virtual objects} in immersive \OST-\AR.
We consider the effect on user performance and experience of (1) the visual feedback of the virtual hand as augmentation of the real hand and (2) different delocalized haptic feedback of virtual object manipulation with the hand in combination with visual hand augmentations.
First, the visual feedback of the virtual hand is a key element for interacting and manipulating \VOs in \VR \cite{prachyabrued2014visual,grubert2018effects}.
Some work has also investigated the visual feedback of the virtual hand in \AR \cite{piumsomboon2014graspshell,blaga2017usability}, but not in an immersive context of \VO manipulation. % with the bare hand.% from simulating mutual occlusions between the hand and \VOs \cite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay \cite{blaga2017usability,yoon2020evaluating}, augmenting the real hand.
First, the visual feedback of the virtual hand is a key element for interacting and manipulating virtual objects in \VR \cite{prachyabrued2014visual,grubert2018effects}.
Some work has also investigated the visual feedback of the virtual hand in \AR \cite{piumsomboon2014graspshell,blaga2017usability}, but not in an immersive context of virtual object manipulation. % with the bare hand.% from simulating mutual occlusions between the hand and virtual objects \cite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay \cite{blaga2017usability,yoon2020evaluating}, augmenting the real hand.
\OST-\AR also has significant perceptual differences from \VR due to the visibility of the real hand and environment, which can affect user experience and performance \cite{yoon2020evaluating}.
%, and these visual hand augmentations have not been evaluated .
Thus, our fourth objective is to \textbf{the visual feedback of the virtual hand as augmentation of the real hand} for direct hand manipulation of \VOs.
Thus, our fourth objective is to \textbf{the visual feedback of the virtual hand as augmentation of the real hand} for direct hand manipulation of virtual objects.
Second, as described above, the haptic actuators need to be moved away from the fingertips to not impair the hand movements, sensations and interactions with the \RE.
Previous work has shown that wearable haptics that provide feedback on hand manipulation with \VOs in \AR can significantly improve user performance and experience \cite{maisto2017evaluation,meli2018combining}.
Previous work has shown that wearable haptics that provide feedback on hand manipulation with virtual objects in \AR can significantly improve user performance and experience \cite{maisto2017evaluation,meli2018combining}.
However, it is unclear which positioning of the actuator is most beneficial and how delocalized haptic feedback of the hand-object contacts compares or complements visual augmentation of the hand.
Our last objective is to \textbf{investigate the delocalized haptic feedback of \VO manipulation} with the hand, in \textbf{combination with visual augmentations of the hand}, using wearable vibrotactile haptics.
Our last objective is to \textbf{investigate the delocalized haptic feedback of virtual object manipulation} with the hand, in \textbf{combination with visual augmentations of the hand}, using wearable vibrotactile haptics.
\section{Thesis Overview}
\label{thesis_overview}
@@ -268,7 +268,7 @@ With this first current \textit{Introduction} chapter, we have presented the res
In \textbf{\chapref{related_work}}, we then review previous work on the perception and manipulation of virtual and augmented objects, directly with the hand, using either wearable haptics, \AR, or their combination.
First, we overview how the hand perceives and manipulates real objects.
Second, we present wearable haptics and haptic augmentations of the texture and hardness of real objects.
Third, we introduce \AR, and how \VOs can be manipulated directly with the hand.
Third, we introduce \AR, and how virtual objects can be manipulated directly with the hand.
Finally, we describe how visuo-haptic feedback has augmented direct hand interaction in \AR, particularly using wearable haptics.
We then address each of our two research axes in a dedicated part.
@@ -291,12 +291,12 @@ The virtual textures are paired visual and haptic captures of real surfaces \cit
Our objective is to assess the perceived realism, coherence and roughness of the combination of nine representative visuo-haptic texture pairs.
\noindentskip
In \textbf{\partref{manipulation}} we describe our contributions to the second axis of research: improving direct hand manipulation of \VOs using visuo-haptic augmentations of the hand as interaction feedback with \VOs in immersive \OST-\AR.
In \textbf{\partref{manipulation}} we describe our contributions to the second axis of research: improving direct hand manipulation of virtual objects using visuo-haptic augmentations of the hand as interaction feedback with virtual objects in immersive \OST-\AR.
In \textbf{\chapref{visual_hand}} we investigate in a user study six visual feedback as hand augmentations, as a set of the most popular hand augmentation in the \AR literature.
Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate their effect on user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a \VO directly with the hand.
Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate their effect on user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a virtual object directly with the hand.
In \textbf{\chapref{visuo_haptic_hand}} we evaluate in a user study delocalized haptic feedback of hand manipulation with \VO using two vibrotactile contact techniques provided at five different positionings on the hand. %, as haptic rendering of \VO manipulation with the hand.
In \textbf{\chapref{visuo_haptic_hand}} we evaluate in a user study delocalized haptic feedback of hand manipulation with virtual object using two vibrotactile contact techniques provided at five different positionings on the hand. %, as haptic rendering of virtual object manipulation with the hand.
They are compared with the two most representative visual hand augmentations from the previous chapter, resulting in twenty visuo-haptic hand feedbacks that are evaluated within the same experimental setup and design.
\noindentskip