erwan/phd-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Embodiment

Browse Source
This commit is contained in:
Erwan Normand
2024-09-18 10:56:50 +02:00
parent 329e506229
commit e57959007d
4 changed files with 68 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
1-introduction/introduction/introduction.tex
View File

@@ -92,7 +92,7 @@ It is technically and conceptually closely related to \VR, which replaces the \R
%
It describes the degree of \RV of the environment along an axis, with one end being the \RE and the other end being a pure \VE, \ie indistinguishable from the real world (such as \emph{The Matrix} movies).
%
Between these two extremes lies \MR, which comprises \AR and \VR as different levels of mixing real and virtual environments~\cite{skarbez2021revisiting}.
Between these two extremes lies \MR, which comprises \AR and \VR as different levels of mixing real and virtual environments~\cite{skarbez2021revisiting}.\footnote{This is the original and classic definition of \MR, but there is still a debate on defining and characterize \AR and \MR experiences~\cite{speicher2019what,skarbez2021revisiting}.}
%
\AR/\VR is most often understood as addressing only the visual sense, and as haptics, it can take many forms as a user interface.
%

4
1-introduction/related-work/2-wearable-haptics.tex
View File

@@ -1,4 +1,4 @@
\section{Augmenting Objects with Wearable Haptics}
\section{Augmenting Object Perception with Wearable Haptics}
\label{wearable_haptics}
One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in \v-\VE~\cite{maclean2008it,culbertson2018haptics}.
@@ -160,7 +160,7 @@ Several types of vibrotactile actuators are used in haptics, with different trad
\end{subfigs}
\subsection{Tactile Renderings for Modifying Object Properties}
\subsection{Modifying Object Properties with Tactile Renderings}
\label{tactile_rendering}
Tactile rendering of haptic properties consists in modelling and reproducing virtual tactile sensations comparable to those perceived when interacting with real objects.

76
1-introduction/related-work/3-augmented-reality.tex
View File

@@ -1,4 +1,4 @@
\section{Principles and Capabilities of AR}
\section{Manipulating Object with the Hands in AR}
\label{augmented_reality}
The first \AR headset was invented by \textcite{sutherland1968headmounted}: With the technology available at the time, it was already capable of displaying \VOs at a fixed point in space in real time, giving the user the illusion that the content was present in the room.
@@ -65,6 +65,9 @@ Yet, the user experience in \AR is still highly dependent on the display used.
\subsubsection{AR Displays and Perception}
\label{ar_displays}
\AR systems render virtual content registered in the \RE to the user's senses via output \UIs.
To create the perception of a combined rea
\cite{bimber2005spatial}
\paragraph{Spatial Augmented Reality}
@@ -87,25 +90,28 @@ Using a VST-AR headset have notable consequences, as the "real" view of the envi
\subsubsection{Presence and Embodiment in AR}
\label{ar_presence_embodiment}
Despite the clear and acknowledged definition presented in \secref{ar_definition} and the viewpoint of this thesis that \AR and \VR are two type of \MR experience with different levels of mixing real and virtual environments, as presented in \secref[introduction]{visuo_haptic_augmentations}, there is still a debate on defining \AR and \MR as well as how to characterize and categorized such experiences~\cite{speicher2019what,skarbez2021revisiting}.
%Despite the clear and acknowledged definition presented in \secref{ar_definition} and the viewpoint of this thesis that \AR and \VR are two type of \MR experience with different levels of mixing real and virtual environments, as presented in \secref[introduction]{visuo_haptic_augmentations}, there is still a debate on defining \AR and \MR as well as how to characterize and categorized such experiences~\cite{speicher2019what,skarbez2021revisiting}.
Presence and embodiment are two key concepts that characterize the user experience in \AR and \VR.
While there is a large literature on these topics in \VR, they are less defined and studied for \AR~\cite{tran2024survey,genay2022being}.
Still, these concepts are useful to design, evaluate and discuss our contributions in the next chapters.
\paragraph{Presence}
\label{ar_presence}
Presence is one of the key concept to characterize a \VR experience.
\AR and \VR are both essentially illusions as the virtual content does not physically exist but is just digitally simulated and rendered to the user's perception through a user interface and the user's senses.
Such experience of disbelief suspension in \VR is what is called presence, and it can be decomposed into two dimensions: \PI and \PSI~\cite{slater2009place}.
\AR and \VR are both essentially illusions as the virtual content does not physically exist but is just digitally simulated and rendered to the user's senses through display \UIs.
Such experience of disbelief suspension in \VR is what is called \emph{presence}, and it can be decomposed into two dimensions: \PI and \PSI~\cite{slater2009place}.
\PI is the sense of the user of \enquote{being there} in the \VE (\figref{presence-vr}).
It emerges from the real time rendering of the \VE from the user's perspective: to be able to move around inside the \VE and look from different point of views.
\PSI is the illusion that the virtual events are really happening, even if the user knows that they are not real.
It doesn't mean that the virtual events are realistic, but that they are plausible and coherent with the user's expectations.
A third strong illusion in \VR is the \SoE, which is the illusion that the virtual body is one's own~\cite{slater2022separate,guy2023sense}.
The \AR presence is far less defined and studied than for \VR~\cite{tran2024survey}, but it will be useful to design, evaluate and discuss our contributions in the next chapters.
Thereby, \textcite{slater2022separate} proposed to invert \PI to what we can call \enquote{object illusion}, \ie the sense of the \VO to \enquote{feels here} in the \RE (\figref{presence-ar}).
%The \AR presence is far less defined and studied than for \VR~\cite{tran2024survey}
For \AR, \textcite{slater2022separate} proposed to invert \PI to what we can call \enquote{object illusion}, \ie the sense of the \VO to \enquote{feels here} in the \RE (\figref{presence-ar}).
As with VR, \VOs must be able to be seen from different angles by moving the head but also, this is more difficult, be consistent with the \RE, \eg occlude or be occluded by real objects~\cite{macedo2023occlusion}, cast shadows or reflect lights.
The \PSI can be applied to \AR as is, but the \VOs must additionally have knowledge of the \RE and react accordingly to it.
\textcite{skarbez2021revisiting} also named \PI for \AR as \enquote{immersion} and \PSI as \enquote{coherence}, and these terms will be used in the remainder of this thesis.
One main issue with presence is how to measure it both in \VR~\cite{slater2022separate} and \AR~\cite{tran2024survey}.
\begin{subfigs}{presence}{The sense of immersion in virtual and augmented environments. Adapted from \textcite{stevens2002putting}. }[
\item Place Illusion (PI) is the sense of the user of \enquote{being there} in the \VE.
@@ -119,25 +125,28 @@ The \PSI can be applied to \AR as is, but the \VOs must additionally have knowle
\paragraph{Embodiment}
\label{ar_embodiment}
As presence, \SoE in \AR is a recent topic and little is known about its perception on the user experience~\cite{genay2021virtual}.
The \SoE is the \enquote{subjective experience of using and having a body}~\cite{blanke2009fullbody}, \ie the feeling that a body is our own.
In everyday life, we are used to being, seeing and controlling our own body, but it is possible to embody a virtual body as an avatar while in \AR~\cite{genay2022being} or \VR~\cite{guy2023sense}.
This illusion arises when the visual, proprioceptive and (if any) haptic sensations of the virtual body are coherent~\cite{kilteni2012sense}.
It can be decomposed into three subcomponents: \emph{Agency}, which is the feeling of controlling the body; \emph{Ownership}, which is the feeling that \enquote{the body is the source of the experienced sensations}; and \emph{Self-Location}, which is the feeling \enquote{spatial experience of being inside [the] body}~\cite{kilteni2012sense}.
In \AR, it could take the form of body accessorization, \eg wearing virtual clothes or make-up in overlay, of partial avatarization, \eg using a virtual prothesis, or a full avatarization~\cite{genay2022being}.
\subsection{Direct Hand Manipulation in AR}
\label{ar_interaction}
Both \AR/\VR and haptic systems to render \VOs as visual or haptic sensations that are prenseted to the user's senses.
In turn, a user must be able to manipulate the \VOs and environments to complete the interaction loop (\figref[introduction]{interaction-loop}), \eg through a hand-held controller, a tangible object, or even directly with the hands.
An \emph{interaction technique} is then required to map the user input to actions on the \VE~\cite{laviola20173d}.
A user in \AR must be able to interact with the virtual content to fulfil the second point of \textcite{azuma1997survey}'s definition (\secref{ar_definition}) and complete the interaction loop (\figref[introduction]{interaction-loop}).%, \eg through a hand-held controller, a tangible object, or even directly with the hands.
In all examples of \AR applications shown in \secref{ar_applications}, the user interacts with the \VE using their hands, either directly or through a physical input \UI.
\subsubsection{User Interfaces and Interaction Techniques}
\label{interaction_techniques}
For a user to interact with a computer system, they first perceive the state of the system and then acts upon it through an input \UI.
Inputs interfaces can be either an \emph{active sensing}, physically held or worn device, such as a mouse, a touch screen, or a hand-held controller, or a \emph{passive sensing}, that does not require physical contact, such as eye trackers, voice recognition, or hand tracking.
The information gathered from the sensors by the \UI is then translated into actions within the computer system by an interaction technique.
For a user to interact with a computer system (desktop, mobile, \AR, etc.), they first perceive the state of the system and then acts upon it through an input \UI.
Inputs \UI can be either an \emph{active sensing}, physically held or worn device, such as a mouse, a touch screen, or a hand-held controller, or a \emph{passive sensing}, that does not require physical contact, such as eye trackers, voice recognition, or hand tracking~\cite{laviola20173d}.
The information gathered from the sensors by the \UI is then translated into actions within the computer system by an \emph{interaction technique} (\figref{interaction-technique}).
For example, a cursor on a screen can be moved using either with a mouse or with the arrow keys on a keyboard, or a two-finger swipe on a touchscreen can be used to scroll or zoom an image.
Choosing useful and efficient \UIs and interaction techniques is crucial for the user experience and the tasks that can be performed within the system~\cite{laviola20173d}.
Choosing useful and efficient \UIs and interaction techniques is crucial for the user experience and the tasks that can be performed within the system.
\fig[0.5]{interaction-technique}{An interaction technique map user inputs to actions within a computer system. Adapted from \textcite{billinghurst2005designing}.}
@@ -263,35 +272,36 @@ While a visual rendering of the virtual hand in \VR can compensate for these iss
\subsection{Visual Rendering of Hands in AR}
\label{ar_visual_hands}
In \VR, as the user is fully immersed in the \VE and cannot see their real hands, it is necessary to represent their virtually.
In \VR, as the user is fully immersed in the \VE and cannot see their real hands, it is necessary to represent their virtually (\secref{ar_embodiment}).
When interacting using a physics-based virtual hand method (\secref{ar_virtual_hands}), the visual rendering of the virtual hand have an influence on perception, interaction performance, and preference of users~\cite{prachyabrued2014visual,argelaguet2016role,grubert2018effects,schwind2018touch}.
In a pick-and-place manipulation task in \VR, \textcite{prachyabrued2014visual} and \textcite{canales2019virtual} found that the visual hand rendering whose motion was constrained to the surface of the \VOs similar as to \textcite{borst2006spring} (\enquote{Outer Hand} in \figref{prachyabrued2014visual}) performed the worst, while the visual hand rendering following the tracked human hand (thus penetrating the \VOs, \enquote{Inner Hand} in \figref{prachyabrued2014visual}), performed the best, even though it was rather disliked.
\textcite{prachyabrued2014visual} also observed that the best compromise was a double rendering, showing both the virtual hand and the tracked hand (\enquote{2-Hand} in \figref{prachyabrued2014visual}).
While a realistic human hand rendering increase the sense of ownership~\cite{lin2016need}, a skeleton-like rendering provide a stronger sense of control~\cite{argelaguet2016role}, and a minimalistic fingertip rendering reduce errors in typing text~\cite{grubert2018effects}.
While a realistic human hand rendering increase the sense of ownership~\cite{lin2016need}, a skeleton-like rendering provide a stronger sense of agency~\cite{argelaguet2016role} (\secref{ar_embodiment}), and a minimalistic fingertip rendering reduce errors in typing text~\cite{grubert2018effects}.
A visual hand rendering while in \VE also seems to affect how one grasps an object~\cite{blaga2020too}, or how real bumps and holes are perceived~\cite{schwind2018touch}.
\fig{prachyabrued2014visual}{Visual hand renderings affect user experience in \VR~\cite{prachyabrued2014visual}.}
As presented in \secref{ar_displays}, a user sees their hands in \AR, and the mutual occlusion between the hands and the \VOs is a common issue, \ie hiding the \VO when the real hand is in front of it and hiding the real hand when it is behind the \VO.
For example, in \figref{hilliges2012holodesk_2}, the user is pinching a virtual cube in \OST-\AR with their thumb and index fingers, but while the index is behind the cube, it is seen as in front of it.
Conversely, a user sees their own hands in \AR, and the mutual occlusion between the hands and the \VOs is a common issue (\secref{ar_displays}), \ie hiding the \VO when the real hand is in front of it and hiding the real hand when it is behind the \VO (\figref{hilliges2012holodesk_2}).
%For example, in \figref{hilliges2012holodesk_2}, the user is pinching a virtual cube in \OST-\AR with their thumb and index fingers, but while the index is behind the cube, it is seen as in front of it.
While in \VST-\AR, this could be solved as a masking problem by combining the real and virtual images~\cite{battisti2018seamless}, \eg in \figref{suzuki2014grasping}, in \OST-\AR, this is much more difficult because the \VE is displayed as a transparent \TwoD image on top of the \ThreeD \RE, which cannot be easily masked~\cite{macedo2023occlusion}.
%Yet, even in \VST-\AR,
As the \VE is intangible and the hand of the user visible while in \AR, adding a visual rendering of the virtual hand that is physically constrained to the \VOs would achieve a similar result to the promising double-hand rendering of \textcite{prachyabrued2014visual}.
Additionally, \textcite{kahl2021investigation} showed that a \VO overlaying a tangible object in \OST-\AR can vary in size without worsening the users' experience nor the performance when manipulating it.
This suggests that a visual hand rendering superimposed on the real hand could be helpful, but should not impair users.
%An alternative is to render the \VOs and the virtual hand semi-transparents, so that they are partially visible even when one is occluding the other (\figref{buchmann2005interaction}).
%Although perceived as less natural, this seems to be preferred to a mutual visual occlusion in \VST-\AR~\cite{buchmann2005interaction,ha2014wearhand,piumsomboon2014graspshell} and \VR~\cite{vanveldhuizen2021effect}, but has not yet been evaluated in \OST-\AR.
%However, this effect still causes depth conflicts that make it difficult to determine if one's hand is behind or in front of a \VO, \eg the thumb is in front of the virtual cube, but could be perceived to be behind it.
An alternative is to render the \VOs and the virtual hand semi-transparents, so that they are partially visible even when one is occluding the other (\figref{buchmann2005interaction}).
Although perceived as less natural, this seems to be preferred to a mutual visual occlusion in \VST-\AR~\cite{buchmann2005interaction,ha2014wearhand,piumsomboon2014graspshell} and \VR~\cite{vanveldhuizen2021effect}, but has not yet been evaluated in \OST-\AR.
However, this effect still causes depth conflicts that make it difficult to determine if one's hand is behind or in front of a \VO, \eg the thumb is in front of the virtual cube, but could be perceived to be behind it.
As the \VE is intangible, adding a visual rendering of the virtual hand in \AR that is physically constrained to the \VOs would achieve a similar result to the promising double-hand rendering of \textcite{prachyabrued2014visual}.
A \VO overlaying a tangible object in \OST-\AR can vary in size and shape without worsening the users' experience nor the performance when manipulating it~\cite{kahl2021investigation,kahl2023using}.
This suggests that a visual hand rendering superimposed on the real hand as a partial avatarization (\secref{ar_embodiment}) might be helpful without impairing the user.
Few works have compared different visual hand rendering in \AR, nor with wearable haptic feedback.
\textcite{blaga2017usability} evaluated direct hand manipulation in non-immersive \VST-\AR a skeleton-like rendering against no visual hand rendering: while user performance did not improve, participants felt more confident with the virtual hand (\figref{blaga2017usability}).
%\textcite{krichenbauer2018augmented} found participants \percent{22} faster in immersive \VST-\AR than in \VR in the same pick-and-place manipulation task.
%No visual hand rendering was used in \VR while the real hand was visible in \AR.
Few works have compared different visual hand rendering in \AR or with wearable haptic feedback.
Rendering the real hand as a semi-transparent hand in \VST-\AR is perceived as less natural but seems to be preferred to a mutual visual occlusion for interaction with real and virtual objects~\cite{buchmann2005interaction,piumsomboon2014graspshell}.
%Although perceived as less natural, this seems to be preferred to a mutual visual occlusion in \VST-\AR~\cite{buchmann2005interaction,ha2014wearhand,piumsomboon2014graspshell} and \VR~\cite{vanveldhuizen2021effect}, but has not yet been evaluated in \OST-\AR.
Similarly, \textcite{blaga2017usability} evaluated direct hand manipulation in non-immersive \VST-\AR a skeleton-like rendering against no visual hand rendering: while user performance did not improve, participants felt more confident with the virtual hand (\figref{blaga2017usability}).
\textcite{krichenbauer2018augmented} found participants \percent{22} faster in immersive \VST-\AR than in \VR in the same pick-and-place manipulation task, but no visual hand rendering was used in \VR while the real hand was visible in \AR.
In a collaboration task in immersive \OST-\AR \vs \VR, \textcite{yoon2020evaluating} showed that a realistic human hand rendering was the most preferred over a low-polygon hand and a skeleton-like hand for the remote partner.
\textcite{genay2021virtual} found that the \SoE was stronger with robotic hands overlay in \OST-\AR when the environment contains \VOs (\figref{genay2021virtual}).
Finally, \textcite{maisto2017evaluation} and \textcite{meli2018combining} compared visual and haptic rendering of the hand in \AR, as detailed in the next section (\secref{vhar_rings}).
\textcite{genay2021virtual} found that the \SoE was stronger with robotic hands overlay in \OST-\AR when the environment contains both real and virtual objects (\figref{genay2021virtual}).
Finally, \textcite{maisto2017evaluation} and \textcite{meli2018combining} compared visual and haptic rendering of the hand in \VST-\AR, as detailed in the next section (\secref{vhar_rings}).
Taken together, these results suggest that a visual hand rendering in \AR could improve the user experience and performance in direct hand manipulation tasks, but the best rendering is still to be determined.
%\cite{chan2010touching} : cues for touching (selection) \VOs.
%\textcite{saito2021contact} found that masking the real hand with a textured 3D opaque virtual hand did not improve performance in a reach-to-grasp task but displaying the points of contact on the \VO did.
@@ -304,7 +314,7 @@ Taken together, these results suggest that a visual hand rendering in \AR could
\item Skeleton rendering overlaying the real hand in \VST-\AR~\cite{blaga2017usability}.
\item Robotic rendering overlaying the real hands in \OST-\AR~\cite{genay2021virtual}.
]
\subfigsheight{29mm}
\subfigsheight{29.5mm}
\subfig{hilliges2012holodesk_2}
\subfig{suzuki2014grasping}
\subfig{buchmann2005interaction}

22
references.bib
View File

@@ -479,6 +479,17 @@
doi = {10/gs4p4x}
}
@article{blanke2009fullbody,
title = {Full-Body Illusions and Minimal Phenomenal Selfhood},
author = {Blanke, Olaf and Metzinger, Thomas},
date = {2009},
journaltitle = {Trends Cogn. Sci.},
volume = {13},
number = {1},
pages = {7--13},
doi = {10/cbqtpc}
}
@article{blausen2014medical,
title = {Medical Gallery of {{Blausen Medical}}},
author = {Blausen, Bruce},
@@ -1751,6 +1762,17 @@
doi = {10/bzjh8b}
}
@article{kilteni2012sense,
title = {The {{Sense}} of {{Embodiment}} in {{Virtual Reality}}},
author = {Kilteni, Konstantina and Groten, Raphaela and Slater, Mel},
date = {2012},
journaltitle = {Presence Teleoperators Virtual Environ.},
volume = {21},
number = {4},
pages = {373--387},
doi = {10/ggzn8w}
}
@inproceedings{kim2015physicsbased,
title = {Physics-Based Hand Interaction with Virtual Objects},
booktitle = {{{IEEE Int}}. {{Conf}}. {{Robot}}. {{Autom}}.},