erwan/phd-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Corrections from Claudio's comments

Browse Source
This commit is contained in:
Erwan Normand
2024-10-21 11:24:14 +02:00
parent 7022765ae3
commit bb25e3db38
5 changed files with 35 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
1-introduction/introduction.tex
View File

@@ -4,8 +4,8 @@
\chaptertoc
%This PhD manuscript shows how wearable haptics, worn on the outside of the hand, can improve direct hand interaction in immersive \AR by augmenting the perception of the virtual content and its manipulation.
This thesis shows how immersive \AR, which integrates visual virtual content into the real world perception, and wearable haptics, worn on the outside of the hand, can improve the free and direct interaction of the hand with virtual and augmented objects.
Our goal is to enable users to perceive and interact with wearable visuo-haptic augmentations in a more realistically and effectively, as if they were real. %interaction of the hand with the virtual content.%, moving towards a seamless integration of the virtual into the real world.
This thesis shows how immersive \AR, which integrates visual virtual content into the real world perception, and wearable haptics, which provides tactile sensations on the skin, can improve the free and direct interaction of virtual objects with the hand.
Our goal is to enable users to perceive and interact with wearable visuo-haptic augmentations in a more realistic and effective way, as if they were real. %interaction of the hand with the virtual content.%, moving towards a seamless integration of the virtual into the real world.
%We are particularly interested in enabling direct contact of virtual and augmented objects with the bare hand.
%The aim of this thesis is to understand how immersive visual and wearable haptic augmentations complement each other in the context of direct hand perception and manipulation with virtual and augmented objects.
@@ -24,7 +24,7 @@ We then \textbf{instinctively construct a unified perception of the properties o
The sense of touch also allows us to perceive our environment, but also to represent ourselves in space and interact with the surrounding objects.
This is due to the many sensory receptors distributed throughout our hands and body.
These receptors can be divided into two modalities: \emph{kinesthetic} (or proprioception), which are the forces felt by muscles and tendons, and \emph{cutaneous} (or tactile), which are the pressures, stretches, vibrations and temperatures felt by the skin.
This rich and complex variety of actions and sensations makes it particularly \textbf{difficult to artificially recreate capabilities of touch}, for example in virtual or remote operating environments \cite{culbertson2018haptics}.
This rich and complex variety of actions and sensations makes it particularly \textbf{difficult to artificially recreate capabilities of touch}, for example in virtual or remote operating environments \cite{culbertson2018haptics,pacchierotti2023cutaneous}.
\subsectionstarbookmark{Wearable Haptics Promise Everyday Use}
@@ -175,9 +175,11 @@ With a better understanding of \textbf{how visual factors can influence the perc
\subsectionstarbookmark{Challenge II: Enabling Effective Manipulation of the Augmented Environment}
Touching, \textbf{grasping and manipulating virtual objects are fundamental interactions for \AR} \cite{kim2018revisiting}, \VR \cite{bergstrom2021how} and \VEs in general \cite[p.385]{laviolajr20173d}.
Since the hand is not occupied or covered with a haptic device so as to not impair interaction with the \RE, as described in the previous section, one can expect a seamless and direct manipulation of the hand with the virtual content as if it were real.
When augmenting a real object, the user's hand is physically constrained by the object, allowing for easy and natural interaction.
However, \textbf{manipulating a purely virtual object with the bare hand can be challenging} without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
In the previous challenge section, we described how the user's hand should be free to interact with the \RE using a wearable haptic device.
We can then expect a seamless and direct manipulation of the virtual content with the hands, as if it were real.
%Since the hand is not occupied or covered with a haptic device so as to not impair interaction with the \RE, as described in the previous section, one can expect a seamless and direct manipulation of the hand with the virtual content as if it were real.
When touching a visually augmenting a real object, the user's hand is physically constrained by the object, allowing for easy and natural interaction.
However, \textbf{manipulating a purely virtual object with the bare hand can be challenging}, especially without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
In addition, current \AR systems have visual rendering limitations that also affect interaction with virtual objects. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency.
\AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
@@ -194,11 +196,11 @@ Yet, it is unclear which type of visual and haptic feedback, or their combinatio
\label{contributions}
%The aim of this thesis is to understand how immersive visual and wearable haptic augmentations complement each other in the context of direct hand perception and manipulation with virtual and augmented objects.
As described in the research challenges section, providing a coherent and effective visuo-haptic augmented environment to a user is complex and raises many issues.
As we described in \secref{research_challenges}, providing a coherent and effective visuo-haptic augmented environment to a user is complex and raises many issues.
Our approach is to:
\begin{enumerate*}[label=(\arabic*)]
\item design immersive and wearable visuo-haptic renderings that augment both the objects being interacted with and the hand interacting with them, and
\item evaluate in user studies how these visuo-haptic renderings affect the interaction of the hand with these objects using psychophysical, performance and user experience methods.
\item evaluate in user studies how these visuo-haptic renderings affect the interaction of these objects with the hand using psychophysical, performance and user experience methods.
\end{enumerate*}
We consider two main axes of research, each addressing one of the research challenges identified above:
@@ -222,7 +224,7 @@ Wearable haptic devices have proven effective in modifying the perception of a t
However, wearable haptic augmentation with \AR has been little explored, as well as the visuo-haptic augmentation of texture.
Texture is indeed one of the most fundamental perceived properties of a surface material \cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) augmentation \cite{unger2011roughness,culbertson2014modeling,asano2015vibrotactile,strohmeier2017generating,friesen2024perceived}.
%Coherently substitute the visuo-haptic texture of a surface directly touched by a finger is an important step towards a \AR capable of visually and haptically augmenting the \RE of a user in a plausible way.
For this first axis of research, we propose to design and evaluate the \textbf{perception of wearable virtual visuo-haptic textures augmenting real surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
For this first axis of research, we propose to \textbf{design and evaluate the perception of wearable virtual visuo-haptic textures augmenting real surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
To this end, we (1) design a system for rendering wearable visuo-haptic texture augmentations, to (2) evaluate how the perception of haptic texture augmentations is affected by visual feedback of the virtual hand and the environment, and (3) investigate the perception of co-localized visuo-haptic texture augmentations.
First, an effective approach to render haptic textures is to generate a vibrotactile signal that represents the finger-texture interaction \cite{culbertson2014modeling,asano2015vibrotactile}.
@@ -245,14 +247,14 @@ Hence, a user can expect natural and direct contact and manipulation of virtual
However, the intangibility of the visual \VE, the display limitations of current visual \OST-\AR systems and the inherent spatial and temporal discrepancies between the user's hand actions and the visual feedback in the \VE can make interaction with virtual objects particularly challenging.
%However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current \AR systems, as well as the spatial and temporal discrepancies between the \RE, the visual feedback, and the haptic feedback, can make the interaction with virtual objects with bare hands particularly challenging.
Two particular sensory feedbacks are known to improve such direct virtual object manipulation, but have not been properly investigated in immersive \AR: visual feedback of the virtual hand \cite{piumsomboon2014graspshell,prachyabrued2014visual} and delocalized haptic feedback \cite{lopes2018adding,teng2021touch}.
For this second axis of research, we propose to design and evaluate \textbf{visuo-haptic augmentations of the hand as interaction feedback with virtual objects} in immersive \OST-\AR.
For this second axis of research, we propose to \textbf{design and evaluate visuo-haptic augmentations of the hand as interaction feedback with virtual objects} in immersive \OST-\AR.
We consider the effect on user performance and experience of (1) the visual feedback of the virtual hand as augmentation of the real hand and (2) different delocalized haptic feedback of virtual object manipulation with the hand in combination with visual hand augmentations.
First, the visual feedback of the virtual hand is a key element for interacting and manipulating virtual objects in \VR \cite{prachyabrued2014visual,grubert2018effects}.
Some work has also investigated the visual feedback of the virtual hand in \AR \cite{piumsomboon2014graspshell,blaga2017usability}, but not in an immersive context of virtual object manipulation. % with the bare hand.% from simulating mutual occlusions between the hand and virtual objects \cite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay \cite{blaga2017usability,yoon2020evaluating}, augmenting the real hand.
\OST-\AR also has significant perceptual differences from \VR due to the visibility of the real hand and environment, which can affect user experience and performance \cite{yoon2020evaluating}.
%, and these visual hand augmentations have not been evaluated .
Thus, our fourth objective is to \textbf{the visual feedback of the virtual hand as augmentation of the real hand} for direct hand manipulation of virtual objects.
Thus, our fourth objective is to \textbf{investigate the visual feedback of the virtual hand as augmentation of the real hand} for direct hand manipulation of virtual objects.
Second, as described above, the haptic actuators need to be moved away from the fingertips to not impair the hand movements, sensations and interactions with the \RE.
Previous work has shown that wearable haptics that provide feedback on hand manipulation with virtual objects in \AR can significantly improve user performance and experience \cite{maisto2017evaluation,meli2018combining}.
@@ -264,7 +266,7 @@ Our last objective is to \textbf{investigate the delocalized haptic feedback of
%This thesis is divided in four parts.
%In \textbf{\partref{background}}, we first describe the context and background of our research, within which
With this first current \textit{Introduction} chapter, we have presented the research challenges, objectives, approach and contributions of this thesis.
With this current \textit{Introduction} chapter, we have presented the research challenges, objectives, approach and contributions of this thesis.
In \textbf{\chapref{related_work}}, we then review previous work on the perception and manipulation of virtual and augmented objects, directly with the hand, using either wearable haptics, \AR, or their combination.
First, we overview how the hand perceives and manipulates real objects.
@@ -278,27 +280,27 @@ We then address each of our two research axes in a dedicated part.
In \textbf{\partref{perception}} we present our contributions to the first axis of research: modifying the visuo-haptic texture perception of real surfaces.
We evaluate how the visual feedback of the hand (real or virtual), the environment (\AR or \VR) and the textures (coherent, different or not shown) affect the perception of virtual vibrotactile textures rendered on real surfaces and touched directly with the index finger.
In \textbf{\chapref{vhar_system}} we design and implement a system for rendering visuo-haptic virtual textures that augment real surfaces. %, using an immersive \OST-\AR headset and a wearable vibrotactile device.
In \textbf{\chapref{vhar_system}}, we design and implement a system for rendering visuo-haptic virtual textures that augment real surfaces. %, using an immersive \OST-\AR headset and a wearable vibrotactile device.
The haptic textures represent a periodical patterned texture rendered by a wearable vibrotactile actuator worn on the middle phalanx of the finger touching the surface.
The tracking of the real hand and the environment is achieved using a marker-based technique.
The visual rendering is done using the immersive \OST-\AR headset Microsoft HoloLens~2.
The system allows free visual and haptic exploration of the textures, as if they were real, and forms the basis of the next two chapters.
In \textbf{\chapref{xr_perception}} we investigate in a user study how different the perception of haptic texture augmentations is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
In \textbf{\chapref{xr_perception}}, we investigate in a user study how different the perception of haptic texture augmentations is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
We use psychophysical methods to measure user perception and extensive questionnaires to understand how this perception is affected by the visual feedback of the virtual hand and the environment (real, augmented or virtual).
In \textbf{\chapref{vhar_textures}} we evaluate in a user study the perception of visuo-haptic texture augmentations directly touched with the real hand in \AR.
In \textbf{\chapref{vhar_textures}}, we evaluate in a user study the perception of visuo-haptic texture augmentations directly touched with the real hand in \AR.
The virtual textures are paired visual and haptic captures of real surfaces \cite{culbertson2014one}, which we render as visual and haptic overlays on the touched augmented surfaces.
Our objective is to assess the perceived realism, coherence and roughness of the combination of nine representative visuo-haptic texture pairs.
\noindentskip
In \textbf{\partref{manipulation}} we describe our contributions to the second axis of research: improving direct hand manipulation of virtual objects using visuo-haptic augmentations of the hand as interaction feedback with virtual objects in immersive \OST-\AR.
In \textbf{\partref{manipulation}}, we describe our contributions to the second axis of research: improving direct hand manipulation of virtual objects using visuo-haptic augmentations of the hand as interaction feedback with virtual objects in immersive \OST-\AR.
In \textbf{\chapref{visual_hand}} we investigate in a user study six visual feedback as hand augmentations, as a set of the most popular hand augmentation in the \AR literature.
In \textbf{\chapref{visual_hand}}, we investigate in a user study six visual feedback as hand augmentations, as a set of the most popular hand augmentation in the \AR literature.
Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate their effect on user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a virtual object directly with the hand.
In \textbf{\chapref{visuo_haptic_hand}} we evaluate in a user study delocalized haptic feedback of hand manipulation with virtual object using two vibrotactile contact techniques provided at five different positionings on the hand. %, as haptic rendering of virtual object manipulation with the hand.
In \textbf{\chapref{visuo_haptic_hand}}, we evaluate in a user study delocalized haptic feedback of hand manipulation with virtual object using two vibrotactile contact techniques provided at five different positionings on the hand. %, as haptic rendering of virtual object manipulation with the hand.
They are compared with the two most representative visual hand augmentations from the previous chapter, resulting in twenty visuo-haptic hand feedbacks that are evaluated within the same experimental setup and design.
\noindentskip
In \textbf{\chapref{conclusion}} we conclude this thesis and discuss short-term future work and long-term perspectives for each of our contributions and research axes.
In \textbf{\chapref{conclusion}}, we conclude this thesis and discuss short-term future work and long-term perspectives for each of our contributions and research axes.