|
|
|
@@ -93,24 +93,24 @@ Many visual displays have been explored, from projection systems to hand-held di
|
|
|
|
\end{subfigs}
|
|
|
|
\end{subfigs}
|
|
|
|
|
|
|
|
|
|
|
|
%Concepts of virtuality and augmentation can also be applied for sensory modalities other than vision.
|
|
|
|
%Concepts of virtuality and augmentation can also be applied for sensory modalities other than vision.
|
|
|
|
\textcite{jeon2009haptic} proposed to describe visuo-haptic \AR/\VR systems with two orthogonal reality-virtuality continuum, one for vision and one for touch, as illustrated in \figref{visuo-haptic-rv-continuum5}.
|
|
|
|
\textcite{jeon2009haptic} proposed to describe visuo-haptic \AR/\VR systems with two orthogonal reality-virtuality continuums, one for vision and one for touch, as shown in \figref{visuo-haptic-rv-continuum5}.
|
|
|
|
The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of virtuality for each visual or haptic feedback: real, augmented and virtual.
|
|
|
|
The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of virtuality for each visual or haptic feedback: real, augmented and virtual.
|
|
|
|
For example, (visual) \AR that uses a real object as a proxy to manipulate a \VO is considered as \emph{haptic reality} (\eg \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum5}), whereas a device that provides synthetic haptic feedback when touching a \VO is considered as \emph{haptic virtuality} (\eg \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
For example, (visual) \AR using a real object as a proxy to manipulate a \VO is considered to be \emph{haptic reality} (\eg \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum5}), whereas a device that provides synthetic haptic feedback when touching a \VO is considered to be \emph{haptic virtuality} (\eg \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
\textbf{A \emph{haptic augmentation} is then the combination of real and virtual haptic stimuli} \cite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
\textbf{A \emph{haptic augmentation} is then the combination of real and virtual haptic stimuli} \cite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
In particular, it has been implemented by augmenting the haptic perception of real objects by providing timely virtual tactile stimuli using wearable haptics:
|
|
|
|
In particular, it has been implemented by augmenting the haptic perception of real objects by providing timely virtual tactile stimuli using wearable haptics:
|
|
|
|
\figref{salazar2020altering} shows an example of modifying the perceived stiffness of a real object in \VR using simultaneous pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
\figref{salazar2020altering} shows an example of modifying the perceived stiffness of a real object in \VR using simultaneous pressure feedback on the finger (left middle cell in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
\figref{bau2012revel} shows another example of visuo-haptic augmentation of virtual texture when running the finger on a real surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
\figref{bau2012revel} shows another example of visuo-haptic augmentation of virtual texture when running the finger on a real surface (middle cell in the two axes in \figref{visuo-haptic-rv-continuum5}).
|
|
|
|
|
|
|
|
|
|
|
|
When a (visual) \AR system lacks haptic feedback, it creates a deceptive and incomplete user experience when reaching the \VE with the hand.
|
|
|
|
If a (visual) \AR system lacks haptic feedback, it creates a deceptive and incomplete user experience when the hand reaches the virtual content.
|
|
|
|
All visual \VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties and interact with them with confidence and efficiency.
|
|
|
|
All visual \VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties and interact with them with confidence and efficiency.
|
|
|
|
It is therefore necessary to provide haptic feedback that is consistent with the visual \VOs and ensures the best possible user experience, as we argue in the next section.
|
|
|
|
It is therefore necessary to provide haptic feedback that is consistent with the visual \VOs and ensures the best possible user experience, as we argue in the next section.
|
|
|
|
The \textbf{integration of wearable haptics with \AR} seems to be one of the most promising solutions, but it \textbf{remains challenging due to their many respective characteristics and the additional constraints of combining them}.
|
|
|
|
The \textbf{integration of wearable haptics with \AR} seems to be one of the most promising solutions, but it \textbf{remains challenging due to their many respective characteristics and the additional constraints of combining them}.
|
|
|
|
|
|
|
|
|
|
|
|
\begin{subfigs}{visuo-haptic-environments}{Visuo-haptic environments with different degrees of reality-virtuality. }[][
|
|
|
|
\begin{subfigs}{visuo-haptic-environments}{Visuo-haptic environments with varying degrees of reality-virtuality. }[][
|
|
|
|
\item \AR environment with a real haptic object used as a proxy to manipulate a \VO \cite{kahl2023using}.
|
|
|
|
\item \AR environment with a real haptic object used as a proxy to manipulate a \VO \cite{kahl2023using}.
|
|
|
|
\item \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a \VO \cite{meli2018combining}.
|
|
|
|
\item \AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a \VO \cite{meli2018combining}.
|
|
|
|
\item A real object seen in a visual \VR environment whose haptic perception of stiffness is augmented with the hRing haptic device \cite{detinguy2018enhancing,salazar2020altering}.
|
|
|
|
\item A real object seen in a visual \VR environment whose haptic perception of stiffness is augmented with the hRing haptic device \cite{detinguy2018enhancing,salazar2020altering}.
|
|
|
|
\item Visuo-haptic texture augmentation of real object being touch, using a hand-held \AR display and haptic electrovibration feedback \cite{bau2012revel}.
|
|
|
|
\item Visuo-haptic texture augmentation of a real object to be touched using a handheld \AR display and haptic electrovibration feedback \cite{bau2012revel}.
|
|
|
|
]
|
|
|
|
]
|
|
|
|
\subfigsheight{31mm}
|
|
|
|
\subfigsheight{31mm}
|
|
|
|
\subfig{kahl2023using}
|
|
|
|
\subfig{kahl2023using}
|
|
|
|
@@ -124,7 +124,7 @@ The \textbf{integration of wearable haptics with \AR} seems to be one of the mos
|
|
|
|
|
|
|
|
|
|
|
|
The integration of wearable haptics with \AR to create a visuo-haptic augmented environment is complex and presents many perceptual and interaction challenges.
|
|
|
|
The integration of wearable haptics with \AR to create a visuo-haptic augmented environment is complex and presents many perceptual and interaction challenges.
|
|
|
|
% \ie sensing the \AE and acting effectively upon it.
|
|
|
|
% \ie sensing the \AE and acting effectively upon it.
|
|
|
|
In this thesis, we propose to \textbf{represent the experience of a user with such a visuo-haptic augmented environment as an interaction loop}, shown in \figref{interaction-loop}.
|
|
|
|
In this thesis, we propose to \textbf{represent the user's experience with such a visuo-haptic augmented environment as an interaction loop}, shown in \figref{interaction-loop}.
|
|
|
|
It is based on the interaction loops of users with \ThreeD systems \cite[p.84]{laviolajr20173d}.
|
|
|
|
It is based on the interaction loops of users with \ThreeD systems \cite[p.84]{laviolajr20173d}.
|
|
|
|
The \RE and the user's hand are tracked in real time by sensors and reconstructed in visual and haptic \VEs.
|
|
|
|
The \RE and the user's hand are tracked in real time by sensors and reconstructed in visual and haptic \VEs.
|
|
|
|
The interactions between the virtual hand and objects are then simulated, and rendered as feedback to the user using a \AR/\VR headset and wearable haptics.
|
|
|
|
The interactions between the virtual hand and objects are then simulated, and rendered as feedback to the user using a \AR/\VR headset and wearable haptics.
|
|
|
|
@@ -132,7 +132,7 @@ Because the visuo-haptic \VE is displayed in real time and aligned with the \RE,
|
|
|
|
|
|
|
|
|
|
|
|
\fig{interaction-loop}{The interaction loop between a user and a visuo-haptic augmented environment as proposed in this thesis.}[
|
|
|
|
\fig{interaction-loop}{The interaction loop between a user and a visuo-haptic augmented environment as proposed in this thesis.}[
|
|
|
|
A user interacts with the visual (in blue) and haptic (in red) \VEs through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with \VOs.
|
|
|
|
A user interacts with the visual (in blue) and haptic (in red) \VEs through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with \VOs.
|
|
|
|
The visual and haptic \VEs are rendered back to the user registered and co-localized with the \RE (in gray) using a immersive \AR headset and wearable haptics.
|
|
|
|
The visual and haptic \VEs are rendered back using an immersive \AR headset and wearable haptics, and are felt by the user to be registered and co-localized with the \RE (in gray)
|
|
|
|
\protect\footnotemark
|
|
|
|
\protect\footnotemark
|
|
|
|
]
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
@@ -158,31 +158,31 @@ Although closely related, \AR and \VR have key differences in their respective r
|
|
|
|
|
|
|
|
|
|
|
|
Many wearable haptic devices take the form of controllers, gloves or exoskeletons, all of which cover the fingertips and are therefore not suitable for \AR.
|
|
|
|
Many wearable haptic devices take the form of controllers, gloves or exoskeletons, all of which cover the fingertips and are therefore not suitable for \AR.
|
|
|
|
The \textbf{user's hand must be indeed free to touch and interact with the \RE while wearing a wearable haptic device}.
|
|
|
|
The \textbf{user's hand must be indeed free to touch and interact with the \RE while wearing a wearable haptic device}.
|
|
|
|
It is possible instead to place the haptic actuator close to the point of contact with the \RE, \eg providing haptic feedback on the nail \cite{ando2007fingernailmounted,teng2021touch}, another phalanx \cite{asano2015vibrotactile,detinguy2018enhancing,salazar2020altering} or the wrist \cite{pezent2022design,sarac2022perceived} for rendering fingertip contacts with virtual content.
|
|
|
|
Instead, it is possible to place the haptic actuator close to the point of contact with the \RE, \eg providing haptic feedback on the nail \cite{ando2007fingernailmounted,teng2021touch}, another phalanx \cite{asano2015vibrotactile,detinguy2018enhancing,salazar2020altering} or the wrist \cite{pezent2022design,sarac2022perceived} for rendering fingertip contacts with virtual content.
|
|
|
|
Therefore, when touching a virtual or augmented object, \textbf{the real and virtual visual sensations are seen as co-localized, but the virtual haptic feedback is not}.
|
|
|
|
Therefore, when touching a virtual or augmented object, \textbf{the real and virtual visual sensations are perceived as co-localized, but the virtual haptic feedback is not}.
|
|
|
|
It remains to be investigated how such potential discrepancies affect the overall perception to design visuo-haptic augmentations adapted to \AR.
|
|
|
|
It remains to be investigated how such potential discrepancies affect the overall perception to design visuo-haptic augmentations adapted to \AR.
|
|
|
|
|
|
|
|
|
|
|
|
%So far, most of the \AR studies and applications only add visual and haptic sensations to the user's overall perception of the environment, but conversely it is more difficult to remove sensations.
|
|
|
|
%So far, most of the \AR studies and applications only add visual and haptic sensations to the user's overall perception of the environment, but conversely it is more difficult to remove sensations.
|
|
|
|
%Visual and haptic augmentations of the \RE add sensations to the user's overall perception.
|
|
|
|
%Visual and haptic augmentations of the \RE add sensations to the user's overall perception.
|
|
|
|
The \textbf{added visual and haptic virtual sensations may be perceived as inconsistent} with the sensations of the \RE, for example with a lower rendering quality, a temporal latency, a spatial shift, or a combination of these.
|
|
|
|
The \textbf{added visual and haptic virtual sensations may be perceived as inconsistent} with the sensations of the \RE, for example with a lower rendering quality, a temporal latency, a spatial shift, or a combination of these.
|
|
|
|
Moreover, in \AR, the user can still see the real world surroundings, including their hands, the augmented real objects and the worn haptic devices, unlike \VR where there is total control over the visual rendering. % of the hand and \VE.
|
|
|
|
Moreover, in \AR, the user can still see the real world environment, including their hands, the augmented real objects and the worn haptic devices, unlike \VR where there is total control over the visual rendering. % of the hand and \VE.
|
|
|
|
It is therefore unclear to what extent the real and virtual visuo-haptic sensations will be perceived as realistic or even plausible, and to what extent they will conflict or complement each other. % in the perception of the \AE.
|
|
|
|
It is therefore unclear to what extent the real and virtual visuo-haptic sensations will be perceived as realistic or even plausible, and to what extent they will conflict or complement each other. % in the perception of the \AE.
|
|
|
|
With a better understanding of \textbf{how visual factors can influence the perception of haptic augmentations}, the many wearable haptic systems that already exist but have not yet been fully explored with \AR can be better applied and new visuo-haptic augmentations adapted to \AR can be designed.
|
|
|
|
With a better understanding of \textbf{how visual factors can influence the perception of haptic augmentations}, the many wearable haptic systems that already exist but have not yet been fully explored with \AR can be better applied and new visuo-haptic augmentations adapted to \AR can be designed.
|
|
|
|
|
|
|
|
|
|
|
|
\subsectionstarbookmark{Challenge II: Enabling Effective Manipulation of the Augmented Environment}
|
|
|
|
\subsectionstarbookmark{Challenge II: Enabling Effective Manipulation of the Augmented Environment}
|
|
|
|
|
|
|
|
|
|
|
|
Touching, \textbf{grasping and manipulating \VOs are fundamental interactions for \AR} \cite{kim2018revisiting}, \VR \cite{bergstrom2021how} and \VEs in general \cite[p.385]{laviolajr20173d}.
|
|
|
|
Touching, \textbf{grasping and manipulating \VOs are fundamental interactions for \AR} \cite{kim2018revisiting}, \VR \cite{bergstrom2021how} and \VEs in general \cite[p.385]{laviolajr20173d}.
|
|
|
|
As the hand is not occupied or covered with a haptic device to not impair interaction with the \RE, as described in the previous section, one can expect a seamless and direct manipulation of the hand with the virtual content as if it were real.
|
|
|
|
Since the hand is not occupied or covered with a haptic device so as to not impair interaction with the \RE, as described in the previous section, one can expect a seamless and direct manipulation of the hand with the virtual content as if it were real.
|
|
|
|
When augmenting a real object, the user's hand is physically constrained by the object, allowing for easy and natural interaction.
|
|
|
|
When augmenting a real object, the user's hand is physically constrained by the object, allowing for easy and natural interaction.
|
|
|
|
However, \textbf{manipulating a purely \VO with the bare hand can be challenging} without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
|
|
|
|
However, \textbf{manipulating a purely \VO with the bare hand can be challenging} without good haptic feedback \cite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
|
|
|
|
|
|
|
|
|
|
|
|
In addition, current \AR systems have visual rendering limitations that also affect interaction with \VOs. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency.
|
|
|
|
In addition, current \AR systems have visual rendering limitations that also affect interaction with \VOs. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency.
|
|
|
|
\AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
|
|
|
|
\AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
|
|
|
|
But the depth perception of the \VOs is often underestimated \cite{peillard2019studying,adams2022depth}.
|
|
|
|
However, the depth perception of the \VOs is often underestimated \cite{peillard2019studying,adams2022depth}.
|
|
|
|
There is also often \textbf{a lack of mutual occlusion between the hand and a \VO}, \ie that the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
|
|
|
|
There is also often \textbf{a lack of mutual occlusion between the hand and a \VO}, \ie that the hand can hide the object or be hidden by the object \cite{macedo2023occlusion}.
|
|
|
|
Finally, as illustrated in \figref{interaction-loop}, interacting with a \VO is an illusion, because in fact the real hand is controlling in real time a virtual hand, like an avatar, whose contacts with \VOs are then simulated in the \VE.
|
|
|
|
Finally, as illustrated in \figref{interaction-loop}, interaction with a \VO is an illusion, because the real hand controls in real time a virtual hand, like an avatar, whose contacts with \VOs are then simulated in the \VE.
|
|
|
|
Therefore, there is inevitably a latency delay between the real hand's movements and the \VO's return movements, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the touched \VO \cite{prachyabrued2014visual}.
|
|
|
|
Therefore, there is inevitably a latency delay between the real hand's movements and the return movements of the \VO, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the \VO touched \cite{prachyabrued2014visual}.
|
|
|
|
This makes it \textbf{difficult to perceive the position of the fingers relative to the object} before touching or grasping it, but also to estimate the force required to grasp and move the object to a desired location.
|
|
|
|
These three rendering limitations make it \textbf{difficult to perceive the position of the fingers relative to the object} before touching or grasping it, but also to estimate the force required to grasp the \VO and move it to a desired location.
|
|
|
|
|
|
|
|
|
|
|
|
Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a \VO with the hand.
|
|
|
|
Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a \VO with the hand.
|
|
|
|
Yet, it is unclear which type of visual and haptic feedback, or their combination, is the best suited to guide the \VO manipulation.%, and whether one or the other of a combination of the two is most beneficial for users.
|
|
|
|
Yet, it is unclear which type of visual and haptic feedback, or their combination, is the best suited to guide the \VO manipulation.%, and whether one or the other of a combination of the two is most beneficial for users.
|
|
|
|
@@ -206,18 +206,18 @@ We consider two main axes of research, each addressing one of the research chall
|
|
|
|
Our contributions in these two axes are summarized in \figref{contributions}.
|
|
|
|
Our contributions in these two axes are summarized in \figref{contributions}.
|
|
|
|
|
|
|
|
|
|
|
|
\fig[0.95]{contributions}{Summary of our contributions through the simplified interaction loop.}[
|
|
|
|
\fig[0.95]{contributions}{Summary of our contributions through the simplified interaction loop.}[
|
|
|
|
The contributions are represented in dark gray boxes, and the research axes in light green circles.
|
|
|
|
The contributions are represented in dark grey boxes, and the research axes in light green circles.
|
|
|
|
The first axis is \textbf{(I)} the design and evaluation of the perception of visuo-haptic texture augmentations of real surfaces, directly touched by the hand.
|
|
|
|
The first axis is \textbf{(I)} the design and evaluation of the perception of visuo-haptic texture augmentations of real surfaces, directly touched by the hand.
|
|
|
|
The second axis focuses on \textbf{(II)} improving the manipulation of \VOs with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.
|
|
|
|
The second axis focuses on \textbf{(II)} improving the manipulation of \VOs with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.
|
|
|
|
]
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
\subsectionstarbookmark{Axis I: Augmenting the Texture Perception of Real Surfaces}
|
|
|
|
\subsectionstarbookmark{Axis I: Augmenting the Texture Perception of Real Surfaces}
|
|
|
|
|
|
|
|
|
|
|
|
Wearable haptic devices have proven to be effective in altering the perception of a touched real surface, without modifying the object nor covering the fingertip, forming a haptic \AE \cite{bau2012revel,detinguy2018enhancing,salazar2020altering}.
|
|
|
|
Wearable haptic devices have proven to be effective in altering the perception of a touched real surface, without modifying the object or covering the fingertip, forming a haptic \AE \cite{bau2012revel,detinguy2018enhancing,salazar2020altering}.
|
|
|
|
%It is achieved by placing the haptic actuator close to the fingertip, to let it free to touch the surface, and rendering tactile stimuli timely synchronised with the finger movement.
|
|
|
|
%It is achieved by placing the haptic actuator close to the fingertip, to let it free to touch the surface, and rendering tactile stimuli timely synchronised with the finger movement.
|
|
|
|
%It enables rich haptic feedback as the combination of kinesthetic sensation from the real and cutaneous sensation from the actuator.
|
|
|
|
%It enables rich haptic feedback as the combination of kinesthetic sensation from the real and cutaneous sensation from the actuator.
|
|
|
|
However, wearable haptic augmentations have been little explored with \AR, as well as the visuo-haptic augmentation of textures.
|
|
|
|
However, wearable haptic augmentations with \AR have been little explored, as well as the visuo-haptic augmentation of texture.
|
|
|
|
Texture is indeed one of the fundamental perceived property of a surface's material \cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by both sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) augmentation \cite{unger2011roughness,culbertson2014modeling,strohmeier2017generating}.
|
|
|
|
Texture is indeed one of the most fundamental perceived property of a surface's material \cite{hollins1993perceptual,okamoto2013psychophysical}, perceived equally well by both sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual}, and one of the most studied haptic (only, without visual) augmentation \cite{unger2011roughness,culbertson2014modeling,strohmeier2017generating}.
|
|
|
|
Being able to coherently substitute the visuo-haptic texture of a surface directly touched by a finger is an important step towards \AR capable of visually and haptically augmenting the \RE of a user in a plausible way.
|
|
|
|
Being able to coherently substitute the visuo-haptic texture of a surface directly touched by a finger is an important step towards \AR capable of visually and haptically augmenting the \RE of a user in a plausible way.
|
|
|
|
For this first axis of research, we propose to design and evaluate the \textbf{perception of wearable virtual visuo-haptic textures augmenting real surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
|
|
|
|
For this first axis of research, we propose to design and evaluate the \textbf{perception of wearable virtual visuo-haptic textures augmenting real surfaces}. %, using an immersive \AR headset and a wearable vibrotactile device.
|
|
|
|
To this end, we (1) design a system for rendering wearable visuo-haptic texture augmentations, to (2) evaluate how the perception of haptic texture augmentations is affected by the visual feedback of the virtual hand and the environment, and (3) investigate the perception of co-localized visuo-haptic texture augmentations.
|
|
|
|
To this end, we (1) design a system for rendering wearable visuo-haptic texture augmentations, to (2) evaluate how the perception of haptic texture augmentations is affected by the visual feedback of the virtual hand and the environment, and (3) investigate the perception of co-localized visuo-haptic texture augmentations.
|
|
|
|
@@ -227,11 +227,11 @@ Yet, to achieve the natural interaction with the hand and a coherent visuo-hapti
|
|
|
|
Thus, our first objective is to \textbf{design an immersive, real time system} that allows free exploration with the bare hand of \textbf{wearable visuo-haptic texture augmentations} on real surfaces.
|
|
|
|
Thus, our first objective is to \textbf{design an immersive, real time system} that allows free exploration with the bare hand of \textbf{wearable visuo-haptic texture augmentations} on real surfaces.
|
|
|
|
It will form the basis of the next two chapters in this section.
|
|
|
|
It will form the basis of the next two chapters in this section.
|
|
|
|
|
|
|
|
|
|
|
|
Second, many works have investigated the haptic augmentation of textures, but none have integrated them with \AR and \VR, or have considered the influence of the visual feedback on their perception.
|
|
|
|
Second, many works have investigated the haptic augmentations of texture, but none have integrated them with \AR and \VR, or have considered the influence of the visual feedback on their perception.
|
|
|
|
Still, it is known that the visual feedback can alter the perception of real and virtual haptic sensations \cite{schwind2018touch,choi2021augmenting} but also that the force feedback perception of grounded haptic devices is not the same in \AR and \VR \cite{diluca2011effects,gaffary2017ar}.
|
|
|
|
Still, it is known that the visual feedback can alter the perception of real and virtual haptic sensations \cite{schwind2018touch,choi2021augmenting} but also that the force feedback perception of grounded haptic devices is not the same in \AR and \VR \cite{diluca2011effects,gaffary2017ar}.
|
|
|
|
Hence, our second objective is to \textbf{evaluate how the perception of wearable haptic texture augmentation is affected by the visual feedback of the virtual hand and the environment} (real, augmented, or virtual).
|
|
|
|
Hence, our second objective is to \textbf{evaluate how the perception of wearable haptic texture augmentation is affected by the visual feedback of the virtual hand and the environment} (real, augmented, or virtual).
|
|
|
|
|
|
|
|
|
|
|
|
Finally, some visuo-haptic texture databases have been created from real texture captures \cite{culbertson2014penn,balasubramanian2024sens3}, to be rendered as virtual textures with hand-held haptic devices that are perceived as similar to real textures \cite{culbertson2015should,friesen2024perceived}.
|
|
|
|
Finally, some visuo-haptic texture databases have been created from real texture captures \cite{culbertson2014penn,balasubramanian2024sens3} to be rendered as virtual textures with hand-held haptic devices that are perceived as similar to real textures \cite{culbertson2015should,friesen2024perceived}.
|
|
|
|
However, the rendering of these textures in an immersive and natural visuo-haptic \AR using wearable haptics remains to be investigated.
|
|
|
|
However, the rendering of these textures in an immersive and natural visuo-haptic \AR using wearable haptics remains to be investigated.
|
|
|
|
Our third objective is to \textbf{evaluate the perception of simultaneous and co-localized wearable visuo-haptic texture augmentations} of real surfaces in \AR.%, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
|
|
|
|
Our third objective is to \textbf{evaluate the perception of simultaneous and co-localized wearable visuo-haptic texture augmentations} of real surfaces in \AR.%, and to understand to what extent each sensory modality contributes to the overall perception of the augmented texture.
|
|
|
|
|
|
|
|
|
|
|
|
@@ -246,15 +246,15 @@ For this second axis of research, we propose to design and evaluate \textbf{visu
|
|
|
|
We consider the effect on the user performance an experience of (1) the visual rendering as hand augmentation and (2) combination of different visuo-haptic rendering of the hand manipulation with \VOs
|
|
|
|
We consider the effect on the user performance an experience of (1) the visual rendering as hand augmentation and (2) combination of different visuo-haptic rendering of the hand manipulation with \VOs
|
|
|
|
|
|
|
|
|
|
|
|
First, the visual rendering of the virtual hand is a key element for interacting and manipulating \VOs in \VR \cite{prachyabrued2014visual,grubert2018effects}.
|
|
|
|
First, the visual rendering of the virtual hand is a key element for interacting and manipulating \VOs in \VR \cite{prachyabrued2014visual,grubert2018effects}.
|
|
|
|
A few works have also investigated the visual rendering of the virtual hand in \AR \cite{piumsomboon2014graspshell,blaga2017usability} but not in an immersive context of \VO manipulation. % with the bare hand.% from simulating mutual occlusions between the hand and \VOs \cite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay \cite{blaga2017usability,yoon2020evaluating}, augmenting the real hand.
|
|
|
|
Some work has also investigated the visual rendering of the virtual hand in \AR \cite{piumsomboon2014graspshell,blaga2017usability}, but not in an immersive context of \VO manipulation. % with the bare hand.% from simulating mutual occlusions between the hand and \VOs \cite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay \cite{blaga2017usability,yoon2020evaluating}, augmenting the real hand.
|
|
|
|
\OST-\AR has also significant perceptual differences from \VR due to the visibility of the real hand and environment that can affect the user experience and performance \cite{yoon2020evaluating}.
|
|
|
|
\OST-\AR also has significant perceptual differences from \VR due to the visibility of the real hand and environment, which can affect the user experience and performance \cite{yoon2020evaluating}.
|
|
|
|
%, and these visual hand augmentations have not been evaluated .
|
|
|
|
%, and these visual hand augmentations have not been evaluated .
|
|
|
|
Thus, our fourth objective is to \textbf{investigate the visual rendering as hand augmentation} for direct hand manipulation of \VOs in \OST-\AR.
|
|
|
|
Thus, our fourth objective is to \textbf{investigate the visual rendering as a hand augmentation} for direct hand manipulation of \VOs in \OST-\AR.
|
|
|
|
|
|
|
|
|
|
|
|
Second, as described above, the haptic actuators need to be moved away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE.
|
|
|
|
Second, as described above, the haptic actuators need to be moved away from the fingertips to not impair the hand movements, sensations, and interactions with the \RE.
|
|
|
|
Previous works have shown that wearable haptics that provide feedback on the hand manipulation with \VOs in \AR can significantly improve the user performance and experience \cite{maisto2017evaluation,meli2018combining}.
|
|
|
|
Previous works have shown that wearable haptics that provide feedback on the hand manipulation with \VOs in \AR can significantly improve the user performance and experience \cite{maisto2017evaluation,meli2018combining}.
|
|
|
|
However, it is unclear which positioning of the actuator is the most beneficial nor how a haptic augmentation of the hand compares or complements with a visual augmentation of the hand.
|
|
|
|
However, it is unclear which positioning of the actuator is the most beneficial nor how a haptic augmentation of the hand compares or complements with a visual augmentation of the hand.
|
|
|
|
Our last objective is to \textbf{investigate the visuo-haptic rendering of hand manipulation with \VOs} in \OST-\AR using wearable vibrotactile haptic.
|
|
|
|
Our last objective is to \textbf{investigate the visuo-haptic rendering of \VO manipulation with the hand} in \OST-\AR using wearable vibrotactile haptics.
|
|
|
|
|
|
|
|
|
|
|
|
\section{Thesis Overview}
|
|
|
|
\section{Thesis Overview}
|
|
|
|
\label{thesis_overview}
|
|
|
|
\label{thesis_overview}
|
|
|
|
@@ -265,7 +265,7 @@ With this first current \textit{Introduction} chapter, we have presented the res
|
|
|
|
|
|
|
|
|
|
|
|
In \textbf{\chapref{related_work}}, we then review previous work on the perception and manipulation of virtual and augmented objects, directly with the hand, using either wearable haptics, \AR, or their combination.
|
|
|
|
In \textbf{\chapref{related_work}}, we then review previous work on the perception and manipulation of virtual and augmented objects, directly with the hand, using either wearable haptics, \AR, or their combination.
|
|
|
|
First, we overview how the hand perceives and manipulates real objects.
|
|
|
|
First, we overview how the hand perceives and manipulates real objects.
|
|
|
|
Second, we present wearable haptics and haptic augmentations of texture and hardness of real objects.
|
|
|
|
Second, we present wearable haptics and haptic augmentations of the texture and hardness of real objects.
|
|
|
|
Third, we introduce \AR, and how \VOs can be manipulated directly with the hand.
|
|
|
|
Third, we introduce \AR, and how \VOs can be manipulated directly with the hand.
|
|
|
|
Finally, we describe how visuo-haptic feedback has augmented direct hand interaction in \AR, particularly using wearable haptics.
|
|
|
|
Finally, we describe how visuo-haptic feedback has augmented direct hand interaction in \AR, particularly using wearable haptics.
|
|
|
|
|
|
|
|
|
|
|
|
@@ -284,8 +284,8 @@ The system allows free visual and haptic exploration of the textures, as if they
|
|
|
|
In \textbf{\chapref{xr_perception}} we investigate in a user study how different the perception of haptic texture augmentations is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
|
|
|
|
In \textbf{\chapref{xr_perception}} we investigate in a user study how different the perception of haptic texture augmentations is in \AR \vs \VR and when touched by a virtual hand \vs one's own hand.
|
|
|
|
We use psychophysical methods to measure the user perception, and extensive questionnaires to understand how this perception is affected by the visual feedback of the virtual hand and the environment (real, augmented, or virtual).
|
|
|
|
We use psychophysical methods to measure the user perception, and extensive questionnaires to understand how this perception is affected by the visual feedback of the virtual hand and the environment (real, augmented, or virtual).
|
|
|
|
|
|
|
|
|
|
|
|
In \textbf{\chapref{vhar_textures}} we evaluate in a user study the perception of visuo-haptic texture augmentations, touched directly with one's own hand in \AR.
|
|
|
|
In \textbf{\chapref{vhar_textures}} we evaluate in a user study the perception of visuo-haptic texture augmentations directly touched with the real hand in \AR.
|
|
|
|
The virtual textures are paired visual and tactile models of real surfaces \cite{culbertson2014one} that we render as visual and haptic overlays on the touched augmented surfaces.
|
|
|
|
The virtual textures are paired visual and haptic captures of real surfaces \cite{culbertson2014one}, which we render as visual and haptic overlays on the touched augmented surfaces.
|
|
|
|
Our objective is to assess the perceived realism, coherence and roughness of the combination of nine representative visuo-haptic texture pairs.
|
|
|
|
Our objective is to assess the perceived realism, coherence and roughness of the combination of nine representative visuo-haptic texture pairs.
|
|
|
|
|
|
|
|
|
|
|
|
\noindentskip
|
|
|
|
\noindentskip
|
|
|
|
@@ -294,7 +294,7 @@ In \textbf{\partref{manipulation}} we describe our contributions to the second a
|
|
|
|
In \textbf{\chapref{visual_hand}} we investigate in a user study of six visual renderings as hand augmentations, as a set of the most popular hand renderings in the \AR literature.
|
|
|
|
In \textbf{\chapref{visual_hand}} we investigate in a user study of six visual renderings as hand augmentations, as a set of the most popular hand renderings in the \AR literature.
|
|
|
|
Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate their effect on the user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a \VO directly with the hand.
|
|
|
|
Using the \OST-\AR headset Microsoft HoloLens~2, we evaluate their effect on the user performance and experience in two representative manipulation tasks: push-and-slide and grasp-and-place a \VO directly with the hand.
|
|
|
|
|
|
|
|
|
|
|
|
In \textbf{\chapref{visuo_haptic_hand}} we evaluate in a user study the visuo-haptic rendering of manual object manipulation with two vibrotactile contact techniques, provided at four different positionings on the user's hand, as haptic rendering of the hand manipulation with \VOs.
|
|
|
|
In \textbf{\chapref{visuo_haptic_hand}} we evaluate in a user study the visuo-haptic rendering of direct hand manipulation of \VO with two vibrotactile contact techniques, provided at four different positionings on the user's hand. %, as haptic rendering of \VO manipulation with the hand.
|
|
|
|
They are compared to the two most representative visual hand renderings from the previous chapter, resulting in sixteen visuo-haptic hand renderings that are evaluated within the same experimental setup and design.
|
|
|
|
They are compared to the two most representative visual hand renderings from the previous chapter, resulting in sixteen visuo-haptic hand renderings that are evaluated within the same experimental setup and design.
|
|
|
|
|
|
|
|
|
|
|
|
\noindentskip
|
|
|
|
\noindentskip
|
|
|
|
|