Add Virtual Object acronym

This commit is contained in:
2024-08-06 15:09:18 +02:00
parent 502191c8a3
commit e6934e42dd
2 changed files with 37 additions and 36 deletions

View File

@@ -12,4 +12,5 @@
\item[vRE, hRE, vhRE] Visual/Haptic/Visuo-Haptic Real Environment \item[vRE, hRE, vhRE] Visual/Haptic/Visuo-Haptic Real Environment
\item[vVE, hVE, vhVE] Visual/Haptic/Visuo-Haptic Virtual Environment \item[vVE, hVE, vhVE] Visual/Haptic/Visuo-Haptic Virtual Environment
\item[VCA] Voice-Coil Actuator \item[VCA] Voice-Coil Actuator
\item[VO] Virtual Object
\end{description} \end{description}

View File

@@ -123,7 +123,7 @@ In particular, \textcite{jeon2009haptic} proposed extending the RV continuum to
% %
The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of RV for each visual or haptic axis (real, augmented, virtual). The combination of the two axes defines 9 types of visuo-haptic environments, with 3 possible levels of RV for each visual or haptic axis (real, augmented, virtual).
% %
For example, a visual augmented environment (vAE) that uses a tangible (touchable) object as a proxy to manipulate virtual content is considered a haptic real environment (hRE; see \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum3}), whereas a device that provides synthetic haptic feedback when touching a virtual object is considered a haptic virtual environment (hVE; see \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum3}). For example, a visual augmented environment (vAE) that uses a tangible (touchable) object as a proxy to manipulate virtual content is considered a haptic real environment (hRE; see \figref{kahl2023using}; bottom middle cell in \figref{visuo-haptic-rv-continuum3}), whereas a device that provides synthetic haptic feedback when touching a virtual object (VO) is considered a haptic virtual environment (hVE; see \figref{meli2018combining}; top middle cell in \figref{visuo-haptic-rv-continuum3}).
% %
Haptic augmented reality (hAR) is then the combination of real and virtual haptic stimuli~\autocite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum3}). Haptic augmented reality (hAR) is then the combination of real and virtual haptic stimuli~\autocite{bhatia2024augmenting} (middle row in \figref{visuo-haptic-rv-continuum3}).
% %
@@ -135,7 +135,7 @@ In particular, it has been implemented by enhancing the haptic perception of tan
Current (visual) AR systems often lack haptic feedback, creating a deceptive and incomplete user experience when reaching the VE with the hand. Current (visual) AR systems often lack haptic feedback, creating a deceptive and incomplete user experience when reaching the VE with the hand.
% %
All visual virtual objects are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties congruently and interact with them with confidence and efficiency. All visual VOs are inherently intangible and cannot physically constrain a user's hand, making it difficult to perceive their properties congruently and interact with them with confidence and efficiency.
% %
It is therefore necessary to provide haptic feedback that is consistent with the vAE and ensures the best possible user experience. It is therefore necessary to provide haptic feedback that is consistent with the vAE and ensures the best possible user experience.
% %
@@ -144,8 +144,8 @@ The integration of wearable haptics with AR seems to be one of the most promisin
\begin{subfigs}{visuo-haptic-environments}{% \begin{subfigs}{visuo-haptic-environments}{%
Visuo-haptic environments with different degrees of reality-virtuality. Visuo-haptic environments with different degrees of reality-virtuality.
}[% }[%
\item Visual AR environment with a real, tangible haptic object used as a proxy to manipulate a virtual object~\autocite{kahl2023using}. \item Visual AR environment with a real, tangible haptic object used as a proxy to manipulate a VO~\autocite{kahl2023using}.
\item Visual AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a virtual object~\autocite{meli2018combining}. \item Visual AR environment with a wearable haptic device that provides virtual, synthetic feedback from contact with a VO~\autocite{meli2018combining}.
\item A tangible object seen in a visual VR environment whose haptic perception of stiffness is augmented with the hRing haptic device~\autocite{salazar2020altering}. \item A tangible object seen in a visual VR environment whose haptic perception of stiffness is augmented with the hRing haptic device~\autocite{salazar2020altering}.
\item Visuo-haptic rendering of texture on a touched tangible object with a visual AR display and haptic electrovibration feedback~\autocite{bau2012revel}. \item Visuo-haptic rendering of texture on a touched tangible object with a visual AR display and haptic electrovibration feedback~\autocite{bau2012revel}.
] ]
@@ -176,7 +176,7 @@ Because the visuo-haptic VE is displayed in real time, colocalized and aligned w
\fig{interaction-loop}{% \fig{interaction-loop}{%
The interaction loop between a user and a visuo-haptic augmented environment.% The interaction loop between a user and a visuo-haptic augmented environment.%
}[% }[%
One interact with the visual (in blue) and haptic (in red) virtual environment through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with virtual objects. % One interact with the visual (in blue) and haptic (in red) virtual environment through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with VOs. %
The virtual environment is rendered back to the user colocalized with the real one (in gray) using a visual AR headset and a wearable haptic device. % The virtual environment is rendered back to the user colocalized with the real one (in gray) using a visual AR headset and a wearable haptic device. %
] ]
@@ -195,7 +195,7 @@ Each of these challenges also raises numerous design, technical and human issues
\subsectionstartoc{Provide Plausible and Coherent Visuo-Haptic Augmentations} \subsectionstartoc{Provide Plausible and Coherent Visuo-Haptic Augmentations}
Many haptic devices have been designed and evaluated specifically for use in VR, providing realistic and varied kinesthetic and tactile feedback to virtual objects. Many haptic devices have been designed and evaluated specifically for use in VR, providing realistic and varied kinesthetic and tactile feedback to VOs.
% %
Although closely related, (visual) AR and VR have key differences in their respective renderings that can affect user perception. Although closely related, (visual) AR and VR have key differences in their respective renderings that can affect user perception.
@@ -203,7 +203,7 @@ Firstly, the user's hand and RE are visible in AR, unlike VR where there is tota
% (unless specifically overlaid with virtual visual content) % (unless specifically overlaid with virtual visual content)
% %
As such, in VR, visual sensations are particularly dominant in perception, and conflicts with haptic sensations are also specifically created to influence the user's perception, for example to create pseudo-haptic~\autocite{ujitoko2021survey} or haptic retargeting~\autocite{azmandian2016haptic} effects. As such, in VR, visual sensations are particularly dominant in perception, and conflicts with haptic sensations are also specifically created to influence the user's perception, for example to create pseudo-haptic~\autocite{ujitoko2021survey} or haptic retargeting~\autocite{azmandian2016haptic} effects.
%enabling techniques such as pseudo-haptic feedback that induce haptic feedback with visual stimuli~\autocite{ujitoko2021survey} or haptic retargeting that associate a single tangible object with multiple virtual objects without the user noticing~\autocite{azmandian2016haptic}. %enabling techniques such as pseudo-haptic feedback that induce haptic feedback with visual stimuli~\autocite{ujitoko2021survey} or haptic retargeting that associate a single tangible object with multiple VOs without the user noticing~\autocite{azmandian2016haptic}.
% %
Moreover, many wearable haptic devices take the form of controllers, gloves or exoskeletons, all of which cover the fingertips and are therefore not suitable for AR. Moreover, many wearable haptic devices take the form of controllers, gloves or exoskeletons, all of which cover the fingertips and are therefore not suitable for AR.
% %
@@ -227,34 +227,34 @@ It is therefore unclear to what extent the real and virtual visuo-haptic sensati
\subsectionstartoc{Enable Effective Interaction with the Augmented Environment} \subsectionstartoc{Enable Effective Interaction with the Augmented Environment}
Touching, grasping and manipulating virtual objects are fundamental interactions for AR~\autocite{kim2018revisiting}, VR~\autocite{bergstrom2021how} and VEs in general~\autocite{laviola20173d}. Touching, grasping and manipulating VOs are fundamental interactions for AR~\autocite{kim2018revisiting}, VR~\autocite{bergstrom2021how} and VEs in general~\autocite{laviola20173d}.
% %
As the hand is not occupied or covered with a haptic device to not impair interaction with the RE, as described in the previous section, one can expect a a seamless and direct interaction of the hand with the virtual content as if it were real. As the hand is not occupied or covered with a haptic device to not impair interaction with the RE, as described in the previous section, one can expect a a seamless and direct interaction of the hand with the virtual content as if it were real.
% %
Thus, augmenting a tangible object has the advantage of physically constraining the hand, allowing for easy and natural interaction, but manipulating a purely virtual object with the bare hand can be challenging without good haptic feedback~\autocite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction. Thus, augmenting a tangible object has the advantage of physically constraining the hand, allowing for easy and natural interaction, but manipulating a purely VO with the bare hand can be challenging without good haptic feedback~\autocite{maisto2017evaluation,meli2018combining}. %, and one will rely on visual and haptic feedback to guide the interaction.
In addition, current AR systems have visual rendering limitations that also affect interaction with virtual objects. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency. In addition, current AR systems have visual rendering limitations that also affect interaction with VOs. %, due to depth underestimation, a lack of mutual occlusions, and hand tracking latency.
% %
Visual AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world. Visual AR is the display of superimposed images of the virtual world, synchronized with the user's current view of the real world.
% %
But the depth perception of the virtual objects is often underestimated~\autocite{peillard2019studying,adams2022depth}, and there is often a lack of mutual occlusion between the hand and a virtual object, \ie that the hand can hide the object or be hidden by the object~\autocite{macedo2023occlusion}. But the depth perception of the VOs is often underestimated~\autocite{peillard2019studying,adams2022depth}, and there is often a lack of mutual occlusion between the hand and a VO, \ie that the hand can hide the object or be hidden by the object~\autocite{macedo2023occlusion}.
% %
Finally, as illustrated in \figref{interaction-loop}, interacting with a virtual object is an illusion, because in fact the real hand is controlling in real time a virtual hand, like an avatar, whose contacts with virtual objects are then simulated in the VE. Finally, as illustrated in \figref{interaction-loop}, interacting with a VO is an illusion, because in fact the real hand is controlling in real time a virtual hand, like an avatar, whose contacts with VOs are then simulated in the VE.
% %
Therefore, there is inevitably a latency delay between the real hand's movements and the virtual object's return movements, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the touched virtual object~\autocite{prachyabrued2014visual}. Therefore, there is inevitably a latency delay between the real hand's movements and the VO's return movements, and a spatial shift between the real hand and the virtual hand, whose movements are constrained to the touched VO~\autocite{prachyabrued2014visual}.
% %
This makes it difficult to perceive the position of the fingers relative to the object before touching or grasping it, and also to estimate the force required to grasp and move the object to a desired location. This makes it difficult to perceive the position of the fingers relative to the object before touching or grasping it, and also to estimate the force required to grasp and move the object to a desired location.
Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a virtual object with the hand. Hence, it is necessary to provide visual and haptic feedback that allows the user to efficiently contact, grasp and manipulate a VO with the hand.
% %
Yet, it is unclear which type of visual and haptic feedback is the best suited to guide the virtual object manipulation, and whether one or the other of a combination of the two is most beneficial for users. Yet, it is unclear which type of visual and haptic feedback is the best suited to guide the VO manipulation, and whether one or the other of a combination of the two is most beneficial for users.
% comme on laisse la main libre, et quelle visible, il n'y a pas de contrôleur tracké comme en VR, donc on s'attend à des interactions naturelles directement avec la main, de manière seamless entre physique et virtuel. On tracking additionel des actuateurs qui couvre la main. Question de où placer lactuateur, encore une fois le feedback n'est pas colocalisé avec l'action, contrairement au visuel (+- lag). Il y a également des décalages dans le temps et l'espace entre la main physique et son replicat virtuel, or c'est ce dernier qui agit avec les objets virtuel, est-ce qu'il faut le montrer ? Ou des rendus haptiques suffisent, complémentaire, en contradiction ? % comme on laisse la main libre, et quelle visible, il n'y a pas de contrôleur tracké comme en VR, donc on s'attend à des interactions naturelles directement avec la main, de manière seamless entre physique et virtuel. On tracking additionel des actuateurs qui couvre la main. Question de où placer lactuateur, encore une fois le feedback n'est pas colocalisé avec l'action, contrairement au visuel (+- lag). Il y a également des décalages dans le temps et l'espace entre la main physique et son replicat virtuel, or c'est ce dernier qui agit avec les objets virtuel, est-ce qu'il faut le montrer ? Ou des rendus haptiques suffisent, complémentaire, en contradiction ?
\sectionstartoc{Approach and Contributions} \sectionstartoc{Approach and Contributions}
The aim of this thesis is to understand how immersive visual and wearable haptic feedback compare and complement each other in the context of direct hand perception and interaction with augmented and virtual objects. The aim of this thesis is to understand how immersive visual and wearable haptic feedback compare and complement each other in the context of direct hand perception and interaction with augmented and VOs.
% %
As described in the Research Challenges section above, providing a convincing, consistent and effective vhAE to a user is complex and raises many issues. As described in the Research Challenges section above, providing a convincing, consistent and effective vhAE to a user is complex and raises many issues.
% %
@@ -277,7 +277,7 @@ Our contributions in these two axes are summarized in \figref{contributions}.
\fig[0.95]{contributions}{Summary of our contributions through the simplified interaction loop.}[% \fig[0.95]{contributions}{Summary of our contributions through the simplified interaction loop.}[%
The contributions are represented in dark gray boxes, and the research axes in light green circles.% The contributions are represented in dark gray boxes, and the research axes in light green circles.%
The first (I) axis designs and evaluates the perception of visuo-haptic texture augmentations of tangible surfaces, directly touched by the hand.% The first (I) axis designs and evaluates the perception of visuo-haptic texture augmentations of tangible surfaces, directly touched by the hand.%
The second (II) axis focuses on improving the manipulation of virtual objects with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.% The second (II) axis focuses on improving the manipulation of VOs with the bare hand using visuo-haptic augmentations of the hand as interaction feedback.%
] ]
@@ -325,32 +325,32 @@ Our third objective is to evaluate the perception of simultaneous and co-localiz
\subsectionstartoc{Improving Virtual Object Interaction with Visuo-Haptic Augmentations of the Hand} \subsectionstartoc{Improving Virtual Object Interaction with Visuo-Haptic Augmentations of the Hand}
In immersive and wearable visuo-haptic AR, the hand is free to touch and interact seamlessly with real, augmented, and virtual objects, and one can expect natural and direct contact and manipulation of virtual objects with the bare hand. In immersive and wearable visuo-haptic AR, the hand is free to touch and interact seamlessly with real, augmented, and virtual objects, and one can expect natural and direct contact and manipulation of VOs with the bare hand.
% %
However, the intangibility of the vVE, the many display limitations of current visual AR systems and wearable haptic devices, and the potential discrepancies between these two types of feedback can make the interaction with virtual objects particularly challenging. However, the intangibility of the vVE, the many display limitations of current visual AR systems and wearable haptic devices, and the potential discrepancies between these two types of feedback can make the interaction with VOs particularly challenging.
%However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current AR systems, as well as the spatial and temporal discrepancies between the real environment, the visual feedback, and the haptic feedback, can make the interaction with virtual objects with bare hands particularly challenging. %However, the intangibility of the virtual visual environment, the lack of kinesthetic feedback of wearable haptics, the visual rendering limitations of current AR systems, as well as the spatial and temporal discrepancies between the real environment, the visual feedback, and the haptic feedback, can make the interaction with VOs with bare hands particularly challenging.
% %
Still two types of sensory feedback are known to improve such direct virtual object interaction, but they have not been studied in combination in immersive vAE: visual rendering of the hand~\autocite{piumsomboon2014graspshell,prachyabrued2014visual} and hand-object interaction rendering with wearable haptics~\autocite{lopes2018adding,teng2021touch}. Still two types of sensory feedback are known to improve such direct VO interaction, but they have not been studied in combination in immersive vAE: visual rendering of the hand~\autocite{piumsomboon2014graspshell,prachyabrued2014visual} and hand-object interaction rendering with wearable haptics~\autocite{lopes2018adding,teng2021touch}.
% %
For this second axis of research, we propose to design and evaluate the role of visuo-haptic augmentations of the hand as interaction feedback with virtual objects. For this second axis of research, we propose to design and evaluate the role of visuo-haptic augmentations of the hand as interaction feedback with VOs.
% %
We consider (1) the effect of different visual augmentations of the hand as AR avatars and (2) the effect of combination of different visuo-haptic augmentations of the hand. We consider (1) the effect of different visual augmentations of the hand as AR avatars and (2) the effect of combination of different visuo-haptic augmentations of the hand.
First, the visual rendering of the virtual hand is a key element for interacting and manipulating virtual objects in VR~\autocite{prachyabrued2014visual,grubert2018effects}. First, the visual rendering of the virtual hand is a key element for interacting and manipulating VOs in VR~\autocite{prachyabrued2014visual,grubert2018effects}.
% %
A few works have also investigated the visual rendering of the virtual hand in AR, from simulating mutual occlusions between the hand and virtual objects~\autocite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay~\autocite{blaga2017usability,yoon2020evaluating}, augmenting the real hand. A few works have also investigated the visual rendering of the virtual hand in AR, from simulating mutual occlusions between the hand and VOs~\autocite{piumsomboon2014graspshell,al-kalbani2016analysis} to displaying the virtual hand as an avatar overlay~\autocite{blaga2017usability,yoon2020evaluating}, augmenting the real hand.
% %
But visual AR has significant perceptual differences from VR due to the visibility of the real hand and environment, and these visual hand augmentations have not been evaluated in the context of virtual object manipulation. But visual AR has significant perceptual differences from VR due to the visibility of the real hand and environment, and these visual hand augmentations have not been evaluated in the context of VO manipulation.
% %
Thus, our fourth objective is to evaluate and compare the effect of different visual hand augmentations on direct manipulation of virtual objects in AR. Thus, our fourth objective is to evaluate and compare the effect of different visual hand augmentations on direct manipulation of VOs in AR.
Finally, as described above, wearable haptics for visual AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the RE. Finally, as described above, wearable haptics for visual AR rely on moving the haptic actuator away from the fingertips to not impair the hand movements, sensations, and interactions with the RE.
% %
Previous works have shown that wearable haptics that provide feedback on the hand interaction with virtual objects in AR can significantly improve the user performance and experience~\autocite{maisto2017evaluation,meli2018combining}. Previous works have shown that wearable haptics that provide feedback on the hand interaction with VOs in AR can significantly improve the user performance and experience~\autocite{maisto2017evaluation,meli2018combining}.
% %
However, it is unclear which positioning of the actuator is the most beneficial nor how a haptic augmentation of the hand compares or complements with a visual augmentation of the hand. However, it is unclear which positioning of the actuator is the most beneficial nor how a haptic augmentation of the hand compares or complements with a visual augmentation of the hand.
% %
Our last objective is to investigate the role of visuo-haptic augmentations of the hand in manipulating virtual objects directly with the hand in AR. Our last objective is to investigate the role of visuo-haptic augmentations of the hand in manipulating VOs directly with the hand in AR.
\sectionstartoc{Thesis Overview} \sectionstartoc{Thesis Overview}
@@ -363,11 +363,11 @@ This thesis is divided in four parts.
% %
\chapref{related_work} then provides an overview of related work on the perception of and interaction with visual and haptic augmentations of objects. \chapref{related_work} then provides an overview of related work on the perception of and interaction with visual and haptic augmentations of objects.
% %
Firstly, it gives an overview of existing wearable haptic devices and renderings, and how they have been used to enhance the touch perception with haptic augmentations and to improve the virtual object interaction, with a focus on vibrotactile feedback and haptic textures. Firstly, it gives an overview of existing wearable haptic devices and renderings, and how they have been used to enhance the touch perception with haptic augmentations and to improve the VO interaction, with a focus on vibrotactile feedback and haptic textures.
% %
Secondly, it introduces the principles and user perception of augmented reality, and describes the 3D interaction techniques used in AR and VR environments to interact with virtual and augmented objects, in particular using the visual rendering of the user's hand. Secondly, it introduces the principles and user perception of augmented reality, and describes the 3D interaction techniques used in AR and VR environments to interact with virtual and augmented objects, in particular using the visual rendering of the user's hand.
% %
Finally, it shows how multimodal visuo-haptic feedback has been used in AR and VR to alter the perception of tangible objects and to improve the interaction with virtual objects. Finally, it shows how multimodal visuo-haptic feedback has been used in AR and VR to alter the perception of tangible objects and to improve the interaction with VOs.
% %
Then, we address each of our two research axes in a dedicated part. Then, we address each of our two research axes in a dedicated part.
@@ -399,17 +399,17 @@ Our objective is to assess the perceived realism, plausibility and roughness of
\bigskip \bigskip
\partref{manipulation} describes our contributions to the second axis of research, improving virtual object interaction with visuo-haptic augmentations of the hand. \partref{manipulation} describes our contributions to the second axis of research, improving VO interaction with visuo-haptic augmentations of the hand.
% %
We evaluate how the visual and haptic augmentation of the hand can improve the interaction with virtual objects directly with the hand. We evaluate how the visual and haptic augmentation of the hand can improve the interaction with VOs directly with the hand.
\chapref{visual_hand} explores in a first user study the effect of six visual hand augmentations that provide contact feedback with the virtual object, as a set of the most popular hand renderings in the AR literature. \chapref{visual_hand} explores in a first user study the effect of six visual hand augmentations that provide contact feedback with the VO, as a set of the most popular hand renderings in the AR literature.
% %
Using the OST-AR headset Microsoft HoloLens~2, the user performance and experience are evaluated in two representative manipulation tasks, \ie push-and-slide and grasp-and-place of a virtual object directly with the hand. Using the OST-AR headset Microsoft HoloLens~2, the user performance and experience are evaluated in two representative manipulation tasks, \ie push-and-slide and grasp-and-place of a VO directly with the hand.
\chapref{visuo_haptic_hand} evaluates in a second user study two vibrotactile contact techniques, provided at four different locations on the real hand, as haptic rendering of the hand-object interaction. \chapref{visuo_haptic_hand} evaluates in a second user study two vibrotactile contact techniques, provided at four different locations on the real hand, as haptic rendering of the hand-object interaction.
% %
They are compared to the two most representative visual hand augmentations from the previous study, and the user performance and experience are evaluateds within the same OST-AR setup and manipulation tasks. They are compared to the two most representative visual hand augmentations from the previous study, and the user performance and experience are evaluated within the same OST-AR setup and manipulation tasks.
\bigskip \bigskip