diff --git a/1-introduction/related-work/1-haptic-hand.tex b/1-introduction/related-work/1-haptic-hand.tex index 51e957c..721b0db 100644 --- a/1-introduction/related-work/1-haptic-hand.tex +++ b/1-introduction/related-work/1-haptic-hand.tex @@ -29,6 +29,14 @@ In contrast, Merkel and Ruffini receptors, known as slow-adapting (SA), have a s The \emph{size of the receptor} determines the area of skin that can be sensed by a single nerve ending. Meissner and Merkel receptors have a small detection area (named Type I) and are sensitive to fine skin deformations, while Ruffini and Pacinian receptors have a larger detection area (named Type II). +The density of mechanoreceptors varies according to skin type and body region. +\emph{Glabrous skin}, especially on the face, feet, hands, and more importantly, the fingers, is particularly rich in cutaneous receptors, giving these regions great tactile sensitivity. +The density of the Meissner and Merkel receptors, which are the most sensitive, is notably high in the fingertips~\cite{johansson2009coding}. +Conversely, \emph{hairy skin} is less sensitive and does not contain Meissner receptors, but has additional receptors at the base of the hairs, as well as receptors known as C-tactile, which are involved in pleasantness and affective touch~\cite{ackerley2014touch}. + +There are also two types of thermal receptors implanted in the skin, which respond to increases or decreases in skin temperature, respectively, providing sensations of warmth or cold~\cite{lederman2009haptic}. +Finally, free nerve endings (without specialized receptors) provide information about pain~\cite{mcglone2007discriminative}. + \begin{tab}{cutaneous_receptors}{Characteristics of the cutaneous mechanoreceptors.}[ Adaptation rate is the speed and duration of the receptor's response to a stimulus. Receptive size is the area of skin detectable by a single receptor. Sensitivities are the stimuli detected by the receptor. Adapted from \textcite{mcglone2007discriminative} and \textcite{johansson2009coding}. ] @@ -44,14 +52,6 @@ Meissner and Merkel receptors have a small detection area (named Type I) and are \end{tabularx} \end{tab} -The density of mechanoreceptors varies according to skin type and body region. -\emph{Glabrous skin}, especially on the face, feet, hands, and more importantly, the fingers, is particularly rich in cutaneous receptors, giving these regions great tactile sensitivity. -The density of the Meissner and Merkel receptors, which are the most sensitive, is notably high in the fingertips~\cite{johansson2009coding}. -Conversely, \emph{hairy skin} is less sensitive and does not contain Meissner receptors, but has additional receptors at the base of the hairs, as well as receptors known as C-tactile, which are involved in pleasantness and affective touch~\cite{ackerley2014touch}. - -There are also two types of thermal receptors implanted in the skin, which respond to increases or decreases in skin temperature, respectively, providing sensations of warmth or cold~\cite{lederman2009haptic}. -Finally, free nerve endings (without specialized receptors) provide information about pain~\cite{mcglone2007discriminative}. - \subsubsection{Kinesthetic Sensitivity} \label{kinesthetic_sensitivity} @@ -162,7 +162,7 @@ This can be explained by the sensitivity of the fingertips (see \secref{haptic_s \subsection{Haptic Perception of Object Properties} \label{object_properties} -The active exploration of an object with the hand is performed as a sensorimotor loop: The exploratory movements guide the search for and adapt to sensory information, allowing to construct a haptic perception of the object's properties. +The active exploration of an object with the hand is performed as a sensorimotor loop: The exploratory movements (see \secref{exploratory_procedures}) guide the search for and adapt to sensory information (see \secref{haptic_sense}), allowing to construct a haptic perception of the object's properties. There are two main types of \emph{perceptual properties}. The \emph{material properties} are the perception of the roughness, hardness, temperature and friction of the surface of the object~\cite{bergmanntiest2010tactual}. The \emph{spatial properties} are the perception of the weight, shape and size of the object~\cite{lederman2009haptic}. @@ -296,7 +296,9 @@ The amplitude of the frictional force $F_s$ is proportional to the normal force \label{eq:friction} F_s = \mu \, F_n \end{equation} -The perceived intensity of friction is thus that of the friction coefficient $\mu$~\cite{smith1996subjective}. +The perceived intensity of friction is thus roughly related to the friction coefficient $\mu$~\cite{smith1996subjective}. +However, it is a complex perception because it is more determined by the micro-scale interactions between the surface and the skin: It depends on many factors such as the normal force applied, the speed of movement, the contact area and the moisture of the skin and the surface~\cite{adams2013finger,messaoud2016relation}. +In this sense, the perception of friction is still poorly understood~\cite{okamoto2013psychophysical}. \begin{subfigs}{smith1996subjective}{Perceived intensity of friction of different materials by active exploration with the finger~\cite{smith1996subjective}. }[ \item Measurements of normal $F_n$ and tangential $F_t$ forces when exploring two surfaces: one smooth (glass) and one rough (nyloprint). The fluctuations in the tangential force are due to the stick-slip phenomenon. The coefficient of friction $\mu$ can be estimated as the slope of the relationship between the normal and tangential forces. @@ -307,13 +309,10 @@ The perceived intensity of friction is thus that of the friction coefficient $\m \subfig{smith1996subjective_2} \end{subfigs} -Mais la perception de la friction est complexe car ce n'est pas seulement une propriété haptique de la surface. -Elle est, en effet, déterminée par les interactions à l'échelle micro entre la surface et la peau, et dépend donc de la force normale appliquée, de la vitesse du mouvement, de l'aire du contact et de l'humidité de la peau et de la surface~\cite{adams2013finger,messaoud2016relation}. -En ce sens, la perception de cette propriété est encore mal comprise~\cite{okamoto2013psychophysical}. - -C'est pourtant une perception fondamentale pour la saisie et la manipulation d'objets : les forces de frottements permettent de tenir fermement l'objet en main pour éviter qu'il ne glisse, et la perception de la friction permet également d'ajuster automatiquement et très rapidement la force à appliquer à l'objet pour le saisir~\cite{johansson1984roles}. -Si le doigt est anesthésié, l'absence de sensations cutanées empêche d'ajuster efficacement la force de préhension: Les forces de l'objet sur le doigt ne sont plus correctement perçues et les doigts appuient alors plus fermement sur l'objet en compensation mais sans réaliser une bonne opposition des doigts~\cite{witney2004cutaneous}. - +Yet, it is a fundamental perception for grasping and manipulating objects. +The forces of friction make it indeed possible to hold the object firmly in the hand and prevent it from slipping +The perception of friction also allows us to automatically and very quickly adjust the force we apply to the object in order to grasp it~\cite{johansson1984roles}. +If the finger is anaesthetized, the lack of cutaneous sensation prevents effective adjustment of the gripping force: the forces of the object on the finger are no longer correctly perceived, and the fingers then press harder on the object in compensation, but without achieving good opposition of the fingers~\cite{witney2004cutaneous}. \subsubsection{Temperature} \label{temperature} diff --git a/1-introduction/related-work/2-wearable-haptics.tex b/1-introduction/related-work/2-wearable-haptics.tex index 378322a..195e1f3 100644 --- a/1-introduction/related-work/2-wearable-haptics.tex +++ b/1-introduction/related-work/2-wearable-haptics.tex @@ -163,16 +163,18 @@ Several types of vibrotactile actuators are used in haptics, with different trad \subsection{Tactile Renderings for Modifying Object Properties} \label{tactile_rendering} -Le rendu tactile des propriétés haptiques consiste à modéliser et reproduire des sensations cutanées virtuelles comparables à celles perçues lors de l'interaction avec des objets réels. +Tactile rendering of haptic properties consists in modelling and reproducing virtual tactile sensations comparable to those perceived when interacting with real objects. +By adding such tactile rendering as feedback to the touch actions of the hand on a real object~\cite{bhatia2024augmenting}, both the real and virtual haptic sensations are integrated into a single property perception~\cite{ernst2004merging}. +Therefore, the visual rendering of a touched object can also greatly influence the perception of its haptic properties, \eg by modifying its visual texture in \AR or \VR, as discussed in the \secref{visuo_haptic_ar}. -En particulier, nous nous intéressons aux actuateurs portables stimulant les méchano-récepteurs de la peau (voir \secref{haptic_sense}) et n'empêchant pas de toucher et interagir avec l'environnement réel et aux rendus de propriétés haptiques d'objets virtuels ou augmentés. +\textcite{bhatia2024augmenting} categorize the tactile augmentations of real objects into three types: direct touch, touch-through, and tool mediated. +In direct touch, the haptic device does not cover the interior of the hand to not impair the user to interact with the \RE. +We are interested in direct touch augmentations with wearable haptic devices (see \secref{wearable_haptic_devices}), as their integration with \AR is particularly promising for direct hand interaction with visuo-haptic augmentations. +We also focus tactile augmentations stimulating the mechanoreceptors of the skin (see \secref{haptic_sense}), thus excluding temperature perception, as they are the most common existing haptic interfaces. -In the context of integrating \WHs with \AR to create a \vh-\AE (see \chapref{introduction}), the definition of \textcite{pacchierotti2017wearable} can be extended to an additional criterion: The wearable haptic interface should not impair the interaction with the \RE, \ie the user should be able to touch and manipulate objects in the real world while wearing the haptic device. - -\cite{bhatia2024augmenting}. Types of interfaces : direct touch, through touch, through tool. Focus on direct touch, but when no rendering done, will overview possibilities with the other types of interfaces. - -\cite{klatzky2003feeling} : rendering roughness, friction, deformation, temperatures -\cite{girard2016haptip} : renderings with a tangential motion actuator +% \cite{bhatia2024augmenting}. Types of interfaces : direct touch, through touch, through tool. Focus on direct touch, but when no rendering done, +% \cite{klatzky2003feeling} : rendering roughness, friction, deformation, temperatures +% \cite{girard2016haptip} : renderings with a tangential motion actuator \subsubsection{Textures} \label{texture_rendering} @@ -201,17 +203,13 @@ A common method vibrotactile rendering of texture is to use a sinusoidal signal \cite{jeon2009haptic} \cite{jeon2012extending} \cite{hachisu2012augmentation} - \cite{kildal20103dpress} - \cite{park2019realistic} \cite{choi2021perceived} \cite{park2023perceptual} - \cite{detinguy2018enhancing} \cite{salazar2020altering} \cite{yim2021multicontact} - \cite{park2017compensation} \cite{tao2021altering} diff --git a/1-introduction/related-work/3-augmented-reality.tex b/1-introduction/related-work/3-augmented-reality.tex index 405f4e4..b067c1f 100644 --- a/1-introduction/related-work/3-augmented-reality.tex +++ b/1-introduction/related-work/3-augmented-reality.tex @@ -72,11 +72,11 @@ Yet, the user experience in \AR is still highly dependent on the display used. Vergence-accommodation conflict. -Using a VST-AR headset have notable consequences, as the "real" view of the environment and the hand is actually a visual stream from a camera, which has a noticeable delay and lower quality (\eg resolution, frame rate, field of view) compared to the direct view of the real environment with OST-AR~\cite{macedo2023occlusion}.1 +Using a VST-AR headset have notable consequences, as the "real" view of the environment and the hand is actually a visual stream from a camera, which has a noticeable delay and lower quality (\eg resolution, frame rate, field of view) compared to the direct view of the real environment with OST-AR~\cite{macedo2023occlusion}. \paragraph{Optical See-Through Headsets} -Distances are underestimated~\cite{adams2022depth,peillard2019studying}. +%Distances are underestimated~\cite{adams2022depth,peillard2019studying}. \subsection{Presence and Embodiment in AR} @@ -165,6 +165,18 @@ Quand l'objet est lointain, la sélection peut se faire avec des techniques de p Problèmes d'occultation, les objets virtuels doivent toujours êtres visibles : soit en utilisant une main virtuelle transparente plutôt qu’opaque, soit en affichant leurs contours si elle les cache \cite{piumsomboon2014graspshell}. +\subsubsection{Manipulating with Tangibles} + +\cite{issartel2016tangible} +\cite{englmeier2020tangible} +en OST-AR \cite{kahl2021investigation,kahl2022influence,kahl2023using} + +Triple problème : +il faut un tangible par objet, problème de l'association qui ne fonctionne pas toujours (\cite{hettiarachchi2016annexing}) et du nombre de tangibles à avoir +et l'objet visuellement peut ne pas correspondre aux sensations haptiques du tangible manipulé (\cite{tinguy2019how}). +C'est pourquoi utiliser du wearable pour modifier les sensations cutanées du tangible est une solution qui fonctionne en VR (\cite{detinguy2018enhancing,salazar2020altering}) et pourrait être adaptée à la RA. +Mais, spécifique à la RA vs RV, le tangible et la main sont visibles, du moins partiellement, même si caché par un objet virtuel : comment va fonctionner l'augmentation haptique en RA vs RV ? Biais perceptuels ? Le fait de voir toucher avec sa propre main le tangible vs en RV où il est caché, donc illusion potentiellement plus forte en RV ? + \subsection{Visual Rendering of Hands in AR} @@ -198,18 +210,6 @@ To the best of our knowledge, evaluating the role of a visual rendering of the h Mais se pose la question de la représentation, qui a montré des effets sur la performance et expérience utilisateur en RV mais reste peu étudiée en RA. -\subsubsection{Manipulating with Tangibles} - -\cite{issartel2016tangible} -\cite{englmeier2020tangible} -en OST-AR \cite{kahl2021investigation,kahl2022influence,kahl2023using} - -Triple problème : -il faut un tangible par objet, problème de l'association qui ne fonctionne pas toujours (\cite{hettiarachchi2016annexing}) et du nombre de tangibles à avoir -et l'objet visuellement peut ne pas correspondre aux sensations haptiques du tangible manipulé (\cite{tinguy2019how}). -C'est pourquoi utiliser du wearable pour modifier les sensations cutanées du tangible est une solution qui fonctionne en VR (\cite{detinguy2018enhancing,salazar2020altering}) et pourrait être adaptée à la RA. -Mais, spécifique à la RA vs RV, le tangible et la main sont visibles, du moins partiellement, même si caché par un objet virtuel : comment va fonctionner l'augmentation haptique en RA vs RV ? Biais perceptuels ? Le fait de voir toucher avec sa propre main le tangible vs en RV où il est caché, donc illusion potentiellement plus forte en RV ? - \subsection{Conclusion} \label{ar_conclusion} diff --git a/1-introduction/related-work/4-visuo-haptic-ar.tex b/1-introduction/related-work/4-visuo-haptic-ar.tex index d2c4c14..cd8a82f 100644 --- a/1-introduction/related-work/4-visuo-haptic-ar.tex +++ b/1-introduction/related-work/4-visuo-haptic-ar.tex @@ -64,8 +64,9 @@ While a large literature has investigated these differences in visual perception \subsection{Wearable Haptics for AR} \label{vhar_haptics} -A few wearable haptic devices have been specifically designed for direct hand interaction in immersive \AR. +Some wearable haptic devices have been specifically designed for direct hand interaction in immersive \AR. The main challenge of wearable haptics for \AR is to provide haptic sensations of virtual or augmented objects that are touched and manipulated directly with the fingers while keeping the fingertips free to interact with the \RE. +%In this context of integrating \WHs with \AR to create a \vh-\AE (see \chapref{introduction}), the definition of \textcite{pacchierotti2017wearable} can be extended to an additional criterion: The wearable haptic interface should not impair the interaction with the \RE, \ie the user should be able to touch and manipulate objects in the real world while wearing the haptic device. %Many approaches have been proposed and they differ greatly in the actuators used (see \secref{wearable_haptic_devices}), the type of rendered object (real or virtual), the rendered haptic property (contact, hardness, texture, see \secref{tactile_rendering}), and the placement of the haptic rendering. As with the general overview of wearable haptic devices for the hand in \secref{wearable_haptics}, the type of the actuator used strongly determines the haptic feedback. @@ -110,16 +111,15 @@ However, as for \textcite{teng2021touch}, finger speed was not taken into accoun \subfig{maeda2022fingeret} \end{subfigs} -\paragraph{Wrist-Mounted Devices} +\paragraph{Bracelet Devices} -With their \enquote{Tactile And Squeeze Bracelet Interface} (Tasbi), already mentioned in \secref{belt_actuators}, \textcite{pezent2019tasbi} explored the use of a wrist-worn belt actuator. +With their \enquote{Tactile And Squeeze Bracelet Interface} (Tasbi), already mentioned in \secref{belt_actuators}, \textcite{pezent2019tasbi} explored the use of a wrist-worn bracelet actuator. It is capable of providing a uniform pressure sensation (up to \qty{15}{\N} and \qty{10}{\Hz}) and vibration with six \LRAs (\qtyrange{150}{200}{\Hz} bandwidth). Although the device has not been tested in \AR, a user study conducted in \VR to compare the perception of haptic and visual stiffness rendering of a virtual button is detailed in the \secref{vhar_interaction}. \subsubsection{Electrical Based Rendering} -[@Bau2010Teslatouch] created a touch-based surface rendering textures using electrovibration and friction feedback between the surface and the user's finger. -They extended this prototype to in [@Bau2012REVEL] to alter the texture of touched real objects using reverse electrovibration. They call this kind of haptic devices that can alter the touch perception of any object without any setup as *intrinsic haptic displays*. +\cite{bau2012revel} alterer the texture of touched real objects using reverse electrovibration. They call this kind of haptic devices that can alter the touch perception of any object without any setup as *intrinsic haptic displays*. \cite{lopes2018adding} @@ -127,15 +127,16 @@ They extended this prototype to in [@Bau2012REVEL] to alter the texture of touch \subsection{Improving the Interactions with Virtual Objects} \label{vhar_interaction} +\cite{scheggi2010shape} \cite{pacchierotti2015cutaneous} -Conjointly, a few studies have explored and compared the effects of visual and haptic feedback in tasks involving the manipulation of virtual objects with the hand. -\textcite{sarac2022perceived} and \textcite{palmer2022haptic} studied the effects of providing haptic feedback about contacts at the fingertips using haptic devices worn at the wrist, testing different mappings. -Results proved that moving the haptic feedback away from the point(s) of contact is possible and effective, and that its impact is more significant when the visual feedback is limited. - In pick-and-place tasks in AR involving both virtual and real objects, \textcite{maisto2017evaluation} and \textcite{meli2018combining} showed that having a haptic {rendering of the} fingertip interactions with the virtual objects led to better performance and perceived effectiveness than having only a visual rendering of the hand. Moreover, employing the haptic ring of~\cite{pacchierotti2016hring} on the proximal finger phalanx led to an improved performance with respect to more standard fingertip haptic devices~\cite{chinello2020modular}. However, the measured difference in performance could be attributed to either the device or the device position (proximal vs fingertip), or both. +Conjointly, a few studies have explored and compared the effects of visual and haptic feedback in tasks involving the manipulation of virtual objects with the hand. +\textcite{sarac2022perceived} and \textcite{palmer2022haptic} studied the effects of providing haptic feedback about contacts at the fingertips using haptic devices worn at the wrist, testing different mappings. +Results proved that moving the haptic feedback away from the point(s) of contact is possible and effective, and that its impact is more significant when the visual feedback is limited. + Furthermore, all of these studies were conducted in non-immersive setups, where users looked at a screen displaying the visual interactions, and only compared haptic and visual feedback, but did not examine them together. The improved performance and perceived effectiveness of a delocalized haptic feedback over a visual feedback alone, or their multimodal combination, remains to be verified in an immersive OST-AR setup. diff --git a/1-introduction/related-work/5-conclusion.tex b/1-introduction/related-work/5-conclusion.tex index 1e33aba..0a02776 100644 --- a/1-introduction/related-work/5-conclusion.tex +++ b/1-introduction/related-work/5-conclusion.tex @@ -1,7 +1,6 @@ \section{Conclusion} \label{conclusion} - La complexité de la perception des propriétés haptiques des objets et la diversité des interactions possibles rendent donc particulièrement difficile de concevoir des dispositifs et rendus haptiques réalistes. D'autant plus que le sens du toucher réparti sur l'ensemble de la main et du corps et que les sensations haptiques sont nécessairement produites par un contact direct de la peau avec l'objet, donc liées à un mouvement de la main sur l'objet. Il n'existe donc pas de système haptique générique pouvant adresser tous les aspects du sens haptique, mais une grande variété de dispositifs et de rendus haptiques avec différents objectifs, contraintes et compromis. @@ -17,14 +16,7 @@ De façon intéressante, les deux sections précédentes, présentant l'haptique C'est de cette manière que chacun des deux domaines est souvent introduit dans la littérature, par exemple avec les travaux de \textcite{choi2013vibrotactile,culbertson2018haptics} pour l'haptique et de \textcite{bimber2005spatial,kim2018revisiting} pour la RA. Mais il est également intéressant de noter que ces deux domaines sont à des stades de maturité différents. -% En effet, pouvoir contribuer pour ces deux domaines soulève, entre autres, des défis techniques importants, comme détaillé dans la \secref[introduction]{research_challenges}. -% Et il y a un besoin de standardisation en haptique portable~\cite{culbertson2018haptics}, notamment en terme de dispositifs et de rendus, alors que l'industrie est plutôt bien établie en RA, par exemple avec les casques HoloLens~2 de Microsoft~\footnoteurl{https://www.microsoft.com/hololens} et Vision~Pro d'Apple~\footnoteurl{https://www.apple.com/apple-vision-pro/} ou bien les frameworks ARCore de Google~\footnoteurl{https://developers.google.com/ar} et ARKit d'Apple~\footnoteurl{https://developer.apple.com/augmented-reality/}. -% Cela peut en partie d'une part s'expliquer par la maturité de l'industrie de la RV, qui entraîne celle de la RA, et avec une tendance annoncée à la convergence de ces deux technologies~\cite{speicher2019what}, mais aussi d'autre part par la plus grande complexité et les particularité du sens haptique~\cite{culbertson2018haptics}. -% À l'inverse, définir et caractériser la RA/RM, dans une bien moindre mesure la RV, reste étonnamment un sujet ouvert~\cite{speicher2019what}. - -Il faut donc également, dans cette thèse, tenir compte des deux éléphants dans la pièce: l'haptique grounded et la RV. -% diff --git a/1-introduction/related-work/related-work.tex b/1-introduction/related-work/related-work.tex index a3e5ead..28382a8 100644 --- a/1-introduction/related-work/related-work.tex +++ b/1-introduction/related-work/related-work.tex @@ -3,10 +3,10 @@ \chaptertoc -This chapter reviews previous work on the perception and manipulation of \AEs directly with the hand using \WHs, \AR and their combination. +This chapter reviews previous work on the perception and manipulation of \AEs directly with the hand using wearable haptics, \AR and their combination. %Experiencing a visual, haptic, or visuo-haptic \AE relies on one to many interaction loops between a user and the environment, as shown in \figref[introduction]{interaction-loop}, and each main step must be addressed and understood: the tracking and modelling of the \RE into a \VE, the interaction techniques to act on the \VE, the rendering of the \VE to the user through visual and haptic user interfaces, and, finally, the user's perception and actions on the overall \AE. -To achieve this, we first describe how the hand senses and acts on its environment to perceive the haptic properties of real everyday objects. -Secondly, we present how \WH devices and renderings have been used to augment the haptic perception with augmented objects, with a focus on vibrotactile feedback and haptic textures. +To achieve this, we first describe how the hand senses and acts on its environment to perceive the haptic properties of real everyday objects, and how the hand grasps and manipulates them. +Secondly, we present how wearable haptic devices and renderings have been used to augment the haptic perception with real, tangible objects, with a focus on vibrotactile feedback and haptic textures. Thirdly, we introduce the principles and user experience of \AR, and overview the main interaction techniques used to manipulate virtual and augmented objects. Finally, we present how multimodal visual and haptic feedback have been combined in \AR to modify the user perception of tangible objects, and to improve the user interaction with \VOs.