From 9ba5d344a5c9991d7ffb4f167a64400dc7e94100 Mon Sep 17 00:00:00 2001 From: Erwan Normand Date: Tue, 24 Sep 2024 11:49:30 +0200 Subject: [PATCH] Clean related work --- .../related-work/2-wearable-haptics.tex | 183 +++++++++--------- .../related-work/3-augmented-reality.tex | 8 +- .../related-work/4-visuo-haptic-ar.tex | 113 +++++------ 3 files changed, 144 insertions(+), 160 deletions(-) diff --git a/1-introduction/related-work/2-wearable-haptics.tex b/1-introduction/related-work/2-wearable-haptics.tex index f970aaf..76681fb 100644 --- a/1-introduction/related-work/2-wearable-haptics.tex +++ b/1-introduction/related-work/2-wearable-haptics.tex @@ -2,10 +2,12 @@ \label{wearable_haptics} One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in a visual \VE \cite{maclean2008it,culbertson2018haptics}. +Due to the high complexity of the haptic sense and the variety of sensations it can feel, haptic actuators and renderings are designed to only address a subset of these sensations. +While it is challenging to create a realistic haptic experience, it is more important to provide the right sensory stimulus \enquote{at the right moment and at the right place} \cite{hayward2007it}. + Moreover, a haptic augmentation system should \enquote{modulating the feel of a real object by virtual [haptic] feedback} \cite{jeon2009haptic}, \ie a touch interaction with a real object whose perception is modified by the addition of virtual haptic feedback. The haptic system should be hand-held or worn, \eg on the hand, and \enquote{not permanently attached to or integrated in the object} \cite{bhatia2024augmenting}. - \subsection{Level of Wearability} \label{wearability_level} @@ -14,16 +16,16 @@ Different types of haptic devices can be worn on the hand, but only some of them An increasing wearability resulting in the loss of the system's kinesthetic feedback capability. \begin{subfigs}{pacchierotti2017wearable}{ - Schematic wearability level of haptic devices for the hand \cite{pacchierotti2017wearable}. - }[ - \item World-grounded haptic devices are fixed on the environment to provide kinesthetic feedback to the user. - \item Exoskeletons are body-grounded kinesthetic devices. - \item Wearable haptic devices are grounded on the point of application of the tactile stimulus. - ] - \subfigsheight{34mm} - \subfig{pacchierotti2017wearable_1} - \subfig{pacchierotti2017wearable_2} - \subfig{pacchierotti2017wearable_3} + Schematic wearability level of haptic devices for the hand \cite{pacchierotti2017wearable}. + }[ + \item World-grounded haptic devices are fixed on the environment to provide kinesthetic feedback to the user. + \item Exoskeletons are body-grounded kinesthetic devices. + \item Wearable haptic devices are grounded on the point of application of the tactile stimulus. + ] + \subfigsheight{34mm} + \subfig{pacchierotti2017wearable_1} + \subfig{pacchierotti2017wearable_2} + \subfig{pacchierotti2017wearable_3} \end{subfigs} Haptic research comes from robotics and teleoperation, and historically led to the design of haptic systems that are \emph{world-grounded} to an external support in the environment, such as a table (\figref{pacchierotti2017wearable_1}). @@ -40,21 +42,20 @@ An approach is then to move the grounding point very close to the end-effector ( Moreover, as detailed in \secref{object_properties}, cutaneous sensations are necessary and often sufficient for the perception of the haptic properties of an object explored with the hand, as also argued by \textcite{pacchierotti2017wearable}. \begin{subfigs}{grounded_to_wearable}{ - Haptic devices for the hand with different wearability levels. - }[ - \item Teleoperation of a virtual cube grasped with the thumb and index fingers each attached to a grounded haptic device \cite{pacchierotti2015cutaneous}. - \item A passive exoskeleton for fingers simulating stiffness of a trumpet's pistons \cite{achibet2017flexifingers}. - \item Manipulation of a virtual cube with the thumb and index fingers each attached with the 3-RSR wearable haptic device \cite{leonardis20173rsr}. - ] - \subfigsheight{38mm} - \subfig{pacchierotti2015cutaneous} - \subfig{achibet2017flexifingers} - \subfig{leonardis20173rsr} + Haptic devices for the hand with different wearability levels. + }[ + \item Teleoperation of a virtual cube grasped with the thumb and index fingers each attached to a grounded haptic device \cite{pacchierotti2015cutaneous}. + \item A passive exoskeleton for fingers simulating stiffness of a trumpet's pistons \cite{achibet2017flexifingers}. + \item Manipulation of a virtual cube with the thumb and index fingers each attached with the 3-RSR wearable haptic device \cite{leonardis20173rsr}. + ] + \subfigsheight{38mm} + \subfig{pacchierotti2015cutaneous} + \subfig{achibet2017flexifingers} + \subfig{leonardis20173rsr} \end{subfigs} % Tradeoff realistic and cost + analogy with sound, Hi-Fi costs a lot and is realistic, but 40$ BT headphone is more practical and enough, as cutaneous feedback without kinesthesic could be enough for wearable haptics and far more affordable and comfortable than world- or body-grounded haptics + cutaneous even better than kine for rendering surface curvature and fine manipulation - \subsection{Wearable Haptic Devices for the Hand} \label{wearable_haptic_devices} @@ -82,18 +83,18 @@ Multiple membranes are often used in a grid to simulate edges and textures, as i Although these two types of effector can be considered wearable, their actuation requires a high level of mechanical and electronic complexity that makes the system as a whole not portable. \begin{subfigs}{normal_actuators}{ - Normal indentation actuators for the fingertip. - }[ - \item A moving platform actuated with cables \cite{gabardi2016new}. - \item A moving platform actuated by articulated limbs \cite{perez2017optimizationbased}. - \item Diagram of a pin-array of tactors \cite{sarakoglou2012high}. - \item A pneumatic system composed of a \numproduct{12 x 10} array of air cylinders \cite{ujitoko2020development}. - ] - \subfigsheight{37mm} - \subfig{gabardi2016new} - \subfig{perez2017optimizationbased} - \subfig{sarakoglou2012high} - \subfig{ujitoko2020development} + Normal indentation actuators for the fingertip. + }[ + \item A moving platform actuated with cables \cite{gabardi2016new}. + \item A moving platform actuated by articulated limbs \cite{perez2017optimizationbased}. + \item Diagram of a pin-array of tactors \cite{sarakoglou2012high}. + \item A pneumatic system composed of a \numproduct{12 x 10} array of air cylinders \cite{ujitoko2020development}. + ] + \subfigsheight{37mm} + \subfig{gabardi2016new} + \subfig{perez2017optimizationbased} + \subfig{sarakoglou2012high} + \subfig{ujitoko2020development} \end{subfigs} \subsubsection{Tangential Motion Actuators} @@ -112,17 +113,17 @@ Conversely, by turning simultaneously in the same direction, the belt pulls on t The simplicity of this approach allows the belt to be placed anywhere on the hand, leaving the fingertip free to interact with the \RE, \eg the hRing on the proximal phalanx in \figref{pacchierotti2016hring} \cite{pacchierotti2016hring} or Tasbi on the wrist in \figref{pezent2022design} \cite{pezent2022design}. \begin{subfigs}{tangential_belts}{Tangential motion actuators and compression belts. }[ - \item A skin strech actuator for the fingertip \cite{leonardis2015wearable}. - \item A 3 \DoF actuator capable of normal and tangential motion on the fingertip \cite{schorr2017fingertip}. - %\item A shearing belt actuator for the fingertip \cite{minamizawa2007gravity}. - \item The hRing, a shearing belt actuator for the proximal phalanx of the finger \cite{pacchierotti2016hring}. - \item Tasbi, a wristband capable of pressure and vibrotactile feedback \cite{pezent2022design}. - ] - \subfigsheight{33.5mm} - \subfig{leonardis2015wearable} - \subfig{schorr2017fingertip} - \subfig{pacchierotti2016hring} - \subfig{pezent2022design} + \item A skin strech actuator for the fingertip \cite{leonardis2015wearable}. + \item A 3 \DoF actuator capable of normal and tangential motion on the fingertip \cite{schorr2017fingertip}. + %\item A shearing belt actuator for the fingertip \cite{minamizawa2007gravity}. + \item The hRing, a shearing belt actuator for the proximal phalanx of the finger \cite{pacchierotti2016hring}. + \item Tasbi, a wristband capable of pressure and vibrotactile feedback \cite{pezent2022design}. + ] + \subfigsheight{33.5mm} + \subfig{leonardis2015wearable} + \subfig{schorr2017fingertip} + \subfig{pacchierotti2016hring} + \subfig{pezent2022design} \end{subfigs} \subsubsection{Vibrotactile Actuators} @@ -137,12 +138,12 @@ Several types of vibrotactile actuators are used in haptics, with different trad An \ERM is a \DC motor that rotates an off-center mass when a voltage or current is applied (\figref{precisionmicrodrives_erm}). \ERMs are easy to control, inexpensive and can be encapsulated in a few millimeters cylinder or coin form factor. However, they have only one \DoF because both the frequency and amplitude of the vibration are coupled to the speed of the rotation, \eg low (high) frequencies output at low (high) amplitudes, as shown on \figref{precisionmicrodrives_erm_performances}. \begin{subfigs}{erm}{Diagram and performance of \ERMs. }[ - \item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives~\footnotemark. - \item Amplitude and frequency output of an \ERM as a function of the input voltage. - ] - \subfigsheight{45mm} - \subfig{precisionmicrodrives_erm} - \subfig{precisionmicrodrives_erm_performances} + \item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives~\footnotemark. + \item Amplitude and frequency output of an \ERM as a function of the input voltage. + ] + \subfigsheight{45mm} + \subfig{precisionmicrodrives_erm} + \subfig{precisionmicrodrives_erm_performances} \end{subfigs} \footnotetext{\url{https://www.precisionmicrodrives.com/}} @@ -158,15 +159,14 @@ They are very small and thin and provide two \DoFs of amplitude and frequency co However, they require high voltages to operate, limiting their use in wearable devices. \begin{subfigs}{lra}{Diagram and performance of \LRAs. }[ - \item Diagram. From Precision Microdrives~\footnotemarkrepeat. - \item Force generated by two \LRAs as a function of sinusoidal wave input with different frequencies: both their maximum force and resonant frequency are different \cite{azadi2014vibrotactile}. - ] - \subfigsheight{50mm} - \subfig{precisionmicrodrives_lra} - \subfig{azadi2014vibrotactile} + \item Diagram. From Precision Microdrives~\footnotemarkrepeat. + \item Force generated by two \LRAs as a function of sinusoidal wave input with different frequencies: both their maximum force and resonant frequency are different \cite{azadi2014vibrotactile}. + ] + \subfigsheight{50mm} + \subfig{precisionmicrodrives_lra} + \subfig{azadi2014vibrotactile} \end{subfigs} - \subsection{Modifying Perceived Haptic Roughness and Hardness} \label{tactile_rendering} @@ -202,7 +202,7 @@ There are two main approaches to modify virtual textures perception: \emph{simul The simplest texture simulation model is a 1D sinusoidal grating $v(t)$ with spatial period $\lambda$ and amplitude $A$ that is scanned by the user at velocity $\dot{x}(t)$: \begin{equation}{grating_rendering} - v(t) = A \sin(\frac{2 \pi \dot{x}(t)}{\lambda}) + v(t) = A \sin(\frac{2 \pi \dot{x}(t)}{\lambda}) \end{equation} That is, this model generates a periodic signal whose frequency is modulated and proportional to the user's velocity, implementing the speed-frequency ratio observed with real patterned textures (\eqref{grating_vibrations}). It gives the user the illusion of a texture with a \emph{fixed spatial period} that approximate the real manufactured grating textures (\secref{roughness}). @@ -239,19 +239,18 @@ When comparing real textures felt through a stylus with their virtual models ren \textcite{culbertson2015should} further showed that the perceived realism of the virtual textures, and similarity to the real textures, depended mostly on the user's speed but not on the user's force as inputs to the model, \ie responding to speed is sufficient to render isotropic virtual textures. \begin{subfigs}{textures_rendering_data}{Augmentating haptic texture perception with voice-coil actuators. }[ - \item Increasing and decreasing the perceived roughness of a real patterned texture in direct touch \cite{asano2015vibrotactile}. - \item Comparing real patterned texture with virtual texture augmentation in direct touch \cite{friesen2024perceived}. - \item Rendering virtual contacts in direct touch with the virtual texture \cite{ando2007fingernailmounted}. - \item Rendering an isotropic virtual texture over a real surface while sliding a hand-held stylus across it \cite{culbertson2012refined}. - ] - \subfigsheight{36mm} - \subfig{asano2015vibrotactile_2} - \subfig{friesen2024perceived} - \subfig{ando2007fingernailmounted} - \subfig{culbertson2012refined} + \item Increasing and decreasing the perceived roughness of a real patterned texture in direct touch \cite{asano2015vibrotactile}. + \item Comparing real patterned texture with virtual texture augmentation in direct touch \cite{friesen2024perceived}. + \item Rendering virtual contacts in direct touch with the virtual texture \cite{ando2007fingernailmounted}. + \item Rendering an isotropic virtual texture over a real surface while sliding a hand-held stylus across it \cite{culbertson2012refined}. + ] + \subfigsheight{36mm} + \subfig{asano2015vibrotactile_2} + \subfig{friesen2024perceived} + \subfig{ando2007fingernailmounted} + \subfig{culbertson2012refined} \end{subfigs} - \subsubsection{Hardness} \label{hardness_rendering} @@ -264,7 +263,7 @@ This was first proposed by \textcite{jeon2008modulating} who augmented a real su When the haptic end-effector contacts the object at time $t$, the object's surface deforms by displacement $x_r(t)$ and opposes a real reaction force $f_r(t)$. The virtual force of the device $\tilde{f_r}(t)$ is then controlled to: \begin{equation}{stiffness_augmentation} - \tilde{f_r}(t) = f_r(t) - \tilde{k} x_r(t) + \tilde{f_r}(t) = f_r(t) - \tilde{k} x_r(t) \end{equation} A force sensor embedded in the device measures the reaction force $f_r(t)$. The displacement $x_r(t)$ is estimated with the reaction force and the tapping velocity using a predefined model of different materials as described in \textcite{jeon2011extensions}. @@ -272,11 +271,11 @@ As shown in \figref{jeon2009haptic_2}, the force $\tilde{f_r}(t)$ perceived by t This stiffness augmentation technique was then extended to allow tapping and pressing with 3 \DoFs \cite{jeon2010stiffness}, to render friction and weight augmentations \cite{jeon2011extensions}, and to grasp and squeez the real object with two contact points \cite{jeon2012extending}. \begin{subfigs}{stiffness_rendering_grounded}{Augmenting the perceived stiffness of a real surface with a hand-held force-feedback device. }[% - \item Diagram of a user tapping the surface \cite{jeon2009haptic}. - \item Displacement-force curves of a real rubber ball (dashed line) and when its perceived stiffness $\tilde{k}$ is modulated \cite{jeon2009haptic}. - ] - \subfig[0.38]{jeon2009haptic_1} - \subfig[0.42]{jeon2009haptic_2} + \item Diagram of a user tapping the surface \cite{jeon2009haptic}. + \item Displacement-force curves of a real rubber ball (dashed line) and when its perceived stiffness $\tilde{k}$ is modulated \cite{jeon2009haptic}. + ] + \subfig[0.38]{jeon2009haptic_1} + \subfig[0.42]{jeon2009haptic_2} \end{subfigs} \textcite{detinguy2018enhancing} transposed this stiffness augmentation technique with the hRing device (\secref{belt_actuators}): While pressing a real piston with the fingertip by displacement $x_r(t)$, the belt compressed the finger with a virtual force $\tilde{k}\,x_r(t)$ where $\tilde{k}$ is the added stiffness (\eqref{stiffness_augmentation}), increasing the perceived stiffness of the piston (\figref{detinguy2018enhancing}). @@ -285,20 +284,19 @@ Conversely, the technique allowed to \emph{decrease} the perceived stiffness by \textcite{tao2021altering} proposed instead to restrict the deformation of the fingerpad by pulling a hollow frame around it to decrease perceived stiffness (\figref{tao2021altering}): it augments the finger contact area and thus the perceived Young's modulus of the object (\secref{hardness}). \begin{subfigs}{stiffness_rendering_wearable}{Modifying the perceived stiffness with wearable pressure devices. }[% - \item Modify the perceived stiffness of a piston by pressing the finger during or prior the contact \cite{detinguy2018enhancing,salazar2020altering}. - \item Decrease perceived stiffness of hard object by restricting the fingerpad deformation \cite{tao2021altering}. - ] - \subfigsheight{35mm} - \subfig{detinguy2018enhancing} - \subfig{tao2021altering} + \item Modify the perceived stiffness of a piston by pressing the finger during or prior the contact \cite{detinguy2018enhancing,salazar2020altering}. + \item Decrease perceived stiffness of hard object by restricting the fingerpad deformation \cite{tao2021altering}. + ] + \subfigsheight{35mm} + \subfig{detinguy2018enhancing} + \subfig{tao2021altering} \end{subfigs} - \paragraph{Vibrations Augmentations} \textcite{okamura2001realitybased} measured impact vibrations $v(t)$ when tapping on real objects and found they can be modeled as exponential decaying sinusoid: \begin{equation}{contact_transient} - v(t) = A \, |v_{in}| \, e^{- \tau t} sin(2 \pi f t) + v(t) = A \, |v_{in}| \, e^{- \tau t} sin(2 \pi f t) \end{equation} With $A$ the amplitude slope, $\tau$ the decay rate and $f$ the frequency, which are measured material properties, and $v_{in}$ the impact velocity. It has been shown that these material properties perceptually express the stiffness (\secref{hardness}) of real \cite{higashi2019hardness} and virtual surface \cite{choi2021perceived}. @@ -306,14 +304,14 @@ Therefore, when contacting or tapping a real object through an indirect feel-thr A challenge with this technique is to provide the vibration feedback at the right time to be felt simultaneously with the real contact \cite{park2023perceptual}. \begin{subfigs}{contact_vibrations}{Augmenting perceived stiffness using vibrations when touching a real surface \cite{choi2021augmenting}. }[% - %\item Experimental setup with a voice-coil actuator attached to a touch-through interface. - \item Voltage inputs (top) to the voice-coil for soft, medium, and hard vibrations, with the corresponding displacement (middle) and force (bottom) outputs of the actuator. - \item Perceived stiffness intensity of real sponge ("Sp") and wood ("Wd") surfaces without added vibrations ("N") and modified by soft ("S"), medium ("M") and hard ("H") vibrations. - ] - %\subfig[.15]{choi2021augmenting_demo} - \subfigsheight{50mm} - \subfig{choi2021augmenting_control} - \subfig{choi2021augmenting_results} + %\item Experimental setup with a voice-coil actuator attached to a touch-through interface. + \item Voltage inputs (top) to the voice-coil for soft, medium, and hard vibrations, with the corresponding displacement (middle) and force (bottom) outputs of the actuator. + \item Perceived stiffness intensity of real sponge ("Sp") and wood ("Wd") surfaces without added vibrations ("N") and modified by soft ("S"), medium ("M") and hard ("H") vibrations. + ] + %\subfig[.15]{choi2021augmenting_demo} + \subfigsheight{50mm} + \subfig{choi2021augmenting_control} + \subfig{choi2021augmenting_results} \end{subfigs} Vibrations on contact have been employed with wearable haptics, but to the best of our knowledge only to render \VOs \cite{pezent2019tasbi,teng2021touch,sabnis2023haptic}. @@ -351,7 +349,6 @@ We describe them in the \secref{vhar_haptics}. %\cite{choi2017grabity} %\cite{culbertson2017waves} - \subsection{Conclusion} \label{wearable_haptics_conclusion} diff --git a/1-introduction/related-work/3-augmented-reality.tex b/1-introduction/related-work/3-augmented-reality.tex index 41f0ef7..6b105d9 100644 --- a/1-introduction/related-work/3-augmented-reality.tex +++ b/1-introduction/related-work/3-augmented-reality.tex @@ -100,7 +100,7 @@ Finally, \AR displays can be head-worn like \VR \emph{headsets} or glasses, prov %Despite the clear and acknowledged definition presented in \secref{ar_definition} and the viewpoint of this thesis that \AR and \VR are two type of \MR experience with different levels of mixing real and virtual environments, as presented in \secref[introduction]{visuo_haptic_augmentations}, there is still a debate on defining \AR and \MR as well as how to characterize and categorized such experiences \cite{speicher2019what,skarbez2021revisiting}. Presence and embodiment are two key concepts that characterize the user experience in \AR and \VR. -While there is a large literature on these topics in \VR, they are less defined and studied for \AR \cite{tran2024survey,genay2022being}. +While there is a large literature on these topics in \VR, they are less defined and studied for \AR \cite{genay2022being,tran2024survey}. Still, these concepts are useful to design, evaluate and discuss our contributions in the next chapters. \paragraph{Presence} @@ -110,15 +110,15 @@ Still, these concepts are useful to design, evaluate and discuss our contributio Such experience of disbelief suspension in \VR is what is called \emph{presence}, and it can be decomposed into two dimensions: place illusion and plausibility \cite{slater2009place}. Place illusion is the sense of the user of \enquote{being there} in the \VE (\figref{presence-vr}). It emerges from the real time rendering of the \VE from the user's perspective: to be able to move around inside the \VE and look from different point of views. -plausibility is the illusion that the virtual events are really happening, even if the user knows that they are not real. +Plausibility is the illusion that the virtual events are really happening, even if the user knows that they are not real. It doesn't mean that the virtual events are realistic, but that they are plausible and coherent with the user's expectations. %The \AR presence is far less defined and studied than for \VR \cite{tran2024survey} For \AR, \textcite{slater2022separate} proposed to invert place illusion to what we can call \enquote{object illusion}, \ie the sense of the \VO to \enquote{feels here} in the \RE (\figref{presence-ar}). As with VR, \VOs must be able to be seen from different angles by moving the head but also, this is more difficult, be consistent with the \RE, \eg occlude or be occluded by real objects \cite{macedo2023occlusion}, cast shadows or reflect lights. The plausibility can be applied to \AR as is, but the \VOs must additionally have knowledge of the \RE and react accordingly to it. -\textcite{skarbez2021revisiting} also named place illusion for \AR as \enquote{immersion} and plausibility as \enquote{coherence}, and these terms will be used in the remainder of this thesis. -One main issue with presence is how to measure it both in \VR \cite{slater2022separate} and \AR \cite{tran2024survey}. +%\textcite{skarbez2021revisiting} also named place illusion for \AR as \enquote{immersion} and plausibility as \enquote{coherence}, and these terms will be used in the remainder of this thesis. +%One main issue with presence is how to measure it both in \VR \cite{slater2022separate} and \AR \cite{tran2024survey}. \begin{subfigs}{presence}{The sense of immersion in virtual and augmented environments. Adapted from \textcite{stevens2002putting}. }[ \item Place illusion is the sense of the user of \enquote{being there} in the \VE. diff --git a/1-introduction/related-work/4-visuo-haptic-ar.tex b/1-introduction/related-work/4-visuo-haptic-ar.tex index 4611b01..fe5b217 100644 --- a/1-introduction/related-work/4-visuo-haptic-ar.tex +++ b/1-introduction/related-work/4-visuo-haptic-ar.tex @@ -3,22 +3,13 @@ Everyday perception and manipulation of objects with the hand typically involves both the visual and haptic senses. Each sense has unique capabilities for perceiving certain object properties, such as color for vision or temperature for touch, but they are equally capable for many properties, such as roughness, hardness, or geometry \cite{baumgartner2013visual}. -Both \AR and wearable haptic systems integrate virtual content into the user's perception as sensory illusions. -It is essential to understand how a multimodal visuo-haptic rendering of a \VO is perceived.%, especially in immersive \AR where the haptic actuator is moved away so as not to cover the inside of the hand. -% Answer the following four questions: “Who else has done work with relevance to this work of yours? What did they do? What did they find? And how is your work here different?” +Both \AR and wearable haptic systems integrate virtual content into the user's perception as sensory illusions. +It is essential to understand how a multimodal visuo-haptic rendering of a \VO is perceived as a coherent object property, and how wearable haptics have been integrated with immersive \AR.%, especially in immersive \AR where the haptic actuator is moved away so as not to cover the inside of the hand. % spatial and temporal integration of visuo-haptic feedback as perceptual cues vs proprioception and real touch sensations % delocalized : not at the point of contact = difficult to integrate with other perceptual cues ? -%Go back to the main objective "to understand how immersive visual and wearable haptic feedback compare and complement each other in the context of direct hand perception and manipulation with augmented objects" and the two research challenges: "providing plausible and coherent visuo-haptic augmentations, and enabling effective manipulation of the augmented environment." -%Also go back to the \figref[introduction]{visuo-haptic-rv-continuum3} : we present previous work that either did haptic AR (the middle row), or haptic VR with visual AR, or visuo-haptic AR. - -% One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in visual \VE \cite{maclean2008it,culbertson2018haptics}. Moreover, a haptic \AR system should \enquote{modulating the feel of a real object by virtual [haptic] feedback} \cite{jeon2009haptic}, \ie a touch interaction with a real object whose perception is modified by the addition of virtual haptic feedback. - -% Finally, we present how multimodal visual and haptic feedback have been combined in \AR to modify the user perception of tangible objects, and to improve the user interaction with \VOs. - - \subsection{Visuo-Haptic Perception of Virtual and Augmented Objects} \label{sensations_perception} @@ -33,15 +24,15 @@ No sensory information is completely reliable and may give different answers to Therefore, each sensation $i$ is said to be an estimate $\tilde{s}_i$ with variance $\sigma_i^2$ of the property $s$. The \MLE model then predicts that the integrated estimated property $\tilde{s}$ is the weighted sum of the individual sensory estimates: \begin{equation}{MLE} - \tilde{s} = \sum_i w_i \tilde{s}_i \quad \text{with} \quad \sum_i w_i = 1 + \tilde{s} = \sum_i w_i \tilde{s}_i \quad \text{with} \quad \sum_i w_i = 1 \end{equation} Where the individual weights $w_i$ are proportional to their inverse variances: \begin{equation}{MLE_weights} - w_i = \frac{1/\sigma_i^2}{\sigma^2} + w_i = \frac{1/\sigma_i^2}{\sigma^2} \end{equation} And the integrated variance $\sigma^2$ is the inverse of the sum of the individual variances: \begin{equation}{MLE_variance} - \sigma^2 = \left( \sum_i \frac{1}{\sigma_i^2} \right)^{-1} + \sigma^2 = \left( \sum_i \frac{1}{\sigma_i^2} \right)^{-1} \end{equation} This was demonstrated by \textcite{ernst2002humans} in a user study where participants estimated the height of a virtual bar using a fixed-window \OST-\AR display (\secref{ar_displays}) and force-feedback devices worn on the thumb and index finger (\secref{wearability_level}), as shown in \figref{ernst2002humans_setup}. @@ -53,16 +44,16 @@ The objective was to determine a \PSE between the comparison and reference bars, \figref{ernst2004merging_results} shows that when the visual noise was low, the visual feedback had more weight, but as visual noise increased, haptic feedback gained more weight, as predicted by the \MLE model. \begin{subfigs}{ernst2002humans}{Visuo-haptic perception of height of a virtual bar \cite{ernst2002humans}. }[ - \item Experimental setup.%: Participants estimated height visually with an \OST-\AR display and haptically with force-feedback devices worn on the thumb and index fingers. - %\item with only haptic feedback (red) or only visual feedback (blue, with different added noise), - %\item combined visuo-haptic feedback (purple, with different visual noises). - \item Proportion of trials (vertical axis) where the comparison bar (horizontal axis) was perceived taller than the reference bar as function of increase variance (inverse of reliability) of the visual feedback (colors). - The reference had different conflicting visual $s_v$ and haptic $s_h$ heights. - ] - \subfig[.34]{ernst2002humans_setup} - \subfig[.64]{ernst2004merging_results} - %\subfig{ernst2002humans_within} - %\subfig{ernst2002humans_visuo-haptic} + \item Experimental setup.%: Participants estimated height visually with an \OST-\AR display and haptically with force-feedback devices worn on the thumb and index fingers. + %\item with only haptic feedback (red) or only visual feedback (blue, with different added noise), + %\item combined visuo-haptic feedback (purple, with different visual noises). + \item Proportion of trials (vertical axis) where the comparison bar (horizontal axis) was perceived taller than the reference bar as function of increase variance (inverse of reliability) of the visual feedback (colors). + The reference had different conflicting visual $s_v$ and haptic $s_h$ heights. + ] + \subfig[.34]{ernst2002humans_setup} + \subfig[.64]{ernst2004merging_results} + %\subfig{ernst2002humans_within} + %\subfig{ernst2002humans_visuo-haptic} \end{subfigs} %Hence, the \MLE model explains how a (visual) \VO in \AR can be perceived as coherent when combined with real haptic sensations of a tangible or a wearable haptic feedback. @@ -70,7 +61,6 @@ The \MLE model implies that when seeing and touching a \VO in \AR, the combinati %As long as the user is able to associate the sensations as the same object property, and even if there are discrepancies between the sensations, the overall perception can be influenced by changing one of the stimuli, as discussed in the next sections. %for example by including tangible objects, wearable haptic feedback, or even by altering the visual rendering of the \VO, as discussed in the next sections. - \subsubsection{Influence of Visual Rendering on Tangible Perception} \label{visual_haptic_influence} @@ -93,12 +83,12 @@ For example, in a fixed \VST-\AR screen (\secref{ar_displays}), by visually defo \textcite{ujitoko2019modulating} increased the perceived roughness of a virtual patterned texture rendered as vibrations through a hand-held stylus (\secref{texture_rendering}) by adding small oscillations to the visual feedback of the stylus on a screen. \begin{subfigs}{pseudo_haptic}{Pseudo-haptic feedback in \AR. }[ - \item A virtual soft texture projected on a table and that deforms when pressed by the hand \cite{punpongsanon2015softar}. - \item Modifying visually a tangible object and the hand touching it in \VST-\AR to modify its perceived shape \cite{ban2014displaying}. - ] - \subfigsheight{42mm} - \subfig{punpongsanon2015softar} - \subfig{ban2014displaying} + \item A virtual soft texture projected on a table and that deforms when pressed by the hand \cite{punpongsanon2015softar}. + \item Modifying visually a tangible object and the hand touching it in \VST-\AR to modify its perceived shape \cite{ban2014displaying}. + ] + \subfigsheight{42mm} + \subfig{punpongsanon2015softar} + \subfig{ban2014displaying} \end{subfigs} %In all of these studies, the visual expectations of participants influenced their haptic perception. @@ -115,17 +105,17 @@ One had a reference stiffness but an additional visual or haptic delay, while th Adding a visual delay increased the perceived stiffness of the reference piston, while adding a haptic delay decreased it, and adding both delays cancelled each other out (\figref{knorlein2009influence_2}). \begin{subfigs}{visuo-haptic-stiffness}{Perception of haptic stiffness in \VST-\AR \cite{knorlein2009influence}. }[ - \item Participant pressing a virtual piston rendered by a force-feedback device with their hand. - \item Proportion of comparison piston perceived as stiffer than reference piston (vertical axis) as a function of the comparison stiffness (horizontal axis) and visual and haptic delays of the reference (colors). - ] - \subfig[.44]{knorlein2009influence_1} - \subfig[.55]{knorlein2009influence_2} + \item Participant pressing a virtual piston rendered by a force-feedback device with their hand. + \item Proportion of comparison piston perceived as stiffer than reference piston (vertical axis) as a function of the comparison stiffness (horizontal axis) and visual and haptic delays of the reference (colors). + ] + \subfig[.44]{knorlein2009influence_1} + \subfig[.55]{knorlein2009influence_2} \end{subfigs} %explained how these delays affected the integration of the visual and haptic perceptual cues of stiffness. The stiffness $\tilde{k}(t)$ of the piston is indeed estimated at time $t$ by both sight and proprioception as the ratio of the exerted force $F(t)$ and the displacement $D(t)$ of the piston, following \eqref{stiffness}, but with potential visual $\Delta t_v$ or haptic $\Delta t_h$ delays: \begin{equation}{stiffness_delay} - \tilde{k}(t) = \frac{F(t + \Delta t_h)}{D(t + \Delta t_v)} + \tilde{k}(t) = \frac{F(t + \Delta t_h)}{D(t + \Delta t_v)} \end{equation} Therefore, the perceived stiffness $\tilde{k}(t)$ increases with a haptic delay in force and decreases with a visual delay in displacement \cite{diluca2011effects}. @@ -136,13 +126,13 @@ This suggests that the haptic stiffness of \VOs feels \enquote{softer} in an \AE %Two differences that could be worth investigating with the two previous studies are the type of \AR (visuo or optical) and to see the hand touching the \VO. \begin{subfigs}{gaffary2017ar}{Perception of haptic stiffness in \OST-\AR \vs \VR \cite{gaffary2017ar}. }[ - \item Experimental setup: a virtual piston was pressed with a force-feedback placed to the side of the participant. - \item View of the virtual piston seen in front of the participant in \OST-\AR and - \item in \VR. - ] - \subfig[0.35]{gaffary2017ar_1} - \subfig[0.3]{gaffary2017ar_3} - \subfig[0.3]{gaffary2017ar_4} + \item Experimental setup: a virtual piston was pressed with a force-feedback placed to the side of the participant. + \item View of the virtual piston seen in front of the participant in \OST-\AR and + \item in \VR. + ] + \subfig[0.35]{gaffary2017ar_1} + \subfig[0.3]{gaffary2017ar_3} + \subfig[0.3]{gaffary2017ar_4} \end{subfigs} Finally, \textcite{diluca2019perceptual} investigated the perceived simultaneity of visuo-haptic contact with a \VO in \VR. @@ -153,7 +143,6 @@ No participant (out of 19) was able to detect a \qty{50}{\ms} visual lag and a \ These studies have shown how the latency of the visual rendering of a \VO or the type of environment (\VE or \RE) can affect the perceived haptic stiffness of the object, rendered with a grounded force-feedback device. We describe in the next section how wearable haptics have been integrated with immersive \AR. - \subsection{Wearable Haptics for Direct Hand Interaction in AR} \label{vhar_haptics} @@ -166,7 +155,6 @@ Other wearable haptic actuators have been proposed for \AR, but are not discusse A first reason is that they permanently cover the fingertip and affect the interaction with the \RE, such as thin-skin tactile interfaces \cite{withana2018tacttoo,teng2024haptic} or fluid-based interfaces \cite{han2018hydroring}. Another category of actuators relies on systems that cannot be considered as portable, such as REVEL \cite{bau2012revel}, which provide friction sensations with reverse electrovibration that must modify the real objects to augment, or Electrical Muscle Stimulation (EMS) devices \cite{lopes2018adding}, which provide kinesthetic feedback by contracting the muscles. - \subsubsection{Nail-Mounted Devices} \label{vhar_nails} @@ -191,19 +179,18 @@ Finally, \textcite{preechayasomboon2021haplets} (\figref{preechayasomboon2021hap However, no proper user study has been conducted to evaluate these devices in \AR. \begin{subfigs}{ar_wearable}{Nail-mounted wearable haptic devices designed for \AR. }[ - %\item A voice-coil rendering a virtual haptic texture on a real sheet of paper \cite{ando2007fingernailmounted}. - \item Touch\&Fold provide contact pressure and vibrations on demand to the fingertip \cite{teng2021touch}. - \item Fingeret is a finger-side wearable haptic device that pulls and pushs the fingertip skin \cite{maeda2022fingeret}. - \item Haplets is a very compact nail device with integrated sensing and vibrotactile feedback \cite{preechayasomboon2021haplets}. - ] - \subfigsheight{33mm} - %\subfig{ando2007fingernailmounted} - \subfig{teng2021touch} - \subfig{maeda2022fingeret} - \subfig{preechayasomboon2021haplets} + %\item A voice-coil rendering a virtual haptic texture on a real sheet of paper \cite{ando2007fingernailmounted}. + \item Touch\&Fold provide contact pressure and vibrations on demand to the fingertip \cite{teng2021touch}. + \item Fingeret is a finger-side wearable haptic device that pulls and pushs the fingertip skin \cite{maeda2022fingeret}. + \item Haplets is a very compact nail device with integrated sensing and vibrotactile feedback \cite{preechayasomboon2021haplets}. + ] + \subfigsheight{33mm} + %\subfig{ando2007fingernailmounted} + \subfig{teng2021touch} + \subfig{maeda2022fingeret} + \subfig{preechayasomboon2021haplets} \end{subfigs} - \subsubsection{Belt Devices} \label{vhar_rings} @@ -225,12 +212,12 @@ However, the measured difference in performance could be due to either the devic These two studies were also conducted in non-immersive setups, where users viewed a screen displaying the visual interactions, and only compared the haptic and visual rendering of the hand-object contacts, but did not examine them together. \begin{subfigs}{ar_rings}{Wearable haptic ring devices for \AR. }[ - \item Rendering weight of a virtual cube placed on a real surface \cite{scheggi2010shape}. - \item Rendering the contact force exerted by the fingers on a virtual cube \cite{maisto2017evaluation,meli2018combining}. - ] - \subfigsheight{57mm} - \subfig{scheggi2010shape} - \subfig{maisto2017evaluation} + \item Rendering weight of a virtual cube placed on a real surface \cite{scheggi2010shape}. + \item Rendering the contact force exerted by the fingers on a virtual cube \cite{maisto2017evaluation,meli2018combining}. + ] + \subfigsheight{57mm} + \subfig{scheggi2010shape} + \subfig{maisto2017evaluation} \end{subfigs} %\subsubsection{Wrist Bracelet Devices}