WIP

1-introduction/related-work/2-wearable-haptics.tex

@@ -1,7 +1,7 @@
 \section{Augmenting Object Perception with Wearable Haptics}
 \label{wearable_haptics}
 
-One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in \v-\VE~\cite{maclean2008it,culbertson2018haptics}.
+One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in a visual \VE~\cite{maclean2008it,culbertson2018haptics}.
 Moreover, a haptic augmentation system should \enquote{modulating the feel of a real object by virtual [haptic] feedback}~\cite{jeon2009haptic}, \ie a touch interaction with a real object whose perception is modified by the addition of virtual haptic feedback.
 The haptic system should be hand-held or worn, \eg on the hand, and \enquote{not permanently attached to or integrated in the object}~\cite{bhatia2024augmenting}.
 
@@ -9,9 +9,9 @@ The haptic system should be hand-held or worn, \eg on the hand, and \enquote{not
 \subsection{Level of Wearability}
 \label{wearability_level}
 
-Different types of haptic devices can be worn on the hand, but only some of them can be considered \emph{wearable}.
+Different types of haptic devices can be worn on the hand, but only some of them can be considered wearable.
 \textcite{pacchierotti2017wearable} classify them into three levels of wearability, as illustrated in the \figref{pacchierotti2017wearable}.
-An increasing \emph{wearability} resulting in the loss of the system's kinesthetic feedback capability.
+An increasing wearability resulting in the loss of the system's kinesthetic feedback capability.
 
 \begin{subfigs}{pacchierotti2017wearable}{
         Schematic wearability level of haptic devices for the hand~\cite{pacchierotti2017wearable}.
@@ -26,17 +26,17 @@ An increasing \emph{wearability} resulting in the loss of the system's kinesthet
     \subfig{pacchierotti2017wearable_3}
 \end{subfigs}
 
-Haptic research comes from robotics and teleoperation, and historically led to the design of haptic systems that are \emph{grounded} to an external support in the environment, such as a table (\figref{pacchierotti2017wearable_1}).
+Haptic research comes from robotics and teleoperation, and historically led to the design of haptic systems that are \emph{world-grounded} to an external support in the environment, such as a table (\figref{pacchierotti2017wearable_1}).
 These are robotic arms whose end-effector is either held in the hand or worn on a finger and which simulate interactions with a \VE by providing kinesthetic forces and torques feedback (\figref{pacchierotti2015cutaneous}).
 They provide high fidelity haptic feedback but are heavy, bulky and limited to small workspaces~\cite{culbertson2018haptics}.
 
 More portable designs have been developed by moving the grounded part to the user's body.
 The entire robotic system is thus mounted on the user, forming an exoskeleton capable of providing kinesthetic feedback to the finger, \eg in \figref{achibet2017flexifingers}.
 However, it cannot constrain the movements of the wrist and the reaction force is transmitted to the user where the device is grounded (\figref{pacchierotti2017wearable_2}).
-They are often heavy and bulky and cannot be considered wearable.
+Such \emph{body-grounded} devices are often heavy and bulky and cannot be considered wearable.
 
 \textcite{pacchierotti2017wearable} defined that : \enquote{A wearable haptic interface should also be small, easy to carry, comfortable, and it should not impair the motion of the wearer}.
-An approach is then to move the grounding point very close to the end-effector (\figref{pacchierotti2017wearable_3}): the interface is limited to cutaneous haptic feedback, but its design is more compact, lightweight and comfortable, \eg in \figref{leonardis20173rsr}, and the system is wearable.
+An approach is then to move the grounding point very close to the end-effector (\figref{pacchierotti2017wearable_3}): the interface is limited to cutaneous haptic feedback, but its design is more compact, lightweight, comfortable and portable, \eg in \figref{grounded_to_wearable}.
 Moreover, as detailed in \secref{object_properties}, cutaneous sensations are necessary and often sufficient for the perception of the haptic properties of an object explored with the hand, as also argued by \textcite{pacchierotti2017wearable}.
 
 \begin{subfigs}{grounded_to_wearable}{
@@ -130,33 +130,35 @@ The simplicity of this approach allows the belt to be placed anywhere on the han
 
 Vibrotactile actuators are the most common and simplest wearable haptic interfaces, and are available as consumer products.
 They are small, lightweight and can be placed directly on any part of the hand.
-\textcite{choi2013vibrotactile} provide a detailed review.
+\textcite{choi2013vibrotactile} provided a detailed review.
 All vibrotactile actuators are based on the same principle: generating an oscillating motion from an electric current with a frequency and amplitude high enough to be perceived by cutaneous mechanoreceptors.
-Several types of vibrotactile actuators are used in haptics, with different trade-offs between size, proposed \DoFs and application constraints:
+Several types of vibrotactile actuators are used in haptics, with different trade-offs between size, proposed \DoFs and application constraints.
-\begin{itemize}
-    \item An \ERM is a \DC motor that rotates an off-center mass when a voltage or current is applied (\figref{precisionmicrodrives_erm}). \ERMs are easy to control, inexpensive and can be encapsulated in a few millimeters cylinder or coin form factor. However, they have only one \DoF because both the frequency and amplitude of the vibration are coupled to the speed of the rotation, \eg low (high) frequencies output at low (high) amplitudes, as shown on \figref{precisionmicrodrives_erm_performances}.
-    \item A \LRA consists of a coil that creates a magnetic field from an \AC to oscillate a magnet attached to a spring, as an audio loudspeaker (\figref{precisionmicrodrives_lra}). They are more complex to control and a bit larger than \ERMs. Each \LRA is designed to vibrate with maximum amplitude at a given frequency, but won't vibrate efficiently at other frequencies, \ie their bandwidth is narrow, as shown on \figref{azadi2014vibrotactile}.
-    \item A \VCA is a \LRA but capable of generating vibration at two \DoF, with an independent control of the frequency and amplitude of the vibration on a wide bandwidth. They are larger in size than \ERMs and \LRAs, but can generate more complex renderings.
-    \item Piezoelectric actuators deform a solid material when a voltage is applied. They are very small and thin, and allow two \DoFs of amplitude and frequency control. However, they require high voltages to operate thus limiting their use in wearable devices.
-\end{itemize}
 
-\begin{subfigs}{vibrotactile_actuators}{Diagrams of vibrotactile acuators. }[
+An \ERM is a \DC motor that rotates an off-center mass when a voltage or current is applied (\figref{precisionmicrodrives_erm}). \ERMs are easy to control, inexpensive and can be encapsulated in a few millimeters cylinder or coin form factor. However, they have only one \DoF because both the frequency and amplitude of the vibration are coupled to the speed of the rotation, \eg low (high) frequencies output at low (high) amplitudes, as shown on \figref{precisionmicrodrives_erm_performances}.
-        \item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives.~\footnotemark
+
-        \item Diagram of a \LRA. From Precision Microdrives.~\footnotemarkrepeat
+\begin{subfigs}{erm}{Diagram and performance of \ERMs. }[
+        \item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives~\footnotemark.
+        \item Amplitude and frequency output of an \ERM as a function of the input voltage.
     ]
-    \subfigsheight{50mm}
+    \subfigsheight{45mm}
     \subfig{precisionmicrodrives_erm}
-    \subfig{precisionmicrodrives_lra}
+    \subfig{precisionmicrodrives_erm_performances}
 \end{subfigs}
 
 \footnotetext{\url{https://www.precisionmicrodrives.com/}}
 
-\begin{subfigs}{vibrotactile_performances}{Performances of vibrotactile acuators. }[
+A \LRA consists of a coil that creates a magnetic field from an \AC to oscillate a magnet attached to a spring, as an audio loudspeaker (\figref{precisionmicrodrives_lra}). They are more complex to control and a bit larger than \ERMs. Each \LRA is designed to vibrate with maximum amplitude at a given resonant frequency, but won't vibrate efficiently at other frequencies, \ie their bandwidth is narrow, as shown on \figref{azadi2014vibrotactile}.
-        \item Amplitude and frequency output of an \ERM as a function of the input voltage. From Precision Microdrives.~\footnotemarkrepeat
+A \VCA is a \LRA but capable of generating vibration at two \DoF, with an independent control of the frequency and amplitude of the vibration on a wide bandwidth. They are larger in size than \ERMs and \LRAs, but can generate more complex renderings.
-        \item Force generated by two \LRAs as a function of sine wave input with different frequencies: both their maximum force and frequency are different~\cite{azadi2014vibrotactile}.
+
+Piezoelectric actuators deform a solid material when a voltage is applied. They are very small and thin, and allow two \DoFs of amplitude and frequency control. However, they require high voltages to operate thus limiting their use in wearable devices.
+
+\begin{subfigs}{lra}{Diagram and performance of \LRAs. }[
+        \item Diagram. From Precision Microdrives~\footnotemarkrepeat.
+        \item Force generated by two \LRAs as a function of sine wave input with different frequencies: both their maximum force and resonant frequency are different~\cite{azadi2014vibrotactile}.
     ]
-    \subfig[.58]{precisionmicrodrives_erm_performances}
+    \subfigsheight{50mm}
-    \subfig[.38]{azadi2014vibrotactile}
+    \subfig{precisionmicrodrives_lra}
+    \subfig{azadi2014vibrotactile}
 \end{subfigs}
 
 
@@ -164,7 +166,7 @@ Several types of vibrotactile actuators are used in haptics, with different trad
 \label{tactile_rendering}
 
 Tactile rendering of haptic properties consists in modelling and reproducing virtual tactile sensations comparable to those perceived when interacting with real objects~\cite{klatzky2013haptic}.
-By adding such tactile rendering as feedback to the touch actions of the hand on a real object~\cite{bhatia2024augmenting}, the perception of the object's haptic property is modified.
+By adding such tactile rendering as feedback to the touch actions of the hand on a real object~\cite{bhatia2024augmenting}, the perception of the object's haptic property can be modified.
 The integration of the real and virtual haptic sensations into a single property perception is discussed in more details in \secref{sensations_perception}.
 %, both the real and virtual haptic sensations are integrated into a single property perception, as presented in \secref{sensations_perception}, \ie the perceived haptic property is modulated by the added virtual feedback.
 In particular, the visual rendering of a touched object can also greatly influence the perception of its haptic properties, \eg by modifying its visual texture in \AR or \VR, as discussed in the \secref{visuo_haptic}.
@@ -176,39 +178,50 @@ In touch-through and tool-mediated, or \emph{indirect feel-through}, the haptic
 Many haptic augmentations were first developed with grounded haptic devices and later transposed to wearable haptic devices.
 %We also focus on tactile augmentations stimulating the mechanoreceptors of the skin (\secref{haptic_sense}), thus excluding temperature perception, as they are the most common existing haptic interfaces.
 
+As we chose in \secref{object_properties} to focus on the haptic perception of the roughness and hardness of objects, we present bellow the methods to modify these properties with wearable haptic devices.
+Of course, wearable haptics can also be used to modify the perceived friction \cite{konyo2008alternative,salazar2020altering}, weight \cite{minamizawa2007gravity}, or local deformation \cite{salazar2020altering} of real objects, but they are less common \cite{bhatia2024augmenting} and will not be detailed here.
+
 % \cite{klatzky2003feeling} : rendering roughness, friction, deformation, temperatures
 % \cite{girard2016haptip} : renderings with a tangential motion actuator
 
-\subsubsection{Textures}
+\subsubsection{Roughness}
 \label{texture_rendering}
 
-Several approaches have been proposed to render virtual haptic texture~\cite{culbertson2018haptics}.
+To modify the perception of haptic roughness (or texture, see \secref{roughness}) of a real object, vibrations are provided to the skin by the wearable haptic device when running the finger over the surface.
-As more traditional force feedback systems are unable to accurately render such micro-details on a simulated surface, vibrotactile devices attached to the end effector instead generate vibrations to simulate interaction with the virtual texture~\cite{campion2005fundamental,culbertson2018haptics}.
+Two approaches are used to render virtual textures: \emph{physics-based models} and \emph{data-driven models}~\cite{culbertson2018haptics}.
-In this way, physics-based models~\cite{chan2021hasti,okamura1998vibration,guruswamy2011iir} and data-based models~\cite{culbertson2015should,romano2010automatic} have been developed and evaluated, the former being simpler but more approximate to real textures, and the latter being more realistic but limited to the captured textures.
 
 \paragraph{Physics-based Models}
 
+As more traditional force feedback systems are unable to accurately render such micro-details on a simulated surface, vibrotactile devices attached to the end effector instead generate vibrations to simulate interaction with the virtual texture~\cite{campion2005fundamental,culbertson2018haptics}.
 High-fidelity force feedback devices can reproduce patterned textures with great precision and provide similar perceptions to real textures, but they are expensive, have a limited workspace, and impose to hold a probe to explore the texture~\cite{unger2011roughness}.
 Notably, \textcite{okamura1998vibration} rendered grating textures with exponentially decaying sinudoids that simulated the strokes of the grooves and ridges of the surface, while \textcite{culbertson2014modeling} captured and modelled the roughness of real surfaces to render them using the speed and force of the user.
+\cite{chan2021hasti,guruswamy2011iir}
 
-\paragraph{Data-driven Models}
+\textcite{ando2007fingernailmounted} were the first to propose this approach that they experimented with a voice-coil mounted on the index nail (\figref{ando2007fingernailmounted}).
-
+The sensation of crossing edges of a virtual patterned texture (\secref{texture_rendering}) on a real sheet of paper were rendered with \qty{20}{\ms} vibration impulses at \qty{130}{\Hz}.
-An effective approach to rendering virtual roughness is to generate vibrations to simulate interaction with the virtual texture~\cite{culbertson2018haptics}, relying on the user's real-time measurements of position, velocity and force to modulate the frequencies and amplitudes of the vibrations, with position and velocity being the most important parameters~\cite{culbertson2015should}.
+Participants were able to match the virtual patterns to their real counterparts of height \qty{0.25}{\mm} and width \qtyrange{1}{10}{\mm}, but systematically overestimated the virtual width to be \qty{4}{\mm} longer.
-For example, when comparing the same virtual texture pairwise, but with different parameters, \textcite{culbertson2015should} showed that the roughness vibrations generated should vary with user speed, but not necessarily with user force.
-Virtual data-driven textures were perceived as similar to real textures, except for friction, which was not rendered properly.
 
 The perceived roughness of real surfaces can be then modified when touched by a tool with a vibrotactile actuator attached~\cite{culbertson2014modeling,ujitoko2019modulating} or directly with the finger wearing the vibrotactile actuator~\cite{asano2015vibrotactile}, creating a haptic texture augmentation.
 Of course, the fingertip skin is not deformed by the virtual texture and only vibrations are felt, but it has been shown that the vibrations produced on the fingertip skin running over a real surface are texture specific and similar between individuals~\cite{delhaye2012textureinduced,manfredi2014natural}.
 A common method vibrotactile rendering of texture is to use a sinusoidal signal whose frequency is modulated by the finger position or velocity~\cite{asano2015vibrotactile,friesen2024perceived,strohmeier2017generating,ujitoko2019modulating}.
 
+\paragraph{Data-driven Models}
+
+Because physics-based models to render realistic textures can be very complex to design and to render in real-time, direct capture of real textures have been used instead to model the produced vibrations~\cite{culbertson2018haptics}.
+
+An effective approach to rendering virtual roughness is to generate vibrations to simulate interaction with the virtual texture~\cite{culbertson2018haptics}, relying on the user's real-time measurements of position, velocity and force to modulate the frequencies and amplitudes of the vibrations, with position and velocity being the most important parameters~\cite{culbertson2015should}.
+For example, when comparing the same virtual texture pairwise, but with different parameters, \textcite{culbertson2015should} showed that the roughness vibrations generated should vary with user speed, but not necessarily with user force.
+Virtual data-driven textures were perceived as similar to real textures, except for friction, which was not rendered properly.
+
+
 \subsubsection{Hardness}
 \label{hardness_rendering}
 
-The two main approaches to modulate the perceived hardness of a real surface with wearable haptics are to render forces or vibrations.
+The perceived hardness (\secref{hardness}) of a real surface can be modified by rendering forces or vibrations.
 
 \paragraph{Modulating Forces}
 
-When tapping or pressing a real object with a tool, the perceived stiffness $\tilde{k}$ (\secref{hardness}) of its surface can be modulated with force feedback~\cite{jeon2015haptic}.
+When tapping or pressing a real object, the perceived stiffness $\tilde{k}$ of its surface can be modulated with force feedback~\cite{jeon2015haptic}.
 This was first proposed by \textcite{jeon2008modulating} who augmented a real surface tapped in 1 \DoF with a grounded force-feedback device held in hand (\figref{jeon2009haptic_1}).
 When the haptic end-effector contacts the object at time $t$, the object's surface deforms by displacement $x_r(t)$ and opposes a real reaction force $f_r(t)$.
 The virtual force of the device $\tilde{f_r}(t)$ is then controlled to:
@@ -220,38 +233,67 @@ The displacement $x_r(t)$ is estimated with the reaction force and tapping veloc
 As shown in \figref{jeon2009haptic_2}, the force $\tilde{f_r}(t)$ perceived by the user being modulated, but not the displacement $x_r(t)$, the perceived stiffness is $\tilde{k}(t)$.
 This stiffness augmentation technique was then extended to enable tapping and pressing with 3 \DoFs~\cite{jeon2010stiffness}, to render friction and weight augmentations~\cite{jeon2011extensions}, and to grasping and squeezing the real object with two contact points~\cite{jeon2012extending}.
 
-\begin{subfigs}{stiffness_rendering}{Augmenting perceived stiffness of a real surface. }[%
+\begin{subfigs}{stiffness_rendering_grounded}{Augmenting the perceived stiffness of a real surface with a hand-held force-feedback device. }[%
-        \item Diagram of a user tapping a real surface with a hand-held force-feedback device~\cite{jeon2009haptic}.
+        \item Diagram of a user tapping the surface \cite{jeon2009haptic}.
-        \item Displacement-force curves of a real rubber ball (dashed line) and when its perceived stiffness $\tilde{k}$ is modulated~\cite{jeon2009haptic}.
+        \item Displacement-force curves of a real rubber ball (dashed line) and when its perceived stiffness $\tilde{k}$ is modulated \cite{jeon2009haptic}.
     ]
-    \subfigsheight{35mm}
+    \subfig[0.38]{jeon2009haptic_1}
-    \subfig[0.2]{jeon2009haptic_1}
+    \subfig[0.42]{jeon2009haptic_2}
-    \subfig[0.4]{jeon2009haptic_2}
 \end{subfigs}
 
-\cite{detinguy2018enhancing}
+\textcite{detinguy2018enhancing} transposed this stiffness augmentation technique with the hRing device (\secref{belt_actuators}): While pressing a real piston with the fingertip by displacement $x_r(t)$, the belt compressed the finger by a virtual force $\tilde{k}\,x_r(t)$ where $\tilde{k}$ is the added stiffness (\eqref{stiffness_augmentation}), increasing the perceived stiffness of the piston (\figref{detinguy2018enhancing}).
-\cite{salazar2020altering}
+%This enables to \emph{increase} the perceived stiffness of the real piston up to \percent{+14}.
+More importantly, the augmentation proved to be robust to the placement of the device, as the increased stiffness was perceived the same on the fingertip, the middle phalanx and the proximal.
+Conversely, the technique allowed to \emph{decrease} the perceived stiffness by compressing the phalanx prior the contact and diminish the belt pressure as the user pressed the piston~\cite{salazar2020altering}.
+\textcite{tao2021altering} proposed instead to restrict the deformation of the fingerpad by pulling a hollow frame around it to decrease perceived stiffness (\figref{tao2021altering}): it augments the finger contact area thus the perceived Young modulus of the object (\secref{hardness}).
 
-\cite{kildal20103dpress}
+\begin{subfigs}{stiffness_rendering_wearable}{Modifying the perceived stiffness with wearable pressure devices. }[%
-\cite{tao2021altering} % wearable softness
+        \item Modify the perceived stiffness of a piston by pressing the finger during or prior the contact \cite{detinguy2018enhancing,salazar2020altering}.
+        \item Decrease perceived stiffness of hard object by restricting the fingerpad deformation \cite{tao2021altering}.
+    ]
+    \subfigsheight{35mm}
+    \subfig{detinguy2018enhancing}
+    \subfig{tao2021altering}
+\end{subfigs}
 
 
 \paragraph{Vibrations Augmentations}
 
-The second main approach is to modulate the vibrations felt when tapping a real surface with a tool~\cite{okamura1998vibration}.
+\textcite{okamura2001realitybased} measured impact vibrations $Q(t)$ when tapping on real objects and found they can be modelled as exponential decaying sine wave:
-Tapping with a tool on a real surface augmented with a vibrotactile actuator generating exponential decaying sinusoids
-
 \begin{equation}{contact_transient}
-    \tilde{f}_c(t) = a \, |v_{in}| \, e^{- \tau t} sin(2 \pi f t)
+    Q(t) = A \, |v_{in}| \, e^{- \tau t} sin(2 \pi f t)
 \end{equation}
+With $A$ the amplitude slope, $\tau$ the sine decay rate and $f$ the sine frequency, which are measured material properties, and $v_{in}$ the impact velocity.
+It has been shown that these material properties perceptually express the stiffness (\secref{hardness}) of real~\cite{higashi2019hardness} and virtual surface~\cite{choi2021perceived}.
+Therefore, when contacting or tapping a real object through an indirect feel-through interface that provide such vibrations (\figref{choi2021augmenting_control}) using a voice-coil (\secref{vibrotactile_actuators}), the perceived stiffness can be increased or decreased \cite{kuchenbecker2006improving,hachisu2012augmentation,choi2021augmenting}, \eg sponge feeling stiffer or wood feeling softer (\figref{choi2021augmenting_results}).
+A challenge with this technique is to provide the vibration feedback at the right time, to be felt simultaneous with the real contact~\cite{park2023perceptual}.
 
-\cite{kuchenbecker2006improving}
+\begin{subfigs}{contact_vibrations}{Augmenting perceived stiffness using vibrations when touching a real surface \cite{choi2021augmenting}. }[%
-\cite{hachisu2012augmentation}
+        %\item Experimental setup with a voice-coil actuator attached to a touch-through interface.
-\cite{park2019realistic}
+        \item Voltage inputs (top) to the voice-coil for soft, medium and hard vibrations, with the corresponding displacement (middle) and force (bottom) outputs of the actuator.
-\cite{park2023perceptual}
+        \item Perceived intensity of stiffness of real sponge ("Sp") and wood ("Wd") surfaces without added vibrations ("N") and modified by soft ("S"), medium ("M") and hard ("H") vibrations.
+    ]
+    %\subfig[.15]{choi2021augmenting_demo}
+    \subfigsheight{50mm}
+    \subfig{choi2021augmenting_control}
+    \subfig{choi2021augmenting_results}
+\end{subfigs}
+
+Vibrations on contact have been employed with wearable haptics but, to the best of our knowledge, only to render \VOs~\cite{pezent2019tasbi,teng2021touch,sabnis2023haptic}.
+We describe them in the \secref{vhar_haptics}.
+
+%A promising alternative approach
+%\cite{kildal20103dpress}
+
+%\begin{subfigs}{vibtration_grains}{Augmenting perceived stiffness of a real surface with vibrations. }
+%    \subfigsheight{35mm}
+%    \subfig{sabnis2023haptic_device}
+%    \subfig{sabnis2023haptic_control}
+%\end{subfigs}
 
 %\textcite{choi2021perceived} combined and compared these two rendering approaches (spring-damper and exponential decaying sinusoids) but to render purely virtual surfaces.
 %They found that the perceived intensity of the virtual hardness $\tilde{h}$ followed a power law, similarly to \eqref{hardness_intensity}, with the amplitude $a$, the %frequency $f$ and the damping $b$ of the vibration, but not the decay time $\tau$.
+%\cite{park2023perceptual}
 
 %\subsubsection{Friction}
 %\label{friction_rendering}

1-introduction/related-work/4-visuo-haptic-ar.tex

@@ -66,62 +66,41 @@ As long as the user is able to match the sensations as the same object property,
 %for example by including tangible objects, wearable haptic feedback, or even by altering the visual rendering of the \VO, as discussed in the next sections.
 
 
-\subsubsection{Influence of Visual Rendering on Haptic Perception}
+\subsubsection{Influence of Visual Rendering on Tangible Perception}
 \label{visual_haptic_influence}
 
-A visuo-haptic perception of the property of a real or virtual object is robust to a certain difference between the two sensory modalities.
+A visuo-haptic perception of an object's property is thus robust to a certain difference between the two sensory modalities, as long as one can match their respective sensations to the same property.
-Yet, as the visual sense
+In particular, the texture perception of everyday objects is known to be constructed from both vision and touch~\cite{klatzky2010multisensory}.
+More precisely, when evaluating surfaces with vision or touch only, both senses mainly discriminate their materials by the same properties of roughness, hardness and friction, and with similar performance~\cite{bergmanntiest2007haptic,baumgartner2013visual,vardar2019fingertip}.
 
-Une perception est donc robuste à une certaine différence entre les sensations visuelles et haptiques, réelles ou virtuelle, tant qu'un utilisateur peut les faire correspondre à un même objet augmenté.
+Overall perception can be then modified by changing one of the sensory modality, as shown by \textcite{yanagisawa2015effects}, who altered perceived roughness, stiffness and friction of real tactile materials touched by the finger by superimposing different real visual textures using a half-mirror.
+With a similar setup but in immersive \VST-\AR, \textcite{kitahara2010sensory} overlaid visual textures on real textured surfaces touched but through a glove: Participants matched visual textures to real textures when their respective hardness felt similar.
+\textcite{degraen2019enhancing} and \textcite{gunther2022smooth} also combined in \VR multiple \VOs with \ThreeD-printed hair structures or with everyday real surfaces, respectively.
+They found that the visual perception of roughness and hardness influenced the haptic perception, and that only a few tangibles seemed to sufficient to match all the visual \VOs (\figref{gunther2022smooth}).
+%Taken together, these studies suggest that a set of haptic textures, real or virtual, can be perceived as coherent with a larger set of visual virtual textures.
 
-Particularly for real textures, it is known that both touch and sight individually perceive textures equally well and similarly~\cite{bergmanntiest2007haptic,baumgartner2013visual,vardar2019fingertip}.
+\fig{gunther2022smooth}{In a passive touch context in \VR, only a smooth and a rough real surfaces were found to match all the visual \VOs \cite{gunther2022smooth}.}
-%
-Thus, the overall perception can be modified by changing one of the modalities, as shown by \textcite{yanagisawa2015effects}, who altered the perception of roughness, stiffness and friction of some real tactile textures touched by the finger by superimposing different real visual textures using a half-mirror.
 
-% Spring compliance is perceived by combining the sensed force exerted by the spring with the displacement caused by the action (sensed through vision and proprioception). diluca2011effects
+The visual feedback can even be designed on purpose to influence the haptic perception, usually by deforming the visual representation of a user input, creating a \enquote{pseudo-haptic feedback}~\cite{ujitoko2021survey}.
-% The ability to discriminate whether two stimuli are simultaneous is important to determine whether stimuli should be bound together and form a single multisensory perceptual object. diluca2019perceptual
+For example, in a fixed \VST-\AR screen (\secref{ar_displays}), by deforming visually the geometry of a tangible object touched by the hand, as well as the touching hand, the visuo-haptic perception of the size, shape or curvature can be altered~\cite{ban2013modifying,ban2014displaying}.
+\textcite{punpongsanon2015softar} used this technique in spatial \AR (\secref{ar_displays}) to induce a softness illusion of a hard tangible object by superimposing a virtual texture that deforms when pressed by the hand (\figref{punpongsanon2015softar}).
 
-Similarly but in VR, \textcite{degraen2019enhancing} combined visual textures with different passive haptic hair-like structure that were touched with the finger to induce a larger set of visuo-haptic materials perception.
+\begin{subfigs}{pseudo_haptic}{Pseudo-haptic feedback in \AR. }[
-\textcite{gunther2022smooth} studied in a complementary way how the visual rendering of a \VO touching the arm with a tangible object influenced the perception of roughness.
+        \item A virtual soft texture projected on a table and that deforms when pressed by the hand~\cite{punpongsanon2015softar}.
-A common finding of these studies is that haptic sensations seem to dominate the perception of roughness, suggesting that a smaller set of haptic textures can support a larger set of visual textures.
+        \item Modifying visually a tangible object and the hand touching it in \VST-\AR to modify its perceived shape \cite{ban2014displaying}.
-When performing a precision grasp (\secref{grasp_types}) on a tangible in \VR, some discrepancy in spatial properties between a tangible and a \VO is not noticeable by users: it took a relative difference of \percent{6} for the object width, \percent{44} for the surface orientation, and \percent{67} for the surface curvature to be perceived~\cite{detinguy2019how}.
+    ]
-
+    \subfigsheight{42mm}
-
-\subsubsection{Pseudo-Haptic Feedback}
-\label{pseudo_haptic}
-
-Le modèle \MLE implique également que la perception peut être croisée entre les modalités sensorielles: une perception haptique peut être volontairement influencé par un stimuli virtuel visuel, quand la vision est dite "dominante" dans la perception.
-Quand il est employé avec un \VE, ce phénomène est appelé \emph{pseudo-haptic feedback}~\cite{ujitoko2021survey}.
-
-See \textcite{ujitoko2021survey} for a detailed survey.
-
-% Visual feedback in VR and AR is known to influence haptic perception [13]. The phenomenon of ”visual dominance” was notably observed when estimating the stiffness of \VOs. L´ecuyer et al. [13] based their ”pseudo-haptic feedback” approach on this notion of visual dominance gaffary2017ar
-
-A few works have also used pseudo-haptic feedback to change the perception of haptic stimuli to create richer feedback by deforming the visual representation of a user input~\cite{ujitoko2021survey}.
-the perceived softness of tangible objects can be altered by superimposing in AR a virtual texture that deforms when pressed by the hand~\cite{punpongsanon2015softar}, or in combination with vibrotactile rendering in VR~\cite{choi2021augmenting}.
-
-\cite{ban2012modifying,ban2014displaying}
-
-\cite{costes2019touchy,ujitoko2019presenting,ota2020surface}
-
-The vibrotactile sinusoidal rendering of virtual texture cited above was also combined with visual oscillations of a cursor on a screen to increase the roughness perception of the texture~\cite{ujitoko2019modulating}.
-However, the visual representation was a virtual cursor seen on a screen while the haptic feedback was felt with a hand-held device.
-Conversely, as discussed by \textcite{ujitoko2021survey} in their review, a co-localised visuo-haptic rendering can cause the user to notice the mismatch between their real movements and the visuo-haptic feedback.
-
-Even before manipulating a visual representation to induce a haptic sensation, shifts and latencies between user input and co-localised visuo-haptic feedback can be experienced differently in AR and VR, which we aim to investigate in this work.
-
-\begin{subfigs}{pseudo_haptic}{Pseudo haptic. }
-    \subfigsheight{35mm}
     \subfig{punpongsanon2015softar}
     \subfig{ban2014displaying}
-    \subfig{ota2020surface}
 \end{subfigs}
 
+In all these studies, the visual expectations of participants influenced their haptic perception.
+In particular in \AR and \VR, the perception of a haptic rendering or augmentation can be influenced by the visual rendering of the \VO.
 
 \subsubsection{Perception of Visuo-Haptic Rendering in AR and VR}
 \label{AR_vs_VR}
 
-Some studies have investigated the visuo-haptic perception of \VOs in \AR and \VR with grounded force-feedback devices.
+Some studies have investigated in \AR and \VR the visuo-haptic perception of \VOs rendered with force-feedback and vibrotactile feedback.
 
 In an immersive \VST-\AR setup, \textcite{knorlein2009influence} rendered a virtual piston using force-feedback haptics that participants pressed directly with their hand (\figref{visuo-haptic-stiffness}).
 In a \TIFC task (\secref{sensations_perception}), participants pressed two pistons and indicated which was stiffer.
@@ -159,12 +138,12 @@ This suggests that the haptic stiffness of \VOs feels \enquote{softer} in an \AE
     \subfig[0.3]{gaffary2017ar_4}
 \end{subfigs}
 
-Finally, \textcite{diluca2019perceptual} investigated perceived simultaneity of visuo-haptic feedback in \VR.
+Finally, \textcite{diluca2019perceptual} investigated perceived simultaneity of visuo-haptic contact with a \VO in \VR.
-In a user study, participants touched a virtual cube with a virtual hand: The contact was both rendered with a vibrotactile piezo-electric device on the fingertip and a visual change in the cube color.
+The contact was both rendered with a vibrotactile piezo-electric device on the fingertip and a visual change in the \VO color.
-The visuo-haptic simultaneity varied by either adding a visual delay or by triggering earlier the haptic feedback.
+But the visuo-haptic simultaneity varied by either adding a visual delay or by triggering earlier the haptic feedback.
 No participant (out of 19) was able to detect a \qty{50}{\ms} visual lag and a \qty{15}{\ms} haptic lead and only half of them detected a \qty{100}{\ms} visual lag and a \qty{70}{\ms} haptic lead.
 
-They have shown how the latency of the visual rendering of a \VO or the type of environment (\VE or \RE) can affect the perceived haptic stiffness of the object, rendered with a grounded force-feedback device.
+These studies have shown how the latency of the visual rendering of a \VO or the type of environment (\VE or \RE) can affect the perceived haptic stiffness of the object, rendered with a grounded force-feedback device.
 We describe in the next section how wearable haptics have been integrated with immersive \AR.
 
 
@@ -172,7 +151,7 @@ We describe in the next section how wearable haptics have been integrated with i
 \label{vhar_haptics}
 
 A few wearable haptic devices have been specifically designed or experimentally tested for direct hand interaction in immersive \AR.
-The main challenge of wearable haptics for \AR is to provide haptic sensations of virtual or augmented objects that are touched and manipulated directly with the fingers while keeping the fingertips free to interact with the \RE.
+As virtual or augmented objects are naturally touched, grasped and manipulated directly with the fingertips (\secref{exploratory_procedures} and \secref{grasp_types}), the main challenge of wearable haptics for \AR is to provide haptic sensations of these interactions while keeping the fingertips free to interact with the \RE.
 Several approaches have been proposed to move the actuator away to another location on the hand.
 Yet, they differ greatly in the actuators used (\secref{wearable_haptic_devices}) thus the haptic feedback (\secref{tactile_rendering}), and the placement of the haptic rendering.
 
@@ -183,10 +162,7 @@ Another category of actuators relies on systems that cannot be considered as por
 \subsubsection{Nail-Mounted Devices}
 \label{vhar_nails}
 
-\textcite{ando2007fingernailmounted} were the first to propose this approach that they experimented with a voice-coil mounted on the index nail (\figref{ando2007fingernailmounted}).
+\textcite{ando2007fingernailmounted} were the first to propose to move away the actuator from the fingertip, as described in \secref{texture_rendering}.
-The sensation of crossing edges of a virtual patterned texture (\secref{texture_rendering}) on a real sheet of paper were rendered with \qty{20}{\ms} vibration impulses at \qty{130}{\Hz}.
-Participants were able to match the virtual patterns to their real counterparts of height \qty{0.25}{\mm} and width \qtyrange{1}{10}{\mm}, but systematically overestimated the virtual width to be \qty{4}{\mm} longer.
-
 This approach was later extended by \textcite{teng2021touch} with Touch\&Fold, a haptic device mounted on the nail but able to unfold its end-effector on demand to make contact with the fingertip when touching \VOs (\figref{teng2021touch}).
 This moving platform also contains a \LRA (\secref{moving_platforms}) and provides contact pressure (\qty{0.34}{\N} force) and texture (\qtyrange{150}{190}{\Hz} bandwidth) sensations.
 %The whole system is very compact (\qtyproduct{24 x 24 x 41}{\mm}), lightweight (\qty{9.5}{\g}), and fully portable by including a battery and Bluetooth wireless communication. \qty{20}{\ms} for the Bluetooth
@@ -196,22 +172,27 @@ Still, there is a high (\qty{92}{\ms}) latency for the folding mechanism and thi
 % teng2021touch: (5.27+3.03+5.23+5.5+5.47)/5 = 4.9
 % ando2007fingernailmounted: (2.4+2.63+3.63+2.57+3.2)/5 = 2.9
 
-To always keep the fingertip, \textcite{maeda2022fingeret} with Fingeret proposed to adapt the belt actuators (\secref{belt_actuators}) to design a \enquote{finger-side actuator} instead (\figref{maeda2022fingeret}).
+With Fingeret, \textcite{maeda2022fingeret} adapted the belt actuators (\secref{belt_actuators}) as a \enquote{finger-side actuator} that lets the fingertip free (\figref{maeda2022fingeret}).
-Mounted on the nail, the device actuates two rollers, one on each side of the fingertip, to deform the skin: When the rollers both rotate inwards (towards the pad) they pull the skin, simulating a contact sensation, and when they both rotate outwards (towards the nail) they push the skin, simulating a release sensation.
+Two rollers, one per side, can deform the skin: When rotating inwards, they pull the skin, simulating a contact sensation, and when rotating outwards, they push the skin, simulating a release sensation.
-By doing quick rotations, the rollers can also simulate a texture sensation.
+By doing quick in and out rotations, they can also simulate a texture sensation.
 %The device is also very compact (\qty{60 x 25 x 36}{\mm}), lightweight (\qty{18}{\g}), and portable with a battery and Bluetooth wireless communication with \qty{83}{\ms} latency.
-In a user study not in \AR, but involving touching different images on a tablet, Fingeret was found to be more realistic (4/7) than a \LRA at \qty{100}{\Hz} on the nail (3/7) for rendering buttons and a patterned texture (\secref{texture_rendering}), but not different from vibrations for rendering high-frequency textures (3.5/7 for both).
+In a user study not in \AR, but directly touching images on a tablet, Fingeret was found to be more realistic (4/7) than a \LRA at \qty{100}{\Hz} on the nail (3/7) for rendering buttons and a patterned texture (\secref{texture_rendering}), but not different from vibrations for rendering high-frequency textures (3.5/7 for both).
 However, as for \textcite{teng2021touch}, finger speed was not taken into account for rendering vibrations, which may have been detrimental to texture perception (\secref{texture_rendering}).
 
+Finally, \textcite{preechayasomboon2021haplets} (\figref{preechayasomboon2021haplets}) and \textcite{sabnis2023haptic} designed Haplets and Haptic Servo, respectively: They are very compact and lightweight vibrotactile \LRA devices designed to feature both integrated sensing of the finger movements and very latency haptic feedback (\qty{<5}{ms}).
+But no proper user study were conducted to evaluate these devices in \AR.
+
 \begin{subfigs}{ar_wearable}{Nail-mounted wearable haptic devices designed for \AR. }[
-        \item A voice-coil rendering a virtual haptic texture on a real sheet of paper~\cite{ando2007fingernailmounted}.
+        %\item A voice-coil rendering a virtual haptic texture on a real sheet of paper~\cite{ando2007fingernailmounted}.
         \item Touch\&Fold provide contact pressure and vibrations on demand to the fingertip~\cite{teng2021touch}.
         \item Fingeret is a finger-side wearable haptic device that pulls and pushs the fingertip skin~\cite{maeda2022fingeret}.
+        \item Haplets is a very compact nail device with integrated sensing and vibrotactile feedback~\cite{preechayasomboon2021haplets}.
     ]
     \subfigsheight{33mm}
-    \subfig{ando2007fingernailmounted}
+    %\subfig{ando2007fingernailmounted}
     \subfig{teng2021touch}
     \subfig{maeda2022fingeret}
+    \subfig{preechayasomboon2021haplets}
 \end{subfigs}
 
 
@@ -219,6 +200,7 @@ However, as for \textcite{teng2021touch}, finger speed was not taken into accoun
 \label{vhar_rings}
 
 The haptic ring belt devices of \textcite{minamizawa2007gravity} and \textcite{pacchierotti2016hring}, presented in \secref{belt_actuators}, have been employed to improve the manipulation of \VOs in \AR (\secref{ar_interaction}).
+Recall that these devices have also been used to modify the perceived stiffness, softness, friction and localized bumps and holes on smooth real surfaces (\secref{hardness_rendering}) \cite{detinguy2018enhancing,salazar2020altering}, but have not been tested in \AR.
 
 In a \VST-\AR setup, \textcite{scheggi2010shape} explored the effect of rendering the weight of a virtual cube placed on a real surface hold with the thumb, index, and middle fingers (\figref{scheggi2010shape}).
 The middle phalanx of each of these fingers was equipped with a haptic ring of \textcite{minamizawa2007gravity}.
@@ -268,7 +250,7 @@ A short vibration (\qty{25}{\ms} \qty{175}{\Hz} square-wave) was also rendered w
     \subfig{pezent2019tasbi_4}
 \end{subfigs}
 
-\cite{sarac2022perceived,palmer2022haptic} not in AR but studies on relocating to the wrist the haptic feedback of the fingertip-object contacts.
+% \cite{sarac2022perceived,palmer2022haptic} not in AR but studies on relocating to the wrist the haptic feedback of the fingertip-object contacts.
 
 %\subsection{Conclusion}
 %\label{visuo_haptic_conclusion}

2-perception/xr-perception/1-introduction.tex

@@ -11,6 +11,8 @@
     Participants explored this haptic roughness augmentation with (Real) their real hand alone, (Mixed) a realistic virtual hand overlay in AR, and (Virtual) the same virtual hand in VR.
 }
 
+% Even before manipulating a visual representation to induce a haptic sensation, shifts and latencies between user input and co-localised visuo-haptic feedback can be experienced differently in AR and VR, which we aim to investigate in this work.
+
 %Imagine you're an archaeologist or in a museum, and you want to examine an ancient object.
 %
 %But it is too fragile to touch directly.

references.bib

@@ -171,6 +171,17 @@
   doi = {10/gnx4k5}
 }
 
+@article{ban2013modifying,
+  title = {Modifying {{Perceived Size}} of a {{Handled Object}} through {{Hand Image Deformation}}},
+  author = {Ban, Yuki and Narumi, Takuji and Tanikawa, Tomohiro and Hirose, Michitaka},
+  date = {2013},
+  journaltitle = {Presence Teleoperators Virtual Environ.},
+  volume = {22},
+  number = {3},
+  pages = {255--270},
+  doi = {10/f5t28x}
+}
+
 @inproceedings{ban2014displaying,
   title = {Displaying Shapes with Various Types of Surfaces Using Visuo-Haptic Interaction},
   booktitle = {{{ACM Symp}}. {{Virtual Real}}. {{Softw}}. {{Technol}}.},
@@ -596,6 +607,15 @@
   doi = {10/f4zvmf}
 }
 
+@inproceedings{campion2005fundamental,
+  title = {Fundamental {{Limits}} in the {{Rendering}} of {{Virtual Haptic Textures}}},
+  booktitle = {First {{Jt}}. {{Eurohaptics Conf}}. {{Symp}}. {{Haptic Interfaces Virtual Environ}}. {{Teleoperator Syst}}.},
+  author = {Campion, G. and Hayward, V.},
+  date = {2005},
+  pages = {263--270},
+  doi = {10/b5hzfn}
+}
+
 @inproceedings{canales2019virtual,
   title = {Virtual {{Grasping Feedback}} and {{Virtual Hand Ownership}}},
   booktitle = {{{ACM Symp}}. {{Appl}}. {{Percept}}.},
@@ -1139,6 +1159,15 @@
   doi = {10/gm5pxb}
 }
 
+@inproceedings{gabardi2018development,
+  title = {Development of a Miniaturized Thermal Module Designed for Integration in a Wearable Haptic Device},
+  booktitle = {{{IEEE Haptics Symp}}.},
+  author = {Gabardi, Massimiliano and Leonardis, Daniele and Solazzi, Massimiliano and Frisoli, Antonio},
+  date = {2018},
+  pages = {100--105},
+  doi = {10/g3qf24}
+}
+
 @article{gaffary2017ar,
   title = {{{AR Feels}} “{{Softer}}” than {{VR}}: {{Haptic Perception}} of {{Stiffness}} in {{Augmented}} versus {{Virtual Reality}}},
   shorttitle = {{{AR Feels}} “{{Softer}}” than {{VR}}},
@@ -2538,6 +2567,17 @@
   doi = {10/dfqxs9}
 }
 
+@article{okamura2001realitybased,
+  title = {Reality-Based Models for Vibration Feedback in Virtual Environments},
+  author = {Okamura, Allison M. and Cutkosky, Mark R. and Dennerlein, Jack T.},
+  date = {2001},
+  journaltitle = {IEEEASME Trans. Mechatron.},
+  volume = {6},
+  number = {3},
+  pages = {245--252},
+  doi = {10/fnjt3h}
+}
+
 @incollection{ota2020surface,
   title = {Surface {{Roughness Judgment During Finger Exploration Is Changeable}} by {{Visual Oscillations}}},
   booktitle = {Haptics: {{Science}}, {{Technology}}, {{Applications}}},
@@ -2793,6 +2833,16 @@
   doi = {10/f4dbwv}
 }
 
+@article{preechayasomboon2021haplets,
+  title = {Haplets: {{Finger-Worn Wireless}} and {{Low-Encumbrance Vibrotactile Haptic Feedback}} for {{Virtual}} and {{Augmented Reality}}},
+  shorttitle = {Haplets},
+  author = {Preechayasomboon, Pornthep and Rombokas, Eric},
+  date = {2021},
+  journaltitle = {Front. Virtual Real.},
+  volume = {2},
+  doi = {10/grq5g9}
+}
+
 @article{pressman2007perception,
   title = {Perception of {{Delayed Stiffness}}},
   author = {Pressman, Assaf and Welty, Leah J. and Karniel, Amir and Mussa-Ivaldi, Ferdinando A.},