erwan/phd-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP

Browse Source
This commit is contained in:
Erwan Normand
2024-09-21 22:57:47 +02:00
parent 6daf361654
commit 363108d03e
16 changed files with 188 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files

150
1-introduction/related-work/2-wearable-haptics.tex
View File

@@ -1,7 +1,7 @@
\section{Augmenting Object Perception with Wearable Haptics}
\label{wearable_haptics}
One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in \v-\VE~\cite{maclean2008it,culbertson2018haptics}.
One of the roles of haptic systems is to render virtual interactions and sensations that are \emph{similar and comparable} to those experienced by the haptic sense with real objects, particularly in a visual \VE~\cite{maclean2008it,culbertson2018haptics}.
Moreover, a haptic augmentation system should \enquote{modulating the feel of a real object by virtual [haptic] feedback}~\cite{jeon2009haptic}, \ie a touch interaction with a real object whose perception is modified by the addition of virtual haptic feedback.
The haptic system should be hand-held or worn, \eg on the hand, and \enquote{not permanently attached to or integrated in the object}~\cite{bhatia2024augmenting}.
@@ -9,9 +9,9 @@ The haptic system should be hand-held or worn, \eg on the hand, and \enquote{not
\subsection{Level of Wearability}
\label{wearability_level}
Different types of haptic devices can be worn on the hand, but only some of them can be considered \emph{wearable}.
Different types of haptic devices can be worn on the hand, but only some of them can be considered wearable.
\textcite{pacchierotti2017wearable} classify them into three levels of wearability, as illustrated in the \figref{pacchierotti2017wearable}.
An increasing \emph{wearability} resulting in the loss of the system's kinesthetic feedback capability.
An increasing wearability resulting in the loss of the system's kinesthetic feedback capability.
\begin{subfigs}{pacchierotti2017wearable}{
Schematic wearability level of haptic devices for the hand~\cite{pacchierotti2017wearable}.
@@ -26,17 +26,17 @@ An increasing \emph{wearability} resulting in the loss of the system's kinesthet
\subfig{pacchierotti2017wearable_3}
\end{subfigs}
Haptic research comes from robotics and teleoperation, and historically led to the design of haptic systems that are \emph{grounded} to an external support in the environment, such as a table (\figref{pacchierotti2017wearable_1}).
Haptic research comes from robotics and teleoperation, and historically led to the design of haptic systems that are \emph{world-grounded} to an external support in the environment, such as a table (\figref{pacchierotti2017wearable_1}).
These are robotic arms whose end-effector is either held in the hand or worn on a finger and which simulate interactions with a \VE by providing kinesthetic forces and torques feedback (\figref{pacchierotti2015cutaneous}).
They provide high fidelity haptic feedback but are heavy, bulky and limited to small workspaces~\cite{culbertson2018haptics}.
More portable designs have been developed by moving the grounded part to the user's body.
The entire robotic system is thus mounted on the user, forming an exoskeleton capable of providing kinesthetic feedback to the finger, \eg in \figref{achibet2017flexifingers}.
However, it cannot constrain the movements of the wrist and the reaction force is transmitted to the user where the device is grounded (\figref{pacchierotti2017wearable_2}).
They are often heavy and bulky and cannot be considered wearable.
Such \emph{body-grounded} devices are often heavy and bulky and cannot be considered wearable.
\textcite{pacchierotti2017wearable} defined that : \enquote{A wearable haptic interface should also be small, easy to carry, comfortable, and it should not impair the motion of the wearer}.
An approach is then to move the grounding point very close to the end-effector (\figref{pacchierotti2017wearable_3}): the interface is limited to cutaneous haptic feedback, but its design is more compact, lightweight and comfortable, \eg in \figref{leonardis20173rsr}, and the system is wearable.
An approach is then to move the grounding point very close to the end-effector (\figref{pacchierotti2017wearable_3}): the interface is limited to cutaneous haptic feedback, but its design is more compact, lightweight, comfortable and portable, \eg in \figref{grounded_to_wearable}.
Moreover, as detailed in \secref{object_properties}, cutaneous sensations are necessary and often sufficient for the perception of the haptic properties of an object explored with the hand, as also argued by \textcite{pacchierotti2017wearable}.
\begin{subfigs}{grounded_to_wearable}{
@@ -130,33 +130,35 @@ The simplicity of this approach allows the belt to be placed anywhere on the han
Vibrotactile actuators are the most common and simplest wearable haptic interfaces, and are available as consumer products.
They are small, lightweight and can be placed directly on any part of the hand.
\textcite{choi2013vibrotactile} provide a detailed review.
\textcite{choi2013vibrotactile} provided a detailed review.
All vibrotactile actuators are based on the same principle: generating an oscillating motion from an electric current with a frequency and amplitude high enough to be perceived by cutaneous mechanoreceptors.
Several types of vibrotactile actuators are used in haptics, with different trade-offs between size, proposed \DoFs and application constraints:
\begin{itemize}
\item An \ERM is a \DC motor that rotates an off-center mass when a voltage or current is applied (\figref{precisionmicrodrives_erm}). \ERMs are easy to control, inexpensive and can be encapsulated in a few millimeters cylinder or coin form factor. However, they have only one \DoF because both the frequency and amplitude of the vibration are coupled to the speed of the rotation, \eg low (high) frequencies output at low (high) amplitudes, as shown on \figref{precisionmicrodrives_erm_performances}.
\item A \LRA consists of a coil that creates a magnetic field from an \AC to oscillate a magnet attached to a spring, as an audio loudspeaker (\figref{precisionmicrodrives_lra}). They are more complex to control and a bit larger than \ERMs. Each \LRA is designed to vibrate with maximum amplitude at a given frequency, but won't vibrate efficiently at other frequencies, \ie their bandwidth is narrow, as shown on \figref{azadi2014vibrotactile}.
\item A \VCA is a \LRA but capable of generating vibration at two \DoF, with an independent control of the frequency and amplitude of the vibration on a wide bandwidth. They are larger in size than \ERMs and \LRAs, but can generate more complex renderings.
\item Piezoelectric actuators deform a solid material when a voltage is applied. They are very small and thin, and allow two \DoFs of amplitude and frequency control. However, they require high voltages to operate thus limiting their use in wearable devices.
\end{itemize}
Several types of vibrotactile actuators are used in haptics, with different trade-offs between size, proposed \DoFs and application constraints.
\begin{subfigs}{vibrotactile_actuators}{Diagrams of vibrotactile acuators. }[
\item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives.~\footnotemark
\item Diagram of a \LRA. From Precision Microdrives.~\footnotemarkrepeat
An \ERM is a \DC motor that rotates an off-center mass when a voltage or current is applied (\figref{precisionmicrodrives_erm}). \ERMs are easy to control, inexpensive and can be encapsulated in a few millimeters cylinder or coin form factor. However, they have only one \DoF because both the frequency and amplitude of the vibration are coupled to the speed of the rotation, \eg low (high) frequencies output at low (high) amplitudes, as shown on \figref{precisionmicrodrives_erm_performances}.
\begin{subfigs}{erm}{Diagram and performance of \ERMs. }[
\item Diagram of a cylindrical encapsulated \ERM. From Precision Microdrives~\footnotemark.
\item Amplitude and frequency output of an \ERM as a function of the input voltage.
]
\subfigsheight{50mm}
\subfigsheight{45mm}
\subfig{precisionmicrodrives_erm}
\subfig{precisionmicrodrives_lra}
\subfig{precisionmicrodrives_erm_performances}
\end{subfigs}
\footnotetext{\url{https://www.precisionmicrodrives.com/}}
\begin{subfigs}{vibrotactile_performances}{Performances of vibrotactile acuators. }[
\item Amplitude and frequency output of an \ERM as a function of the input voltage. From Precision Microdrives.~\footnotemarkrepeat
\item Force generated by two \LRAs as a function of sine wave input with different frequencies: both their maximum force and frequency are different~\cite{azadi2014vibrotactile}.
A \LRA consists of a coil that creates a magnetic field from an \AC to oscillate a magnet attached to a spring, as an audio loudspeaker (\figref{precisionmicrodrives_lra}). They are more complex to control and a bit larger than \ERMs. Each \LRA is designed to vibrate with maximum amplitude at a given resonant frequency, but won't vibrate efficiently at other frequencies, \ie their bandwidth is narrow, as shown on \figref{azadi2014vibrotactile}.
A \VCA is a \LRA but capable of generating vibration at two \DoF, with an independent control of the frequency and amplitude of the vibration on a wide bandwidth. They are larger in size than \ERMs and \LRAs, but can generate more complex renderings.
Piezoelectric actuators deform a solid material when a voltage is applied. They are very small and thin, and allow two \DoFs of amplitude and frequency control. However, they require high voltages to operate thus limiting their use in wearable devices.
\begin{subfigs}{lra}{Diagram and performance of \LRAs. }[
\item Diagram. From Precision Microdrives~\footnotemarkrepeat.
\item Force generated by two \LRAs as a function of sine wave input with different frequencies: both their maximum force and resonant frequency are different~\cite{azadi2014vibrotactile}.
]
\subfig[.58]{precisionmicrodrives_erm_performances}
\subfig[.38]{azadi2014vibrotactile}
\subfigsheight{50mm}
\subfig{precisionmicrodrives_lra}
\subfig{azadi2014vibrotactile}
\end{subfigs}
@@ -164,7 +166,7 @@ Several types of vibrotactile actuators are used in haptics, with different trad
\label{tactile_rendering}
Tactile rendering of haptic properties consists in modelling and reproducing virtual tactile sensations comparable to those perceived when interacting with real objects~\cite{klatzky2013haptic}.
By adding such tactile rendering as feedback to the touch actions of the hand on a real object~\cite{bhatia2024augmenting}, the perception of the object's haptic property is modified.
By adding such tactile rendering as feedback to the touch actions of the hand on a real object~\cite{bhatia2024augmenting}, the perception of the object's haptic property can be modified.
The integration of the real and virtual haptic sensations into a single property perception is discussed in more details in \secref{sensations_perception}.
%, both the real and virtual haptic sensations are integrated into a single property perception, as presented in \secref{sensations_perception}, \ie the perceived haptic property is modulated by the added virtual feedback.
In particular, the visual rendering of a touched object can also greatly influence the perception of its haptic properties, \eg by modifying its visual texture in \AR or \VR, as discussed in the \secref{visuo_haptic}.
@@ -176,39 +178,50 @@ In touch-through and tool-mediated, or \emph{indirect feel-through}, the haptic
Many haptic augmentations were first developed with grounded haptic devices and later transposed to wearable haptic devices.
%We also focus on tactile augmentations stimulating the mechanoreceptors of the skin (\secref{haptic_sense}), thus excluding temperature perception, as they are the most common existing haptic interfaces.
As we chose in \secref{object_properties} to focus on the haptic perception of the roughness and hardness of objects, we present bellow the methods to modify these properties with wearable haptic devices.
Of course, wearable haptics can also be used to modify the perceived friction \cite{konyo2008alternative,salazar2020altering}, weight \cite{minamizawa2007gravity}, or local deformation \cite{salazar2020altering} of real objects, but they are less common \cite{bhatia2024augmenting} and will not be detailed here.
% \cite{klatzky2003feeling} : rendering roughness, friction, deformation, temperatures
% \cite{girard2016haptip} : renderings with a tangential motion actuator
\subsubsection{Textures}
\subsubsection{Roughness}
\label{texture_rendering}
Several approaches have been proposed to render virtual haptic texture~\cite{culbertson2018haptics}.
As more traditional force feedback systems are unable to accurately render such micro-details on a simulated surface, vibrotactile devices attached to the end effector instead generate vibrations to simulate interaction with the virtual texture~\cite{campion2005fundamental,culbertson2018haptics}.
In this way, physics-based models~\cite{chan2021hasti,okamura1998vibration,guruswamy2011iir} and data-based models~\cite{culbertson2015should,romano2010automatic} have been developed and evaluated, the former being simpler but more approximate to real textures, and the latter being more realistic but limited to the captured textures.
To modify the perception of haptic roughness (or texture, see \secref{roughness}) of a real object, vibrations are provided to the skin by the wearable haptic device when running the finger over the surface.
Two approaches are used to render virtual textures: \emph{physics-based models} and \emph{data-driven models}~\cite{culbertson2018haptics}.
\paragraph{Physics-based Models}
As more traditional force feedback systems are unable to accurately render such micro-details on a simulated surface, vibrotactile devices attached to the end effector instead generate vibrations to simulate interaction with the virtual texture~\cite{campion2005fundamental,culbertson2018haptics}.
High-fidelity force feedback devices can reproduce patterned textures with great precision and provide similar perceptions to real textures, but they are expensive, have a limited workspace, and impose to hold a probe to explore the texture~\cite{unger2011roughness}.
Notably, \textcite{okamura1998vibration} rendered grating textures with exponentially decaying sinudoids that simulated the strokes of the grooves and ridges of the surface, while \textcite{culbertson2014modeling} captured and modelled the roughness of real surfaces to render them using the speed and force of the user.
\cite{chan2021hasti,guruswamy2011iir}
\paragraph{Data-driven Models}
An effective approach to rendering virtual roughness is to generate vibrations to simulate interaction with the virtual texture~\cite{culbertson2018haptics}, relying on the user's real-time measurements of position, velocity and force to modulate the frequencies and amplitudes of the vibrations, with position and velocity being the most important parameters~\cite{culbertson2015should}.
For example, when comparing the same virtual texture pairwise, but with different parameters, \textcite{culbertson2015should} showed that the roughness vibrations generated should vary with user speed, but not necessarily with user force.
Virtual data-driven textures were perceived as similar to real textures, except for friction, which was not rendered properly.
\textcite{ando2007fingernailmounted} were the first to propose this approach that they experimented with a voice-coil mounted on the index nail (\figref{ando2007fingernailmounted}).
The sensation of crossing edges of a virtual patterned texture (\secref{texture_rendering}) on a real sheet of paper were rendered with \qty{20}{\ms} vibration impulses at \qty{130}{\Hz}.
Participants were able to match the virtual patterns to their real counterparts of height \qty{0.25}{\mm} and width \qtyrange{1}{10}{\mm}, but systematically overestimated the virtual width to be \qty{4}{\mm} longer.
The perceived roughness of real surfaces can be then modified when touched by a tool with a vibrotactile actuator attached~\cite{culbertson2014modeling,ujitoko2019modulating} or directly with the finger wearing the vibrotactile actuator~\cite{asano2015vibrotactile}, creating a haptic texture augmentation.
Of course, the fingertip skin is not deformed by the virtual texture and only vibrations are felt, but it has been shown that the vibrations produced on the fingertip skin running over a real surface are texture specific and similar between individuals~\cite{delhaye2012textureinduced,manfredi2014natural}.
A common method vibrotactile rendering of texture is to use a sinusoidal signal whose frequency is modulated by the finger position or velocity~\cite{asano2015vibrotactile,friesen2024perceived,strohmeier2017generating,ujitoko2019modulating}.
\paragraph{Data-driven Models}
Because physics-based models to render realistic textures can be very complex to design and to render in real-time, direct capture of real textures have been used instead to model the produced vibrations~\cite{culbertson2018haptics}.
An effective approach to rendering virtual roughness is to generate vibrations to simulate interaction with the virtual texture~\cite{culbertson2018haptics}, relying on the user's real-time measurements of position, velocity and force to modulate the frequencies and amplitudes of the vibrations, with position and velocity being the most important parameters~\cite{culbertson2015should}.
For example, when comparing the same virtual texture pairwise, but with different parameters, \textcite{culbertson2015should} showed that the roughness vibrations generated should vary with user speed, but not necessarily with user force.
Virtual data-driven textures were perceived as similar to real textures, except for friction, which was not rendered properly.
\subsubsection{Hardness}
\label{hardness_rendering}
The two main approaches to modulate the perceived hardness of a real surface with wearable haptics are to render forces or vibrations.
The perceived hardness (\secref{hardness}) of a real surface can be modified by rendering forces or vibrations.
\paragraph{Modulating Forces}
When tapping or pressing a real object with a tool, the perceived stiffness $\tilde{k}$ (\secref{hardness}) of its surface can be modulated with force feedback~\cite{jeon2015haptic}.
When tapping or pressing a real object, the perceived stiffness $\tilde{k}$ of its surface can be modulated with force feedback~\cite{jeon2015haptic}.
This was first proposed by \textcite{jeon2008modulating} who augmented a real surface tapped in 1 \DoF with a grounded force-feedback device held in hand (\figref{jeon2009haptic_1}).
When the haptic end-effector contacts the object at time $t$, the object's surface deforms by displacement $x_r(t)$ and opposes a real reaction force $f_r(t)$.
The virtual force of the device $\tilde{f_r}(t)$ is then controlled to:
@@ -220,38 +233,67 @@ The displacement $x_r(t)$ is estimated with the reaction force and tapping veloc
As shown in \figref{jeon2009haptic_2}, the force $\tilde{f_r}(t)$ perceived by the user being modulated, but not the displacement $x_r(t)$, the perceived stiffness is $\tilde{k}(t)$.
This stiffness augmentation technique was then extended to enable tapping and pressing with 3 \DoFs~\cite{jeon2010stiffness}, to render friction and weight augmentations~\cite{jeon2011extensions}, and to grasping and squeezing the real object with two contact points~\cite{jeon2012extending}.
\begin{subfigs}{stiffness_rendering}{Augmenting perceived stiffness of a real surface. }[%
\item Diagram of a user tapping a real surface with a hand-held force-feedback device~\cite{jeon2009haptic}.
\item Displacement-force curves of a real rubber ball (dashed line) and when its perceived stiffness $\tilde{k}$ is modulated~\cite{jeon2009haptic}.
\begin{subfigs}{stiffness_rendering_grounded}{Augmenting the perceived stiffness of a real surface with a hand-held force-feedback device. }[%
\item Diagram of a user tapping the surface \cite{jeon2009haptic}.
\item Displacement-force curves of a real rubber ball (dashed line) and when its perceived stiffness $\tilde{k}$ is modulated \cite{jeon2009haptic}.
]
\subfigsheight{35mm}
\subfig[0.2]{jeon2009haptic_1}
\subfig[0.4]{jeon2009haptic_2}
\subfig[0.38]{jeon2009haptic_1}
\subfig[0.42]{jeon2009haptic_2}
\end{subfigs}
\cite{detinguy2018enhancing}
\cite{salazar2020altering}
\textcite{detinguy2018enhancing} transposed this stiffness augmentation technique with the hRing device (\secref{belt_actuators}): While pressing a real piston with the fingertip by displacement $x_r(t)$, the belt compressed the finger by a virtual force $\tilde{k}\,x_r(t)$ where $\tilde{k}$ is the added stiffness (\eqref{stiffness_augmentation}), increasing the perceived stiffness of the piston (\figref{detinguy2018enhancing}).
%This enables to \emph{increase} the perceived stiffness of the real piston up to \percent{+14}.
More importantly, the augmentation proved to be robust to the placement of the device, as the increased stiffness was perceived the same on the fingertip, the middle phalanx and the proximal.
Conversely, the technique allowed to \emph{decrease} the perceived stiffness by compressing the phalanx prior the contact and diminish the belt pressure as the user pressed the piston~\cite{salazar2020altering}.
\textcite{tao2021altering} proposed instead to restrict the deformation of the fingerpad by pulling a hollow frame around it to decrease perceived stiffness (\figref{tao2021altering}): it augments the finger contact area thus the perceived Young modulus of the object (\secref{hardness}).
\cite{kildal20103dpress}
\cite{tao2021altering} % wearable softness
\begin{subfigs}{stiffness_rendering_wearable}{Modifying the perceived stiffness with wearable pressure devices. }[%
\item Modify the perceived stiffness of a piston by pressing the finger during or prior the contact \cite{detinguy2018enhancing,salazar2020altering}.
\item Decrease perceived stiffness of hard object by restricting the fingerpad deformation \cite{tao2021altering}.
]
\subfigsheight{35mm}
\subfig{detinguy2018enhancing}
\subfig{tao2021altering}
\end{subfigs}
\paragraph{Vibrations Augmentations}
The second main approach is to modulate the vibrations felt when tapping a real surface with a tool~\cite{okamura1998vibration}.
Tapping with a tool on a real surface augmented with a vibrotactile actuator generating exponential decaying sinusoids
\textcite{okamura2001realitybased} measured impact vibrations $Q(t)$ when tapping on real objects and found they can be modelled as exponential decaying sine wave:
\begin{equation}{contact_transient}
\tilde{f}_c(t) = a \, |v_{in}| \, e^{- \tau t} sin(2 \pi f t)
Q(t) = A \, |v_{in}| \, e^{- \tau t} sin(2 \pi f t)
\end{equation}
With $A$ the amplitude slope, $\tau$ the sine decay rate and $f$ the sine frequency, which are measured material properties, and $v_{in}$ the impact velocity.
It has been shown that these material properties perceptually express the stiffness (\secref{hardness}) of real~\cite{higashi2019hardness} and virtual surface~\cite{choi2021perceived}.
Therefore, when contacting or tapping a real object through an indirect feel-through interface that provide such vibrations (\figref{choi2021augmenting_control}) using a voice-coil (\secref{vibrotactile_actuators}), the perceived stiffness can be increased or decreased \cite{kuchenbecker2006improving,hachisu2012augmentation,choi2021augmenting}, \eg sponge feeling stiffer or wood feeling softer (\figref{choi2021augmenting_results}).
A challenge with this technique is to provide the vibration feedback at the right time, to be felt simultaneous with the real contact~\cite{park2023perceptual}.
\cite{kuchenbecker2006improving}
\cite{hachisu2012augmentation}
\cite{park2019realistic}
\cite{park2023perceptual}
\begin{subfigs}{contact_vibrations}{Augmenting perceived stiffness using vibrations when touching a real surface \cite{choi2021augmenting}. }[%
%\item Experimental setup with a voice-coil actuator attached to a touch-through interface.
\item Voltage inputs (top) to the voice-coil for soft, medium and hard vibrations, with the corresponding displacement (middle) and force (bottom) outputs of the actuator.
\item Perceived intensity of stiffness of real sponge ("Sp") and wood ("Wd") surfaces without added vibrations ("N") and modified by soft ("S"), medium ("M") and hard ("H") vibrations.
]
%\subfig[.15]{choi2021augmenting_demo}
\subfigsheight{50mm}
\subfig{choi2021augmenting_control}
\subfig{choi2021augmenting_results}
\end{subfigs}
Vibrations on contact have been employed with wearable haptics but, to the best of our knowledge, only to render \VOs~\cite{pezent2019tasbi,teng2021touch,sabnis2023haptic}.
We describe them in the \secref{vhar_haptics}.
%A promising alternative approach
%\cite{kildal20103dpress}
%\begin{subfigs}{vibtration_grains}{Augmenting perceived stiffness of a real surface with vibrations. }
% \subfigsheight{35mm}
% \subfig{sabnis2023haptic_device}
% \subfig{sabnis2023haptic_control}
%\end{subfigs}
%\textcite{choi2021perceived} combined and compared these two rendering approaches (spring-damper and exponential decaying sinusoids) but to render purely virtual surfaces.
%They found that the perceived intensity of the virtual hardness $\tilde{h}$ followed a power law, similarly to \eqref{hardness_intensity}, with the amplitude $a$, the %frequency $f$ and the damping $b$ of the vibration, but not the decay time $\tau$.
%\cite{park2023perceptual}
%\subsubsection{Friction}
%\label{friction_rendering}