136 lines
15 KiB
TeX
136 lines
15 KiB
TeX
\section{Related Work}
|
|
\label{related_work}
|
|
|
|
\subsection{Haptics in AR}
|
|
|
|
As in VR, the addition of haptic feedback in AR has been explored through numerous approaches, including %
|
|
grounded force feedback devices~\autocite{jeon2009haptic,knorlein2009influence,hachisu2012augmentation,gaffary2017ar}, %
|
|
exoskeletons~\autocite{lee2021wearable}, %
|
|
wearable haptic devices~\autocite{maisto2017evaluation,detinguy2018enhancing,lopes2018adding,meli2018combining,pezent2019tasbi,teng2021touch},
|
|
tangible objects~\autocite{punpongsanon2015softar,hettiarachchi2016annexing,kahl2021investigation}, and %
|
|
mid-air haptics~\autocite{ochiai2016crossfield}. %
|
|
%
|
|
Most have been used to provide haptic feedback to virtual objects.
|
|
%
|
|
While this may seem similar to haptic feedback in VR, there are significant differences in terms of perception, as in AR the real world and the hand of the user remain visible, but also because the virtual content may be less realistic or inconsistent with the real world~\autocite{kim2018revisiting,macedo2023occlusion}.
|
|
%
|
|
Indeed, the same haptic stimuli can be perceived differently in AR and VR, \eg the perceived stiffness of a piston seemed higher in AR than in VR~\autocite{gaffary2017ar} or was altered in the presence of a delay between the haptic and visual feedback~\autocite{knorlein2009influence}.
|
|
%
|
|
It might be therefore interesting to study how haptic and visual augmentations of textures of tangible surfaces are perceived in AR.
|
|
|
|
An additional challenge in AR is to let the hand of the user free to touch, feel, and interact with the real objects~\autocite{maisto2017evaluation,detinguy2018enhancing,teng2021touch}.
|
|
%
|
|
For example, mounted on the nail, the haptic device of \textcite{teng2021touch} can be quickly unfolded on demand to the fingertip to render haptic feedback of virtual objects.
|
|
%
|
|
It is however not suitable for rendering haptic feedback when touching real objects.
|
|
%
|
|
In this respect, some wearable haptic devices were specifically designed to provide haptic feedback about fingertip interactions with the virtual content, but delocalized elsewhere on the body: on the proximal finger phalanx with the hRing haptic ring device~\autocite{pacchierotti2016hring,ferro2023deconstructing}, on the wrist with the Tasbi bracelet~\autocite{pezent2019tasbi}, or on the arm~\autocite{lopes2018adding}.
|
|
%
|
|
Compared to a fingertip worn device, the hRing was even preferred by participants and perceived as more effective in virtual object manipulation task in AR~\autocite{maisto2017evaluation,meli2018combining}.
|
|
%
|
|
This device has been then taken further to alter cutaneous perception of touched tangible objects in VR and AR~\autocite{detinguy2018enhancing,salazar2020altering}: by providing normal and shear forces to the proximal phalanx skin in a timely manner, the perceived stiffness, softness, slipperiness, and local deformations (bumps and holes) of the touched tangible object were augmented.
|
|
%
|
|
However, wearable haptic devices have not yet been used in AR to modify the texture perception of a tangible surface.
|
|
|
|
% Jeon2009Haptic, Hachisu2012Augmentation, tapping : Using passive tangible objects provides the kinesthetic feedback because there is a real contact and can modulate the cutaneous stimuli using wearable or hand-held haptic device. Jeon2009Haptic altered the perceived stiffness of a real object when tapped with a grounded haptic system. "Haptic augmented reality (AR) enables the user to feel a real environment augmented with synthetic haptic stimuli". "The user can touch a real object, a virtual object, or a real object augmented with virtual touch". Imagine exploring a table a magic pen with an augmented feel of either "a paintbrush on a smooth piece of paper, or using a marker on a stiff white board".
|
|
|
|
% [@Knorlein2009Influence] studied the influence of visual and haptic delays in the perception of stiffness in AR. Users pressed a virtual stiff using a VST-AR and a force feedback haptic device. Like in previous results, haptic feedback delays caused smaller perceived stiffness. But visual feedback delays caused higher perceived stiffness. Both effect seemed compensated when combined. Integrated force feedback haptic into VST-AR to render stiffness on a virtual piston. While the perceived stiffness was decreased with haptic, it was incresed with visual delay.
|
|
|
|
% [@Hachisu2012Augmentation] proposed to replace the grounded haptic device with a hand-held stylus device. It uses a vibrotactile actuator The haptic stylus changes changes the perceived stiffness of a tapped real object: the original vibration is substracted, absorbed by an elastic and a vibrotactile actuator adds, renders the intended vibration.
|
|
|
|
% but grounded haptic grounded active force feedback device which is expensive, complex, and has a limited workspace.
|
|
|
|
%[@asano2015vibrotactile] augmented and diminished the roughness perception of a touched grating scale surface with vibrations of a voice-coil attached to the finger. Roughness was perceived stronger with small amplitudes, weaker with large amplitudes (saturated the skin sensors or masked the perception of the real surface). But was with a fixed speed and no visual augmentation.
|
|
|
|
%La rigidité est perçue différemment entre RA et RV [@gaffary2017ar].
|
|
|
|
%- evaluated wearable haptic for AR for both augment virtual and tangible objects [[maisto2017evaluation]] [[meli2018combining]]
|
|
%[@Maisto2017Evaluation] compared two wearable haptic devices in different AR applications. The 3-RRS fingertip device of [@Chinello2018Three] provides contact deformation to the fingertip. The hRing wearable finger device of [@Pacchierotti2016hRing] stretchs the skin and provides normal stimuli to the proximal phalanx.
|
|
%In a first experiment, participants wrote a word on a virtual board with a real chunk. As results, kinesthetic feedback provided by grounded haptics leaded to better performances with least errors, least force applied and was perceived as the most effective. There were no difference of error, force and distance between the two wearable feedback conditions but still better than than visual condition and no feedback condition. However hRing was preferred to 3RRS, probably because it renders better the interaction force between the chalk and the whiteboard.
|
|
%Second experiment was to move two real objects to a position and two virtual objects to a target. There were no time differences between the two wearable haptic devices despite their very different type of feedback (tilting platform on fingertip vs shear and normal stimuli on proximal phalanx). But the hRing was perceived the more effective, probably because it let the fingertip free and it renders better the weight of objects.
|
|
%Third experiment was to move a maximum of virtual sphere on a plane to a target by inclining a cardboard using the thumb and the index. No difference of time between all the conditions. 3RRS is perceived the more effective, probably because it renders well the inclination of the cardboard.
|
|
|
|
%[@Tinguy2018Enhancing; @Salazar2020Altering] combined tangible tangible objects and wearable haptics to modulate the perceived cutaneous stimuli of the tangible object. They have been able to increase and decrease perceived stiffness, to render bump and holes and friction on touched surfaces.
|
|
|
|
%[@Teng2021Touch] developed a nail-mounted haptic device for augmented reality that folds quickly (92 ms) on the nail when the user interacts with real objects. It renders on demand haptic feedback of virtual objects against fingerpad: Touch on virtual surfaces and buttons with pressure, sense of progression (slider detent) on AR slider with vibration, low frequency texture with tapping back and forth and high frequency textures with vibration.
|
|
|
|
%[@Lee2021Wearable] compared different haptic feedback (pseudo-haptic, vibrations, force-feedback) to render stiffness of a grasped virtual object in AR. Both rigid and elastic kinesthetic feedback rendered a perceived stiffness close to the rigid and elastic real sphere, respectively. They also seemed quite realistic (~4.5/7). Vibration feedback rendered well elastic stiffness but is not perceived realistic enough (~3.2/7). Pseudo-haptics was perceived as quite realistic as kinesthetic feedback (with large difference between users though) even if it rendered no stiffness.
|
|
|
|
|
|
\subsection{Virtual Texture Perception}
|
|
|
|
% Explain how different from \autocite{konyo2005tactile, asano2012vibrotactile, asano2015vibrotactile}, \autocite{ando2007fingernailmounted}, \autocite{bau2012revel}, \autocite{chan2021hasti}, and culbertson
|
|
%
|
|
%Tactile perception of a real texture involves multiple sensations, among them roughness being one of the most important.
|
|
%
|
|
%When running a finger over a surface, the perception of its roughness is due to the deformation of the skin caused by the micro height differences of the material \autocite{klatzky1999tactile,klatzky2003feeling}.
|
|
%
|
|
%Interestingly, visual perception of material roughness seems almost as good as haptic perception of roughness.
|
|
%
|
|
%However, there is a greater variability between individuals for visual perception than for haptic perception of roughness \autocite{bergmanntiest2007haptic}.
|
|
|
|
Many approaches have been used to generate realistic haptic virtual textures.
|
|
%
|
|
Ultrasonic vibrating screens are capable of modulating their friction~\autocite{rekik2017localized,ito2019tactile}, but their use in AR is limited.
|
|
%
|
|
By simulating the roughness of a surface instead, force feedback devices can reproduce perceptions of patterned textures identical to those of real textures~\autocite{unger2011roughness}, but they are expensive and have a limited workspace.
|
|
%
|
|
An alternative is to reproduce the vibrations that occur when a tool or the finger is moved across a surface using a vibrotactile device attached to a hand-held tool~\autocite{culbertson2018haptics}.
|
|
%
|
|
Several physical models have been proposed to represent such vibrations~\autocite{okamura1998vibration,guruswamy2011iir,chan2021hasti}.
|
|
%
|
|
However, as they can be difficult to tune, measurement-based models have been developed to record, model, and render these vibrations~\autocite{culbertson2014modeling,culbertson2017ungrounded}.
|
|
%
|
|
In this work, we employed such data-driven haptic models to augment and studied the visuo-haptic texture perception of tangible surfaces in AR.%\CP{Here the original sentence was: ``We use these data-driven haptic models to augment [...].''. It was not clear what ``we use'' meant. Check that the new sentence is correct.}
|
|
|
|
To evaluate the perception of virtual haptic textures, the same psycho-physical methods as for real materials are often used, as described by \textcite{okamoto2013psychophysical}.
|
|
%
|
|
For example, when comparing the same virtual texture pairwise, but with different parameters, \textcite{culbertson2015should} showed that the roughness vibrations generated should vary with user speed, but not necessarily with user force.
|
|
%
|
|
Similarly, \textcite{culbertson2014modeling} compared the similarity of all possible pairs between five real textures and their data-driven virtual equivalents, and rated their perceived properties in terms of hardness, roughness, friction, and smoothness.
|
|
%
|
|
Virtual data-driven textures were perceived as similar to real textures, except for friction, which was not rendered properly.
|
|
%
|
|
For grating textures, an arbitrary roughness rating is used to determine a psycho-physical curve as a function of pattern spacing~\autocite{unger2011roughness,asano2015vibrotactile,degraen2019enhancing}.
|
|
%
|
|
Another common method is to identify a given haptic texture among visual representations of all haptic textures~\autocite{ando2007fingernailmounted,rekik2017localized,degraen2019enhancing,chan2021hasti}.
|
|
%
|
|
In this user study, participants matched the pairs of visual and haptic textures they find most coherent and ranked the textures according to their perceived roughness.
|
|
%\CP{Do you refer to the one in our paper? Not super clear.}
|
|
|
|
A few studies have explored vibrotactile haptic devices worn directly on the finger to render virtual textures on real surfaces.
|
|
%
|
|
\textcite{ando2007fingernailmounted} mounted a vibrotactile actuator on the index nail, which generated impulse vibrations to render virtual edges and gaps on a real surface.
|
|
%
|
|
%This rendering method was compared later to providing the vibrations with pressure directly on the fingertip in AR and was found more realistic to render virtual objects and textures~\autocite{teng2021touch}.
|
|
%
|
|
%Covering the fingertip is however not suitable for rendering haptic feedback when touching real objects.
|
|
%
|
|
Using a voice-coil actuator worn on the middle index phalanx, \textcite{asano2015vibrotactile} altered the roughness perception of a grating surface with a \qty{250}{\Hz} vibrotactile stimulus.
|
|
%
|
|
Small amplitudes as a function of finger speed increased perceived roughness, whereas large constant amplitudes decreased it.
|
|
%
|
|
We used a similar approach, but to augment in AR the visuo-haptic texture perception of \emph{real} surfaces.
|
|
|
|
%As alternative, \textcite{teng2021touch} have designed a wearable haptic device specifically for AR scenarios mounted on the nail that can unfold on demand on the finger pad.%
|
|
%While it as been perceived more realistic in rendering virtual textures, covering the finger pad is only suitable for rendering mid-air virtual objects.
|
|
|
|
%[[chan2021hasti]] tried to combine homogenous textures with patterned textures with vibrotactile in VR.
|
|
|
|
When the same object property is sensed simultaneously by vision and touch, the two modalities are integrated into one single perception.
|
|
%
|
|
The well-known phycho-physical model of \textcite{ernst2002humans} established that the sense with the least variability dominates perception.
|
|
%
|
|
This effect has been used to alter the texture perception in AR and VR.
|
|
%
|
|
For example, superimposed virtual visual opaque textures on real surfaces in AR can be perceived as coherent together even though they have very different roughnesses~\autocite{kitahara2010sensory}.
|
|
%
|
|
\textcite{fradin2023humans} explored this effect further, finding that a superimposed AR visual texture slightly different from a colocalized haptic texture affected the ability to recognize the haptic texture.
|
|
%
|
|
Similarly, \textcite{punpongsanon2015softar} altered the softness perception of a tangible surface using AR-projected visual textures whereas \textcite{chan2021hasti} evaluated audio-haptic texture perception in VR.
|
|
%
|
|
Conversely, colocalized 3D-printed real hair structures were able to correctly render several virtual visual textures seen in VR in terms of haptic hardness and roughness~\autocite{degraen2019enhancing}.
|
|
%
|
|
This study investigated how virtual roughness haptic texture can be used to enhance touched real surfaces augmented with visual AR textures.
|
|
%Dans cet article, les textures haptiques sont senties co-localisées avec des textures visuelles
|