erwan/phd-thesis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

WIP

Browse Source
This commit is contained in:
Erwan Normand
2024-09-22 15:28:36 +02:00
parent 363108d03e
commit ee2b739ddb
30 changed files with 502 additions and 473 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
2-perception/ar-textures/1-introduction.tex
View File

@@ -1,31 +1,31 @@
\section{Introduction}
\label{introduction}
When we look at the surface of an everyday object, we then touch it to confirm or contrast our initial visual impression and to estimate the properties of the object~\cite{ernst2002humans}.
When we look at the surface of an everyday object, we then touch it to confirm or contrast our initial visual impression and to estimate the properties of the object \cite{ernst2002humans}.
%
One of the main characteristics of a textured surface is its roughness, \ie the micro-geometry of the material~\cite{klatzky2003feeling}, which is perceived equally well and similarly by both sight and touch~\cite{bergmanntiest2007haptic,baumgartner2013visual,vardar2019fingertip}.
One of the main characteristics of a textured surface is its roughness, \ie the micro-geometry of the material \cite{klatzky2003feeling}, which is perceived equally well and similarly by both sight and touch \cite{bergmanntiest2007haptic,baumgartner2013visual,vardar2019fingertip}.
%
Many haptic devices and rendering methods have been used to generate realistic virtual rough textures~\cite{culbertson2018haptics}.
Many haptic devices and rendering methods have been used to generate realistic virtual rough textures \cite{culbertson2018haptics}.
%
One of the most common approaches is to reproduce the vibrations that occur when running across a surface, using a vibrotactile device attached to a hand-held tool~\cite{culbertson2014modeling,culbertson2015should} or worn on the finger~\cite{asano2015vibrotactile,friesen2024perceived}.
One of the most common approaches is to reproduce the vibrations that occur when running across a surface, using a vibrotactile device attached to a hand-held tool \cite{culbertson2014modeling,culbertson2015should} or worn on the finger \cite{asano2015vibrotactile,friesen2024perceived}.
%
By providing timely vibrations synchronized with the movement of the tool or the finger moving on a real object, the perceived roughness of the surface can be augmented~\cite{culbertson2015should,asano2015vibrotactile}.
By providing timely vibrations synchronized with the movement of the tool or the finger moving on a real object, the perceived roughness of the surface can be augmented \cite{culbertson2015should,asano2015vibrotactile}.
%
In that sense, data-driven haptic textures have been developed as captures and models of real surfaces, resulting in the Penn Haptic Texture Toolkit (HaTT) database~\cite{culbertson2014one}.
In that sense, data-driven haptic textures have been developed as captures and models of real surfaces, resulting in the Penn Haptic Texture Toolkit (HaTT) database \cite{culbertson2014one}.
%
While these virtual haptic textures are perceived as similar to real textures~\cite{culbertson2015should}, they have been evaluated using hand-held tools and not yet in a direct finger contact with the surface context, in particular combined with visual textures in an immersive virtual environment.
While these virtual haptic textures are perceived as similar to real textures \cite{culbertson2015should}, they have been evaluated using hand-held tools and not yet in a direct finger contact with the surface context, in particular combined with visual textures in an immersive virtual environment.
Combined with virtual reality (VR), where the user is immersed in a visual virtual environment, wearable haptic devices have also proven to be effective in modifying the visuo-haptic perception of tangible objects touched with the finger, without needing to modify the object~\cite{asano2012vibrotactile,asano2015vibrotactile,salazar2020altering}.
Combined with virtual reality (VR), where the user is immersed in a visual virtual environment, wearable haptic devices have also proven to be effective in modifying the visuo-haptic perception of tangible objects touched with the finger, without needing to modify the object \cite{asano2012vibrotactile,asano2015vibrotactile,salazar2020altering}.
%
Worn on the finger, but not directly on the fingertip to keep it free to interact with tangible objects, they have been used to alter perceived stiffness, softness, friction and local deformations~\cite{detinguy2018enhancing,salazar2020altering}.
Worn on the finger, but not directly on the fingertip to keep it free to interact with tangible objects, they have been used to alter perceived stiffness, softness, friction and local deformations \cite{detinguy2018enhancing,salazar2020altering}.
%
However, the use of wearable haptic devices has been little explored in Augmented Reality (AR), where visual virtual content is integrated into the real-world environment, especially for augmenting texture sensations~\cite{punpongsanon2015softar,maisto2017evaluation,meli2018combining,chan2021hasti,teng2021touch,fradin2023humans,normand2024visuohaptic}.
However, the use of wearable haptic devices has been little explored in Augmented Reality (AR), where visual virtual content is integrated into the real-world environment, especially for augmenting texture sensations \cite{punpongsanon2015softar,maisto2017evaluation,meli2018combining,chan2021hasti,teng2021touch,fradin2023humans,normand2024visuohaptic}.
%
A key difference in AR compared to VR is that the user can still see the real-world surroundings, including their hands, the augmented tangible objects and the worn haptic devices.
%
One additional issue of current AR systems is their visual display limitations, or virtual content that may not be seen as consistent with the real world~\cite{kim2018revisiting,macedo2023occlusion}.
One additional issue of current AR systems is their visual display limitations, or virtual content that may not be seen as consistent with the real world \cite{kim2018revisiting,macedo2023occlusion}.
%
These two factors have been shown to influence the perception of haptic stiffness rendering~\cite{knorlein2009influence,gaffary2017ar}.
These two factors have been shown to influence the perception of haptic stiffness rendering \cite{knorlein2009influence,gaffary2017ar}.
%
It remains to be investigated whether simultaneous and co-localized visual and haptic texture augmentation of tangible surfaces in AR can be perceived in a coherent and realistic manner, and to what extent each sensory modality would contribute to the overall perception of the augmented texture.
%
@@ -33,4 +33,4 @@ Being able to coherently substitute the visuo-haptic texture of an everyday surf
In this paper, we investigate how users perceive a tangible surface touched with the index finger when it is augmented with a visuo-haptic roughness texture using immersive optical see-through AR (OST-AR) and wearable vibrotactile stimuli provided on the index.
%
In a user study, twenty participants freely explored and evaluated the coherence, realism and roughness of the combination of nine representative pairs of visuo-haptic texture augmentations (\figref{setup}, left) from the HaTT database~\cite{culbertson2014one}.
In a user study, twenty participants freely explored and evaluated the coherence, realism and roughness of the combination of nine representative pairs of visuo-haptic texture augmentations (\figref{setup}, left) from the HaTT database \cite{culbertson2014one}.

14
2-perception/ar-textures/2-experiment.tex
View File

@@ -4,7 +4,7 @@
\begin{subfigs}{setup}{%
User Study.
}[%
\item The nine visuo-haptic textures used in the user study, selected from the HaTT database~\cite{culbertson2014one}. %
\item The nine visuo-haptic textures used in the user study, selected from the HaTT database \cite{culbertson2014one}. %
The texture names were never shown, so as to prevent the use of the user's visual or haptic memory of the textures.
\item Experimental setup. %
Participant sat in front of the tangible surfaces, which were augmented with visual textures displayed by the HoloLens~2 AR headset and haptic roughness textures rendered by the vibrotactile haptic device placed on the middle index phalanx. %
@@ -20,7 +20,7 @@
The user study aimed at analyzing the user perception of tangible surfaces when augmented through a visuo-haptic texture using AR and vibrotactile haptic feedback provided on the finger touching the surfaces.
%
Nine representative visuo-haptic texture pairs from the HaTT database~\cite{culbertson2014one} were investigated in two tasks:
Nine representative visuo-haptic texture pairs from the HaTT database \cite{culbertson2014one} were investigated in two tasks:
%
(1) a matching task, where participants had to find the haptic texture that best matched a given visual texture; and (2) a ranking task, where participants had to rank only the haptic textures, only the visual textures, and the visuo-haptic texture pairs according to their perceived roughness.
%
@@ -30,7 +30,7 @@ Our objective is to assess which haptic textures were associated with which visu
\subsection{The textures}
\label{textures}
The 100 visuo-haptic texture pairs of the HaTT database~\cite{culbertson2014one} were preliminary tested and compared using AR and vibrotactile haptic feedback on the finger on a tangible surface.
The 100 visuo-haptic texture pairs of the HaTT database \cite{culbertson2014one} were preliminary tested and compared using AR and vibrotactile haptic feedback on the finger on a tangible surface.
%
These texture models were chosen as they are visuo-haptic representations of a wide range of real textures that are publicly available online.
%
@@ -54,11 +54,11 @@ Positioned \qty{20}{\cm} above the surfaces, a webcam (StreamCam, Logitech) film
%
The visual textures were displayed on the tangible surfaces using the HoloLens~2 OST-AR headset (\figref{setup}, middle and right) within a \qtyproduct{43 x 29}{\degree} field of view at \qty{60}{\Hz}; a set of empirical tests enabled us to choose the best rendering characteristics in terms of transparency and brightness for the visual textures, that were used throughout the user study.
%
When a haptic texture was touched, a \qty{48}{kHz} audio signal was generated using the corresponding HaTT haptic texture model and the measured tangential speed of the finger, using the rendering procedure described in Culbertson \etal~\cite{culbertson2014modeling}.
When a haptic texture was touched, a \qty{48}{kHz} audio signal was generated using the corresponding HaTT haptic texture model and the measured tangential speed of the finger, using the rendering procedure described in Culbertson \etal \cite{culbertson2014modeling}.
%
The normal force on the texture was assumed to be constant at \qty{1.2}{\N} to generate the audio signal from the model, as Culbertson \etal~\cite{culbertson2015should}, who found that the HaTT textures can be rendered using only the speed as input without decreasing their perceived realism.
The normal force on the texture was assumed to be constant at \qty{1.2}{\N} to generate the audio signal from the model, as Culbertson \etal \cite{culbertson2015should}, who found that the HaTT textures can be rendered using only the speed as input without decreasing their perceived realism.
%
An amplifier (XY-502, not branded) converted this audio signal to a current transmitted to the vibrotactile voice-coil actuator (HapCoil-One, Actronika), that was encased in a 3D-printed plastic shell firmly attached to the middle index phalanx of the participant's dominant hand, similarly to previous studies~\cite{asano2015vibrotactile,friesen2024perceived}.
An amplifier (XY-502, not branded) converted this audio signal to a current transmitted to the vibrotactile voice-coil actuator (HapCoil-One, Actronika), that was encased in a 3D-printed plastic shell firmly attached to the middle index phalanx of the participant's dominant hand, similarly to previous studies \cite{asano2015vibrotactile,friesen2024perceived}.
%
This voice-coil actuator was chosen for its wide frequency range (\qtyrange{10}{1000}{\Hz}) and its relatively low acceleration distortion, as specified by the manufacturer\footnoteurl{https://www.actronika.com/haptic-solutions}.
%
@@ -92,7 +92,7 @@ The placement of the haptic textures was randomized before each trial.
%
Participants were instructed to look closely at the details of the visual textures and explore the haptic textures with a constant pressure and various speeds to find the haptic texture that best matched the visual texture, \ie choose the surface with the most coherent visual-haptic texture pair.
%
The texture names were never given or shown to prevent the use of visual or haptic memory of the textures, nor a definition of what roughness is was given, so as to let participants complete the task as naturally as possible, similarly to Bergmann Tiest \etal~\cite{bergmanntiest2007haptic}.
The texture names were never given or shown to prevent the use of visual or haptic memory of the textures, nor a definition of what roughness is was given, so as to let participants complete the task as naturally as possible, similarly to Bergmann Tiest \etal \cite{bergmanntiest2007haptic}.
Then, participants performed the \emph{ranking task}, employing the same setup as the matching task and the same 9 textures.
%

2
2-perception/ar-textures/3-results.tex
View File

@@ -117,7 +117,7 @@ The first dimension was similar to the rankings (\figref{results_matching_rankin
%
It seems that the second dimension opposed textures that were perceived as hard with those perceived as softer, as also reported by participants.
%
Stiffness is indeed an important perceptual dimension of a material~\cite{okamoto2013psychophysical,culbertson2014modeling}.
Stiffness is indeed an important perceptual dimension of a material \cite{okamoto2013psychophysical,culbertson2014modeling}.
\figref{results_similarity} (right) shows the dendrograms of the two hierarchical clusterings of the haptic and visual textures, constructed using the Euclidean distance and the Ward's method on squared distance.
%

24
2-perception/ar-textures/4-discussion.tex
View File

@@ -3,13 +3,13 @@
In this study, we investigated the perception of visuo-haptic texture augmentation of tangible surfaces touched directly with the index fingertip, using visual texture overlays in AR and haptic roughness textures generated by a vibrotactile device worn on the middle index phalanx.
%
The nine evaluated pairs of visuo-haptic textures, taken from the HaTT database~\cite{culbertson2014one}, are models of real texture captures.
The nine evaluated pairs of visuo-haptic textures, taken from the HaTT database \cite{culbertson2014one}, are models of real texture captures.
%
Their perception was evaluated in a two-task user study in which participants chose the most coherent combinations of visual and haptic textures (matching task), and ranked all textures according to their perceived roughness (ranking task).
%
The visual textures were displayed statically on the tangible surface, while the haptic textures adapted in real time to the speed of the finger on the surface, giving the impression that the visuo-haptic textures were integrated into the tangible surface.
%
In addition, the interaction with the textures was designed to be as natural as possible, without imposing a specific speed of finger movement, as in similar studies~\cite{asano2015vibrotactile,friesen2024perceived}.
In addition, the interaction with the textures was designed to be as natural as possible, without imposing a specific speed of finger movement, as in similar studies \cite{asano2015vibrotactile,friesen2024perceived}.
In the matching task, participants were not able to effectively match the original visual and haptic texture pairs (\figref{results_matching_ranking}, left), except for the Coffee Filter texture, which was the smoothest both visually and haptically.
%
@@ -25,9 +25,9 @@ Several strategies were used, as some participants reported using vibration freq
%
It should be noted that the task was rather difficult (\figref{results_questions}), as participants had no prior knowledge of the textures, there were no additional visual cues such as the shape of an object, and the term \enquote{roughness} had not been used by the experimenter prior to the ranking task.
The correspondence analysis (\figref{results_similarity}, left) highlighted that participants did indeed match visual and haptic textures primarily on the basis of their perceived roughness (60\% of variance), which is in line with previous perception studies on real~\cite{baumgartner2013visual} and virtual~\cite{culbertson2014modeling} textures.
The correspondence analysis (\figref{results_similarity}, left) highlighted that participants did indeed match visual and haptic textures primarily on the basis of their perceived roughness (60\% of variance), which is in line with previous perception studies on real \cite{baumgartner2013visual} and virtual \cite{culbertson2014modeling} textures.
%
The rankings (\figref{results_matching_ranking}, right) confirmed that the participants all perceived the roughness of haptic textures very similarly, but that there was less consensus for visual textures, which is also in line with roughness rankings for real haptic and visual textures~\cite{bergmanntiest2007haptic}.
The rankings (\figref{results_matching_ranking}, right) confirmed that the participants all perceived the roughness of haptic textures very similarly, but that there was less consensus for visual textures, which is also in line with roughness rankings for real haptic and visual textures \cite{bergmanntiest2007haptic}.
%
These results made it possible to identify and name groups of textures in the form of clusters, and to construct confusion matrices between these clusters and between visual texture ranks with haptic clusters, showing that participants consistently identified and matched haptic and visual textures (\figref{results_clusters}).
%
@@ -35,17 +35,17 @@ Interestingly, 30\% of the matching variance was captured with a second dimensio
%
One hypothesis is that this dimension could be the perceived stiffness of the textures, with Metal Mesh and smooth textures appearing stiffer than the other textures, whose granularity could have been perceived as bumps on the surface that could deform under finger pressure.
%
Stiffness is, with roughness, one of the main characteristics perceived by the vision and touch of real materials~\cite{baumgartner2013visual,vardar2019fingertip}, but also on virtual haptic textures~\cite{culbertson2014modeling,degraen2019enhancing}.
Stiffness is, with roughness, one of the main characteristics perceived by the vision and touch of real materials \cite{baumgartner2013visual,vardar2019fingertip}, but also on virtual haptic textures \cite{culbertson2014modeling,degraen2019enhancing}.
%
The last visuo-haptic roughness ranking (\figref{results_matching_ranking}, right) showed that both haptic and visual sensory information were well integrated as the resulting roughness ranking was being in between the two individual haptic and visual rankings.
%
Several strategies were reported: some participants first classified visually and then corrected with haptics, others classified haptically and then integrated visuals.
%
While visual sensation did influence perception, as observed in previous haptic AR studies~\cite{punpongsanon2015softar,gaffary2017ar,fradin2023humans}, haptic sensation dominated here.
While visual sensation did influence perception, as observed in previous haptic AR studies \cite{punpongsanon2015softar,gaffary2017ar,fradin2023humans}, haptic sensation dominated here.
%
This indicates that participants were more confident and relied more on the haptic roughness perception than on the visual roughness perception when integrating both in one coherent perception.
%
Several participants also described attempting to identify visual and haptic textures using spatial breaks, edges or patterns, that were not observed when these textures were displayed in non-immersive virtual environments with a screen~\cite{culbertson2014modeling,culbertson2015should}.
Several participants also described attempting to identify visual and haptic textures using spatial breaks, edges or patterns, that were not observed when these textures were displayed in non-immersive virtual environments with a screen \cite{culbertson2014modeling,culbertson2015should}.
%
A few participants even reported that they clearly sensed patterns on haptic textures.
%
@@ -55,19 +55,19 @@ Overall, the haptic device was judged to be comfortable, and the visual and hapt
These results have of course some limitations as they addressed a small set of visuo-haptic textures augmenting the perception of smooth white tangible surfaces.
%
Indeed, the increase in visuo-haptic texture perception may be limited on surfaces that already have strong visual or haptic patterns~\cite{asano2012vibrotactile}, or on objects with complex shapes.
Indeed, the increase in visuo-haptic texture perception may be limited on surfaces that already have strong visual or haptic patterns \cite{asano2012vibrotactile}, or on objects with complex shapes.
%
In addition, the haptic textures used were modelled from the vibrations of a probe sliding over the captured surfaces.
%
The perception of surface roughness with the finger is actually more complex because it involves both the perception of vibrations and the spatial deformation of the skin~\cite{klatzky2003feeling}, but also because the sensations generated when exploring a surface depend on factors other than the speed of the finger alone, such as the force of contact, the angle, the posture or the surface of the contact~\cite{schafer2017transfer}, and the integration of these sensory information into one unified perception is not yet fully understood~\cite{richardson2022learning}.
The perception of surface roughness with the finger is actually more complex because it involves both the perception of vibrations and the spatial deformation of the skin \cite{klatzky2003feeling}, but also because the sensations generated when exploring a surface depend on factors other than the speed of the finger alone, such as the force of contact, the angle, the posture or the surface of the contact \cite{schafer2017transfer}, and the integration of these sensory information into one unified perception is not yet fully understood \cite{richardson2022learning}.
%
Another limitation that may have affected the perception of haptic textures is the lack of compensation for the frequency response of the actuator and amplifier~\cite{asano2012vibrotactile,culbertson2014modeling,friesen2024perceived}.
Another limitation that may have affected the perception of haptic textures is the lack of compensation for the frequency response of the actuator and amplifier \cite{asano2012vibrotactile,culbertson2014modeling,friesen2024perceived}.
%
Finally, the visual textures used were also simple color captures not meant to be used in an immersive virtual environment.
%
However, our objective was not to accurately reproduce real textures, but to alter the perception of simultaneous visual and haptic roughness augmentation of a real surface directly touched by the finger in AR.
%
In addition of these limitations, both visual and haptic texture models should be improved by integrating the rendering of spatially localized breaks, edges or patterns, like real textures~\cite{richardson2022learning}, and by being adaptable to individual sensitivities, as personalized haptics is a promising approach~\cite{malvezzi2021design,young2020compensating}.
In addition of these limitations, both visual and haptic texture models should be improved by integrating the rendering of spatially localized breaks, edges or patterns, like real textures \cite{richardson2022learning}, and by being adaptable to individual sensitivities, as personalized haptics is a promising approach \cite{malvezzi2021design,young2020compensating}.
%
More generally, a wide range of haptic feedbacks should be integrated to form rich and complete haptic augmentations in AR~\cite{maisto2017evaluation,detinguy2018enhancing,salazar2020altering,normand2024visuohaptic,pacchierotti2024haptics}.
More generally, a wide range of haptic feedbacks should be integrated to form rich and complete haptic augmentations in AR \cite{maisto2017evaluation,detinguy2018enhancing,salazar2020altering,normand2024visuohaptic,pacchierotti2024haptics}.