Remove background folder

3-perception/vhar-textures/1-introduction.tex

@@ -0,0 +1,27 @@
+\section{Introduction}
+\label{intro}
+
+When we look at the surface of an everyday object, we then touch it to confirm or contrast our initial visual impression and to estimate the properties of the object, particularly its texture \secref[related_work]{visual_haptic_influence}.
+Among the various haptic texture augmentations, data-driven methods allow to capture, model and reproduce the roughness perception of real surfaces when touched touched by a hand-held stylus \secref[related_work]{texture_rendering}.
+Databases of visuo-haptic textures have been developed in this way \cite{culbertson2014one,balasubramanian2024sens3}, but they have not yet been explored in an immersive and direct touch context with \AR and wearable haptics.
+
+In this chapter, we investigate whether simultaneous and \textbf{co-localized visual and wearable haptic texture augmentation of real surfaces} in \AR can be perceived in a coherent and realistic manner, and to what extent each sensory modality would contribute to the overall perception of the augmented texture.
+We used nine pairs of \textbf{data-driven visuo-haptic textures} from the \HaTT database \cite{culbertson2014one}, which we rendered using the wearable visuo-haptic augmentatio nsystem presented in \chapref{vhar_system}. %, an \OST-\AR headset, and a wearable voice-coil device worn on the finger.
+In a \textbf{user study}, 20 participants freely explored in direct touch the combination of the visuo-haptic texture pairs to rate their coherence, realism and perceived roughness.
+We aimed to assess \textbf{which haptic textures were matched with which visual textures}, how the roughness of the visual and haptic textures was perceived, and whether \textbf{the perceived roughness} could explain the matches made between them.
+
+\noindentskip The contributions of this chapter are:
+\begin{itemize}
+  \item Transposition of data-driven visuo-haptic textures to augment real objects in a direct touch context in immersive \AR.
+  \item A user study evaluating with 20 participants the coherence, realism and perceived roughness of nine pairs of these visuo-haptic texture augmentations.
+\end{itemize}
+
+\noindentskip In the next sections, we first describe the apparatus of the user study experimental design, including the two tasks performed. We then present the results obtained and discuss them before concluding.
+
+\bigskip
+
+\fig[0.65]{experiment/view}{First person view of the user study.}[
+  As seen through the immersive \AR headset Microsoft HoloLens~2.
+  The visual texture overlays were statically displayed on the surfaces, allowing the user to move around to view them from different angles.
+  The haptic texture augmentations were generated based on \HaTT data-driven texture models and finger speed, and were rendered on the middle index phalanx as it slides on the considered surface.
+]

3-perception/vhar-textures/2-experiment.tex

@@ -0,0 +1,123 @@
+\section{User Study}
+\label{experiment}
+
+%The user study aimed at analyzing the user perception of real surfaces when augmented through a visuo-haptic texture using \AR and vibrotactile haptic feedback provided on the finger touching the surfaces.
+%Nine representative visuo-haptic texture pairs from the \HaTT database \cite{culbertson2014one} were investigated in two tasks:
+%\begin{enumerate}
+%  \item \level{Matching} task: participants had to find the haptic texture that best matched a given visual texture; and
+%  \item \level{Ranking} task: participants had to rank the haptic textures, the visual textures, and the visuo-haptic texture pairs according to their perceived roughness.
+%\end{enumerate}
+%Our objective is to assess which haptic textures were associated with which visual textures, how the roughness of the visual and haptic textures are perceived, and whether the perceived roughness can explain the matches made between them.
+
+\subsection{The textures}
+\label{textures}
+
+The 100 visuo-haptic texture pairs of the \HaTT database \cite{culbertson2014one} were preliminary tested and compared using the apparatus described in \secref{apparatus} to select the most representative textures for the user study.
+% visuo-haptic system presented in \chapref{vhar_system}, and with the vibrotactile haptic feedback provided on the middle-phalanx of the finger touching a real surface. on the finger on a real surface
+These texture models were chosen as they are visuo-haptic representations of a wide range of real textures that are publicly available online.
+Nine texture pairs were selected (\figref{experiment/textures}) to cover various perceived roughness, from rough to smooth, as named on the database: \level{Metal Mesh}, \level{Sandpaper~100}, \level{Brick~2}, \level{Cork}, \level{Sandpaper~320}, \level{Velcro Hooks}, \level{Plastic Mesh~1}, \level{Terra Cotta}, \level{Coffee Filter}.
+All these visual and haptic textures are isotropic: their rendering (appearance or roughness) is the same whatever the direction of the movement on the surface, \ie there are no local deformations (holes, bumps, or breaks).
+
+\subsection{Apparatus}
+\label{apparatus}
+
+\figref{experiment/setup} shows the experimental setup, and \figref{experiment/view} the first person view of participants during the user study.
+The user study was held in a quiet room with no windows, with one light source of \qty{800}{\lumen} placed \qty{70}{\cm} above the table.
+
+Nine \qty{5}{\cm} square cardboards with smooth, white melamine surface, arranged in a \numproduct{3 x 3} grid, were used as real surfaces to augment.
+Their poses were estimated with three \qty{2}{\cm} AprilTag fiducial markers glued on the surfaces grid.
+Similarly, a \qty{2}{\cm} fiducial marker was glued on top of the vibrotactile actuator to detect the finger pose.
+Positioned \qty{20}{\cm} above the surfaces, a webcam (StreamCam, Logitech) filmed the markers to track finger movements relative to the surfaces, as described in \secref[vhar_system]{virtual_real_alignment}.
+The visual textures were displayed on the real surfaces using the \OST-\AR headset Microsoft HoloLens~2 running a custom application at \qty{60}{FPS} made with Unity 2021.1 and Mixed Reality Toolkit (MRTK) 2.7.2.
+A set of empirical tests enabled us to choose the best rendering characteristics in terms of transparency and brightness for the visual textures, that were used throughout the user study.
+
+When a virtual haptic texture was touched, a \qty{48}{kHz} audio signal was generated using the rendering procedure described in \cite{culbertson2014modeling} from the corresponding \HaTT haptic texture model and the measured tangential speed of the finger (\secref[vhar_system]{texture_generation}).
+The normal force on the texture was assumed to be constant at \qty{1.2}{\N} to generate the audio signal from the model, as \textcite{culbertson2015should}, who found that the \HaTT textures can be rendered using only the speed as input without decreasing their perceived realism.
+The rendering of the virtual texture is described in \secref[vhar_system]{texture_generation}.
+The vibrotactile voice-coil actuator (HapCoil-One, Actronika) was firmly attached to the middle index phalanx of the participant's dominant hand using a Velcro strap, similarly to previous studies \cite{asano2015vibrotactile,friesen2024perceived}.
+%An amplifier (XY-502, not branded) converted this audio signal to a current transmitted to the vibrotactile voice-coil actuator (HapCoil-One, Actronika), that was encased in a \ThreeD-printed plastic shell firmly attached to the middle index phalanx of the participant's dominant hand, similarly to previous studies \cite{asano2015vibrotactile,friesen2024perceived}.
+%This voice-coil actuator was chosen for its wide frequency range (\qtyrange{10}{1000}{\Hz}) and its relatively low acceleration distortion, as specified by the manufacturer\footnoteurl{https://www.actronika.com/haptic-solutions}.
+%Overall latency was measured to \qty{46 \pm 6}{\ms}, as a result of latency in image acquisition \qty{16 \pm 1}{\ms}, fiducial marker detection \qty{8 \pm 3}{\ms}, network synchronization \qty{4 \pm 1}{\ms}, audio sampling \qty{3 \pm 1}{\ms}, and the vibrotactile actuator latency (\qty{15}{\ms}, as specified by the manufacturer\footnotemark[5]).
+%This latency was below the \qty{60}{\ms} threshold for vibrotactile feedback \cite{okamoto2009detectability} and was not noticed by the participants.
+
+\begin{subfigs}{setup}{Textures used and experimental setup of the user study. }[][
+  \item The nine visuo-haptic textures used in the user study, selected from the \HaTT database \cite{culbertson2014one}.
+    The texture names were never shown, to prevent the use of the user's visual or haptic memory of the textures.
+  \item Experimental setup.
+    Participant sat in front of the real surfaces, which were augmented with visual textures displayed by the Microsoft HoloLens~2 \AR headset and haptic roughness textures rendered by the vibrotactile haptic device placed on the middle index phalanx.
+    A webcam above the surfaces tracked the finger movements.
+  ]
+  \subfig[0.49]{experiment/textures}
+  \subfig[0.49]{experiment/setup}
+\end{subfigs}
+
+\subsection{Procedure and Collected Data}
+\label{procedure}
+
+Participants were first given written instructions about the experimental setup, the tasks, and the procedure of the user study.
+Then, after having signed an informed consent form, they were asked to seat in front of the table with the experimental setup and to wear the \AR headset.
+%The experimenter firmly attached the plastic shell encasing the vibrotactile actuator to the middle index phalanx of their dominant hand.
+As the haptic textures generated no audible noise, participants did not wear any noise reduction headphones.
+A calibration of both the HoloLens~2 and the hand tracking was performed to ensure the correct alignment of the visual and haptic textures on the real surfaces.
+Finally, participants familiarized with the augmented surface in a \qty{2}{min} training session with textures different from the ones used in the user study.
+
+Participants started with the \level{Matching} task.
+They were informed that the user study involved nine pairs of corresponding visual and haptic textures that were separated and shuffled.
+On each trial, the same visual texture was displayed on the nine real surfaces, while the nine haptic textures were rendered on only one of the surfaces at a time, \ie all surfaces were augmented by the same visual texture, but each surface was augmented by a different haptic texture.
+The placement of the haptic textures was randomized before each trial.
+Participants were instructed to look closely at the details of the visual textures and explore the haptic textures with a constant pressure and various speeds to find the haptic texture that best matched the visual texture, \ie choose the surface with the most coherent visual-haptic texture pair.
+The texture names were never given or shown to prevent the use of visual or haptic memory of the textures, nor a definition of what roughness is was given, to let participants complete the task as naturally as possible, similarly to \textcite{bergmanntiest2007haptic}.
+
+Then, participants performed the \level{Ranking} task, employing the same setup as the matching task and the same 9 textures.
+In this case, participants were asked to rank the textures according to their perceived roughness.
+First, they ranked all the haptic textures (without any visual augmentation given), then all the visual textures (without any haptic augmentation given), and finally all the visuo-haptic texture pairs together, being informed that they were the correct matches as per the original \HaTT database.
+The placement of the textures was also randomized before each trial.
+
+The user study took on average 1 hour to complete.
+
+\subsection{Experimental Design}
+\label{design}
+
+The user study was a within-subjects design with two tasks: \level{Matching} and \level{Ranking}.
+
+In the \level{Matching} task, participants had to find the haptic texture that best matched a given visual texture.
+It had one within-subjects factor, \factor{Visual Texture} with the following levels:
+\level{Metal Mesh}, \level{Sandpaper~100}, \level{Brick~2}, \level{Cork}, \level{Sandpaper~320}, \level{Velcro Hooks}, \level{Plastic Mesh~1}, \level{Terra Cotta}, \level{Coffee Filter}.
+To account for learning and fatigue effects, the order of \factor{Visual Texture} was counterbalanced using a balanced \numproduct{18 x 18} Latin square design.
+A total of 9 textures \x 3 repetitions = 18 matching trials were performed per participant.
+
+In the \level{Ranking} task, participants had to rank the haptic textures, the visual textures, and the visuo-haptic texture pairs according to their perceived roughness.
+It had one within-subjects factor, \factor{Modality} with the following levels: \level{Visual}, \level{Haptic}, \level{Visuo-Haptic}.
+Each modality level was ranked once per participant following the fixed order listed above (\secref{procedure}).
+%The order of \level{Modality} was fixed as listed above, and.
+%A total of 3 modalities = 60 ranking trials were collected.
+
+\subsection{Participants}
+\label{participants}
+
+Twenty participants took part in the user study (12 males, 7 females, 1 preferred not to say), aged between 20 and 60 years (\mean{29.1}, \sd{9.4}).
+One participant was left-handed, all others were right-handed; they all performed the user study with their dominant hand.
+All participants had normal or corrected-to-normal vision and none of them had a known hand or finger impairment.
+They rated their experience with haptics, \AR, and \VR (\enquote{I use it every month or more}); 10 were experienced with haptics, 2 with \AR, and 10 with \VR.
+Experiences were correlated between haptics and \AR (\spearman{0.53}), haptics and \VR (\spearman{0.61}), and \AR and \VR (\spearman{0.74}); but not with age (\spearman{-0.06} to \spearman{-0.05}) or gender (\spearman{0.10} to \spearman{0.27}).
+Participants were recruited at the university on a voluntary basis.
+They all signed an informed consent form before the user study.
+
+\subsection{Collected Data}
+\label{collected_data}
+
+For each trial of the \level{Matching} task, the chosen \response{Haptic Texture} for the given displayed \factor{Visual Texture} was recorded.
+The \response{Completion Time} was also measured as the time between the visual texture display and the haptic texture selection.
+For each modality of the \level{Ranking} task, the \response{Rank} of each of the visual, haptic, or visuo-haptic pairs of the textures presented was recorded.
+
+\noindentskip After each of the two tasks, participants answered to the following 7-item Likert scale questions (1=Not at all, 7=Extremely):
+\begin{itemize}
+  \item \response{Haptic Difficulty}: How difficult was it to differentiate the tactile textures?
+  \item \response{Visual Difficulty}: How difficult was it to differentiate the visual textures?
+  \item \response{Textures Match}: For the visual-tactile pairs you have chosen, how coherent were the tactile textures with the corresponding visual textures?
+  \item \response{Haptic Realism}: How realistic were the tactile textures?
+  \item \response{Visual Realism}: How realistic were the visual textures?
+  \item \response{Uncomfort}: How uncomfortable was to use the haptic device?
+\end{itemize}
+
+\noindentskip In an open question, participants also commented on their strategy for completing the \level{Matching} task (\enquote{How did you associate the tactile textures with the visual textures?}) and the \level{Ranking} task (\enquote{How did you rank the textures?}).

3-perception/vhar-textures/3-results.tex

@@ -0,0 +1,157 @@
+\section{Results}
+\label{results}
+
+\subsection{Textures Matching}
+\label{results_matching}
+
+\paragraph{Confusion Matrix}
+\label{results_matching_confusion_matrix}
+
+\figref{results/matching_confusion_matrix} shows the confusion matrix of the \level{Matching} task with the visual textures and the proportion of haptic texture selected in response, \ie the proportion of times the corresponding haptic texture was selected in response to the presentation of the corresponding visual texture.
+A two-sample Pearson Chi-Squared test (\chisqr{64}{540}{420}, \pinf{0.001}) and Holm-Bonferroni adjusted binomial tests indicated that the following (\factor{Visual Texture}, \response{Haptic Texture}) pairs have proportion selections statistically significantly higher than chance (\ie \percent{11} each):
+\begin{itemize}
+  \item (\level{Sandpaper~320}, \level{Coffee Filter}), (\level{Terra Cotta}, \level{Coffee Filter}), and (\level{Coffee Filter}, \level{Coffee Filter}) (\pinf{0.001} each);
+  \item (\level{Cork}, \level{Sandpaper~320}), (\level{Brick~2}, \level{Plastic Mesh~1}), (\level{Brick~2}, \level{Sandpaper~320}), (\level{Plastic Mesh~1}, \level{Sandpaper~320}), and (\level{Sandpaper~320}, \level{Plastic Mesh~1}) (\pinf{0.01}); and
+  \item (\level{Metal Mesh}, \level{Cork}), (\level{Cork}, \level{Velcro Hooks}), (\level{Velcro Hooks}, \level{Plastic Mesh~1}), (\level{Velcro Hooks}, \level{Sandpaper~320}), and (\level{Coffee Filter}, \level{Terra Cotta}) (\pinf{0.05} each).
+\end{itemize}
+
+Except for one visual texture (\level{Sandpaper~100}) and 4 haptic textures (\level{Metal Mesh}, \level{Sandpaper~100}, \level{Brick~2}, and \level{Terra Cotta}), all haptic and visual textures were matched statistically significantly higher than chance with at least one visual and haptic texture, respectively.
+However, many mistakes were made: the expected haptic texture was selected on average only \percent{20} of the time for five of the visual textures, and even around \percent{5} for (visual) \level{Sandpaper~100}, \level{Brick~2}, and \level{Sandpaper~320}.
+Only haptic \level{Coffee Filter} was correctly selected \percent{59} of the time, and was also particularly matched with the visual \level{Sandpaper~320} and \level{Terra Cotta} (around \percent{45} each).
+Similarly, the haptic textures \level{Sandpaper~320} and \level{Plastic Mesh~1} were also selected for four and three visual textures, respectively (around \percent{25} each).
+Additionally, the Spearman correlations between the trials were computed for each participant and only 21 out of 60 were statistically significant (\pinf{0.05}), with a mean \spearman{0.52} (\ci{0.43}{0.59}).
+
+\fig[0.82]{results/matching_confusion_matrix}{Confusion matrix of the \level{Matching} task.}[
+  With the presented visual textures as columns and the selected haptic texture in proportion as rows.
+  The number in a cell is the proportion of times the corresponding haptic texture was selected in response to the presentation of the corresponding visual texture.
+  The diagonal represents the expected correct answers.
+  Holm-Bonferroni adjusted binomial test results are marked in bold when the proportion is higher than chance (\ie more than \percent{11}, \pinf{0.05}).
+]
+
+These results indicate that the participants hesitated between several haptic textures for a given visual texture, as also reported in several comments, some haptic textures being more favored while some others were almost not selected at all.
+Another explanation could be that the participants had difficulties to estimate the roughness of the visual textures.
+Indeed, many participants explained that they tried to identify or imagine the roughness of a given visual texture then to select the most plausible haptic texture, in terms of frequency and/or amplitude of vibrations.
+
+\paragraph{Completion Time}
+
+To verify that the difficulty with all the visual textures was the same on the \level{Matching} task, the \response{Completion Time} of a trial was analyzed.
+As the \response{Completion Time} results were Gamma distributed, they were transformed with a log to approximate a normal distribution.
+A \LMM on the log \response{Completion Time} with the \factor{Visual Texture} as fixed effect and the participant as random intercept was performed.
+Normality was verified with a QQ-plot of the model residuals.
+No statistical significant effect of \factor{Visual Texture} was found (\anova{8}{512}{1.9}, \p{0.06}) on \response{Completion Time} (\geomean{44}{\s}, \ci{42}{46}), indicating an equal difficulty and participant behaviour for all the visual textures.
+
+\subsection{Textures Ranking}
+\label{results_ranking}
+
+\figref{results/ranking_mean_ci} presents the results of the three rankings of the haptic textures alone, the visual textures alone, and the visuo-haptic texture pairs.
+
+\paragraph{Haptic Textures Ranking}
+
+Almost all the texture pairs in the haptic textures ranking results were statistically significantly different (\chisqr{8}{20}{146}, \pinf{0.001}; \pinf{0.05} for each comparison), except between (\level{Metal Mesh}, \level{Sandpaper~100}), (\level{Cork}, \level{Brick~2}), (\level{Cork}, \level{Sandpaper~320}) (\level{Plastic Mesh~1}, \level{Velcro Hooks}), and (\level{Plastic Mesh~1}, \level{Terra Cotta}).
+Average Kendall's Tau correlations between the participants indicated a high consensus (\kendall{0.82}, \ci{0.81}{0.84}) showing that participants perceived similarly the roughness of the haptic textures.
+
+\paragraph{Visual Textures Ranking}
+
+Most of the texture pairs in the visual textures ranking results were also statistically significantly different (\chisqr{8}{20}{119}, \pinf{0.001}; \pinf{0.05} for each comparison), except for the following groups: \{\level{Metal Mesh}, \level{Cork}, \level{Plastic Mesh~1}\}; \{\level{Sandpaper~100}, \level{Brick~2}, \level{Plastic Mesh~1}, \level{Velcro Hooks}\}; \{\level{Cork}, \level{Velcro Hooks}\}; \{\level{Sandpaper~320}, \level{Terra Cotta}\}; and \{\level{Sandpaper~320}, \level{Coffee Filter}\}.
+Even though the consensus was high (\kendall{0.61}, \ci{0.58}{0.64}), the roughness of the visual textures were more difficult to estimate, in particular for \level{Plastic Mesh~1} and \level{Velcro Hooks}.
+
+\paragraph{Visuo-Haptic Textures Ranking}
+
+Also, almost all the texture pairs in the visuo-haptic textures ranking results were statistically significantly different (\chisqr{8}{20}{140}, \pinf{0.001}; \pinf{0.05} for each comparison), except for the following groups: \{\level{Sandpaper~100}, \level{Cork}\}; \{\level{Cork}, \level{Brick~2}\}; and \{\level{Plastic Mesh~1}, \level{Velcro Hooks}, \level{Sandpaper~320}\}.
+The consezsus between the participants was also high \kendall{0.77}, \ci{0.74}{0.79}.
+Finally, calculating the similarity of the three rankings of each participant, the \textit{Visuo-Haptic Textures Ranking} was on average highly similar to the \textit{Haptic Textures Ranking} (\kendall{0.79}, \ci{0.72}{0.86}) and moderately to the \textit{Visual Textures Ranking} (\kendall{0.48}, \ci{0.39}{0.56}).
+A Wilcoxon signed-rank test indicated that this difference was statistically significant (\wilcoxon{190}, \p{0.002}).
+These results indicate that the two haptic and visual modalities were integrated together, the resulting roughness ranking being between the two rankings of the modalities alone, but with haptics predominating.
+
+\fig[0.6]{results/ranking_mean_ci}{Means with bootstrap \percent{95} \CI of the three rankings of the haptic textures alone, the visual textures alone, and the visuo-haptic texture pairs. }[
+  A lower rank means that the texture was considered rougher, a higher rank means smoother.
+]
+
+\subsection{Perceived Similarity of Visual and Haptic Textures}
+\label{results_clusters}
+
+The high level of agreement between participants on the three haptic, visual and visuo-haptic rankings in the \level{Ranking} task (\secref{results_ranking}), as well as the similarity of the within-participant rankings, suggest that participants perceived the roughness of the textures similarly, but differed in their strategies for matching the haptic and visual textures in the \level{Matching} task (\secref{results_matching}).
+
+To further investigate the perceived similarity of the haptic and visual textures and to identify groups of textures that were perceived as similar on the \level{Matching} task, a correspondence analysis and a hierarchical clustering were performed on the matching task confusion matrix (\figref{results/matching_confusion_matrix}).
+
+\paragraph{Correspondence Analysis}
+
+The correspondence analysis captured \percent{60} and \percent{29} of the variance in the first and second dimensions, respectively, with the remaining dimensions each accounting for less than \percent{5} each.
+\figref{results/matching_correspondence_analysis} shows the first two dimensions with the 18 haptic and visual textures.
+The first dimension was similar to the rankings (\figref{results/ranking_mean_ci}), distributing the textures according to their perceived roughness.
+It seems that the second dimension opposed textures that were perceived as hard with those perceived as softer, as also reported by participants.
+Stiffness is indeed an important perceptual dimension of a material (\secref[related_work]{hardness}).% \cite{okamoto2013psychophysical,culbertson2014modeling}.
+
+\fig[0.6]{results/matching_correspondence_analysis}{
+  Correspondence analysis of the confusion matrix of the \level{Matching} task.
+}[
+  %The haptic textures are represented as green squares, the haptic textures as red circles. %
+  The closer the haptic and visual textures are, the more similar they were judged. %
+  The first dimension (horizontal axis) explains \percent{60} of the variance, the second dimension (vertical axis) explains \percent{30} of the variance.
+  The confusion matrix is \figref{results/matching_confusion_matrix}.
+]
+
+\paragraph{Hierarchical Clustering}
+
+\figref{results_clusters} shows the dendrograms of the two hierarchical clusterings of the haptic and visual textures, constructed using the Euclidean distance and the Ward's method on squared distance.
+
+The four identified haptic texture clusters were: "Roughest" \{\level{Metal Mesh}, \level{Sandpaper~100}, \level{Brick~2}, \level{Cork}\}; "Rougher" \{\level{Sandpaper~320}, \level{Velcro Hooks}\}; "Smoother" \{\level{Plastic Mesh~1}, \level{Terra Cotta}\}; "Smoothest" \{\level{Coffee Filter}\} (\figref{results/clusters_haptic}).
+Similar to the haptic ranks (\figref{results/ranking_mean_ci}), the clusters could have been named according to their perceived roughness.
+It also shows that the participants compared and ranked the haptic textures during the \level{Matching} task to select the one that best matched the given visual texture.
+
+The five identified visual texture clusters were: "Roughest" \{\level{Metal Mesh}\}; "Rougher" \{\level{Sandpaper~100}, \level{Brick~2}, \level{Velcro Hooks}\}; "Medium" \{\level{Cork}, \level{Plastic Mesh~1}\}; "Smoother" \{\level{Sandpaper~320}, \level{Terra Cotta}\}; "Smoothest" \{\level{Coffee Filter}\} (\figref{results/clusters_visual}).
+They are also easily identifiable on the visual ranking results, which also made it possible to name them.
+
+\begin{subfigs}{results_clusters}{Dendrograms of the hierarchical clusterings of the \level{Matching} task confusion matrix.}[
+    Done with the Euclidean distance and the Ward's method on squared distance.
+    The height of the dendrograms represents the distance between the clusters.
+  ][%
+  \item For the haptic textures.
+  \item For the visual textures.
+  ]
+  \subfig[0.45]{results/clusters_haptic}
+  \subfig[0.45]{results/clusters_visual}
+\end{subfigs}
+
+\paragraph{Confusion Matrices of Clusters}
+
+Based on these results, two alternative confusion matrices were constructed.
+
+\figref{results/haptic_visual_clusters_confusion_matrices} (left) shows the confusion matrix of the \level{Matching} task with visual texture clusters and the proportion of haptic texture clusters selected in response.
+A two-sample Pearson Chi-Squared test (\chisqr{16}{540}{353}, \pinf{0.001}) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Cluster, Haptic Cluster) pairs have proportion selections statistically significantly higher than chance (\ie \percent{20} each): %
+(Roughest, Roughest), (Rougher, Rougher), (Medium, Rougher), (Medium, Smoother), (Smoother, Smoother), (Smoother, Smoothest), and (Smoothest, Smoothest) (\pinf{0.005} each).
+
+\figref{results/haptic_visual_clusters_confusion_matrices} (right) shows the confusion matrix of the \level{Matching} task with visual texture ranks and the proportion of haptic texture clusters selected in response.
+A two-sample Pearson Chi-Squared test (\chisqr{24}{540}{342}, \pinf{0.001}) and Holm-Bonferroni adjusted binomial tests indicated that the following (Visual Texture Rank, Haptic Cluster) pairs have proportion selections statistically significantly higher than chance: %
+(0, Roughest); (1, Rougher); (2, Rougher); (3, Rougher); (4, Rougher); (5, Smoother); (6, Smoother); (7, Smoothest); and (8, Smoothest) (\pinf{0.05} each).
+This shows that the participants consistently identified the roughness of each visual texture and selected the corresponding haptic texture cluster.
+
+\fig{results/haptic_visual_clusters_confusion_matrices}{
+  Confusion matrices of the visual texture (left) or rank (right) with the corresponding haptic texture clusters selected in proportion.
+}[
+  Holm-Bonferroni adjusted binomial test results are marked in bold when the proportion is higher than chance (\ie more than \percent{20}, \pinf{0.05}).
+]
+
+\subsection{Questionnaire}
+\label{results_questions}
+
+\figref{results_questions} presents the questionnaire results of the \level{Matching} and \level{Ranking} tasks.
+A non-parametric \ANOVA on an \ART model was used on the \response{Difficulty} and \response{Realism} question results, while the other question results were analyzed using Wilcoxon signed-rank tests.
+
+On \response{Difficulty}, there were statistically significant effects of \factor{Task} (\anova{1}{57}{13}, \pinf{0.001}) and of \response{Modality} (\anova{1}{57}{8}, \p{0.007}), but no interaction effect \factor{Task} \x \factor{Modality} (\anova{1}{57}{2}, \ns).
+The \level{Ranking} task was found easier (\mean{2.9}, \sd{1.2}) than the \level{Matching} task (\mean{3.9}, \sd{1.5}), and the Haptic textures were found easier to discrimate (\mean{3.0}, \sd{1.3}) than the Visual ones (\mean{3.8}, \sd{1.5}).
+Both haptic and visual textures were judged moderately realistic for both tasks (\mean{4.2}, \sd{1.3}), with no statistically significant effect of \factor{Task}, \factor{Modality} or their interaction on \response{Realism}.
+No statistically significant effects of \factor{Task} on \response{Textures Match} and \response{Uncomfort} were found either.
+The coherence of the texture pairs was considered moderate (\mean{4.6}, \sd{1.2}) and the haptic device was not felt uncomfortable (\mean{2.4}, \sd{1.4}).
+
+\begin{subfigs}{results_questions}{Boxplots of the questionnaire results for each visual hand rendering.}[
+    Pairwise Wilcoxon signed-rank tests with Holm-Bonferroni adjustment: * is \pinf{0.05}, ** is \pinf{0.01} and *** is \pinf{0.001}.
+    Lower is better for Difficulty and Uncomfortable; higher is better for Realism and Textures Match.
+  ][
+  \item By modality.
+  \item By task.
+  ]
+  \subfigsheight{70mm}
+  \subfig{results/questions_modalities}%
+  \subfig{results/questions_tasks}%
+\end{subfigs}

3-perception/vhar-textures/4-discussion.tex

@@ -0,0 +1,35 @@
+\section{Discussion}
+\label{discussion}
+
+In this study, we investigated the perception of visuo-haptic texture augmentation of real surfaces touched directly with the index fingertip, using visual texture overlays in \AR and haptic roughness textures generated by a vibrotactile device worn on the middle index phalanx.
+The nine evaluated pairs of visuo-haptic textures, taken from the \HaTT database \cite{culbertson2014one}, are models of real texture captures (\secref[related_work]{texture_rendering}).
+
+Their perception was evaluated in a two-task user study in which participants chose the most coherent combinations of visual and haptic textures (\level{Matching} task), and ranked all textures according to their perceived roughness (\level{Ranking} task).
+The visual textures were displayed statically on the real surface, while the haptic textures adapted in real time to the speed of the finger on the surface, giving the impression that the visuo-haptic textures were integrated into the surface.
+In addition, the interaction with the textures was designed to be as natural as possible, without imposing a specific speed of finger movement, as in similar studies \cite{asano2015vibrotactile,friesen2024perceived}.
+
+In the \level{Matching} task, participants were not able to effectively match the original visual and haptic texture pairs (\figref{results/matching_confusion_matrix}), except for the \level{Coffee Filter} texture, which was the smoothest both visually and haptically.
+However, almost all visual textures, except \level{Sandpaper~100}, were matched with at least one haptic texture at a level above chance.
+Similarly, five haptic textures were favored over the others to be matched with the visual textures.
+Thus, it seems that not all participants perceived visual textures in the same way and that they also hesitated between several haptic textures for a given visual texture.
+Indeed, the majority of users explained that, based on the roughness, granularity, or imperfections of the visual texture, they matched the haptic texture that seemed most similar or natural to what they imagined.
+Several strategies were used, as some participants reported using vibration frequency and/or amplitude to match a haptic texture.
+It should be noted that the task was rather difficult (\figref{results_questions}), as participants had no prior knowledge of the textures, there were no additional visual cues such as the shape of an object, and the term \enquote{roughness} had not been used by the experimenter prior to the \level{Ranking} task.
+
+The correspondence analysis (\figref{results/matching_correspondence_analysis}) highlighted that participants did indeed match visual and haptic textures primarily on the basis of their perceived roughness (\percent{60} of variance), which is in line with previous perception studies on real \cite{baumgartner2013visual} and virtual \cite{culbertson2014modeling} textures.
+The rankings (\figref{results/ranking_mean_ci}) confirmed that the participants all perceived the roughness of haptic textures very similarly, but that there was less consensus for visual textures, which is also in line with roughness rankings for real haptic and visual textures \cite{bergmanntiest2007haptic}.
+These results made it possible to identify and name groups of textures in the form of clusters (\figref{results_clusters}), and to construct confusion matrices between these clusters and between visual texture ranks with haptic clusters (\figref{results/haptic_visual_clusters_confusion_matrices}), showing that participants consistently identified and matched haptic and visual textures.
+\percent{30} of the matching variance of the correspondence analysis was also captured with a second dimension, opposing the roughest textures (\level{Metal Mesh}, \level{Sandpaper~100}), and to a lesser extent the smoothest (\level{Coffee Filter}, \level{Sandpaper~320}), with all other textures (\figref{results/matching_correspondence_analysis}).
+
+One hypothesis is that this dimension could be the perceived hardness (\secref[related_work]{hardness}) of the virtual materials, with \level{Metal Mesh} and smooth textures appearing harder than the other textures, whose granularity could have been perceived as bumps on the surface that could deform under finger pressure.
+Hardness is, with roughness, one of the main characteristics perceived by the vision and touch of real materials \cite{baumgartner2013visual,vardar2019fingertip}, but also on virtual haptic renderings \cite{culbertson2014modeling,degraen2019enhancing}.
+
+The last visuo-haptic roughness ranking (\figref{results/ranking_mean_ci}) showed that both haptic and visual sensory information were well integrated as the resulting roughness ranking was being in between the two individual haptic and visual rankings.
+Several strategies were reported: some participants first classified visually and then corrected with haptics, others classified haptically and then integrated visuals.
+While visual sensation did influence perception, as observed in previous haptic \AR studies \cite{punpongsanon2015softar,gaffary2017ar,fradin2023humans}, haptic sensation dominated here.
+This indicates that participants were more confident and relied more on the haptic roughness perception than on the visual roughness perception when integrating both in one coherent perception.
+
+Several participants also described attempting to identify visual and haptic textures using spatial breaks, edges or patterns, that were not reported when these textures were displayed in non-immersive \VEs with a screen \cite{culbertson2014modeling,culbertson2015should}.
+A few participants even reported that they clearly sensed patterns on haptic textures.
+However, the visual and haptic textures used were isotropic and homogeneous models of real texture captures, \ie their rendered roughness was constant and did not depend on the direction of movement but only on the speed of the finger (\secref[related_work]{texture_rendering}).
+Overall, the haptic device was judged to be comfortable, and the visual and haptic textures were judged to be fairly realistic and to work well together (\figref{results_questions}).

3-perception/vhar-textures/5-conclusion.tex

@@ -0,0 +1,25 @@
+\section{Conclusion}
+\label{conclusion}
+
+In this chapter, we investigated how users perceived simultaneous and co-localized visuo-haptic texture augmentations of real surfaces seen in immersive \OST-\AR and touched directly with the index finger.
+Using the wearable visuo-haptic augmentation system presented in \chapref{vhar_system}, the haptic roughness texture was rendered with on the \HaTT data-driven models and finger speed.
+In a user study, 20 participants rated the coherence, realism and perceived roughness of the combination of nine representative visuo-haptic texture pairs.
+
+The results showed that participants consistently identified and matched clusters of visual and haptic textures with similar perceived roughness.
+The texture rankings did indeed show that participants perceived the roughness of haptic textures to be very similar, but less so for visual textures, and the haptic roughness perception predominated the final roughness perception ranking of the original visuo-haptic pairs.
+This suggests that \AR visual textures that augments real surfaces can be enhanced with a set of data-driven vibrotactile haptic textures in a coherent and realistic manner.
+
+This paves the way for new \AR applications capable of augmenting a \RE with virtual visuo-haptic textures, such as visuo-haptic painting in artistic or object design context, or viewing and touching virtual objects in a museum or a showroom.
+The latter is illustrated in \figref{experiment/use_case}, where a user applies different visuo-haptic textures to a wall, in an interior design scenario, to compare them visually and by touch.
+
+\noindentskip This work was presented and published at the EuroHaptics 2024 conference:
+
+Erwan Normand, Claudio Pacchierotti, Eric Marchand, and Maud Marchal.
+\enquote{Augmenting the Texture Perception of Tangible Surfaces in Augmented Reality using Vibrotactile Haptic Stimuli}.
+In: \textit{EuroHaptics}. Lille, France, July 2024.
+
+\fig[0.5]{experiment/use_case}{
+  Illustration of the texture augmentation in \AR through an interior design scenario.
+}[
+  A user wearing an \AR headset and a wearable vibrotactile haptic device worn on their index is applying different virtual visuo-haptic textures to a real wall to compare them visually and by touch.
+]

3-perception/vhar-textures/vhar-textures.tex

@@ -0,0 +1,10 @@
+\chapter{Perception of Visual and Haptic Texture Augmentations in Augmented Reality}
+\mainlabel{vhar_textures}
+
+\chaptertoc
+
+\input{1-introduction}
+\input{2-experiment}
+\input{3-results}
+\input{4-discussion}
+\input{5-conclusion}