\section{User Study}
\label{experiment}

\begin{subfigs}{setup}{User Study. }[][
  \item The nine visuo-haptic textures used in the user study, selected from the HaTT database \cite{culbertson2014one}.
    The texture names were never shown, so as to prevent the use of the user's visual or haptic memory of the textures.
  \item Experimental setup.
    Participant sat in front of the tangible surfaces, which were augmented with visual textures displayed by the HoloLens~2 \AR headset and haptic roughness textures rendered by the vibrotactile haptic device placed on the middle index phalanx.
    A webcam above the surfaces tracked the finger movements.
  \item First person view of the user study, as seen through the immersive \AR headset HoloLens~2.
    The visual texture overlays are statically displayed on the surfaces, allowing the user to move around to view them from different angles.
    The haptic roughness texture is generated based on HaTT data-driven texture models and finger speed, and it is rendered on the middle index phalanx as it slides on the considered surface.
  ]
  \subfig[0.32]{experiment/textures}
  \subfig[0.32]{experiment/setup}
  \subfig[0.32]{experiment/view}
\end{subfigs}

The user study aimed at analyzing the user perception of tangible surfaces when augmented through a visuo-haptic texture using \AR and vibrotactile haptic feedback provided on the finger touching the surfaces.
%
Nine representative visuo-haptic texture pairs from the HaTT database \cite{culbertson2014one} were investigated in two tasks:
%
(1) a matching task, where participants had to find the haptic texture that best matched a given visual texture; and (2) a ranking task, where participants had to rank only the haptic textures, only the visual textures, and the visuo-haptic texture pairs according to their perceived roughness.
%
Our objective is to assess which haptic textures were associated with which visual textures, how the roughness of the visual and haptic textures are perceived, and whether the perceived roughness can explain the matches made between them.

\subsection{The textures}
\label{textures}

The 100 visuo-haptic texture pairs of the HaTT database \cite{culbertson2014one} were preliminary tested and compared using \AR and vibrotactile haptic feedback on the finger on a tangible surface.
%
These texture models were chosen as they are visuo-haptic representations of a wide range of real textures that are publicly available online.
%
Nine texture pairs were selected (\figref{setup}, left) to cover various perceived roughness, from rough to smooth, as listed: Metal Mesh, Sandpaper~100, Brick~2, Cork, Sandpaper~320, Velcro Hooks, Plastic Mesh~1, Terra Cotta, Coffee Filter.
%
All these visual and haptic textures are isotropic: their rendering (appearance or roughness) is the same whatever the direction of the movement on the surface, \ie there are no local deformations (holes, bumps, or breaks).

\subsection{Apparatus}
\label{apparatus}

\figref{setup} shows the experimental setup (middle) and the first person view (right) of the user study.
%
Nine 5-cm square cardboards with smooth, white melamine surface, arranged in a 3 \x 3 grid, were used as real tangible surfaces to augment.
%
Their poses were estimated with three 2-cm-square AprilTag fiducial markers glued on the surfaces grid.
%
Similarly, a 2-cm-square fiducial marker was glued on top of the vibrotactile actuator to detect the finger pose.
%
Positioned \qty{20}{\cm} above the surfaces, a webcam (StreamCam, Logitech) filmed the markers to track finger movements relative to the surfaces.
%
The visual textures were displayed on the tangible surfaces using the HoloLens~2 OST-AR headset (\figref{setup}, middle and right) within a \qtyproduct{43 x 29}{\degree} field of view at \qty{60}{\Hz}; a set of empirical tests enabled us to choose the best rendering characteristics in terms of transparency and brightness for the visual textures, that were used throughout the user study.
%
When a haptic texture was touched, a \qty{48}{kHz} audio signal was generated using the corresponding HaTT haptic texture model and the measured tangential speed of the finger, using the rendering procedure described in Culbertson \etal \cite{culbertson2014modeling}.
%
The normal force on the texture was assumed to be constant at \qty{1.2}{\N} to generate the audio signal from the model, as Culbertson \etal \cite{culbertson2015should}, who found that the HaTT textures can be rendered using only the speed as input without decreasing their perceived realism.
%
An amplifier (XY-502, not branded) converted this audio signal to a current transmitted to the vibrotactile voice-coil actuator (HapCoil-One, Actronika), that was encased in a 3D-printed plastic shell firmly attached to the middle index phalanx of the participant's dominant hand, similarly to previous studies \cite{asano2015vibrotactile,friesen2024perceived}.
%
This voice-coil actuator was chosen for its wide frequency range (\qtyrange{10}{1000}{\Hz}) and its relatively low acceleration distortion, as specified by the manufacturer\footnoteurl{https://www.actronika.com/haptic-solutions}.
%
Overall latency was measured to \qty{46 \pm 6}{\ms}, as a result of latency in image acquisition \qty{16 \pm 1}{\ms}, fiducial marker detection \qty{8 \pm 3}{\ms}, network synchronization \qty{4 \pm 1}{\ms}, audio sampling \qty{3 \pm 1}{\ms}, and the vibrotactile actuator latency (\qty{15}{\ms}, as specified by the manufacturer\footnotemark[5]).
%
This latency was below the \qty{60}{\ms} threshold for vibrotactile feedback \cite{okamoto2009detectability} and was not noticed by the participants.
%
The user study was held in a quiet room with no windows, with one light source of \qty{800}{\lumen} placed \qty{70}{\cm} above the table.

\subsection{Procedure and Collected Data}
\label{procedure}

Participants were first given written instructions about the experimental setup, the tasks, and the procedure of the user study.
%
Then, after having signed an informed consent form, they were asked to seat in front of the table with the experimental setup and to wear the HoloLens~2 \AR headset. The experimenter firmly attached the plastic shell encasing the vibrotactile actuator to the middle index phalanx of their dominant hand.
%
As the haptic device generated no audible noise, participants did not wear any noise reduction headphones.
%
A calibration of both the HoloLens~2 and the hand tracking was performed to ensure the correct alignment of the visual and haptic textures on the tangible surfaces.
%
Finally, participants familiarized with the augmented surface in a 2-min training session with textures different from the ones used in the user study.

Participants started with the \emph{matching task}.
%
They were informed that the user study involved nine pairs of corresponding visual and haptic textures that were separated and shuffled.
%
On each trial, the same visual texture was displayed on the nine tangible surfaces, while the nine haptic textures were rendered on only one of the surfaces at a time, \ie all surfaces were augmented by the same visual texture, but each surface was augmented by a different haptic texture.
%
The placement of the haptic textures was randomized before each trial.
%
Participants were instructed to look closely at the details of the visual textures and explore the haptic textures with a constant pressure and various speeds to find the haptic texture that best matched the visual texture, \ie choose the surface with the most coherent visual-haptic texture pair.
%
The texture names were never given or shown to prevent the use of visual or haptic memory of the textures, nor a definition of what roughness is was given, so as to let participants complete the task as naturally as possible, similarly to Bergmann Tiest \etal \cite{bergmanntiest2007haptic}.

Then, participants performed the \emph{ranking task}, employing the same setup as the matching task and the same 9 textures.
%
In this case, participants were asked to rank the textures according to their perceived roughness.
%
First, they ranked all the haptic textures (without any visual augmentation given), then all the visual textures (without any haptic augmentation given), and finally all the visuo-haptic texture pairs together, being informed that they were the correct matches as per the original HaTT database.
%
The placement of the textures was also randomized before each trial.

After each task, participants answered to the following 7-item Likert scale questions (1=Not at all, 7=Extremely):
%
(\textit{Haptic Difficulty}) How difficult was it to differentiate the tactile textures?
(\textit{Visual Difficulty}) How difficult was it to differentiate the visual textures?
(\textit{Textures Match}) For the visual-tactile pairs you have chosen, how coherent were the tactile textures with the corresponding visual textures?
(\textit{Haptic Realism}) How realistic were the tactile textures?
(\textit{Visual Realism}) How realistic were the visual textures?
(\textit{Uncomfort}) How uncomfortable was to use the haptic device?
%
In an open question, participants commented also on their strategy for completing the matching task (How did you associate the tactile textures with the visual textures?) and the ranking task (How did you rank the textures?).
%
The user study took on average 1 hour to complete.

\subsection{Participants}
\label{participants}

Twenty participants took part to the user study (12 males, 7 females, 1 preferred not to say), aged between 20 and 60 years old (M=29.1, SD=9.4).
%
One participant was left-handed, all others were right-handed; they all performed the user study with their dominant hand.
%
All participants had normal or corrected-to-normal vision and none of them had a known hand or finger impairment.
%
They rated their experience with haptics, \AR, and \VR (\enquote{I use it every month or more}); 10 were experienced with haptics, 2 with \AR, and 10 with \VR.
%
Experiences were correlated between haptics and \AR (\spearman{0.53}), haptics and \VR (\spearman{0.61}), and \AR and \VR (\spearman{0.74}); but not with age (\spearman{-0.06} to \spearman{-0.05}) or gender (\spearman{0.10} to \spearman{0.27}).
%
Participants were recruited at the university on a voluntary basis.
%
They all signed an informed consent form before the user study.

\subsection{Design}
\label{design}

The matching task was a single-factor within-subjects design, \textit{Visual Texture}, with the following levels:
%
Metal Mesh, Sandpaper~100, Brick~2, Cork, Sandpaper~320, Velcro Hooks, Plastic Mesh~1, Terra Cotta, Coffee Filter.
%
To account for learning and fatigue effects, the order of \textit{Visual Texture} was counterbalanced using a balanced \numproduct{18 x 18} Latin square design.
%
A total of 20 participants \x 9 textures \x 3 repetitions = 540 matching trials were collected.
%
The ranking task was a single-factor within-subjects design, \textit{Modality}, with the following levels: Visual, Haptic, Visuo-Haptic.
%
The order of \textit{Modality} was fixed as listed above.
%
A total of 20 participants \x 3 modalities = 60 ranking trials were collected.