220 lines
19 KiB
TeX
220 lines
19 KiB
TeX
\section{User Study}
|
|
\sublabel{experiment}
|
|
|
|
\begin{subfigswide}{renderings}{%
|
|
The three visual rendering conditions and the experimental procedure of the two-alternative forced choice (2AFC) psychophysical study.
|
|
%
|
|
During a trial, two tactile textures were rendered on the augmented area of the paper sheet (black rectangle) for 3\,s each, one after the other, then the participant chose which one was the roughest.
|
|
%
|
|
The visual rendering stayed the same during the trial.
|
|
%
|
|
(\level{Real}) The real environment and real hand view without any visual augmentation.
|
|
%
|
|
(\level{Mixed}) The real environment and hand view with the virtual hand.
|
|
%
|
|
(\level{Virtual}) Virtual environment with the virtual hand.
|
|
%
|
|
%The pictures are captured directly from the Microsoft HoloLens 2 headset.
|
|
}
|
|
\hidesubcaption
|
|
\subfig[0.32][]{experiment/real}
|
|
\subfig[0.32][]{experiment/mixed}
|
|
\subfig[0.32][]{experiment/virtual}
|
|
\end{subfigswide}
|
|
|
|
Our visuo-haptic rendering system, described in \secref{xr_perception:method}, allows free exploration of virtual vibrotactile textures on tangible surfaces directly touched with the bare finger to simulate roughness augmentation, while the visual rendering of the hand and environment can be controlled to be in AR or VR.
|
|
%
|
|
The user study aimed to investigate the effect of visual hand rendering in AR or VR on the perception of roughness texture augmentation. % of a touched tangible surface.
|
|
%
|
|
In a two-alternative forced choice (2AFC) task, participants compared the roughness of different tactile texture augmentations in three visual rendering conditions: without any visual augmentation (see \figref{renderings}, \level{Real}), in AR with a realistic virtual hand superimposed on the real hand (see \figref{renderings}, \level{Mixed}), and in VR with the same virtual hand as an avatar (see \figref{renderings}, \level{Virtual}).
|
|
%
|
|
In order not to influence the perception, as vision is an important source of information and influence for the perception of texture~\autocite{bergmanntiest2007haptic,yanagisawa2015effects,normand2024augmenting,vardar2019fingertip}, the touched surface was visually a uniform white; thus only the visual aspect of the hand and the surrounding environment is changed.
|
|
|
|
|
|
\subsection{Participants}
|
|
\sublabel{participants}
|
|
|
|
Twenty participants were recruited for the study (16 males, 3 females, 1 prefer not to say), aged between 18 and 61 years old (\median{26}{}, \iqr{6.8}{}).
|
|
%
|
|
All participants had normal or corrected-to-normal vision, none of them had a known hand or finger impairment.
|
|
%
|
|
One was left-handed while the rest were right-handed; they all performed the task with their right index.
|
|
%
|
|
In rating their experience with haptics, AR and VR (\enquote{I use it several times a year}), 12 were experienced with haptics, 5 with AR, and 10 with VR.
|
|
%
|
|
Experiences were correlated between haptics and VR (\pearson{0.59}), and AR and VR (\pearson{0.67}) but not haptics and AR (\pearson{0.20}) nor haptics, AR, or VR with age (\pearson{0.05} to \pearson{0.12}).
|
|
%
|
|
Participants were recruited at the university on a voluntary basis.
|
|
%
|
|
They all signed an informed consent form before the user study and were unaware of its purpose.
|
|
|
|
|
|
\subsection{Apparatus}
|
|
\sublabel{apparatus}
|
|
|
|
An experimental environment similar as \citeauthorcite{gaffary2017ar} was created to ensure a similar visual rendering in AR and VR (see \figref{renderings}).
|
|
%
|
|
It consisted of a \qtyproduct{300 x 210 x 400}{\mm} medium-density fibreboard (MDF) box with a paper sheet glued inside, and a \qtyproduct{15 x 5}{\mm} rectangle printed on the sheet to delimit the area where the tactile textures were rendered.
|
|
%
|
|
A single light source of \qty{800}{\lumen} placed \qty{70}{\cm} above the table fully illuminated the inside of the box.
|
|
%
|
|
Participants rated the roughness of the paper (without any texture augmentation) before the experiment on a 7-point Likert scale (1 = Extremely smooth, 7 = Extremely rough) as quite smooth (\mean{2.5}, \sd{1.3}).
|
|
|
|
%The visual rendering of the virtual hand and environment was achieved using the Microsoft HoloLens~2, an OST-AR headset with a \qtyproduct{43 x 29}{\degree} field of view (FoV) and a \qty{60}{\Hz} refresh rate, running a custom application made with Unity 2021.1.0f1 and Mixed Reality Toolkit (MRTK) 2.7.2.
|
|
%f
|
|
The virtual environment was carefully reproducing the real environment including the geometry of the box, the textures, the lighting, and the shadows (see \figref{renderings}, \level{Virtual}).
|
|
%
|
|
The virtual hand model was a gender-neutral human right hand with realistic skin texture, similar to the one used by \citeauthorcite{schwind2017these}.
|
|
%
|
|
Its size was adjusted to match the real hand of the participants before the experiment.
|
|
%
|
|
%An OST-AR headset (Microsoft HoloLens~2) was chosen over a VST-AR headset because the former only adds virtual content to the real environment, while the latter streams a real-time video capture of the real environment, and one of our objectives was to directly compare a virtual environment replicating a real one, not to a video feed that introduces many other visual limitations~\autocite{macedo2023occlusion}.
|
|
%
|
|
The visual rendering of the virtual hand and environment is described in \secref{xr_perception:virtual_real_alignment}.
|
|
%
|
|
%In the \level{Virtual} rendering, a cardboard mask (with holes for sensors) was attached to the headset to block the view of the real environment and simulate a VR headset (see \figref{method/headset}).
|
|
%
|
|
To ensure for the same FoV in all \factor{Visual Rendering} condition, a cardboard mask was attached to the AR headset (see \figref{method/headset}).
|
|
%
|
|
In the \level{Virtual} rendering, the mask had only holes for sensors to block the view of the real environment and simulate a VR headset.
|
|
%
|
|
In the \level{Mixed} and \level{Real} conditions, the mask had two additional holes for the eyes that matched the FoV of the HoloLens~2 (see \figref{method/headset}).
|
|
%
|
|
\figref{renderings} shows the resulting views in the three considered \factor{Visual Rendering} conditions.
|
|
|
|
%A vibrotactile voice-coil device (HapCoil-One, Actronika), incased in a 3D-printed plastic shell, was firmly attached to the right index finger of the participants using a Velcro strap (see \figref{method/device}), was used to render the textures
|
|
%
|
|
%This voice-coil was chosen for its wide frequency range (\qtyrange{10}{1000}{\Hz}) and its relatively low acceleration distortion, as specified by the manufacturer\footnotemark[1].
|
|
%
|
|
%It was driven by an audio amplifier (XY-502, not branded) connected to a computer that generated the audio signal of the textures as described in \secref{xr_perception:method}, using the NAudio library and the WASAPI driver in exclusive mode.
|
|
%
|
|
%The position of the finger relative to the sheet was estimated using a webcam placed on top of the box (StreamCam, Logitech) and the OpenCV library by tracking a \qty{2}{\cm} square fiducial marker (AprilTag) glued to top of the vibrotactile actuator.
|
|
%
|
|
%The total texture latency was measured to \qty{36 \pm 4}{\ms}, as a result of latency in image acquisition \qty{16 \pm 1}{\ms}, fiducial marker detection \qty{2 \pm 1}{\ms}, audio sampling \qty{3 \pm 1}{\ms}, and the vibrotactile actuator latency (\qty{15}{\ms}, as specified by the manufacturer\footnotemark[1]), and was below the \qty{60}{\ms} threshold for vibrotactile feedback \autocite{okamoto2009detectability}.
|
|
%
|
|
%The virtual hand followed the position of the fiducial marker with a slightly higher latency due to the network synchronization \qty{4 \pm 1}{\ms} between the computer and the HoloLens~2.
|
|
|
|
Participants sat comfortably in front of the box at a distance of \qty{30}{\cm}, wearing the HoloLens~2 with a cardboard mask attached, so that only the inside of the box was visible, as shown in \figref{method/apparatus}.
|
|
%
|
|
%A vibrotactile voice-coil actuator (HapCoil-One, Actronika) was encased in a 3D printed plastic shell with a \qty{2}{\cm} AprilTag glued to top, and firmly attached to the middle phalanx of the right index finger of the participants using a Velcro strap.
|
|
%
|
|
The generation of the virtual texture and the control of the virtual hand is described in \secref{xr_perception:method}.
|
|
%
|
|
They also wore headphones with a pink noise masking the sound of the voice-coil.
|
|
%
|
|
The user study was held in a quiet room with no windows.
|
|
|
|
|
|
\subsection{Procedure}
|
|
\sublabel{procedure}
|
|
|
|
Participants were first given written instructions about the experimental setup and procedure, the informed consent form to sign, and a demographic questionnaire.
|
|
%
|
|
%They were then asked to sit in front of the box and wear the HoloLens~2 and headphones while the experimenter firmly attached the vibrotactile device to the middle phalanx of their right index finger (see \figref{method/apparatus}).
|
|
%
|
|
A calibration was then performed to adjust the HoloLens~2 to the participant's interpupillary distance, the virtual hand to the real hand size, and the fiducial marker to the finger position.
|
|
%
|
|
They familiarised themselves with the task by completing four training trials with the most different pair of textures.
|
|
%
|
|
The trials were divided into three blocks, one for each \factor{Visual Rendering} condition, with a break and questionnaire between each block.
|
|
%
|
|
Before each block, the experimenter ensured that the virtual environment and the virtual hand were correctly aligned with their real equivalents, that the haptic device was in place, and attached the cardboard mask corresponding to the next \factor{Visual Rendering} condition to the headset.
|
|
|
|
The participant started the trial by clicking the middle button of a mouse with the left hand.
|
|
%
|
|
The first texture was then rendered on the augmented area of the paper sheet for \qty{3}{\s} and, after a \qty{1}{\s} pause, the second texture was also rendered for \qty{3}{\s}.
|
|
%
|
|
The participant then had to decide which texture was the roughest by clicking the left (for the first texture) or right (for the second texture) button of the mouse and confirming their choice by clicking the middle button again.
|
|
%
|
|
If the participant moved their finger away from the texture area, the texture timer was paused until they returned.
|
|
%
|
|
Participants were asked to explore the textures as they would in real life by moving their finger back and forth over the texture area at different speeds.
|
|
|
|
One of the textures in the tested pair was always the reference texture, while the other was the comparison texture.
|
|
%
|
|
Participants were not told that there was a reference and a comparison texture.
|
|
%
|
|
The order of presentation was randomised and not revealed to the participants.
|
|
%
|
|
All textures were rendered as described in \secref{xr_perception:texture_generation} with period $\lambda$ of \qty{2}{\mm}, but with different amplitudes $A$ to create different levels of roughness.
|
|
%
|
|
Preliminary studies allowed us to determine a range of amplitudes that could be felt by the participants and were not too uncomfortable, and the reference texture was chosen to be the one with the middle amplitude.
|
|
|
|
|
|
\subsection{Experimental Design}
|
|
\sublabel{experimental_design}
|
|
|
|
The user study was a within-subjects design with two factors:
|
|
%
|
|
\begin{itemize}
|
|
\item \factor{Visual Rendering}, consisting of the augmented or virtual view of the environment, the hand and the wearable haptic device, with 3 levels: real environment and real hand view without any visual augmentation (see \figref{renderings}, \level{Real}), real environment and hand view with the virtual hand (see \figref{renderings}, \level{Mixed}) and virtual environment with the virtual hand (see \figref{renderings}, \level{Virtual}).
|
|
\item \factor{Amplitude Difference}, consisting of the difference in amplitude between the comparison and the reference textures, with 6 levels: \qtylist{0; +-12.5; +-25.0; +-37.5}{\%}.
|
|
\end{itemize}
|
|
|
|
A trial consisted on a two-alternative forced choice (2AFC) task where a participant had to touch two virtual vibrotactile textures one after the other and decide which one was the roughest.
|
|
%
|
|
To avoid any order effect, the order of \factor{Visual Rendering} conditions was counterbalanced between participants using a balanced Latin square design.
|
|
%
|
|
Within each condition, the order of presentation of the reference and comparison textures was also counterbalanced, and all possible texture pairs were presented in random order and repeated three times.
|
|
%
|
|
A total of 3 visual renderings \x 6 amplitude differences \x 2 texture presentation order \x 3 repetitions = 107 trials were performed by each participant.
|
|
|
|
|
|
\subsection{Collected Data}
|
|
\sublabel{collected_data}
|
|
|
|
For each trial, the \textit{Texture Choice} by the participant as the roughest of the pair was recorded.
|
|
%
|
|
The \textit{Response Time} between the end of the trial and the choice of the participant was also measured as an indicator of the difficulty of the task.
|
|
%
|
|
At each frame the \textit{Finger Position} and \textit{Finger Speed} were recorded to control for possible differences in texture exploration behaviour.
|
|
%
|
|
After each \factor{Visual Rendering} block of trials, participants rated their experience with the vibrotactile textures (all blocks), the vibrotactile device (all blocks), the virtual hand rendering (all except \level{Mixed} block) and the virtual environment (\level{Virtual} block) using the questions shown in \tabref{questions}.
|
|
%
|
|
%They also assessed their workload with the NASA Task Load Index (\textit{NASA-TLX}) questionnaire after each blocks of trials.
|
|
%
|
|
For all questions, participants were shown only labels (\eg \enquote{Not at all} or \enquote{Extremely}) and not the actual scale values (\eg 1 or 5), following the recommendations of \citeauthorcite{muller2014survey}.
|
|
|
|
\newcommand{\scalegroup}[2]{\multirow{#1}{1\linewidth}{#2}}
|
|
\begin{tabwide}{questions}{%
|
|
Questions asked to participants after each \factor{Visual Rendering} block of trials. %
|
|
Unipolar scale questions were 5-point Likert scales (1 = Not at all, 2 = Slightly, 3 = Moderately, 4 = Very and 5 = Extremely), and %
|
|
bipolar scale questions were 7-point Likert scales (1 = Extremely A, 2 = Moderately A, 3 = Slightly A, 4 = Neither A nor B, 5 = Slightly B, 6 = Moderately B, 7 = Extremely B), %
|
|
where A and B are the two poles of the scale (indicated in parentheses in the Scale column of the questions).
|
|
%, and NASA TLX questions were bipolar 100-points scales (0 = Very Low and 100 = Very High, except for Performance where 0 = Perfect and 100 = Failure). %
|
|
Participants were shown only the labels for all questions.
|
|
}
|
|
\begin{tabularx}{\linewidth}{l X p{0.2\linewidth}}
|
|
\toprule
|
|
\textbf{Code} & \textbf{Question} & \textbf{Scale} \\
|
|
\midrule
|
|
Texture Agency & Did the tactile sensations of texture seem to be caused by your movements? & \scalegroup{4}{Unipolar (1-5)} \\
|
|
Texture Realism & How realistic were the tactile textures? & \\
|
|
Texture Plausibility & Did you feel like you were actually touching textures? & \\
|
|
Texture Latency & Did the sensations of texture seem to lag behind your movements? & \\
|
|
\midrule
|
|
Vibration Location & Did the vibrations seem to come from the surface you were touching or did you feel them on the top of your finger? & Bipolar (1=surface, 7=top of finger) \\
|
|
Vibration Strength & Overall, how weak or strong were the vibrations? & Bipolar (1=weak, 7=strong) \\
|
|
Device Distraction & To what extent did the vibrotactile device distract you from the task? & \scalegroup{2}{Unipolar (1-5)} \\
|
|
Device Discomfort & How uncomfortable was it to use the vibrotactile device? & \\
|
|
\midrule
|
|
Hand Agency & Did the movements of the virtual hand seem to be caused by your movements? & \scalegroup{5}{Unipolar (1-5)} \\
|
|
Hand Similarity & How similar was the virtual hand to your own hand in appearance? & \\
|
|
Hand Ownership & Did you feel the virtual hand was your own hand? & \\
|
|
Hand Latency & Did the virtual hand seem to lag behind your movements? & \\
|
|
Hand Distraction & To what extent did the virtual hand distract you from the task? & \\
|
|
Hand Reference & Overall, did you focus on your own hand or the virtual hand to complete the task? & Bipolar (1=own hand, 7=virtual hand) \\
|
|
\midrule
|
|
Virtual Realism & How realistic was the virtual environment? & \scalegroup{2}{Unipolar (1-5)} \\
|
|
Virtual Similarity & How similar was the virtual environment to the real one? & \\
|
|
%\midrule
|
|
%Mental Demand & How mentally demanding was the task? & \scalegroup{6}{Bipolar (0-100)} \\
|
|
%Temporal Demand & How hurried or rushed was the pace of the task? & \\
|
|
%Physical Demand & How physically demanding was the task? & \\
|
|
%Performance & How successful were you in accomplishing what you were asked to do? & \\
|
|
%Effort & How hard did you have to work to accomplish your level of performance? & \\
|
|
%Frustration & How insecure, discouraged, irritated, stressed, and annoyed were you? & \\
|
|
\bottomrule
|
|
\end{tabularx}
|
|
\end{tabwide}
|