diff --git a/3-perception/vhar-system/2-method.tex b/3-perception/vhar-system/2-method.tex index e8c4aa9..49fce30 100644 --- a/3-perception/vhar-system/2-method.tex +++ b/3-perception/vhar-system/2-method.tex @@ -4,7 +4,7 @@ The visuo-haptic texture rendering system is based on: \begin{enumerate}[label=(\arabic*)] \item a real-time interaction loop between the finger movements and a coherent visuo-haptic feedback simulating the sensation of a touched texture, - \item a precise alignment of the \VE with its real counterpart, and + \item a precise registration of the \VE with its real counterpart, and \item a modulation of the signal frequency by the estimated finger speed with a phase matching. \end{enumerate} @@ -44,14 +44,24 @@ The velocity (without angular velocity) of the finger marker, denoted as $\pose{ It is then filtered with another 1€ filter with the same parameters, and denoted as $\pose{c}{\hat{\dot{X}}}{f}$. Finally, this filtered finger velocity is transformed into the augmented surface frame $\poseFrame{s}$ to be used in the vibrotactile signal generation, such as $\pose{s}{\hat{\dot{X}}}{f} = \pose{s}{T}{c} \, \pose{c}{\hat{\dot{X}}}{f}$. -\subsection{Virtual Environment Alignment} -\label{virtual_real_alignment} +\subsection{Virtual Environment Registration} +\label{virtual_real_registration} +\comans{JG}{The registration process between the external camera, the finger, surface and HoloLens could have been described in more detail. Specifically, it could have been described clearer how the HoloLens coordinate system was aligned (e.g., by also tracking the fiducials on the surface and or finger).}{This has been better described.} Before a user interacts with the system, it is necessary to design a \VE that will be registered with the \RE during the experiment. Each real element tracked by a marker is modelled virtually, \eg the hand and the augmented surface (\figref{device}). In addition, the pose and size of the virtual textures were defined on the virtual replicas. -During the experiment, the system uses marker pose estimates to align the virtual models with their real world counterparts. -This allows to detect if a finger touches a virtual texture using a collision detection algorithm (Nvidia PhysX), and to show the virtual elements and textures in real-time, aligned with the \RE, using the considered \AR or \VR headset. +Prior to any usage, it is necessary to register the \VE with the \RE. +First, the coordinate system of the headset is manually aligned with that of the external camera by the experimenter using a raycast from the headset to the origin point of the camera (the center of the black rectangle in the box in \figref{apparatus}). +This resulted in a \qty{\pm .5}{\cm} spatial alignment error between the \RE and the \VE. +While this was sufficient for our use cases, other methods can achieve better accuracy if needed \cite{grubert2018survey}. +The registration of the coordinate systems of the camera and the headset thus allows the use of the marker estimation poses performed with the camera to display in the headset the virtual models aligned with their real-world counterparts. + +\comans{JG}{A description if and how the offset between the lower side of the fingertip touching the surface and the fiducial mounted on the top of the finger was calibrated / compensated is missing}{This has been better described.} +An additional calibration is performed to compensate for the offset between the finger contact point and the estimated marker pose \cite{son2022effect}. +The current user then places the index finger on the origin point, whose respective poses are known from the attached fiducial markers. +The transformation between the marker pose of the finger and the finger contact point can be estimated and compensated with an inverse transformation. +This allows to detect if the calibrated real finger touches a virtual texture using a collision detection algorithm (Nvidia PhysX). In our implementation, the \VE is designed with Unity (v2021.1) and the Mixed Reality Toolkit (v2.7)\footnoteurl{https://learn.microsoft.com/windows/mixed-reality/mrtk-unity}. The visual rendering is achieved using the Microsoft HoloLens~2, an \OST-\AR headset with a \qtyproduct{43 x 29}{\degree} \FoV, a \qty{60}{\Hz} refresh rate, and self-localisation capabilities. diff --git a/3-perception/vhar-textures/2-experiment.tex b/3-perception/vhar-textures/2-experiment.tex index e97e07b..0b0e23a 100644 --- a/3-perception/vhar-textures/2-experiment.tex +++ b/3-perception/vhar-textures/2-experiment.tex @@ -27,7 +27,7 @@ The user study was held in a quiet room with no windows, with one light source o Nine \qty{5}{\cm} square cardboards with smooth, white melamine surface, arranged in a \numproduct{3 x 3} grid, were used as real surfaces to augment. Their poses were estimated with three \qty{2}{\cm} AprilTag fiducial markers glued on the surfaces grid. Similarly, a \qty{2}{\cm} fiducial marker was glued on top of the vibrotactile actuator to detect the finger pose. -Positioned \qty{20}{\cm} above the surfaces, a webcam (StreamCam, Logitech) filmed the markers to track finger movements relative to the surfaces, as described in \secref[vhar_system]{virtual_real_alignment}. +Positioned \qty{20}{\cm} above the surfaces, a webcam (StreamCam, Logitech) filmed the markers to track finger movements relative to the surfaces, as described in \secref[vhar_system]{virtual_real_registration}. The visual textures were displayed on the real surfaces using the \OST-\AR headset Microsoft HoloLens~2 running a custom application at \qty{60}{FPS} made with Unity (v2021.1) and Mixed Reality Toolkit (v2.7). A set of empirical tests enabled us to choose the best rendering characteristics in terms of transparency and brightness for the visual textures, that were used throughout the user study. @@ -58,7 +58,7 @@ Participants were first given written instructions about the experimental setup, Then, after having signed an informed consent form, they were asked to seat in front of the table with the experimental setup and to wear the \AR headset. %The experimenter firmly attached the plastic shell encasing the vibrotactile actuator to the middle index phalanx of their dominant hand. As the haptic textures generated no audible noise, participants did not wear any noise reduction headphones. -A calibration of both the HoloLens~2 and the finger pose estimation was performed to ensure the correct alignment of the visual and haptic textures on the real surfaces. +A calibration of both the HoloLens~2 and the finger pose estimation was performed to ensure the correct registration of the visuo-haptic textures and the real finger with the real surfaces, as described in \secref[vhar_system]{virtual_real_registration}. Finally, participants familiarized with the augmented surface in a \qty{2}{min} training session with textures different from the ones used in the user study. Participants started with the \level{Matching} task. diff --git a/3-perception/xr-perception/3-experiment.tex b/3-perception/xr-perception/3-experiment.tex index b283aa9..a0057af 100644 --- a/3-perception/xr-perception/3-experiment.tex +++ b/3-perception/xr-perception/3-experiment.tex @@ -14,9 +14,9 @@ In order not to influence the perception, as vision is an important source of in The visual rendering stayed the same during the trial. The pictures are captured directly from the Microsoft HoloLens 2 headset. ][ - \item The real environment and real hand view without any visual augmentation. - \item The real environment and hand view with the virtual hand. - \item Virtual environment with the virtual hand. + \item The \RE with real hand view and without any visual augmentation. + \item The \RE with real hand and virtual hand view. + \item The \VE with the virtual hand. ] \subfig[0.326]{experiment/real} \subfig[0.326]{experiment/mixed} @@ -31,13 +31,15 @@ It consisted of a \qtyproduct{30 x 21 x 40}{\cm} medium-density fibreboard box w A single light source of \qty{800}{\lumen} placed \qty{70}{\cm} above the table fully illuminated the inside of the box. Participants rated the roughness of the paper (without any texture augmentation) before the experiment on a 7-point Likert scale (1~=~Extremely smooth, 7~=~Extremely rough) as quite smooth (\mean{2.5}, \sd{1.3}). -The visual rendering of the virtual hand and environment was achieved using the \OST-\AR headset Microsoft HoloLens~2 (\secref[vhar_system]{virtual_real_alignment}) running at \qty{60}{FPS} a custom application made with Unity (v2021.1) and Mixed Reality Toolkit (v2.7). +The visual rendering of the virtual hand and the \VE was achieved using the \OST-\AR headset Microsoft HoloLens~2 running at \qty{60}{FPS} a custom application made with Unity (v2021.1) and Mixed Reality Toolkit (v2.7). An \OST-\AR headset was chosen over a \VST-\AR headset because the former only adds virtual content to the \RE, while the latter streams a real-time video capture of the \RE, and one of our objectives was to directly compare a \VE replicating a real one, not to a video feed that introduces many other visual limitations (\secref[related_work]{ar_displays}). -The \VE carefully reproduced the \RE, including the geometry of the box, textures, lighting, and shadows (\figref{renderings}, \level{Virtual}). +\comans{JG}{In addition, the lag between the real and virtual hand in the Mixed condition could have been quantified (e.g. using a camera filming through the headset) to shed more light on the reported differences, as also noted in Section 4.5, as well as the registration error between the real and the virtual hand (as visible in Figure 4.1, Mixed).}{This has been added.} +We carefully reproduced the \RE in the \VE, including the geometry of the box, textures, lighting, and shadows (\figref{renderings}, \level{Virtual}). The virtual hand model was a gender-neutral human right hand with realistic skin texture, similar to that used by \textcite{schwind2017these}. -Its size was adjusted to match the real hand of the participants before the experiment. -The visual rendering of the virtual hand and environment is described in \secref[vhar_system]{virtual_real_alignment}. +Prior to the experiment, the virtual hand and the \VE were registered to the real hand of the participant and the \RE, respectively, as described in \secref[vhar_system]{virtual_real_registration}. +The size of the virtual hand was also manually adjusted to match the real hand of the participant. +A \qty{\pm .5}{\cm} spatial alignment error (\secref[vhar_system]{virtual_real_registration}) and a \qty{160 \pm 30}{\ms} lag (\secref[vhar_system]{virtual_real_registration}) between the real hand the virtual hand were measured. To ensure the same \FoV in all \factor{Visual Rendering} condition, a cardboard mask was attached to the \AR headset (\figref{experiment/headset}). In the \level{Virtual} rendering, the mask only had holes for sensors to block the view of the \RE and simulate a \VR headset. @@ -63,7 +65,7 @@ The user study was held in a quiet room with no windows. \label{procedure} Participants were first given written instructions about the experimental setup and procedure, the informed consent form to sign, and a demographic questionnaire. -A calibration was then performed to adjust the HoloLens~2 to the participant's interpupillary distance (IPD), the virtual hand to the real hand size, and the fiducial marker to the finger position. +The calibration was then performed to adjust the HoloLens~2 to the participant's interpupillary distance, the fiducial marker to the finger position, and the virtual hand size to the real hand. They familiarized themselves with the task by completing four training trials with the most different pair of textures. The trials were divided into three blocks, one for each \factor{Visual Rendering} condition, with a break and questionnaire between each block. Before each block, the experimenter ensured that the \VE and the virtual hand were correctly aligned with their real equivalents, that the haptic device was in place, and attached the cardboard mask corresponding to the next \factor{Visual Rendering} condition to the headset. diff --git a/4-manipulation/visual-hand/2-method.tex b/4-manipulation/visual-hand/2-method.tex index f643f72..43bbeac 100644 --- a/4-manipulation/visual-hand/2-method.tex +++ b/4-manipulation/visual-hand/2-method.tex @@ -100,7 +100,7 @@ This design led to a total of 2 manipulation tasks \x 6 visual hand augmentation \subsection{Apparatus} \label{apparatus} -We used the \OST-\AR headset HoloLens~2, as described in \secref[vhar_system]{virtual_real_alignment}. +We used the \OST-\AR headset HoloLens~2, as described in \secref[vhar_system]{virtual_real_registration}. It is also able to track the user's fingers. We measured the latency of the hand tracking at \qty{15}{\ms}, independent of the hand movement speed. diff --git a/references.bib b/references.bib index 6f90c98..d831812 100644 --- a/references.bib +++ b/references.bib @@ -3303,6 +3303,15 @@ doi = {10/fj8xvn} } +@inproceedings{son2022effect, + title = {Effect of {{Contact Points Feedback}} on {{Two-Thumb Touch Typing}} in {{Virtual Reality}}}, + booktitle = {{{CHI Conf}}. {{Hum}}. {{Factors Comput}}. {{Syst}}.}, + author = {Son, Jeongmin and Ahn, Sunggeun and Kim, Sunbum and Lee, Geehyuk}, + date = {2022}, + doi = {10/g8234n}, + isbn = {978-1-4503-9156-6} +} + @inproceedings{speicher2019what, title = {What Is {{Mixed Reality}}?}, booktitle = {{{CHI Conf}}. {{Hum}}. {{Factors Comput}}. {{Syst}}.},