WIP xr-perception

2-perception/vhar-system/1-introduction.tex

@@ -1,20 +1,17 @@
 % Even before manipulating a visual representation to induce a haptic sensation, shifts and latencies between user input and co-localised visuo-haptic feedback can be experienced differently in \AR and \VR, which we aim to investigate in this work.
 
-%Imagine you're an archaeologist or in a museum, and you want to examine an ancient object.
-%
-%But it is too fragile to touch directly.
-%
-%What if you could still grasp it and manipulate it through a tangible object in your hand, whose visual appearance has been modified using Augmented Reality (AR)?
-%
-%And what if you could also feel its shape or texture?
-%
-%Such tactile augmentation is made possible by wearable haptic devices, which are worn directly on the finger or hand and can provide a variety of sensations on the skin, while being small, light and discreet \cite{pacchierotti2017wearable}.
-%
-Wearable haptic devices, worn directly on the finger or hand, have been used to render a variety of tactile sensations to virtual objects seen in \VR \cite{choi2018claw,detinguy2018enhancing,pezent2019tasbi} or \AR \cite{maisto2017evaluation,meli2018combining,teng2021touch}.
+Wearable haptic devices have proven to be effective in modifying the perception of a touched tangible surface, without modifying the tangible, nor covering the fingertip, forming a haptic \AE \cite{bau2012revel,detinguy2018enhancing,salazar2020altering}.
+
+Second, many works have investigated the haptic augmentation of textures, but few have integrated them with immersive \VEs or have considered the influence of the visual rendering on their perception.
+Still, it is known that the visual feedback can alter the perception of real and virtual haptic sensations \cite{schwind2018touch,choi2021augmenting} but also that the force feedback perception of grounded haptic devices is not the same in \AR and \VR \cite{diluca2011effects,gaffary2017ar}.
+
+% Insist on the advantage of wearable : augment any surface see bau2012revel
+
+Wearable haptic devices, worn directly on the finger or hand, have been used to render a variety of tactile sensations to \VOs seen in \VR \cite{choi2018claw,detinguy2018enhancing,pezent2019tasbi} or \AR \cite{maisto2017evaluation,meli2018combining,teng2021touch}.
 %
 They have also been used to alter the perception of roughness, stiffness, friction, and local shape perception of real tangible objects \cite{asano2015vibrotactile,detinguy2018enhancing,salazar2020altering}.
 %
-Such techniques place the actuator \emph{close} to the point of contact with the real environment, leaving the user free to directly touch the tangible.
+Such techniques place the actuator \emph{close} to the point of contact with the \RE, leaving the user free to directly touch the tangible.
 %
 This combined use of wearable haptics with tangible objects enables a haptic \emph{augmented} reality (HAR) \cite{bhatia2024augmenting} that can provide a rich and varied haptic feedback.
 
@@ -22,7 +19,7 @@ The degree of reality/virtuality in both visual and haptic sensory modalities ca
 %
 Although \AR and \VR are closely related, they have significant differences that can affect the user experience \cite{genay2021virtual,macedo2023occlusion}.
 %
-%By integrating visual virtual content into the real environment, \AR keeps the hand of the user, the haptic devices worn and the tangibles touched visible, unlike \VR where they are hidden by immersing the user into a visual virtual environment.
+%By integrating visual virtual content into the \RE, \AR keeps the hand of the user, the haptic devices worn and the tangibles touched visible, unlike \VR where they are hidden by immersing the user into a visual virtual environment.
 %
 %Current \AR systems also suffer from display and rendering limitations not present in \VR, affecting the user experience with virtual content that may be less realistic or inconsistent with the real augmented environment \cite{kim2018revisiting,macedo2023occlusion}.
 %
@@ -40,13 +37,15 @@ We focus on the perception of roughness, one of the main tactile sensations of m
 %
 By understanding how these visual factors influence the perception of haptically augmented tangible objects, the many wearable haptic systems that already exist but have not yet been fully explored with \AR can be better applied and new visuo-haptic renderings adapted to \AR can be designed.
 
-Our contributions are:
-%
+\noindentskip The contributions of this chapter are:
 \begin{itemize}
-  \item A system for rendering virtual vibrotactile roughness textures in real time on a tangible surface touched directly with the finger, integrated with an immersive visual AR/VR headset to provide a coherent multimodal visuo-haptic augmentation of the real environment.
-  \item A psychophysical study with 20 participants to evaluate the perception of these virtual roughness textures in three visual rendering conditions: without visual augmentation, with a realistic virtual hand rendering in \AR, and with the same virtual hand in \VR.
+  \item The rendering of virtual vibrotactile roughness textures in real time using webcam to track the finger touching.
+  \item A system to provide a coherent multimodal visuo-haptic texture augmentations of the \RE in direct touch context using an immersive visual AR/VR headset and wearable haptics.
 \end{itemize}
-%First, we present a system for rendering virtual vibrotactile textures in real time without constraints on hand movements and integrated with an immersive visual AR/VR headset to provide a coherent multimodal visuo-haptic augmentation of the real environment.
+
+\noindentskip In the remainder of this chapter, we describe the principles of the system, how the real and virtual environments are registered, the generation of the vibrotactile textures, and measures of visual and haptic rendering latencies.
+
+%First, we present a system for rendering virtual vibrotactile textures in real time without constraints on hand movements and integrated with an immersive visual AR/VR headset to provide a coherent multimodal visuo-haptic augmentation of the \RE.
 %
 %An experimental setup is then presented to compare haptic roughness augmentation with an optical \AR headset (Microsoft HoloLens~2) that can be transformed into a \VR headset using a cardboard mask.
 %

2-perception/vhar-system/2-method.tex

@@ -2,7 +2,7 @@
 %
 In this section, we describe a system for rendering vibrotactile roughness textures in real time, on any tangible surface, touched directly with the index fingertip, with no constraints on hand movement and using a simple camera to track the finger pose.
 %
-We also describe how to pair this tactile rendering with an immersive \AR or \VR headset visual display to provide a coherent, multimodal visuo-haptic augmentation of the real environment.
+We also describe how to pair this tactile rendering with an immersive \AR or \VR headset visual display to provide a coherent, multimodal visuo-haptic augmentation of the \RE.
 
 \section{Principle}
 \label{principle}
@@ -11,19 +11,19 @@ The visuo-haptic texture rendering system is based on
 %
 \begin{enumerate*}[label=(\arabic*)]
 \item a real-time interaction loop between the finger movements and a coherent visuo-haptic feedback simulating the sensation of a touched texture,
-\item a precise alignment of the virtual environment with its real counterpart, and
+\item a precise alignment of the \VE with its real counterpart, and
 \item a modulation of the signal frequency by the estimated finger speed with a phase matching.
 \end{enumerate*}
 %
 \figref{diagram} shows the interaction loop diagram and \eqref{signal} the definition of the vibrotactile signal.
 %
-The system consists of three main components: the pose estimation of the tracked real elements, the visual rendering of the virtual environment, and the vibrotactile signal generation and rendering.
+The system consists of three main components: the pose estimation of the tracked real elements, the visual rendering of the \VE, and the vibrotactile signal generation and rendering.
 
 \figwide[1]{diagram}{Diagram of the visuo-haptic texture rendering system. }[
   Fiducial markers attached to the voice-coil actuator and to tangible surfaces to track are captured by a camera.
   The positions and rotations (the poses) ${}^c\mathbf{T}_i$, $i=1..n$ of the $n$ defined markers in the camera frame $\mathcal{F}_c$ are estimated, then filtered with an adaptive low-pass filter.
-  %These poses are transformed to the \AR/\VR headset frame $\mathcal{F}_h$ and applied to the virtual model replicas to display them superimposed and aligned with the real environment.
-  These poses are used to move and display the virtual model replicas aligned with the real environment.
+  %These poses are transformed to the \AR/\VR headset frame $\mathcal{F}_h$ and applied to the virtual model replicas to display them superimposed and aligned with the \RE.
+  These poses are used to move and display the virtual model replicas aligned with the \RE.
   A collision detection algorithm detects a contact of the virtual hand with the virtual textures.
   If so, the velocity of the finger marker ${}^c\dot{\mathbf{X}}_f$ is estimated using discrete derivative of position and adaptive low-pass filtering, then transformed onto the texture frame $\mathcal{F}_t$.
   The vibrotactile signal $s_k$ is generated by modulating the (scalar) finger velocity ${}^t\hat{\dot{X}}_f$ in the texture direction with the texture period $\lambda$ (\eqref{signal}).
@@ -36,13 +36,13 @@ The system consists of three main components: the pose estimation of the tracked
 
 \begin{subfigs}{setup}{Visuo-haptic texture rendering system setup. }[][
   \item HapCoil-One voice-coil actuator with a fiducial marker on top attached to a participant's right index finger.
-  \item HoloLens~2 \AR headset, the two cardboard masks to switch the real or virtual environments with the same field of view, and the 3D-printed piece for attaching the masks to the headset.
+  \item HoloLens~2 \AR headset, the two cardboard masks to switch the real or virtual environments with the same \FoV, and the \ThreeD-printed piece for attaching the masks to the headset.
   \item User exploring a virtual vibrotactile texture on a tangible sheet of paper.
   ]
   \subfig[0.325]{device}
-  \subfig[0.65]{headset}
-  \par\vspace{2.5pt}
-  \subfig[0.992]{apparatus}
+  %\subfig[0.65]{headset}
+  %\par\vspace{2.5pt}
+  %\subfig[0.992]{apparatus}
 \end{subfigs}
 
 A fiducial marker (AprilTag) is glued to the top of the actuator (\figref{device}) to track the finger pose with a camera (StreamCam, Logitech) which is placed above the experimental setup and capturing \qtyproduct{1280 x 720}{px} images at \qty{60}{\hertz} (\figref{apparatus}).
@@ -63,8 +63,8 @@ The optimal filter parameters were determined using the method of \textcite{casi
 %
 The velocity (without angular velocity) of the marker, denoted as ${}^c\dot{\mathbf{X}}_i$, is estimated using the discrete derivative of the position and an other 1€ filter with the same parameters.
 
-To be able to compare virtual and augmented realities, we then create a virtual environment that closely replicate the real one.
-%Before a user interacts with the system, it is necessary to design a virtual environment that will be registered with the real environment during the experiment.
+To be able to compare virtual and augmented realities, we then create a \VE that closely replicate the real one.
+%Before a user interacts with the system, it is necessary to design a virtual environment that will be registered with the \RE during the experiment.
 %
 Each real element tracked by a marker is modelled virtually, \ie the hand and the augmented tangible surface (\figref{renderings}).
 %
@@ -72,24 +72,24 @@ In addition, the pose and size of the virtual textures are defined on the virtua
 %
 During the experiment, the system uses marker pose estimates to align the virtual models with their real-world counterparts. %, according to the condition being tested.
 %
-This allows to detect if a finger touches a virtual texture using a collision detection algorithm (Nvidia PhysX), and to show the virtual elements and textures in real-time, aligned with the real environment (\figref{renderings}), using the considered \AR or \VR headset.
+This allows to detect if a finger touches a virtual texture using a collision detection algorithm (Nvidia PhysX), and to show the virtual elements and textures in real-time, aligned with the \RE (\figref{renderings}), using the considered \AR or \VR headset.
 
 In our implementation, the virtual hand and environment are designed with Unity and the Mixed Reality Toolkit (MRTK).
 %
 The visual rendering is achieved using the Microsoft HoloLens~2, an \OST-\AR headset with a \qtyproduct{43 x 29}{\degree} \FoV, a \qty{60}{\Hz} refresh rate, and self-localisation capabilities.
 %
-It was chosen over \VST-\AR because \OST-\AR only adds virtual content to the real environment, while \VST-\AR streams a real-time video capture of the real environment \cite{macedo2023occlusion}.
+It was chosen over \VST-\AR because \OST-\AR only adds virtual content to the \RE, while \VST-\AR streams a real-time video capture of the \RE \cite{macedo2023occlusion}.
 %
-Indeed, one of our objectives (\secref{experiment}) is to directly compare a virtual environment that replicates a real one, rather than a video feed that introduces many supplementary visual limitations \cite{kim2018revisiting,macedo2023occlusion}.
+Indeed, one of our objectives (\secref{experiment}) is to directly compare a \VE that replicates a real one, rather than a video feed that introduces many supplementary visual limitations \cite{kim2018revisiting,macedo2023occlusion}.
 %
-To simulate a \VR headset, a cardboard mask (with holes for sensors) is attached to the headset to block the view of the real environment (\figref{headset}).
+To simulate a \VR headset, a cardboard mask (with holes for sensors) is attached to the headset to block the view of the \RE (\figref{headset}).
 
 \section{Vibrotactile Signal Generation and Rendering}
 \label{texture_generation}
 
 A voice-coil actuator (HapCoil-One, Actronika) is used to display the vibrotactile signal, as it allows the frequency and amplitude of the signal to be controlled independently over time, covers a wide frequency range (\qtyrange{10}{1000}{\Hz}), and outputs the signal accurately with relatively low acceleration distortion\footnote{HapCoil-One specific characteristics are described in its data sheet: \url{https://web.archive.org/web/20240228161416/https://tactilelabs.com/wp-content/uploads/2023/11/HapCoil_One_datasheet.pdf}}.
 %
-The voice-coil actuator is encased in a 3D printed plastic shell and firmly attached to the middle phalanx of the user's index finger with a Velcro strap, to enable the fingertip to directly touch the environment (\figref{device}).
+The voice-coil actuator is encased in a \ThreeD printed plastic shell and firmly attached to the middle phalanx of the user's index finger with a Velcro strap, to enable the fingertip to directly touch the environment (\figref{device}).
 %
 The actuator is driven by a class D audio amplifier (XY-502 / TPA3116D2, Texas Instrument). %, which has proven to be an effective type of amplifier for driving moving-coil \cite{mcmahan2014dynamic}.
 %
@@ -154,7 +154,7 @@ The tactile texture is described and rendered in this work as a one dimensional
 \section{System Latency}
 \label{latency}
 
-%As shown in \figref{diagram} and described above, the system includes various haptic and visual sensors and rendering devices linked by software processes for image processing, 3D rendering and audio generation.
+%As shown in \figref{diagram} and described above, the system includes various haptic and visual sensors and rendering devices linked by software processes for image processing, \ThreeD rendering and audio generation.
 %
 Because the chosen \AR headset is a standalone device (like most current \AR/\VR headsets) and cannot directly control the sound card and haptic actuator, the image capture, pose estimation and audio signal generation steps are performed on an external computer.
 %
@@ -166,20 +166,20 @@ The rendering system provides a user with two interaction loops between the move
 %
 Measures are shown as mean $\pm$ standard deviation (when it is known).
 %
-The end-to-end latency from finger movement to feedback is measured at \qty{36 +- 4}{\ms} in the haptic loop and \qty{43 +- 9}{\ms} in the visual loop.
+The end-to-end latency from finger movement to feedback is measured at \qty{36 \pm 4}{\ms} in the haptic loop and \qty{43 \pm 9}{\ms} in the visual loop.
 %
-Both are the result of latency in image capture \qty{16 +- 1}{\ms}, markers tracking \qty{2 +- 1}{\ms} and network communication \qty{4 +- 1}{\ms}.
+Both are the result of latency in image capture \qty{16 \pm 1}{\ms}, markers tracking \qty{2 \pm 1}{\ms} and network communication \qty{4 \pm 1}{\ms}.
 %
-The haptic loop also includes the voice-coil latency \qty{15}{\ms} (as specified by the manufacturer\footnotemark[1]), whereas the visual loop includes the latency in 3D rendering \qty{16 +- 5}{\ms} (60 frames per second) and display \qty{5}{\ms}.
+The haptic loop also includes the voice-coil latency \qty{15}{\ms} (as specified by the manufacturer\footnotemark[1]), whereas the visual loop includes the latency in \ThreeD rendering \qty{16 \pm 5}{\ms} (60 frames per second) and display \qty{5}{\ms}.
 %
 The total haptic latency is below the \qty{60}{\ms} detection threshold in vibrotactile feedback \cite{okamoto2009detectability}.
 %
 The total visual latency can be considered slightly high, yet it is typical for an \AR rendering involving vision-based tracking \cite{knorlein2009influence}.
 
-The two filters also introduce a constant lag between the finger movement and the estimated position and velocity, measured at \qty{160 +- 30}{\ms}.
+The two filters also introduce a constant lag between the finger movement and the estimated position and velocity, measured at \qty{160 \pm 30}{\ms}.
 %
 With respect to the real hand position, it causes a distance error in the displayed virtual hand position, and thus a delay in the triggering of the vibrotactile signal.
 %
-This is proportional to the speed of the finger, \eg distance error is \qty{12 +- 2.3}{\mm} when the finger moves at \qty{75}{\mm\per\second}.
+This is proportional to the speed of the finger, \eg distance error is \qty{12 \pm 2.3}{\mm} when the finger moves at \qty{75}{\mm\per\second}.
 %
 %and of the vibrotactile signal frequency with respect to the finger speed.%, that is proportional to the speed of the finger.

2-perception/vhar-system/6-conclusion.tex

@@ -4,4 +4,4 @@
 %Summary of the research problem, method, main findings, and implications.
 
 We designed and implemented a system for rendering virtual haptic grating textures on a real tangible surface touched directly with the fingertip, using a wearable vibrotactile voice-coil device mounted on the middle phalanx of the finger. %, and allowing free explorative movements of the hand on the surface.
-This tactile feedback was integrated with an immersive visual virtual environment, using an OST-AR headset, to provide users with a coherent multimodal visuo-haptic augmentation of the real environment, that can be switched between an \AR and a \VR view.
+This tactile feedback was integrated with an immersive visual virtual environment, using an OST-AR headset, to provide users with a coherent multimodal visuo-haptic augmentation of the \RE, that can be switched between an \AR and a \VR view.