diff --git a/1-introduction/introduction.tex b/1-introduction/introduction.tex index dbf7704..aebbf50 100644 --- a/1-introduction/introduction.tex +++ b/1-introduction/introduction.tex @@ -7,7 +7,6 @@ In this manuscript thesis, we show how \AR headset, which integrates visual virtual content into the real world perception, and wearable haptics, which provide tactile sensations on the skin, can improve direct hand interaction with virtual and augmented objects. Our goal is to enable users to perceive and interact with wearable visuo-haptic augmentations in a more realistic and effective way, as if they were real. -\comans{JG}{I was wondering what the difference between an immersive AR headset and a non-immersive AR headset should be. If there is a difference (e.g., derived through headset properties by FoV), it should be stated. If there is none, I would suggest not using the term immersive AR headset but simply AR headset. On this account, in Figure 1.5 another term (“Visual AR Headset”) is introduced (and later OST-AR systems, c.f. also section 2.3.1.3).}{The terms "immersive AR headset" and "visual AR headset" have been replaced by the more appropriate term "AR headset".} \section{Visual and Haptic Object Augmentations} \label{visuo_haptic_augmentations} @@ -130,7 +129,6 @@ The \RE and the user's hand are tracked in real time by sensors and reconstructe The interactions between the virtual hand and objects are then simulated, and rendered as feedback to the user using an \AR headset and wearable haptics. It is important that the visuo-haptic \VE is registered with the \RE and rendered in real time. This gives the user the illusion of directly perceiving and interacting with the virtual content as if it were part of the \RE. -\comans{SJ}{The chapter could benefit from some expansion. For instance, the current introduction tries to describe the scope of the research in haptic AR but lacks sufficient background on the general issues in this domain. As a result, it may not be very helpful for readers unfamiliar with the field in understanding the significance of the thesis's focus and positioning it within the broader context of haptic AR research.}{This section has been added to provide a better overview of the general research challenges of visuo-haptic augmentations.} \fig{interaction-loop}{The interaction loop between a user and a visuo-haptic \AE as proposed in this thesis.}[ A user interacts with the visual (in blue) and haptic (in red) \VEs through a virtual hand (in purple) interaction technique that tracks real hand movements and simulates contact with virtual objects. @@ -164,7 +162,6 @@ In this thesis, we focus on two main research challenges: \textbf{(I) providing plausible and coherent visuo-haptic augmentations}, and \textbf{(II) enabling effective manipulation of the \AE}. Each of these challenges also raises numerous design, technical, perceptual and user experience issues specific to wearable haptics and \AR headsets. -\comans{JG}{While these research axes seem valid, it could have been described more clearly how they fit into the overarching research fields of visuo-haptic augmentations.}{The previous section has been expanded to better describe the general research challenges of visuo-haptic augmentations.} %\footnotetext{% % The icons are \href{https://creativecommons.org/licenses/by/3.0/}{CC BY} licensed: diff --git a/2-related-work/3-augmented-reality.tex b/2-related-work/3-augmented-reality.tex index e80b9d7..9bc5f57 100644 --- a/2-related-work/3-augmented-reality.tex +++ b/2-related-work/3-augmented-reality.tex @@ -144,7 +144,6 @@ Choosing useful and efficient \UIs and interaction techniques is crucial for the \textcite{laviolajr20173d} (p.385) classify interaction techniques into three categories based on the tasks they enable users to perform: manipulation, navigation, and system control. \textcite{hertel2021taxonomy} proposed a similar taxonomy of interaction techniques specifically for \AR headsets. -\comans{JG}{In, Figure 2.24 I suggest removing d. or presenting it as separate figure as it shows no interaction technique (The caption is “Interaction techniques in AR” but a visualization of a spatial registration technique).}{It has been removed and replaced by an example of resizing a virtual object.} In this thesis we focus on manipulation tasks of virtual content directly with the hands, more specifically on touching visuo-haptic textures with a finger (\partref{perception}) and positioning and rotating virtual objects pushed and grasp by the hand (\partref{manipulation}). The \emph{manipulation tasks} are the most fundamental tasks in \AR and \VR systems, and the building blocks for more complex interactions. diff --git a/3-perception/vhar-system/1-introduction.tex b/3-perception/vhar-system/1-introduction.tex index 8bd3d84..fc0df28 100644 --- a/3-perception/vhar-system/1-introduction.tex +++ b/3-perception/vhar-system/1-introduction.tex @@ -15,7 +15,6 @@ The visuo-haptic augmentations rendered with this design allow a user to \textbf To ensure both real-time and reliable renderings, the hand and the real surfaces are tracked using a webcam and marker-based pose estimation. The haptic textures are rendered as a vibrotactile signal representing a patterned grating texture that is synchronized with the finger movement on the augmented surface. The goal of this design is to enable new \AR applications capable of augmenting real objects with virtual visuo-haptic textures in a portable, on-demand manner, and without impairing with the interaction of the user with the \RE. -\comans{SJ}{The rationale behind the proposed design is not provided. Since there are multiple ways to implement mechanically transparent haptic devices, the thesis should at least clarify why this design is considered optimal for a specific purpose at this stage.}{This has been better explained in the introduction.} \noindentskip The contributions of this chapter are: \begin{itemize} diff --git a/3-perception/vhar-system/2-method.tex b/3-perception/vhar-system/2-method.tex index 597d213..bc4d66f 100644 --- a/3-perception/vhar-system/2-method.tex +++ b/3-perception/vhar-system/2-method.tex @@ -55,13 +55,11 @@ First, the coordinate system of the headset is manually aligned with that of the This resulted in a \qty{\pm .5}{\cm} spatial alignment error between the \RE and the \VE. While this was sufficient for our use cases, other methods can achieve better accuracy if needed \cite{grubert2018survey}. The registration of the coordinate systems of the camera and the headset thus allows the use of the marker estimation poses performed with the camera to display in the headset the virtual models aligned with their real-world counterparts. -\comans{JG}{The registration process between the external camera, the finger, surface and HoloLens could have been described in more detail. Specifically, it could have been described clearer how the HoloLens coordinate system was aligned (e.g., by also tracking the fiducials on the surface and or finger).}{This has been better described.} An additional calibration is performed to compensate for the offset between the finger contact point and the estimated marker pose \cite{son2022effect}. The current user then places the index finger on the origin point, whose respective poses are known from the attached fiducial markers. The transformation between the marker pose of the finger and the finger contact point can be estimated and compensated with an inverse transformation. This allows to detect if the calibrated real finger touches a virtual texture using a collision detection algorithm (Nvidia PhysX). -\comans{JG}{A description if and how the offset between the lower side of the fingertip touching the surface and the fiducial mounted on the top of the finger was calibrated / compensated is missing}{This has been better described.} In our implementation, the \VE is designed with Unity (v2021.1) and the Mixed Reality Toolkit (v2.7)\footnoteurl{https://learn.microsoft.com/windows/mixed-reality/mrtk-unity}. The visual rendering is achieved using the Microsoft HoloLens~2, an \OST-\AR headset with a \qtyproduct{43 x 29}{\degree} \FoV, a \qty{60}{\Hz} refresh rate, and self-localisation capabilities. diff --git a/3-perception/vhar-textures/2-experiment.tex b/3-perception/vhar-textures/2-experiment.tex index cd13b20..66672a2 100644 --- a/3-perception/vhar-textures/2-experiment.tex +++ b/3-perception/vhar-textures/2-experiment.tex @@ -123,4 +123,3 @@ After each of the two tasks, participants answered to the following 7-item Liker In an open question, participants also commented on their strategy for completing the \level{Matching} task (\enquote{How did you associate the tactile textures with the visual textures?}) and the \level{Ranking} task (\enquote{How did you rank the textures?}). The results were analyzed using R (v4.4) and the packages \textit{afex} (v1.4), \textit{ARTool} (v0.11), \textit{corrr} (v0.4), \textit{FactoMineR} (v2.11), \textit{lme4} (v1.1), and \textit{performance} (v0.13). -\comans{JG}{I suggest to also report on [...] the software packages used for statistical analysis (this holds also for the subsequent chapters).}{This has been added to all chapters where necessary.} diff --git a/3-perception/vhar-textures/3-results.tex b/3-perception/vhar-textures/3-results.tex index a4424c0..7aed1ed 100644 --- a/3-perception/vhar-textures/3-results.tex +++ b/3-perception/vhar-textures/3-results.tex @@ -10,7 +10,6 @@ \figref{results/matching_confusion_matrix} shows the confusion matrix of the \level{Matching} task with the visual textures and the proportion of haptic texture selected in response, \ie the proportion of times the corresponding \response{Haptic Texture} was selected in response to the presentation of the corresponding \factor{Visual Texture}. To determine which haptic textures were selected most often, the repetitions of the trials were first aggregated by counting the number of selections per participant for each (\factor{Visual Texture}, \response{Haptic Texture}) pair. An \ANOVA based on a Poisson regression (no overdispersion was detected) indicated a statistically significant effect on the number of selections of the interaction \factor{Visual Texture} \x \response{Haptic Texture} (\chisqr{64}{180}{414}, \pinf{0.001}). -\comans{JG}{For the two-sample Chi-Squared tests in the matching task, the number of samples reported is 540 due to 20 participants conducting 3 trials for 9 textures each. However, this would only hold true if the repetitions per participant would be independent and not correlated (and then, one could theoretically also run 10 participants with 6 trials each, or 5 participants with 12 trials each). If they are not independent, this would lead to an artificial inflated sample size and Type I error. If the trials are not independent (please double check), I suggest either aggregating data on the participant level or to use alternative models that account for the within-subject correlation (as was done in other chapters).}{Data of the three confusion matrices have been aggregated on the participant level and analyzed using a Poisson regression.} Post-hoc pairwise comparisons using the Tukey's \HSD test then indicated there was statistically significant differences for the following visual textures: \begin{itemize} \item With \level{Sandpaper~320}, \level{Coffee Filter} was more selected than the other haptic textures (\ztest{3.4}, \pinf{0.05} each) except \level{Plastic Mesh~1} and \level{Terra Cotta}. diff --git a/3-perception/xr-perception/3-experiment.tex b/3-perception/xr-perception/3-experiment.tex index 6efce45..308feae 100644 --- a/3-perception/xr-perception/3-experiment.tex +++ b/3-perception/xr-perception/3-experiment.tex @@ -39,7 +39,6 @@ The virtual hand model was a gender-neutral human right hand with realistic skin Prior to the experiment, the virtual hand and the \VE were registered to the real hand of the participant and the \RE, respectively, as described in \secref[vhar_system]{virtual_real_registration}. The size of the virtual hand was also manually adjusted to match the real hand of the participant. A \qty{\pm .5}{\cm} spatial alignment error (\secref[vhar_system]{virtual_real_registration}) and a \qty{160 \pm 30}{\ms} lag (\secref[vhar_system]{virtual_real_registration}) between the real hand the virtual hand were measured. -\comans{JG}{In addition, the lag between the real and virtual hand in the Mixed condition could have been quantified (e.g. using a camera filming through the headset) to shed more light on the reported differences, as also noted in Section 4.5, as well as the registration error between the real and the virtual hand (as visible in Figure 4.1, Mixed).}{This has been added.} To ensure the same \FoV in all \factor{Visual Rendering} condition, a cardboard mask was attached to the \AR headset (\figref{experiment/headset}). In the \level{Virtual} rendering, the mask only had holes for sensors to block the view of the \RE and simulate a \VR headset. diff --git a/4-manipulation/visuo-haptic-hand/5-conclusion.tex b/4-manipulation/visuo-haptic-hand/5-conclusion.tex index a780492..76e2d9c 100644 --- a/4-manipulation/visuo-haptic-hand/5-conclusion.tex +++ b/4-manipulation/visuo-haptic-hand/5-conclusion.tex @@ -18,7 +18,6 @@ Yet, a wrist-mounted haptic device will be able to provide richer feedback by em It could thus provide more complex feedback of the contacts with the virtual objects. Finally, we think that the visual hand augmentation complements the haptic contact rendering well by providing continuous feedback on hand tracking. Such a visual augmentation can be disabled during the grasping phase to avoid redundancy with the haptic feedback of the contact with the virtual object. -\comans{SJ}{Again, it would strengthen the thesis if the authors provided a systematic guideline on how to choose the appropriate haptic feedback or visual augmentation depending on the specific requirements of an application.}{The guidelines paragraph have been expanded in the conclusion.} \noindentskip The work described in \chapref{visual_hand} and \ref{visuo_haptic_hand} was published in Transactions on Haptics: diff --git a/5-conclusion/conclusion.tex b/5-conclusion/conclusion.tex index 1683165..d1da11c 100644 --- a/5-conclusion/conclusion.tex +++ b/5-conclusion/conclusion.tex @@ -68,7 +68,6 @@ A more robust hand pose estimation system would support wearing haptic devices o The spatial registration error \cite{grubert2018survey} and the temporal latency \cite{diluca2019perceptual} between the \RE and \VE should also be reduced to be imperceptible. The effect of these spatial and temporal errors on the perception and manipulation of the virtual object should be systematically investigated. Prediction of hand movements should also be considered to overcome such issues \cite{klein2020predicting,gamage2021predictable}. -\comans{JG}{I [...] also want to highlight the opportunity to study the effect of visual registration error as noted already in chapter 4.}{Sentences along these lines has been added.} A complementary solution would be to embed tracking sensors in the wearable haptic devices, such as an inertial measurement unit (IMU) or cameras \cite{preechayasomboon2021haplets}. This would allow a complete portable and wearable visuo-haptic system to be used in practical applications. @@ -102,12 +101,9 @@ As in the previous chapter, our aim was not to accurately reproduce real texture However, the results also have some limitations, as they addressed a small set of visuo-haptic textures that augmented the perception of smooth and white real surfaces. Visuo-haptic texture augmentation might be difficult on surfaces that already have strong visual or haptic patterns \cite{asano2012vibrotactile}, or on objects with complex shapes. A real surface could be indeed augmented not only to add visuo-haptic textures, but also to amplify, diminish, mask, or replace the existing real texture. -\comans{SJ}{It would be valuable to explore how real texture from a physical surface could be combined with virtual texture, enabling merged, augmented, amplified, or diminished feedback}{This has been better discussed.} In addition, the visual textures used were simple color images not intended for use in an \ThreeD \VE, and enhancing their visual quality could improve the perception of visuo-haptic texture augmentation. -\comans{JG}{As future work, the effect of visual quality of the rendered textures on texture perception could also be of interest.}{A sentence along these lines has been added.} It would also be interesting to replicate the experiment in more controlled visuo-haptic environments, in \VR or with world-grounded haptic devices. This would enable to better understand how the rendering quality, spatial registration and latency of virtual textures can affect their perception. -\comans{SJ}{Moreover, if we only consider the experimental findings, the system could likely be recreated using VR or conventional visuo-haptic setups in a more stable manner. It would be beneficial to emphasize how the experiment is closely tied to the specific domain of haptic AR.}{This has been added.} Finally, the role of visuo-haptic texture augmentation should also be evaluated in more complex tasks, such as object recognition and assembly, or in more concrete use cases, such as displaying and touching a museum object or a 3D printed object before it is manufactured. \paragraph{Specificities of Direct Touch.} @@ -118,7 +114,6 @@ The respective importance of these factors on the haptic texture perception is n It would be interesting to determine the importance of these factors on the perceived realism of virtual vibrotactile textures in the context of bare finger touch. Finger based captures of real textures should also be considered \cite{balasubramanian2024sens3}. Finally, the virtual texture models should also be adaptable to individual sensitivities \cite{malvezzi2021design,young2020compensating}. -\comans{SJ}{Technical concern: As far as I know, the texture rendering algorithm from [Curbertson et al.] is based on rigid-tool-based interactions. The vibration patterns due to texture in a bare-hand interaction scenario (used in this study) should differ significantly from those produced in rigid-tool interactions. I conduct similar research and am confident that the signals involved in bare-hand interactions are far more complex than those in rigid-tool-based interactions. Therefore, the choice of rendering algorithm could negatively affect the experimental results. This issue is critical and should either be revised or extensively discussed in the thesis.}{This has been discussed more in depth in this section.} \subsection*{Visual Augmentation of the Hand for Manipulating virtual objects in AR} @@ -131,7 +126,6 @@ However, the user's visual perception and experience are different with other ty In particular, the mutual occlusion problem and the latency of hand pose estimation could be overcome with a \VST-\AR headset. In this case, the occlusion rendering could be the most natural, realistic and effective augmentation. Yet, a visual hand augmentation could still be beneficial to users by providing depth cues and feedback on hand tracking, and should be evaluated as such. -\comans{SJ}{According to the results, occlusion is the most natural (in terms of realism) but least efficient for manipulation. In some cases, natural visualization is necessary. It would be beneficial to discuss these cases to help guide AR interaction designers in choosing the most appropriate visualization methods.}{This has been discussed more in depth in this section.} \paragraph{More Practical Usages.} @@ -140,7 +134,6 @@ These tasks are indeed fundamental building blocks for more complex manipulation They can indeed require users to perform more complex finger movements and interactions with the virtual object. Depending on the task, the importance of position, orientation and depth information of the hand and the object may vary and affect the choice of visual hand augmentation. More practical applications should also be considered, such as medical, educational or industrial scenarios, which may have different needs and constraints (\eg, the most natural visual hand augmentation for a medical application, or the easiest to understand and use for an educational context). -\comans{SJ}{The task in the experiment is too basic, making it difficult to generalize the results. There are scenarios where depth information may be more important than position, or where positioning may be more critical than orientation. A systematic categorization and analysis of such cases would add depth to the chapter.}{This has been discussed more in depth in this section.} Similarly, a broader experimental study might shed light on the role of gender and age, as our subject pool was not sufficiently diverse in this regard. Finally, all visual hand augmentations received low and high rank rates from different participants, suggesting that users should be able to choose and personalize some aspects of the visual hand augmentation according to their preferences or needs, and this should also be evaluated. diff --git a/config/content.tex b/config/content.tex index 5832258..989e681 100644 --- a/config/content.tex +++ b/config/content.tex @@ -1,14 +1,3 @@ -% Changes -\usepackage[commentmarkup=footnote]{changes} -\definechangesauthor[name=Jens Grubert, color=Dandelion]{JG} -\definechangesauthor[name=Seokhee Jeon, color=Red]{SJ} -\newcommand{\comans}[3]{% - \comment[id=#1]{% - #2\\% - \textbf{Answer: }#3% - }% -} - % Images \usepackage{graphicx} \usepackage{caption}% Point references to the figure not the caption diff --git a/normand2025thesis.tex b/normand2025thesis.tex index 6dc37bd..97df4f0 100644 --- a/normand2025thesis.tex +++ b/normand2025thesis.tex @@ -32,8 +32,6 @@ \frontmatter \import{0-front}{cover} -\pdfbookmark[chapter]{List of changes}{changes} -\listofchanges %\importchapter{0-front}{acknowledgement} \importchapter{0-front}{contents}