Haptics and the 3D Internet
What haptics means for simulation-based training, remote operation, and advanced content design.
Why haptics, and why now?
The recent exponential increase in content creation for the 3D internet is ensuring a vast and rich diversity of experiences to meet the endless needs of those of us seeking to not only see the virtual world, but to touch it too. Such is the function of haptic technology – the electromechanical stimulation of the sense of touch.
This technology has enabled Haply Robotics to create exciting applications in areas like the metaverse, industrial robotics, automation and supervision, and VR (virtual reality) simulation. New technologies like enterprise 3D displays, simulation engines, and 5G are creating a wealth of opportunities to build transformative technologies.
This, in conjunction with the developments surrounding the metaverse and VR, has made for a tech ecosystem that is ready for an advanced 3D mouse that enables users to interact with the growing 3D internet in ways never before possible. Haptic technology gives us a sense of touch when engaging with digital experiences, paving the way towards a more connected future. As a result, we anticipate the main areas of application for haptic devices to include:
1. Simulation-Based Skills Training
2. New Ways to Create and Consume 3D Content
3. Remote Device Operation
For the past year, we have used NVIDIA’s Omniverse™ technology and NVIDIA accelerated computing systems such as the RTXA6000 to build touch-enabled applications for the Mediverse™, or the medical metaverse. We invite you to join our panel session at the virtual NVIDIA GTC on September 21 at 11 AM PDT, where our cofounder, Colin Gallacher, and other thought leaders in medical imaging, genomics, biopharma, and medical devices, will discuss healthcare applications of robotics, AR technologies and more.
Here are all the ways touch-enabled devices will shape our future:
Simulation-Based Skills Training
Training surgeons is difficult, time consuming, and expensive, but there’s no better teacher than experience. This is why a simulation-based approach to skills training is a highly effective way to learn. Providing trainees the opportunity to practice learned skills in real-life situations is paramount to growth, and simulated training scenarios allow students to run through high-stakes procedures with close to zero risk.
Haptic simulation training takes that a step further, facilitating skill mastery by replicating the exact sensations of real-life scenarios and stimulating deep focus by providing a more natural and immersive learning environment compared to traditional simulators. Indeed, the more authentic the simulation environment is, the better the results of training. This is particularly important for skills that require very precise and tactile feedback and movements.
One such example is that of virtual medical training. For instance, the Haply Inverse3 is a state-of-the-art force-feedback haptic controller designed to precisely emulate complex sensations, such as incising soft tissue with a scalpel or drilling through bone tissue. This empowers students, surgeons, and other healthcare professionals to hone and perfect life-saving procedures before ever setting foot in an operating room.
Other applications for haptic simulators include dangerous task training. The use of a haptic controller coupled with a VR headset can transport trainees to highly realistic environments without the risk of endangering themselves or others or the cost of building a physical training space. For this reason, haptic simulators are commonly used for military, pilot, and fire fighter training. Similarly, haptic simulators are great tools for systematically training repetitive tasks, which has proven to be very useful in the manufacturing industry. Much of industrial assembly, disassembly and maintenance is still performed by humans and requires an efficient training method for a large amount of people in a short amount of time. Haptic devices are the perfect tool for this as they can accurately simulate the force feedback received from routine hands-on operations.
New Ways to Create and Consume 3D Content
Unity, Epic Games Unreal Engine, Dassault Systemes SOLIDWORKS and CATIA, Maxon ZBrush, Autodesk Maya, AutoCAD, and Inventor, Blender, McNeel Rhino - the list goes on and on. Since their introduction in the 1960’s, 3D modeling tools have become central to the workflow in fields like engineering, architecture, gaming, and industrial design. They have transformed the way work is done for many industries, and as the 3D internet and metaverse evolves, we will see even more new and exciting ways to interact with 3D content.
Gone are the days where architects had to lug around heavy portfolios of hand-drawn plans and animators had to painstakingly draw each individual frame of a cartoon movie. Companies like Meta, Apple, and HP are pouring billions of dollars into the development of next-gen displays. Their belief, as seen in Mark Zuckerberg's recent appearance on Tested, is that the near-future will see displays that will allow us to comfortably blend the tasks we do in 2D and 3D. This trend, can be taken a step further with the use of haptic technology.
Introducing haptic design has the potential to reinvigorate creative fields with new and improved user interactions. At the same time, it is Haply’s belief that facilitating the seamless transition from 2D to 3D content is vital for any type of input device. This is a fundamental principle driving our product development as a company, as legacy content will exist for years to come. With that said, the new UX paradigms realized by haptic content will enable users to naturally navigate spatial environments and feel what they digitally produce, transforming product development systems and speeding up processes across multiple industries.
With new ways to interact with 3D content steadily emerging, like the convergence of heads-up displays, and one of the most notable companies in this space, Meta, driving more money and effort towards its hardware and software VR offerings, it’s not hard to conclude that the future of content creation and consumption will be focused on making virtual experiences as photorealistic and immersive as possible — and the rising popularity of haptic controllers signify a major step in that direction.
Remote Device Operation
The advent of 5G means more than just faster internet. It also means increased capacity for connected devices and remote performance, such as VR streaming on the NVIDIA’s CloudXR platform. With 5G’s reduced latency, it is finally possible to have reliable wireless communication at rates that can support haptics.
Teleoperation in robotics combines the benefits of robotic manipulation with human mental capabilities and handling strategies. Two-way teleoperation provides haptic feedback that enables a human operator to feel interaction forces and to intuitively control the forces exerted by a remotely-operated robot on its environment. This feedback is vital for delicate tasks that involve handling of fragile or dangerous objects, or require high precision, as it enables the operator to naturally sense and respond to highly controlled forces, such as traction or load, on guided vehicles. For example, remote haptic devices can be used to physically inspect construction sites with drones, control surgical robots, supervise industrial robots at a distance, or conduct a rescue operation in a mine.
Remote-controlled robots, vehicles, and other devices are not at all new to the technological landscape, but introducing fine control precision and using audio channels, visual channels, and now haptic channels (AV to become AVH) can make for exciting applications and possibilities in the world of device-operation at a distance.
Whether it is too dangerous for human supervision, audio-visual information is insufficient, or a robot’s autonomous control fails and needs human intervention, remote control is a proven solution. When used as a controller for a robot, drone, or other vehicle, haptic controllers offer a natural and intuitive way to supervise machinery with an added dimension of accuracy by relaying touch feedback to the user through an extra sensory channel.
5G networks are the infrastructure that replace highways in the modern digital universe, and just as 2D mice and tablets displaced paper and pen, Haply's haptic technologies will blend the advantages of the 2D mouse and tablet with the spatial needs of the 21st century and beyond. Designed and built for a 3D world, our products are at the forefront of technological evolution.
11.2 million AR/VR headsets were sold last year alone, a trend signaling a growing public desire for simulated experiences and realistic virtual environments. The quality of VR spaces are progressing at an exciting rate as it is. However, the added sensory component that haptics provides has wide-ranging benefits from 3D design to complex training and telepresence device manipulation.
At Haply, we aim to support these applications with a focus on precision and ergonomics within an enterprise setting. Using the Inverse3—the world’s first and only lightweight, reconfigurable, and completely portable haptic interface that replicates the sense of touch for special tasks—companies will create revolutionary solutions to:
1. Train Skills Using Simulations
2. Create and Consume 3D Content
3. Operate Remote Devices
To learn more about the future of this exciting technology and how we use NVIDIA technology for advanced graphics and simulation as well as NVIDIA Clara for healthcare applications to facilitate our haptic controllers’ functionality, we invite you to join our panel session at NVIDIA GTC on Wednesday September 21 at 11 AM PDT.
