WebRTC and HoloLens
Social Brothers uses WebRTC for Bureau Veritas' remote assistance application. Bureau Veritas inspects ships all over the world. That is, their inspectors sometimes fly all over the world for a ten-minute inspection. Not only does this cost the inspector time and money, shipping can also miss out on millions. After all, the ship must remain stationary until the inspector is on site.
At Social Brothers we have been working with the HoloLens for a while. We have built an application for Bureau Veritas ourselves to meet this issue of Bureau Veritas and to enable remote inspections. The idea behind our remote Assistanceapplication is that there is someone on the ship wearing the HoloLens. The Bureau Veritas inspector sits remotely behind his own desk and can monitor the ship via the camera image of the HoloLens and communicate with the wearer of the HoloLens. For more information about remote assistance, read our blog about it.
In addition, the application has the option to exchange audio and to exchange extra data, such as the position of the carrier of the HoloLens in the engine room of the ship. This allows the remote inspection to be performed faster and more efficiently. WebRTC turned out to be the best technology for the communication protocol, which is why we started using it for this proof-of-concept. WebRTC makes it possible to connect two devices in a very simple way and to get a stable audio and video connection.
The WebRTC protocol provides a fast connection. If the server is the connection, one sends some to the server and the other gets it. That is an extra step that causes delays. Compare it to a conversation: it is easier to talk directly to the person to whom you want to convey a specific message, than it is to have a conversation through an intermediary. By skipping the extra step in the digital communication process, P2P has up to six times faster connection times. The video quality is better and you have more natural online conversations because there is no delay.
Because there is no server in between, WebRTC is in principle also extra safe, because it is more difficult for a man-in-the-middle attack to take place that captures certain communications. Of course, you have to trust the person you connect with. We therefore verify via a websocket technique whether you are connecting with the right person. When you connect to the server, socket.io creates a room from the server. That room will be given a separate name, so you can only join if you know the name of the room. That way you know who's on the other side.
Data channels at WebRTC
Another advantage of the WebRTC protocol is the data channels. These make it possible to send other data back and forth besides speech and images. For the Bureau Veritas application, this is the position of the HoloLens in the engine room. The inspector has a 3D model of the engine room on his computer screen, next to the video image from the HoloLens. The position and rotation of the HoloLens is constantly transmitted to the inspector through the data channel. We use that data to ensure that the 3D model shows exactly the position of the HoloLens wearer.