The Holodeck and the future of communication from Karlsruhe

A “real” meeting between two people who are in different locations — this is the vision of the future of Fraunhofer IOSB alumnus Miro Taphanel and his collaborators. They want to rescue people from video telephony hell, as a large amount of non-verbal information is lost with conventional technology. In actual fact, this new form of communication is already in use. The founders of the Karlsruhe-based start-up hold regular virtual meetings from different locations and stand opposite each other virtually in full height. It certainly feels that way at least, thanks to Gixel’s augmented reality technology consisting of a camera, special glasses, screen, artificial intelligence and complex software.

US tech giants with billion-dollar budgets are also pushing their way into this market and want to make it possible to place people in different environments — but with completely different approaches. If Gixel achieves the technical breakthrough, then financial success looks to be a given. Before we are able to dive into the future of communication, the Karlsruhe-based company still needs to overcome a few technical challenges.

The prospect of solving a fundamental problem has also impressed the German federal government’s Federal Agency for Disruptive Innovation (SPRIND). SPRIND is therefore investing over 20 million euros in Gixel, an accolade bestowed on just a few start-ups.

Miro Taphanel studied mechanical engineering at Karlsruhe Institute of Technology (KIT). Even during his doctorate at KIT, he was in regular contact with Fraunhofer IOSB to exchange ideas on scientific topics. He then started working as a scientist for the renowned university, before moving to Fraunhofer IOSB to lead the “Variable imaging” group. In the summer of 2019, he founded Gixel together with IOSB scientist and colleague Ding Luo as well as experienced company founder Felix Nienstädt.

Your solution is often compared to the Holodeck, a fictional device from a science fiction series. Can you explain the idea behind the Gixel solution?

Our goal is to have a remote meeting that feels like the real thing. For this reason, we focus on the communication solution as a whole. With the aid of augmented reality, we create the illusion that objects, or in our case people, are in my room, my secure area. This is how we distinguish ourselves clearly from the virtual reality approach and thereby from the Holodeck, too. With our solution, you don’t have to wear a massive headset and you don’t leave your own personal sphere in order to enter another world. We augment our own reality with friends, acquaintances and colleagues, staying in the here and now. In this way, I retain all my abilities and can use my cellphone or notebook or even hammer a nail into the wall. If I was wearing a mixed reality headset, I wouldn’t attempt such a thing!

This limitless nature makes both communication and other scenarios conceivable. What approach are you taking here?

Sight is the most important sense, which we cannot match when we convert information initially into pixels and digital data, in order to then convert it again into something visual or analog. People need to retain their peripheral vision, just like in real life. We want to give people the superpower to incorporate things into this sense. The best thing is spending evenings with friends on the sofa, playing games or doing sport together.

We also use spatial audio. You hear your companion from where they appear in the room. We use face-to-face meetings with people as a reference. Because of this, we go one step further and don’t use volume control. If the person you are talking to in real life is talking too quietly, you can’t turn up the volume; instead, you ask them to speak louder.

If I want to have a meeting with Gixel, what do I need for this?

The only hardware you need for this solution is a special set of glasses. These are designed so that both people still feel like talking to each other. I wouldn’t say it’s a call. We use face-to-face meetings as a reference. In these, I don’t call somewhere, I enter a room. Of course, you need to initiate the meeting, but in terms of vision and sound it feels just like you’re in the same room together. I can see the other person; everyone can see where to focus their attention. I can say something and see how someone responds to what has been said, and I can absorb this information. It’s therefore possible to use all forms of communication that aren’t available to us with all the conventional video tools currently on the market.

Software also plays a significant role. You need a three-dimensional image of the other person so that I can walk around them and also see where I am in relation to them. However, this means that I can’t put a camera in this spot. This image or avatar must be generated by the software. We use deep learning for this. We have a wide variety of options available to us, ranging from cartoon figures to perfect photorealism. We don’t believe in cartoons but rather focus on the much more technically challenging photorealism.

What are the next steps?

Initial meetings are already possible, but only in our offices. With these prototypes, which we developed in our garage with limited resources, the first step was to convince ourselves that we could get behind this form of communication and decide whether we as entrepreneurs wanted to spend the next few years developing it. During the pandemic, we communicated as holograms, while at the same time developing the product from this user experience and testing the avatarization. However, the final mobile solution is not yet ready for the market.

Gixel devotes a large part of its website to company culture. Does that have anything to do with the solution, too?

In the team we have Felix, who has established several companies and is passionate about the topic of company culture. In our company, we want the best possible interaction with each other. Augmented reality is currently one of the most complicated products you can build. This is due to its highly complex nature as well as the large number of experts who are needed for it. If Gixel is successful, it will become a large company. If you don’t lay solid foundations from the outset, it will be difficult to implement a culture later on. This is why we put a lot of thought into it in advance, but we didn’t invent these rules. Ultimately, I learned these things at Fraunhofer. We have embodied this culture for over a year, and it has worked surprisingly well. At the moment, I couldn’t imagine anything better.

Could you give us an idea of your philosophy?

In our company, there is virtually no micromanagement, and we don’t define any processes or rules. In a nutshell: We work with adults. Each employee knows what is needed at a personal level, and everyone also knows what is required for our company. We demand both and every employee is responsible for reconciling both sides. We don’t want time being served in an office in order to make up the agreed eight hours. Time is used as a measurable variable. No employer wants to pay for time but rather for performance. With our culture, we want to solve this dilemma and it works really well. However, I’m not sure whether this concept can be applied to every company.

We are by no means evangelical. We have established our culture for one reason only: because we believe that we can achieve maximum effectiveness with it. It’s timeless, because we were already working with these principles before the pandemic. There’s a significant push for digitalization, a higher acceptance of hybrid working models and working from home. From a technical point of view, these are precisely the areas where our product comes into play. However, the pandemic wasn’t an enabler. Communication is THE basic human need, and we provide a product for this need. I see these things as separate.

Did you have experiences at IOSB that were heading in this direction?

From the outside, Fraunhofer is seen as a very large organization. Generally, you work in smaller units of 30 to 40 people, which means you are heavily shaped by your own department. Personally, I found Fraunhofer to be excellent training. Whether you’re a committed employee or completing a doctorate, you’re in regular contact with the industry and learn very quickly at a very high level. I draw on this a lot, you could even say daily. When we are setting up measurement systems, for example, we come very close to what we did at Fraunhofer. As a result, implementation usually happens very quickly here.

Fraunhofer is renowned for its special culture, but a start-up works completely differently. One example is the speed. One Friday evening, we had the idea to test out an AR headset. On Saturday morning, we had our hands on the model we wanted. In a start-up, ideas are implemented very quickly. At Fraunhofer, there are a number of good reasons that slow down this speed — when it comes to purchasing, for example. Another huge difference is the focus on a single product. As an entrepreneur, you have so much more personal responsibility and you’re motivated to solve problems.

Do you still keep in touch with former colleagues or Fraunhofer?

Of course, I’m still in contact with my old department. As a client, we work together with various institutes of the Fraunhofer-Gesellschaft. I see Fraunhofer as a laboratory that is always available and, as an alumnus, I don’t hesitate to contact them. We have fixed plans with Fraunhofer and don’t need to set up every lab ourselves. We’re also grappling with the shortage of skilled workers, and I’d wager you can’t find a better pool of experts in Germany with the right equipment and enthusiasm for complex projects.

What challenges does Gixel face?

Various manufacturers are currently experimenting with interfaces such as hand controllers, control by means of a ring or gesture control. The same applies to virtual meetings in the room. A well-known company has even changed its own name for this. However, it is primarily a technical issue. One of the key aspects is the “field of view”, which is closely linked to the feeling of immersion. If the person I’m talking to disappears from my field of view when I turn my head, then our communication is not immersive. The feeling that someone is actually there can only be achieved with a wide field of view. Approaches which focus on diffractive waveguides or birdbath optics, for example, make huge compromises necessary and encounter physical restrictions. We were lucky enough to be able to observe tests and are now taking a completely different technical approach.

How does your start-up benefit from SPRIND funding?

It’s a convenient, new way to finance start-ups in Germany. Augmented reality is incredibly complex and costly, as it involves a complicated mixture of software, processor and optics and is usually at the limits of what is technically possible. The money required for this is correspondingly high as is the time required to bring a development out of the lab and onto shelves. The German federal government established SPRIND to support projects that have the potential to set up new industries in Germany. We applied for this special type of financing — it’s not a grant — with the aim of turning a great idea into reality.

Given the incredibly high financial risk with which we started, we would never have received this amount of financing on the regular market, of this I’m certain. We’ve since minimized this risk and we hope that we can use our idea to show how this kind of financing is worth it and how we can create added value for the Federal Republic of Germany.

Mr. Taphanel, thank you for talking to us.