Cognitive Machines

Screwing, drilling, welding or bonding: ANNIE, the mobile assistive robot developed at Fraunhofer IFF, allows direct human-machine cooperation.

Cognitive Systems/Machines: Teaching Machines to Learn

Service Robot

Machines are found in more and more application areas where they help people make better decisions. International Data Corporation (IDC) anticipates global expenditures of more than 40 billion US dollars for cognitive solutions by the year 2020. Fraunhofer wants to achieve more effective pooling of the activities in relevant research fields and systematically promote them.

Shopping assistant Paul from Fraunhofer IPA asks customers what they are looking for and escorts them to the appropriate shelf.
© Photo Saturn

Shopping assistant Paul from Fraunhofer IPA asks customers what they are looking for and escorts them to the appropriate shelf.

Fraunhofer IIS uses affective computing to allow machines to recognize emotions.
© Photo Fotolia

Fraunhofer IIS uses affective computing to allow machines to recognize emotions.

As a shopping assistant, service robot Paul recently welcomed customers in an electronics store, asked which products they wanted, and escorted them to the appropriate shelf. On the way, he chatted about the weather and then asked a few feedback questions, for example, whether the customer was satisfied with his service. “He operates in a dynamic, everyday environment in which he recognizes objects or people and reacts to them,” explains Martin Hägele, head of the Robot and Assistive Systems Department at Fraunhofer IPA. The robot uses sensors to gather information about its environment in order to navigate dependably at all times and to locate people to conduct dialogs.

Paul contains the Care-O-Bot4® robot platform, which Fraunhofer IPA originally developed for active support of people in households, hotels, nursing homes and hospitals. Now plans call for increased deployment in companies.

Understanding environments, planning actions, reacting to obstacles, communicating with people - cognitive systems master these challenges by harnessing machine learning methods. Here, machines learn to solve a task on the basis of example data and to transfer what they have learned to new situations. For example, they can plan and optimize processes, make forecasts, recognize patterns or distinctive features, and analyze image and voice signals. These systems form the basis for important future technologies such as autonomous driving or autonomous robots.

Thanks to the software SHORE from Fraunhofer IIS and "affective computing", shopping assistant Paul can even recognize a person's mood and express his own state of mind. This involves machine recognition of emotions in facial expressions, sensor data merging, and analysis of biosignals, such as pulse, voice, gestures or movement. For example, the stress level of car drivers or factory workers can be determined, as can customer wishes or requirements. “Analysts regard affective computing as the commercially fastest growing market in the machine learning field,” explains Dr. Jens-Uwe Garbas, head of the Intelligent Systems Group at Fraunhofer IIS.

Mastering Uncertainties

Underwater robots must act entirely autonomously.
© Photo karakter Design Studio

Underwater robots must act entirely autonomously.

Blazing their way through unknown terrain is especially challenging for robots. The Fraunhofer Institute of Optronics, System Technologies, and Image Exploitation IOSB is developing probabilistic methods that assess statements not only as “true” or “false”, but additionally with a certain probability that indicates the degree of uncertainty of the statements. “While the technology is highly complex, it plays a crucial role in mastering all possible uncertainties,” explains Professor Jürgen Beyerer. The robot constructs a type of "world model" for itself by observing the real world; it then reaches adjusted conclusions by making associations, interpreting new information and learning new concepts. This “powerful memory structure” even allows robots to be used in danger zones, which Fraunhofer IOSB demonstrated by taking the fully autonomous vehicle “IOSB.amp Q1” as an example. This mobile platform is able to explore even especially rough terrain on its own. A demanding task that cannot be solved based on logic alone.

Underwater robots that operate in depths where no light signals or radio waves penetrate call for what is probably the highest level of autonomy. They glide through the depths without cables, collect data and return to the research ship on their own. The Shell Ocean Discovery XPRIZE competition challenges teams from around the world to participate in the large-scale mapping of the ocean floor at depths of up to 4000 meters. The only German team competing in the field is the Argonauts from Fraunhofer IOSB.

But even without robots, learning systems are making more and more of an impact in many industrial sectors. For example, machine learning methods can also be used to examine complex situations under real conditions, and can, for example, replace elaborate prototype tests. At the Fraunhofer Institute for Algorithms and Scientific Computing SCAI, a team headed by Professor Jochen Garcke, head of the Numerical Data-Driven Prediction Department, is investigating how the various methods can be adapted to specific technical tasks. For instance, in vehicle development, numerical crash simulations are employed during the development phase for the ongoing optimization of components and sheet metal thicknesses. “Our methods are helping structure the large quantities of complex data,” Garcke explains. “Which data are similar, which are different? This is not easy to see when there are 50 different forming steps.” The investigation has produced innovative methods for development engineers in the automotive industry, who can now conduct simultaneous comparative analyses of data from many simulations.

Particularly in the engineering world, the buzz term "grey box" pops up in the machine learning procedure. While black box models do not take the physical model of the problem that is to be learned into account, it is derived as precisely as possible and used in white box algorithms. Grey box combines these two approaches: Here, a data analysis model is enriched with physical knowledge in order to achieve better analyses.

Saving Lives with SENEKA – IOSB.amp Q1 in Action

Five Fraunhofer institutes from different disciplines have joined forces for the sensor network with mobile robots  for disaster management SENEKA project. The objective: swifter rescue of people in the event of a disaster by a reliable and flexible system comprising sensors, communication components and robots. 

Simulating Traffic

The RODOS® system is the first interactive motion simulator based on an industrial robot.
© Photo Fraunhofer ITWM

The RODOS® system is the first interactive motion simulator based on an industrial robot.

Software from Fraunhofer IAIS is learning to recognize an extensive range of traffic information more quickly and efficiently.
© Photo Fraunhofer IAIS

Software from Fraunhofer IAIS is learning to recognize an extensive range of traffic information more quickly and efficiently.

The Fraunhofer Institute for Industrial Mathematics ITWM also works with system simulations for virtual vehicle development and safeguarding. “Today there are various levels of assistance situations, ranging from fully human guided to autonomous, and the number of information systems in use is steadily rising,” explains Dr. Klaus Dreßler, head of the Mathematical Methods in Dynamics and Durability Department at Fraunhofer ITWM. He is examining how drivers and their assistance systems react to external conditions with the help of RODOS, the first interactive motion simulator based on an industrial robot, which extends far beyond a standard simulator's possibilities. This should allow safeguarding concepts for mixed traffic in the future. “We challenge the intelligence of the autonomous vehicles and observe how they react when, for example, they are cut off by a human-controlled car.”

Construction sites pose a further challenge in road traffic. Drivers behave with uncertainty and automated vehicles have problems understanding complex traffic routing with different information on speed and the course of the lanes. AutoConstruct, a project funded with almost two million euros by the German Federal Ministry of Economic Affairs and Energy, aims at real-time environment recognition in construction sites using series and cost-optimized camera sensor systems for highly and fully automated driving. The project kicked off at the beginning of December 2016. The Fraunhofer Institute for Intelligent Analysis and Information Systems IAIS is substantially involved, and has taken on the job of image recognition and processing. ”With deep learning, a key technology for the automotive sector’s future, we are teaching the software to recognize classic patterns more swiftly and more efficiently,” explains Dr. Stefan Eickeler, responsible for object recognition at Fraunhofer IAIS.

Deep learning draws on multi-layer, artificial neural networks which, in an abstract form, are based on information processing in the human brain. Deep learning is used to analyze highly complex data. Today, however, not even researchers can precisely explain how neural networks reach certain results. The process more or less involves feeding values into a black box and obtaining “surprisingly” applicable results. The team headed by Dr. Wojciech Samek, head of the Machine Learning Research Group at the Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute, HHI, has been working with the Technische Universität Berlin to develop software that makes it possible to observe a neural network as it thinks. For example, if genetic data from patients is fed into such a network, it can analyze the probability that the patient has a certain genetic disease. “But it would be even more interesting to know which characteristics the program uses to determine its decisions,” says Samek. This could be a specific genetic defect that supplies information for cancer therapy individually tailored to the patient.

Personalizing Medicine

© Photo istock

At the Fraunhofer Institute for Algorithms and Scientific Computing SCAI, the Bioinformatics Department focuses on speeding up the path from knowledge generation to application for the biotechnological and pharmaceutical industry, primarily in the area of dementia research. This involves combining very different data and distributed knowledge. The department specializes in generating disease models for this purpose. “Machine learning methods help us extract both data and knowledge from the literature on an order of magnitude that extends far beyond the cognitive capabilities of individual scientists,” states Professor Martin Hofmann-Apitius, head of the Bioinformatics Department. “With help from the machines, we are generating models of the world that allow new insights into the mechanisms of dementia that are beyond human capability.”

Self-Learning Software for Better Medical Diagnoses

© Photo Fraunhofer MEVIS

MRI, CT, pathology: doctors have to consider medical image data –increasing in both amount and complexity – to perform diagnoses and monitor therapy. The Fraunhofer Institute for Medical Image Computing MEVIS in Bremen is creating a new approach to provide effective assistance. In the recently started AMI project (Automation in Medical Imaging), self-learning computer algorithms will automatically trawl large volumes of data and search for abnormalities to improve the accuracy of computer-generated diagnoses. MEVIS has partnered with the Radboud University in Nijmegen, the Netherlands, which hosts one of the world’s leading research groups for automated image evaluation.

Optimizing Processes

Industrial production is likewise generating more and more data in the age of Industrie 4.0 and the Internet of Things. “Since 2009, we have been using unsupervised, data-driven learning methods in order to monitor a process's normality and detect errors,” explains Christian Frey, head of the Systems for Measurement, Control and Diagnosis Department at the Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB. “Now we can additionally optimize the processes.” The system searches the data for key performance indices and then maps these to qualities, resource efficiency, material or energy consumption, and similar categories. For example, a universal tool to monitor complex chemical processes is already in use.

At the Fraunhofer Institute for Open Communication Systems FOKUS, the work concentrates on the use of time-variable electricity prices in power-intensive processes as well as on “predictive maintenance” for distributed systems and for infrastructure management of smart cities. “For example, we create forecasts for ‘load shifting’,” explains Dr. Florin Popescu, head of the IT4Energy Center Project at Fraunhofer FOKUS. Load shifting is intended to prevent energy peaks in production and to profit from energy rates that vary over time. This forecast is based both on algorithms and on big data integration, and it leads to the necessity of continuous planning adjustments. “We estimate that careful forecasting can cut the operating costs of large, energy-intensive industries by five to ten percent,” Popescu states. “Wind parks and other fixed systems could save even more.”

A Robot Colleague in Industrial Production

And if humans and machines work directly next to one another and are to cooperate with one another in industrial production? Mobile assistive robots must also recognize the workers' movements and be able to avoid running into them. In addition, they must independently decide which action is necessary in order to carry out diverse tasks with people and within human environments. Researchers from the Fraunhofer Institute for Factory Operation and Automation IFF are developing ANNIE, a mobile assistive robot, as well as the complex software framework and sensor data processing to allow flexible performance of tasks such as screwing, drilling, welding and bonding. “Mobile assistive robots must display fault-tolerant behavior in order to remain capable of acting in unforeseeable situations,” says Christoph Walter from Fraunhofer IFF, who coordinates and directs the research topics on mobile assistive robots at the institute.

Valued at billions, the cognitive systems market calls for major research and development activities, and will likewise necessitate major investments to be committed. In this area, Fraunhofer is a significant system provider with core competences. The objective is to promote the research and development of cognitive machines as a forward-looking field in Germany, to position it prominently, and to generate innovations. A number of joint projects and initiatives have already been launched. The Fraunhofer Big Data Alliance has initiated a far-reaching investigation into market development and potential innovations in artificial intelligence as an internally funded project. The Young Research Class, Fraunhofer's new career program for young scientists, is also dedicated to the topic of cognitive machines. In 2017, the proof of concept should be ready for the “self-determined learning cognitive assistants” idea. Fraunhofer and the Max Planck Society are cooperating on a project on the topic of machine learning. They hope to reduce large, multi-dimensional machine learning models to the parameters relevant for the results, without having to incur considerable quality losses.

There are still a number of issues to clarify, such as how to handle ethical aspects, and how to achieve further efficiency increases in the methods by harnessing revolutionary technologies such as neuromorphic chips. “All the necessary disciplines and competencies are in place at Fraunhofer at high quality levels,” summarizes Jürgen Beyerer from the Fraunhofer Institute of Optronics, System Technologies, and Image Exploitation IOSB. “Now we need to consolidate the previously unnetworked, parallel activities into visionary, symbiotic project teams."

Do We Need an Ethics Committee?

Brief interview with Professor Ina Schieferdecker, Director of the Fraunhofer Institute for Open Communication Systems FOKUS

You are advocating the installation of an ethics committee for digitalization issues. Why? 

Schieferdecker: The digitalization of our society is not only a technical challenge, it also has a moral dimension, such as when it involves topics such as autonomous driving or the use of surgical robots. 

Which ethical questions need to be answered? 

Schieferdecker: How do we teach machines moral behavior? What are corporate groups allowed to do with our data? How can I distance myself from the omnipresent digitalization? How do we regulate digital forgetting? 

Who should be a member of the ethics committee?

Schieferdecker: Philosophers, physicians, lawyers and naturally computer scientists. 

Glossary of Cognitive Systems/Machines Terms

A brief explanation of the most important terms used in cognitive systems/machines.

Cognitive systems/machines are technical systems that take in digital information from sensor data and networks and, on the basis of learning algorithms, use such data to derive conclusions, decisions and actions and verify and optimize these in a dialogue with their environment. 

Machine learning refers to methods in which an algorithm/machine repeats a task in order to learn how to execute it better and better with regard to a quality criterion. 

Artificial Intelligence (AI) is a branch of computer science that deals with equipping machines with capabilities that resemble intelligent (human) behavior. This can be achieved with pre-programmed rules or on the basis of machine learning. Strong or general AI denotes machines that can achieve generalizing intelligence and transfer capacities and are consequently not only limited to very restricted, pre-defined task fields. 

Artificial neural networks are a basis for machine learning methods based on the model of the neural cell networking in the brain. They consist of data nodes and the weighted connections between them. It is possible to implement machine learning methods by changing various parameters in the network. Deep learning refers to neural networks with a greatly increased number of levels enabling the advance into new problem classes. 

Black box, grey box and white box models differ from one another in whether or not and to which degree the algorithm knows the physical model of the problem to be learned and includes it in its learning process. White box models know this as precisely as possible, while black box approaches do not take the model into account at all. Grey box indicates a combination of the two approaches. 

Neuromorphic chips are microchips in which the properties and architecture of nerve cells are modelled at the hardware level. These neuron-like components simulate the learning and association capability of the brain, and can particularly accelerate the recognition of patterns in images or big data structures.