Teaching a computer about harmony

New Cross Currents festival combines art and science


MAASTRICHT. What do Grammy winner Jacob Collier, food trucks, and Maastricht University researchers have in common? They are all part of Cross Currents, a new festival held in the Sphinxkwartier on Saturday 13 May, and organised by Jazz Maastricht. Four art and science collaborations will be presented at the festival, which combines music performances with movie showings and presentations. One of them is a project on artificial intelligence and creativity.

The artists and scientists will spend the next two years researching how they can bridge the gap between disciplines and use their respective expertise to contribute to interdisciplinary developments. Their presentations at Cross Currents can be seen as a kick-off. Gerhard Weiss, professor of Data Science and Artificial Intelligence, was immediately interested when he was asked for the project. “I thought it was exhilarating to explore creativity from the perspective of artificial intelligence (AI). Is it possible to create a computer program that is creative? Can it develop a kind of understanding of the human concept of harmony? This question is relevant with respect to novel innovative AI applications and it allows us to investigate the ultimate boundaries of AI.” He came in contact with Marie van Vollenhoven, a visual artist who wrote her thesis about AI and the arts. “That was ten years ago, and technology has improved a lot since then. I am very curious to see if a computer can understand composition. I want to understand the logic behind harmony”, says Van Vollenhoven.

Van Vollenhoven, Weiss and Jerry Spanakis, assistant professor at DKE use a special composition tool for their project. It shows simple shapes in black and white: a circle, a triangle and a square. They can be made bigger or smaller and placed in a certain order. At the festival, visitors will use the shapes to make a composition. “There will be a piano player who plays the corresponding sounds and a dancer who will make the moves,” says Van Vollenhoven. The question is: what do people find harmonic – in both visual and sound aspects? The tool will be used to collect data about harmonic compositions that Spanakis will then analyse and explore. Spanakis: “We want to make concepts such as harmony and composition clearer. Now we have a general understanding of what is well-balanced, but this will put it into a system.”

Van Vollenhoven also plans to make several compositions herself. “Then we ask people to judge on a scale of 1-10 to see how harmonic they are.” The researchers also want to conduct a worldwide online survey in which people are asked to solve small puzzles. “Which shape do you feel should follow a certain shape?” says Van Vollenhoven. The aim is to collect data to write a computer program. Spanakis: “We want to find out if we can train a computer to recognise harmony based on this data. The next step is letting it compose music by itself.” Future performances could include a computer on stage, says Van Vollenhoven. “While the audience is composing, it will make its own composition, and afterwards we’ll compare the two.”

This project touches on some of the fundamental questions of AI, says Weiss. “What is the difference between man and machine? The machine is already better at making rational decisions in complex situations. People say: we are creative, machines are not. But what if they can be too?” Many people will also argue that in the case of art, emotions are involved, but Weiss thinks there is a way around that. “If a work of art is well-balanced, it will trigger certain emotions. Also, there are already systems that recognise emotions and adapt to your mood. You could build a system that has the purpose, or built-in intention, to create a picture or a melody that makes you feel happy.” As an artist, Van Vollenhoven feels that there is often too much emphasis on the story behind a work of art. “I believe there is a universal sense of what is beautiful or what communicates from one person to the next. If that’s the essence, all the rest is extra layers. You could say the stories are important after the work itself triggers a common interest between the spectator and the maker.”

However, this is something for the distant future. The project – which is part of the International Meeting in Performing Arts and Creative Technologies (Impact) – receives funding from the European subsidy programme Interreg, but as Van Vollenhoven says “It’s not a lot. We can’t work on it full-time, but we will continue with it.”

There are three more collaborations at the festival. Dancer Joost Vrouenraets and his team have joined forces with neuroscientist Beatrice de Gelder to discover what happens when you translate movement into algorithms. Filmmaker Joris Hoefakker and researchers Ron Heeren and Martin Paine are experimenting with new ways of turning complex molecular imaging data into beautiful, fluid imagery on screen. And visual artist Fabio Roncato and neuroscientist Elia Formisano of the Maastricht Brain Imaging Centre have created an audiovisual “sculpture” that will transport you to another world in which aural illusions become a visual reality.

What is Cross Currents?

Since it’s organised by Jazz Maastricht, naturally jazz plays a major role at the festival. All artists performing at the Muziekgieterij combine jazz with another genre. Pop and soul in the case of Jacob Collier, Afrobeat where the Dutch Jungle by Night is concerned. Belgium’s Stuff brings a unique, hard-hitting electro-jazz blend to the stage. Additionally, the best bands of the Maastricht Conservatorium’s jazz department will perform. Pathé and Lumière will show movies in which jazz plays a big role, while their soundtracks are performed live.

Cross Currents, Saturday 13 May, 17.00-1.00 hrs, tickets: €25 (student), €35 (regular), www.crosscurrents.nl

Teaching a computer about harmony