Audio Research Group conducts research in many areas related to audio, speech, and music signals. The research contains both basic research on the core techniques such as acoustic pattern classification, as well as applied research targeting specific applications. 

Content analysis of audio

Content analysis of general audio is concerned with the analysis of audio for automatic extraction of relevant information such as acoustic scene characteristics and sound sources present. Applications of audio content analysis range from simple classification tasks for enabling context awareness, to security and surveillance systems based on detected sounds of interest, to the organization, indexing, tagging, and querying of large audio databases.

Topics

  • Acoustic scene classification
    Classifying recordings into classes that characterize the environment in which it was recorded.
  • Sound event detection and recognition
    Recognizing individual sound events present in the auditory scene.
  • Audio tagging
    Automatically applying descriptive tags for audio signals
  • Audio captioning
    Automatically applying textual description to audio signals
  • Audio-based database query
    Querying large audio databases based on audio content
  • General audio classification
    Classifying audio signals based on many different aspects of the sound.

Spatial audio

Microphone arrays provide a link between the physical locations of sound objects for the computer software and can allow capturing the sound field. Applications of microphone arrays include physical location determination such as speaker localization and speaker position tracking, signal enhancement, and separation.

  • Sound localization
    Automatic determination of the sound source physical location
  • Self-localization
    Automatic localization and synchronization of the device microphones
  • Speaker position tracking
    Tracking the location and detecting the appearance and disappearance of multiple speakers that are temporally overlapping
  • Shooter localization
    Shooter localization and estimation of bullet trajectory, caliber, and speed

Source separation and signal enhancement

Source separation means the tasks of estimating the signal produced by an individual sound source from a mixture signal consisting of several sources. The techniques can be used to process sounds for human listeners for example to improve speech intelligibility, or to a preprocessing step for computational analysis methods, since analysis and processing of isolated sources can be done with much better accuracy than the processing of mixtures of sounds.

Topics

  • Singing voice and music separation
    Separation of instrumental tracks from a song
  • Spatial sound source separation
    Separation of sound sources based on their perceived direction
  • Object-based coding of spatial audio
    Separation of sound objects from spatial audio mixtures
  • Speech enhancement and separation
    Enhancement of target speaker's voice from background noise or interfering voices

Music information retrieval

The on-line and personal music collections nowadays are huge, not being limited by space requirements anymore. Such collections created the need to develop efficient music information retrieval (MIR) techniques to automatically organize and search through collections. Our research team is a world-leading research group in audio-based MIR, where the information about the music is analyzed automatically based on the sound signals of the music pieces.

Topics

  • Automatic musical instrument recognition
    Automatically recognizing musical instruments from polyphonic and multi-timbral music signals.
  • Singer identification
    Identifying the singer from a musical piece
  • Lyrics recognition
    Automatic recognition of the words/phonemes in singing
  • Automatic alignment of singing and lyrics
    Synchronization of the lyrics text with the singing voice in a musical piece
  • Sound source separation
  • Music transcription
    Automatically extracting musical notation for musical signals.
  • Music structure analysis
    Subdividing a musical piece into parts and sections at the largest time-scale. In popular music, for example, it is usually possible to identify parts that we label as the chorus, the verse, an introductory section, and so on.
  • Structured audio coding
    Developing sparse and structured representations for music signals, sound source modeling (parametric and statistical models)
  • Music classification
    Classification of music according to the style, the genre, or the musical instruments involved
  • Musical instrument synthesis Synthesis of two or more musical instruments, resulting in a new instrument that possesses the acoustic characteristics of the instruments involved

Speech recognition

One of the major problems in automatic speech recognition technologies is the sensitivity of recognizers to any interfering sounds. Since natural environments often include other sound sources, the performance of the existing technologies is severely limited. Our research team has been doing pioneering work in the recognition of sounds in mixtures, including speech, music and environmental sounds. This work has resulted in top positions in the international CHiME evaluation campaigns.

Topics

  • Noise-robust automatic speech recognition
    Automatic speech recognition in sound mixtures

Speech synthesis and voice conversion

Speech synthesis refers to the artificial conversion of text into speech. It is needed in applications where the most convenient modality of interaction with the user is the speech: for instance in hands-free user interfaces, voice-driven services, and applications targeted for the blind. The aim of voice conversion is to convert the speech spoken by one speaker to sound like the speech of a specific target speaker while maintaining the content of an utterance. It can be used for example in creating various synthesis voices in text-to-speech systems as well as generating various voices from a speaker for dubbing purposes. In addition, knowledge on how to separate the speaker identity and the speech content can lead to improved results in speaker and speech recognition.

Topics

  • Corpus-based speech synthesis
    Speech synthesis using real recorded speech data as a basis: unit selection and hidden Markov model (HMM) based synthesis
  • Hybrid-form synthesis combining unit selection and HMM-based synthesis
    Multiform synthesis by combining the two methods
  • Exemplar-based voice conversion
    Converting speech of one speaker to sound like speech of a specific target speaker