Make your paper interactiveCreate

Text from the interactive explanation

AudioCLIP: Extending CLIP to Image, Text and Audio

Andrey Guzhov, Federico Raue, Jörn Hees, Andreas Dengel

Original paper

AudioCLIP · Guzhov, Raue, Hees & Dengel · 2021

Hand AudioCLIP a recording and a list of 50 sound names it was never trained to choose between, and it picks the right one about 7 times in 10 (69.40 %, reported in the paper; blind guessing would score 2 %). Its trick is one shared map of meaning, where a sound, a picture and the words for the same thing end up side by side.

Illustrative mapA 3-D sketch of a 1,024-number space; positions are hand-placed, not computed by AudioCLIP. Drag (or focus it and use the arrow keys) to turn the ball.

Drop a new recording onto the map

Tap a sound: it lands on the map, and the closest written word names it.

violet tag = written words

green tile = pictures

coral wave = sounds

A sorting machine with a fixed menu

Most sound-recognition systems are sorting machines. Engineers fix a list of categories — the menu — and train the machine on many labelled recordings until it drops each new sound into one of those bins. UrbanSound8K, one of the two benchmarks in this paper, has exactly ten bins.

Feed this pretend machine a few sounds. Some are on its menu. Some are not.

Feed the machine

Pick a sound above and watch which bin it lands in.

IllustrativeA pretend classifier with UrbanSound8K's ten real category names. Its wrong guesses are invented to show the problem; sounds are synthesised examples.

Adding a new category the old way means collecting labelled recordings of it and retraining. That is where sound has a problem: labelled audio is scarce. ESC-50, the paper's other benchmark, holds 2,000 five-second clips in 50 categories. The text-and-picture model CLIP, which AudioCLIP builds on, learned from a collection of its own.

Predict first

Drag the slider to your guess, then reveal.

So what if sound could borrow from the giant world of words and pictures?

Meaning as a direction

Modern AI models contain encoders — the paper calls them heads . A head reads an input (a sentence, a photo, a recording) and outputs a list of 1,024 numbers, called an embedding . Picture that list as an arrow pointing in some direction in a vast space. The goal: things that mean the same should point the same way. Collect all those arrow tips and you get a map of meaning .

To compare two arrows, measure the angle between them. The score that does this is cosine similarity : 1 when two arrows point the same way, 0 when they are at right angles (unrelated), −1 when they point in opposite directions.

Turn the sound arrow

Drag its round handle, tap anywhere on the dial, or use the slider.

IllustrativeThis is how AudioCLIP names a sound: whichever word points closest wins. A flat 2-D drawing of a 1,024-dimensional idea.

Only the direction of each arrow matters, so arrows are rescaled to the same length (“normalised”). Their tips then all lie on the surface of a ball — a ball in 1,024 dimensions, which nobody can draw. The Meaning Map at the top of this page is a 3-D sketch of that ball.

In AudioCLIP, the word head and the picture head produce embeddings that “are mapped linearly into a multi-modal embedding space of size 1 024”, and training maximises “the cosine similarity between matching textual and visual representations” (paper, Section 3.1). The new audio head is given a 1,024-number output so that it can point into the same space.

Learning by matching

How did CLIP (OpenAI, 2021) learn to point a photo of a dog and the words “a dog” the same way? By playing a matching game on about 400 million captioned pictures. Each round it takes a batch of N picture–caption pairs and scores every caption against every picture — an N×N grid of cosines. Then it nudges all the arrows so the N true pairs (the grid's diagonal) score high and the N×N−N mismatches score low. This is called contrastive learning .

Starting…

Toy simulationReal gradient descent on CLIP's training objective, but tiny: 5 items on a circle instead of 400 M pairs in 1,024 dimensions. The temperature starts soft and eases to 0.1 so pairs can slip past each other on a flat circle.

For each word (a row of the grid), turn its similarities into shares that add up to 100 % (a “softmax”; dividing by a small temperature such as 0.1 makes the shares sharper). Ask: what share went to the right picture? The loss is minus the logarithm of that share, averaged over all rows. Do the same for every picture's column — that is why it is called symmetric cross-entropy. Gradient descent then moves every arrow a small step in the direction that lowers the loss: matches attract, mismatches repel.

The paper puts it this way: CLIP “learns to maximize the cosine similarity between matching textual and visual representations, while minimizing it between incorrect ones, which is achieved using symmetric cross entropy loss over similarity measures.”

The payoff is called zero-shot classification : to sort pictures into new categories, you just write the category names. Each name becomes an arrow, and the name pointing closest to the picture wins — no retraining. AudioCLIP wants exactly this trick for sounds.

Turning sound into a picture

To join the map, sound needs a head of its own. AudioCLIP uses an audio network called ESResNeXt . It first turns a recording into a time–frequency picture — time runs left to right, pitch runs bottom to top, brightness shows loudness — and then reads that picture with a network built like an image recogniser.

IllustrativeSynthesised example sounds (illustrative), not recordings from the paper's datasets. The picture is computed live from the very sound you hear (short-time Fourier transform: 512-sample window, steps of 128 samples, loudness on a log scale).

Can you read a sound by eye?

Which one is the siren?

Tap the picture you think it is.

That is the point: sounds have visual shapes, so a network designed for photos can learn to read them. ESResNeXt even starts from weights trained on ImageNet photographs, and instead of a fixed recipe for making the time–frequency picture, it learns its own during training.

ESResNeXt is based on the ResNeXt-50 image architecture and has about 30 million trainable parameters. Its time–frequency transform is itself trainable, built from “complex frequency B-spline wavelets”. The authors chose it because it performs competitively on AudioSet and reached state-of-the-art-level accuracy on UrbanSound8K and ESC-50, and because it is robust to added white noise and to lower sample rates.

Because the sound collections are small, the training clips were varied (“augmented”) to fight overfitting: time scaling (stretching or squeezing a clip, which changes its pitch too; random factor in [−1.5, 1.5]), time inversion (playing it backwards, probability 0.5 — the audio version of mirroring a photo), random crop or padding to a common length (centred during evaluation), and added white noise at a signal-to-noise ratio between 10 and 120 dB (probability 0.25).

Three heads, one map

AudioCLIP keeps CLIP's two heads — words and pictures — and adds the audio head as a third. The glue is AudioSet : about 1.8 million YouTube snippets of up to 10 seconds in 527 sound classes. Every snippet naturally comes with a soundtrack, video frames and a text label: a ready-made triple.

Training adds two new matching games to CLIP's original one. Besides words ↔ pictures, there are now words ↔ sounds and pictures ↔ sounds — three similarity grids, one shared map. The paper trains in stages; step through them below.

Starting…

Toy simulationSix concepts in 3-D, trained with the same three symmetric matching losses (temperature 0.25). Dashed rings with a padlock = frozen heads. In stage ③, faint outlines mark where CLIP originally put each word and picture. Drag the ball to turn it.

Warm-up (standalone). The audio head, initialised from ImageNet picture weights, was first trained on AudioSet alone — for 30 epochs instead of the original 5 — which improved it (Table 1).

Partial (cooperative) training. Its classification layer was replaced by a randomly initialised one with as many outputs as CLIP's embedding space (1,024). The word and picture heads were frozen and “served as teachers in a multi-modal knowledge distillation setup”: the audio head learns to point where they point.

Full training. Because AudioSet's pictures and labels are distributed differently from CLIP's own data, all three heads were then tuned together on AudioSet.

Fine-tuning (optional). For the classification records, only the audio head was further trained on ESC-50 or UrbanSound8K, using each sound and its text label. All stages used stochastic gradient descent with Nesterov momentum 0.9, weight decay 5·10⁻⁴ and batches of 64; 30 epochs on AudioSet, 50 for fine-tuning.

Name it from words alone

Once sounds live on the map, classifying them needs no new training. Write the candidate names; the word head places each name on the map; the name pointing closest to the sound wins. The cosines are turned into shares that add up to 100 % (a softmax). The menu is just text — so you can change it at will.

Lines join the sound to every word on the menu; thicker = bigger share. Dim words are off the menu.

1 · Pick a sound

2 · Write the menu — tap names to add or remove

IllustrativeWord and sound positions are hand-placed (sounds sit near, not exactly on, their word, with realistic look-alikes nearby); the percentages follow from those positions, not from AudioCLIP.

The real test: zero-shot accuracy (Table 3)

≈ 69 of every 100 ESC-50 clips named correctly from the 50 written names alone (69.40 %). Guessing gets 2 of 100 (yellow rings).

“Zero-shot” is used in CLIP's sense: AudioCLIP was not trained on ESC-50 or UrbanSound8K before being tested on them. It learned beforehand from AudioSet's 527 everyday-sound classes. On UrbanSound8K, 68.78 % (full training) sets a new zero-shot baseline, in the paper's words.

Search in any direction

A shared map works every way round: a word can find sounds, a sound can find pictures, a picture can find sounds. The authors' public demo shows this with the released, fully trained model on 5 labels, 5 ESC-50 sound clips and 10 photos. Below are its actual printed outputs.

1 · Pick a direction (query → result)

2 · Pick a query

Real model outputReal outputs of the released AudioCLIP model (full training) as printed in the authors' demo notebook, MIT-licensed repository. Photos and sounds are not reproduced here; icons stand in for them. Percentages are softmax shares among the demo's candidates only (5 labels, 5 sounds or 10 photos); “—” = not printed in the demo.

At scale: search quality before and after full training (Table 4, mAP)

mAP (mean average precision) measures how high the correct items rank in the results list, averaged over all queries (0–100; higher is better).

Full training helps most AudioSet searches and UrbanSound8K, but hurts word→picture search on ImageNet (52.91 → 33.13), and ESC-50 word→sound search stays about level (77.20 → 76.97).

P@1 / R@1 = top-1 precision / recall. All values in %.

The scoreboard

How does this compare with networks built only for sound? The paper's Table 3 lists accuracy on both benchmarks, from 2015 to 2021. Most entries were trained on the target dataset; two earlier attempts and AudioCLIP's zero-shot runs were not. Tap a bar for details.

Reported in the paperAccuracy in %, Table 3 (sources as cited by the paper). Bars run from 0 to 100 %.

Back to the map

So — can you teach a machine new sound categories just by writing their names? AudioCLIP shows it can work. Given only ESC-50's 50 written names, AudioCLIP picks the right one for 69.40 % of clips without training on that dataset: more than double the previous zero-shot best (33.00 %), and above a 2015 network trained on it (64.50 %). Fine-tuned on the target data, it set new records of 97.15 % (ESC-50) and 90.07 % (UrbanSound8K). (All reported in the paper.)

What it brings closer

One shared map for sounds, words and pictures.

Sorting sounds into new categories by writing their names — no new recordings, no retraining.

Searching sounds with words or photos, and photos with sounds.

Scarce sound labels can lean on the vast text-and-picture world CLIP learned from.

Where it leads next

Training choices shape the map: letting all three heads adapt to AudioSet raised most sound scores, while ImageNet zero-shot picture accuracy went from 40.51 % to 21.79 % — the authors point to AudioSet's pictures and labels being distributed differently from CLIP's.

The authors plan to test AudioCLIP “on a wider variety of datasets and tasks”.

Swapping in “more powerful networks” for the picture and sound heads “could further improve the model performance”.

The code, training settings and trained weights are public (see below), so anyone can put new names on the menu.

Source & credits

Andrey Guzhov, Federico Raue, Jörn Hees, Andreas Dengel. “AudioCLIP: Extending CLIP to Image, Text and Audio.” arXiv:2106.13043 [cs.SD], 2021 (DFKI GmbH & TU Kaiserslautern). arxiv.org/abs/2106.13043 · PDF

Version note: arXiv preprint v1, 24 June 2021 (comments: submitted to GCPR 2021).

Code, hyper-parameters and trained weights: github.com/AndreyGuzhov/AudioCLIP. The query results in Chapter 7 are the outputs printed in that repository's demo notebook (MIT-licensed), run with the released full-training weights; the demo's photos and sound files are not reproduced.

Paper distributed on arXiv under CC BY-NC-ND 4.0; no figures are reproduced here.

All visuals and sounds on this page are original illustrations and toy simulations; numbers quoted from the paper's Tables 1–4 are reported results.

Topics in this iPaper

Computer science
we present an extension of the CLIP model that handles audio in addition to text and images

Abstract, page 1

multimodal learning
The proposed model is able to process all three modalities simultaneously, as well as any pair of them.

Section 3.3, page 4

environmental sound classification
AudioCLIP achieves new state-of-the-art results in the Environmental Sound Classification (ESC) task

Abstract, page 1

zero shot learning
Further it sets new baselines in the zero-shot ESC-task on the same datasets (68.78 % and 69.40 %, respectively).

Abstract, page 1

contrastive learning
our proposed model employs contrastive learning to perform training on textual, visual and audible modalities

Section 2, page 3

cross modal retrieval
we added the support of audio, enabling the model to query between text, images and audio in any combination

Section 5.2, page 11

knowledge distillation
these heads served as teachers in a multi-modal knowledge distillation setup

Section 4.3, page 7

AudioCLIP: Extending CLIP to Image, Text and Audio — Interactive explanation | iPaper.ai