Research / Robust speech recognition and spoken search / [15]
ALBAYZIN 2018: how well can systems find spoken terms in Spanish audio?
ALBAYZIN 2018 spoken term detection evaluation: a multi-domain international evaluation in Spanish. EURASIP Journal on Audio, Speech, and Music Processing, 2019.
An international open evaluation of spoken term detection in Spanish over three speech domains. Three research groups submitted 11 systems, and performance proved highly sensitive to changes in the data domain.
The task
Spoken term detection takes a search term written as text, one word or several, and finds where it is said in a collection of recordings. A system returns the files that contain the term, the start and end time of each occurrence, and a score for its confidence in the detection.
The evaluation
ALBAYZIN 2018 was an internationally open evaluation for this task in Spanish. It used three databases from different domains: MAVIR, a set of talks from workshops; RTVE, broadcast news programmes; and COREMAH, spontaneous conversations between two people.
Three research groups took part and submitted 11 systems. The paper describes the evaluation, the databases, the metric and the systems, and reports the results together with a fusion of the primary systems.
What the analyses show
Results are broken down by properties of the search terms: whether they are inside or outside the system’s vocabulary, one word or several, and native or foreign. The overall conclusion is that the task is far from solved and that performance is highly sensitive to a change of data domain.
Why it matters
A shared evaluation with public conditions is what lets different systems be compared on equal terms, and here it shows how much a system tuned for one kind of speech loses on another.
Cite
Tejedor J., Toledano D.T., Lopez-Otero P., Docío-Fernández L., Montalvo A.R., Ramírez J.M., Peñagarikano M., Rodríguez-Fuentes L.J. ALBAYZIN 2018 spoken term detection evaluation: a multi-domain international evaluation in Spanish. EURASIP Journal on Audio, Speech, and Music Processing, 2019.
PDF distributed under CC BY 4.0; copy of the published version.