Modelling error in query -by -humming applications.

Abstract

Query-by-humming (QBH) applications search audio and multimedia databases for strong matches to sung, whistled or hummed musical queries. A core component of a QBH system is the similarity metric or matcher, used to determine which targets in the database most closely resemble the query. By comprehensively modelling the errors or transformations between an originating target and query object, we can effectively measure similarity in this context. We identify two major modelling concerns for the matcher component: first, the model must anticipate a range of alignment relationships between a query and target, managing the various ways queries temporally correspond with their correct targets; and, second, the model must express the range of transformations observed between corresponding elements of the target and query. We develop systems addressing these concerns, which are generalizations and extensions of the existing state-of-the-art. Relative note-representation Mongeau-Sankoff Edit algorithm (RMSE) is an efficient, probabilistically-grounded model supporting arbitrary alignment constraints---such as global and local alignment---that we term alignment<italic> types</italic>. Johnny Can't Sing (JCS) is a model comprehensively expressing all variety of transformations, or error <italic>classes</italic>: contextual differences in tempo and key; isolated errors in rhythm and pitch, as well as cumulative errors through tempo changes and modulation. JCS is not only expressive, but automatically trainable, or able to learn and generalize from query examples. We present results of experiments measuring the retrieval performance of these models with three query data sets to illustrate the effects of various assumptions about the range of errors in a query. With regards to error classes, we demonstrate that assuming an exclusively cumulative view of error---as is implicitly done with existing relative note models---has a negative impact on retrieval performance, and that the strongest performance is achieved when all classes are explicitly modelled. A comparison of different alignment types reveals the significant impact of system and experiment constraints on performance: under controlled conditions, global alignment significantly outperforms the alternatives, although in general, overlap and local alignment types more effectively model the range of queries observed. Characteristics of the three data sets---database coverage, recording quality, and query familiarity---have a large influence on retrieval performance, but under realistic conditions (untrained singers posing queries against a 10,000-entry database), the correct target is identified as most similar for 78% of queries. This research fosters a greater understanding of the impact---and indeed the existence---of modelling assumptions in query-by-humming system design.