Language & Speech Technology Arjan van Hessen + * Franciska de Jong* Roeland Ordelman* * Computer science, University Twente + Speech & Language group, TeleCats

DRUD Document Retrieval Using Intelligent Disclosure

DRUID “Developing Tools for the Indexing & Retrieval of Multi Media Content”  time-coded indexing with DUTCH speech recogniser  television news broadcast  benchmark international SDR research  parallel sources available (teletext, auto cues)

Druid: what Extract information from non-textual content Classify and index the information Give access to the information via linked time codes

Druid: how –Speech recognition Large vocabulary, speaker independent –Recognition of visual objects –Story detection –Linking to related information

Large vocabulary recognition Indexing & Retrieval

Druid Speech recogniser –ABBOT speech recogniser (Cambridge, Sheffield) –Feature extraction –Phone classification (NN) –Word recognition (HMM)

B roadcast news –Pro’s Easy available Often high quality, undisturbed speech Availability of related sources –(auto-cues, news papers) –Contra’s Mixed languages Different quality of speech (wide & narrow band), mixed together

Development –British English  Dutch TNO-NRC corpus: 10h read speech (newspaper data) –Additional phoneme training Groningen corpus: 20h read speech Speech Styles corpus: 16h spontaneous speech –Final training Broadcast corpus: 50 x “8 o’clock news” broadcasts (10h speech) Corpus Spoken Dutch: 1000h spontaneous speech (to be done in 2002)

Language modelling Acoustic recognition stops at a certain level Recognition can only improve with: –Statistical language models (large vocabulary recognition) –Finite state grammars (small vocabulary recognition)

Large vocabulary recognition Recognition is directed by –Acoustic features –Word frequency (= 65K most used words) –Bi-grams (65K 2 combinations) –Tri-grams (65K 3 combinations)

Large vocabulary recognition Building reliable acoustic feature requires  100 hours of speech Building reliable LM requires  hours of text Different context models (sport, finance, politics etc.)

Language modelling Standard LM procedure text normalisation Dutch diseases: spelling reform 90’s compounding foreign words increase of English

Text collection Nederlandse Persdata bank –Electronic version of 4 major Dutch newspapers ( ) NOS Auto cues –Daily Auto-cues of the 8 o’clock news and the news for children ( ) TeleText –Daily recording of the teletext of the news, discussion & sport programs ( ) WWW –Daily downloading of news providers & papers ( )

Text collection YearNum of wordsNum unique words Spellings reform Number of words of the newspaper collection after normalisation

Phonetic transcriptions Phonetic dictionaries –Celex (300k, SAMPA) –VLIS database (1300k, Van Dale Data Format) –Rule-based decompounded-compounded dictionary (600k, SAMPA) G2P tool –Machine learning algorithm (vd Bosch) –95% correct (without syllable & stress information)

Text normalisation I Cleaning of punctuation marks Expansion –Numbers, abbreviations Statistical capital letter reduction –Rotterdam, rotterdam, ROTTERDAM  Rotterdam –KOK, Kok, kok  kok Spelling correction –Reduction of “doubles” caused by the spelling reform of the nineties (pannekoek  pannenkoek) –Removal, correction, or adding of accentuation marks cafe, café, cafeé, cafë etc.  café hét, hèt  het

Text normalisation II German and Dutch are “compound” languages Increased number of words Relative high number of “new” words –(Eclipsbril = Eclipse glasses) Lowe lexical coverage  High OOV –LC = #word/(#distinct words) –OOV = 1- LC

Text normalisation III drugbeleid drugbestrijding drugbezit drugdealer drugdealers drugdeals drugdelict drugdistributeur druggebruik druggebruiker druggebruikers drughandel drugkartels drugmisbruik drugrunner drugsaanpak drugsactie drugsacties drugsactiviteiten drugsadviseur drugsafdeling drugsaffaire drugsaffaires drugsafrekeningen drugsattributen drugsavonturen drugsavontuur drugsbaas drugsbanden drugsbaron drugsbaronnen drugsbazen drugsbedrijf drugsbeleid drugsbende drugsbendes drugsbestaan drugsbestellingen drugsbestrijdend drugsbestrijder drugsbestrijders drugsbestrijding drugsbezit drugsbezitters drugsboef drugsboeven drugsbonzen drugsbrigade drugsbrigades drugsbron drugsbuisje drugsbureau drugsbusiness drugsbuurt drugscafé drugscafés drugscampagnes drugscare drugscircuit drugsclans drugsclip drugscocktail drugscocktails drugsconferentie drugsconflict drugsconnecties drugsconsument drugsconsumptie drugscontainers drugscontrole drugscontroles drugsconventie drugscriminaliteit drugscrimineel drugscriminelen drugsdaglicht drugsdeal drugsdealen drugsdealend drugsdealende drugsdealer drugsdealers drugsdeals drugsdebat drugsdelict drugsdelicten drugsdeskundige drugsdiscussie drugsdode drugsdoden drugsdollars drugsdominee drugsdood drugsdossier drugsdossiers drugsdraaiboek drugseconomie drugseenheid drugsellende drugsexcessen drugsexperiment drugsexpert drugsexperts drugsexport drugsfabricage drugsfabrikanten drugsfamilie drugsfunctionaris drugsgebied drugsgebruik drugsgebruiker drugsgebruikers drugsgebruikster drugsgeld drugsgelden drugsgelieerde drugsgeschiedenis drugsgeschillen drugsgewoonte drugsgoeroe drugsgroeperingen drugsgrondstoffen drugshaarden drugshandel drugshandelaar drugshandelaars drugshandelaarster drugshandelaren drugshandlangers drugshel drugshoertje drugshol drugshond drugshonden drugshoofdstad drugshuizen drugshulpverleners drugshulpverlening drugsimago drugsimport drugsindustrie drugsinkomsten drugsinstelling drugsinval drugsinvoer drugsjacht drugsjagende drugsjager drugsjaren drugsjongens drugskartel drugskartels

Text normalisation VI Decompounding –Low frequency compounds are decompounded if decompounding improves the Lexical Coverage –50% of the unique words that were not in one of the phonetic dictionaries could be successfully decompounded although some error were made: zeeroverschatten  zeerover + schatten  zeerovers + chatten

Most / least frequent words TOP 10 de van het en een in dat is op te DOWN 10 milko39 miljardenovername39 mifune's39 middeninkomen39 michelingids39 mexx39 metaalnijverheid39 metaaldetectoren39 mesquita39 mervyn39

Language modelling Language corpus UK WSJ IT Sole24 FR Le monde NL PDB D FR #words 37M27M38M22M36M #distinct words 165K200K280K320K650K 20K coverage 97.5%96.3%94.7%93.0%90.0% 65K coverage 99.6%99.0%98.3%97.5%95.1%

Language modelling data# words# unique words ratio Original ,04 After decompounding ,22 change+ 2.1%-32.6%+51.6% Effect on the ratio after decompounding

Different language models First use the general LM to detect the sub-category Use the politic LM to improve recognition results

Segmentation I Full news broadcasts are too long (20 min.) Retrieved items may start and/or stop in the middle of phrases different LM has to be assigned to different “stories”

Segmentation II Segmentation in phrases, sentences, and paragraphs –Prosodic information F0 Pauses –Different LM assigning

Results descriptionOOVWER Basic, 44K words5.07%68.5% +forward/backward training5.07%62.4% + newspaper corpus5.07%53.5% + newspaper corpus + FB training 5.07%50.2% + 65K words3.54%46.3%

Results WERextra Read speech30% (OOV = 2.5%)15 hrs training material Broadcast news36.9% (OOV = 14%)5 hrs training material Historical archives90% (OOV = 20%)1933 Historical archives60% (OOV = 10%)1940 Historical archives43% (OOV = 14%)1960

DRUID “de Israëlische premier Chevron houdt vanavond en televisie toespraak zullen ingaan op de crisis die is ontstaan na de bloedige aanslagen van het weekend in Jeruzalem en hij vaak zo'n kwam vanochtend vroeg terug uit Amerika heeft gesproken met president Bush het ene op het vliegveld van Tel Aviv pasje om met ministers pers en ben een Jezus met weinig gevoel voor huizen vanavond is het kabinet kabinet beraadt geweld gaat ook vanochtend door op de westelijke Jordaan oever bijen is z'n vijven dertig jarige Palestijn door Israëlische militairen gedood die bij controle proberen te vluchten of stonden Shiva heeft pech” “de Israëlische premier Sharon houdt vanavond ‘n televisie toespraak. Hij zal dan ingaan op de crisis die is ontstaan na de bloedige aanslagen van dit weekend in Jeruzalem en Haifa. Sharon kwam vanochtend vervroegd terug uit Amerika; daar heeft hij gesproken met president Bush. Meteen al op het vliegveld van TelAviv sprak Sharon met de ministers Peres en Ben Illiëzer en met veiligheidsfunctionarissen. Vanavond is het kabinet kabinetsberaadt. ‘t geweld gaat ook vanochtend door, op de westelijke Jordaanoever bij Jinien is 'n vijfendertig jarige Palestijn door Israëlische militairen gedood toen ie bij controle probeerden te vluchten. Correspondent: Shivra Hertzberg” December :142 3 December :14

OOV problems  20% (  14k) of the 65k most frequent words (MFW) are not in the phonetic dictionary  86% of these 14k words starts with a capital letter 50% of these 14k words are names (family, geographic, companies) that are not in the phonetic dictionary and are difficult to transcribe by G2P because they often do not follow Dutch transcription rules

Demo 8 o’clock TV news Daily radio news Adjust

DRUID Evaluation –A time consuming, boring, but necessary process!!

Questions ?