Feature Extraction for Token Based Word Alignment for Question Answering Systems

Resumen

Mapping between the source words and the target words in a set of parallel sentences are acrucial part of Question Answering (QA) systems. If an accurate aligner is used in QA systems then efficiency of these systems also gets increased. We purpose the aligner which despite using very less lexical resources gives very good results in terms of precision, recall and F1. Previous aligners either uses more lexical resources or uses very less lexical resources. Hence, we have used POS TAG and WordNet as lexical resources. But some words whose meaning we may not know but these occur in a similar distribution and by observing their distribution these words are similar. Consider two sentences ”Lambodar is the son of Parvati” and ”Ganesha is the son of Parvati”. Here we will not find the meaning of Lambodar and Ganesha in Wordnet but since they have similar distributions so they should be aligned. For these words, we used Distribution Similarity Feature in our word aligner. This distributional similarity helps our aligner in broader coverage of words. Previous aligners were having recall in the range of 75-86 but this aligner has recall in the range of 88.4-93.3. Similarly, Exact match of previous aligners was in the range of21-35.3 but the proposed aligner’s exact match range is 46.1-58.6. Similarly F-measure and precision have also increased.