A Deep Attention based Framework for Image Caption Generation in Hindi Language

Authors

  • Rijul Dhir Indian Institute of Technology Patna
  • Santosh Kumar Mishra Indian Institute of Technology Patna
  • Sriparna Saha Indian Institute of Technology Patna
  • Pushpak Bhattacharyya Indian Institute of Technology Patna

DOI:

https://doi.org/10.13053/cys-23-3-3269

Keywords:

Image captioning, hindi language, convolutional neural network, recurrent neural network, gated recurrent unit, attention mechanism

Abstract

Image captioning refers to the process of generating a textual description for an image which defines the object and activity within the image. It is an intersection of computer vision and natural language processing where computer vision is used to understand the content of an image and language modelling from natural language processing is used to convert an image into words in the right order. A large number of works exist for generating image captioning in English language, but no work exists for generating image captioning in Hindi language. Hindiis the official language of India, and it is the fourth most-spoken language in the world, after Mandarin, Spanish and English. The current paper attempts to bridge this gap. Here an attention-based novel architecture for generating image captioning in Hindi language is proposed. Convolution neural network isused as an encoder to extract features from an input image and gated recurrent unit based neural network isused as a decoder to perform language modelling up to the word level. In between, we have used the attention mechanism which helps the decoder to look into the important portions of the image. In order to show theefficacy of the proposed model, we have first created a manually annotated image captioning training corpusin Hindi corresponding to popular MS COCO English dataset having around 80000 images. Experimental results show that our proposed model attains a BLEU1score of 0.5706 on this data set.

Downloads

Published

2019-09-25

Issue

Vol. 23 No. 3 (2019): Thematic issue: Computational Linguistics (Guest Editor A. Gelbukh)

Section

Articles of the Thematic Issue

License 

Hereby I transfer exclusively to the Journal "Computación y 
Sistemas", published by the Computing Research Center (CIC-IPN),
the Copyright of the aforementioned paper. I also accept that these 
rights will not be transferred to any other publication, in any other 
format, language or other existing means of developing.
I certify that the paper has not been previously disclosed or simultaneo
usly submitted to any other publication, and that it does not contain 
material whose publication would violate the Copyright or other 
proprietary rights of any person, company or institution. I certify that 
I have the permission from the institution or company where I work or 
study to publish this work.
The representative author accepts the responsibility for the publication
of this paper on behalf of each and every one of the authors. 
This transfer is subject to the following conditions:
  • The authors retain all ownership rights (such as patent rights) of this work, except for the publishing rights transferred to the CIC, through this document.
  • Authors retain the right to publish the work in whole or in part in any book they are the authors or publishers. They can also make use of this work in conferences, courses, personal web pages, and so on.
  • Authors may include working as part of his thesis, for non-profit distribution only.