Application of the LDA Model for Obtaining Topics from the WIKICORPUS

Authors

  • Gerardo Martínez Guzmán Benemérita Universidad Autónoma de Puebla
  • María Beatríz Bernábe Loranca Benemérita Universidad Autónoma de Puebla
  • Carmen Cerón Garnica Benemérita Universidad Autónoma de Puebla
  • Jonathan Serrano Pérez Benemérita Universidad Autónoma de Puebla
  • Etelvina Archundia Sierra Benemérita Universidad Autónoma de Puebla

DOI:

https://doi.org/10.13053/cys-26-1-4171

Keywords:

Corpus, generative model, Dirichlet distribution, latent topics, parallelization, algorithm, C/C programming

Abstract

A fundamental problem in text analysis of great amount of information is to discover the topics described in the documents. One of the most useful application involves the extraction of topics from documents corpus. Such is the case of Wikicorpus that consists of approximately 250,000 documents totaling in 250 millions of words. In this work, a system based on the Latent Dirichlet Allocation (LDA) model has been developed to carry out the task of automatically selecting the words of the corpus and, based on their frequency in the documents, it would indicate that they may or not belong to certain topic, classifying words without human intervention. Due to the large amount of information of the corpus, a Serial-Parallel Algorithm (SPA) in C/C++ and OpenMP have been used to perform parallel programming, since in parallel stages all threads must share certain variables, so the design architecture was shared memory

Downloads

Published

2022-03-26

Issue

Vol. 26 No. 1 (2022): Recent Advances in Language & Knowledge Engineering (Guest editors: D. Pinto, B. Beltrán, A. Vázquez, D. Vilariño)

Section

Articles of the Thematic Issue

License 

Hereby I transfer exclusively to the Journal "Computación y 
Sistemas", published by the Computing Research Center (CIC-IPN),
the Copyright of the aforementioned paper. I also accept that these 
rights will not be transferred to any other publication, in any other 
format, language or other existing means of developing.
I certify that the paper has not been previously disclosed or simultaneo
usly submitted to any other publication, and that it does not contain 
material whose publication would violate the Copyright or other 
proprietary rights of any person, company or institution. I certify that 
I have the permission from the institution or company where I work or 
study to publish this work.
The representative author accepts the responsibility for the publication
of this paper on behalf of each and every one of the authors. 
This transfer is subject to the following conditions:
  • The authors retain all ownership rights (such as patent rights) of this work, except for the publishing rights transferred to the CIC, through this document.
  • Authors retain the right to publish the work in whole or in part in any book they are the authors or publishers. They can also make use of this work in conferences, courses, personal web pages, and so on.
  • Authors may include working as part of his thesis, for non-profit distribution only.