Cross-Domain Failures of Fake News Detection

Authors

  • Maria Janicka Samsung R&D Institute Poland
  • Maria Pszona Samsung R&D Institute Poland
  • Aleksander Wawer Samsung R&D Institute Poland

DOI:

https://doi.org/10.13053/cys-23-3-3281

Keywords:

Cross domain, failures, news detection

Abstract

Fake news recognition has become aprominent research topic in natural language processing. Researchers reported significant successes when applying methods based on various stylometric and lexical features and machine learning, with accuracy reaching 90%. This article is focused on answering the question: are the fake news detection models universally applicable or limited to the domain they have been trained on? We used four different, freely available English language Fake News corpora and trained models in both in-domain and cross-domain setting. We also explored and compared features important in eachdomain. We found that the performance in cross-domain setting degrades by 20% and sets of features importantto detect fake texts differ between domains. Our conclusions support the hypothesis that high accuracy of machine learning models applied to fake news detectionmay be related to over-fitting, and models need to betrained and evaluated on mixed types of texts.

Downloads

Published

2019-09-30

Issue

Vol. 23 No. 3 (2019): Thematic issue: Computational Linguistics (Guest Editor A. Gelbukh)

Section

Articles of the Thematic Issue

License 

Hereby I transfer exclusively to the Journal "Computación y 
Sistemas", published by the Computing Research Center (CIC-IPN),
the Copyright of the aforementioned paper. I also accept that these 
rights will not be transferred to any other publication, in any other 
format, language or other existing means of developing.
I certify that the paper has not been previously disclosed or simultaneo
usly submitted to any other publication, and that it does not contain 
material whose publication would violate the Copyright or other 
proprietary rights of any person, company or institution. I certify that 
I have the permission from the institution or company where I work or 
study to publish this work.
The representative author accepts the responsibility for the publication
of this paper on behalf of each and every one of the authors. 
This transfer is subject to the following conditions:
  • The authors retain all ownership rights (such as patent rights) of this work, except for the publishing rights transferred to the CIC, through this document.
  • Authors retain the right to publish the work in whole or in part in any book they are the authors or publishers. They can also make use of this work in conferences, courses, personal web pages, and so on.
  • Authors may include working as part of his thesis, for non-profit distribution only.