Data Triggered Programming Model for Text Processing in Big Data

Abstract

Large volume of text processing becomes a challenge in recent era. Text processing methods drive much of modern data analysis across engineering sciences and commercial applications. Extraction of useful information from text sources refers to text analytics. This term describes tasks from annotating text sources with meta-information such as places mentioned in the text and a wide range of documents. The key/value pair generation of MapReduce program creates memory overhead and deserialization overhead due to data redundancy. Redundancy of data is one of the most important factors that consumes space and affect system performance while using large set of data. This overhead can be avoided considerably by using a novel approach that we developed named Data Triggered Multithreaded Programming (DTMP) model. In this paper, we demonstrate the use of DTMP model using a large dataset with author details and his publications. The Data Triggered Multithreaded Programming can dynamically allocate the resources and can identify the data repetition occurring during computation. DTMP model when applied to the MapReduce programming model brings performance improvement to the system. The major contributions of this work are a simple and scalable processing of text data that enables automatic parallelization and distribution of large-scale computations.