|
|
 |
| |
|
|
|
Real-Time Big Data Processing Framework: Challenges and Solutions |
|
|
|
PP: 3169-3190 |
|
|
|
Author(s) |
|
|
|
Zhigao Zheng,
Ping Wang,
Jing Liu,
Shengli Sun,
|
|
|
|
Abstract |
|
|
| Data type and amount in human society is growing in amazing speed which is caused by emerging new services as cloud
computing, internet of things and location-based services, the era of big data has arrived. As data has been fundamental resource, how to
manage and utilize big data better has attracted much attention. Especially, with the development of internet of things, how to processing
large amount real-time data has become a great challenge in research and applications. Recently, cloud computing technology has
attracted much attention with high-performance, but how to use cloud computing technology for large-scale real-time data processing
has not been studied. This paper studied the challenges of big data firstly and concludes all these challenges into six issues. In order
to improve the performance of real-time processing of large data, this paper builds a kind of real-time big data processing (RTDP)
architecture based on the cloud computing technology and then proposed the four layers of the architecture, and hierarchical computing
model. This paper proposed a multi-level storage model and the LMA-based application deployment method to meet the real-time and
heterogeneity requirements of RTDP system. We use DSMS, CEP, batch-based MapReduce and other processing mode and FPGA,
GPU, CPU, ASIC technologies differently to processing the data at the terminal of data collection. We structured the data and then
upload to the cloud server and MapReduce the data combined with the powerful computing capabilities cloud architecture. This paper
points out the general framework for future RTDP system and calculation methods, is currently the general method RTDP system
design. |
|
|
|
|
|
|
|
