LaSA: A Locality-aware Scheduling Algorithm for Hadoop-MapReduce Resource Assignment
學年 101
學期 2
發表日期 2013-05-20
作品名稱 LaSA: A Locality-aware Scheduling Algorithm for Hadoop-MapReduce Resource Assignment
作品名稱（其他語言）
著者 Chen, Tseng-Yi; Wei, Hsin-Wen; Wei, Ming-Feng; Chen, Ying-Jie; Hsu, Tsan-sheng; Shih, Wei-Kuan
作品所屬單位 淡江大學資訊管理學系
出版者
會議名稱 2013 International Conference on Collaboration Technologies and Systems (CTS 2013)
會議地點 San Diego, California, USA
摘要 Cloud computing has become more popular for a decade; it has been under continuous development with advances in architecture, software, and network. Hadoop-MapReduce is a common software framework processing parallelizable problem across big datasets using a distributed cluster of processors or stand-alone computers. Cloud Hadoop-MapReduce can scale incrementally in the number of processing nodes. Hence, the Hadoop-MapReduce is designed to provide a processing platform with powerful computation. Network traffic is always a most important bottleneck in data-intensive computing and network latency decreases significant performance in data parallel systems. Network bottleneck is caused by network bandwidth and the network speed is much slower than disk data access. So that, good data locality can reduces network traffic and increases performance in data-intensive HPC systems. However, Hadoop's scheduler has a defect of data locality in resource assignment. In this paper, we present a locality-aware scheduling algorithm (LaSA) for Hadoop-MapReduce scheduler. Firstly, we propose a mathematical model of weight of data interference in Hadoop scheduler. Secondly, we present the LaSA algorithm to use weight of data interference to provide data locality-aware resource assignment in Hadoop scheduler. Finally, we build an experimental environment with 3 cluster and 35 VMs to verify the LaSA's performance.
關鍵字
語言 en
收錄於
會議性質 國際
校內研討會地點
研討會時間 20130520~20130524
通訊作者
國別 USA
公開徵稿 Y
出版型式
出處 2013 International Conference on Collaboration Technologies and Systems (CTS), pp.342-346
相關連結

機構典藏連結 ( http://tkuir.lib.tku.edu.tw:8080/dspace/handle/987654321/92999 )

機構典藏連結