實體與虛擬機器之Hadoop分散計算模型之效能分析與評估

Hadoop分散式運算架構為資訊系統帶來分析大型模型數據組的解決方案，而虛擬化技術的發展，將運算節點以VM型式建構也是業界尋求系統整合的方向之一，儘管可達到彈性分配網路效能與硬體資源，但虛擬機器卻難以比擬實體主機的運算效能，也成為Hadoop叢集虛擬化的一大瓶頸，管理者難以評估雲端平臺虛擬化後的效能衝擊。為此，本研究以Hadoop為主體建構分散運算叢集，系統將串連異質伺服器運算資源，採用HBase為主要的資料庫儲存系統，同時運用Hive、Sqoop等套件實現異質環境資料交換能力，將以該架構為基礎運作於虛擬／實體Hadoop平台。為了能有效評估叢集運算的效能表現，本研究基於現有效能模型進行整合，並以此計算模擬效能值，與實測數據進行分析比較，驗證該模型之可用性。結果顯示，修正後的效能模型適用於本研究架構，接著考量虛擬層對Hadoop的影響要素，提出適用於虛擬化Hadoop運算叢集的效能模型，最終驗證兩者在運行上的效能差異，提出改善虛擬化技術運用於雲端系統平臺的方案，確保Hadoop在虛擬化環境下運作能比擬原有效能。

The Hadoop distributed computing architecture provides solutions of Big Data processing for information systems, and development of virtualization technologies, constructing computing nodes based on virtual machine is the goal of industry for system integration. Although it reached high flexible deployment of network performance and hardware resources, the virtualized Hadoop clusters have become a major bottleneck because virtual machine is difficult to acquire high computing performance just like physical host. The administrator is also difficult to assess the impact of performance about virtualization of cloud platform. In this study, to deploy distributed computing clusters based on Hadoop scheme, integrate heterogeneous server computing resources, and using HBase as the main database storage system. We also adopt Hive, Sqoop and other kit to achieve heterogeneous environments data exchange. The Hadoop clusters will apply to physical host and virtual machine clusters. In order to efficiently assess the performance of cluster computing, this research integrate current performance model, and compared execution performance value with predicted performance value to verify the availability of this model. All results show that modified model is applicable to this system architecture. Then, we consider the impact factors of virtualization, putting performance model into virtualized Hadoop cluster, and verify differences in performance between the physical host and virtual host. We propose the improved solution for virtualization technology for cloud platform and make Hadoop in a virtual host can get the same performance just like in a physical host.