|
|
论文:2024,Vol:42,Issue(2):303-309 |
|
|
引用本文： |
|
|
杜轶德, 刘文洁. 基于LSM-Tree的分布式数据库异步融合机制研究与实现[J]. 西北工业大学学报 |
|
|
DU Yide, LIU Wenjie. Research and implementation of asynchronous compaction mechanism of distributed database based on LSM-Tree[J]. Journal of Northwestern Polytechnical University |
|
|
|
|
|
|
|
| 基于LSM-Tree的分布式数据库异步融合机制研究与实现 |
|
| 杜轶德, 刘文洁 |
|
| 西北工业大学 计算机学院, 陕西 西安 710072 |
| 摘要: |
| 信息技术的不断发展,使得分布式数据库成为研究热点。由于NoSQL架构的分布式数据库对SQL支持有限且在事务处理及一致性方面存在缺陷,基于LSM-Tree的NewSQL数据库逐渐成为应用的主流,例如TiDB、OceanBase等。分布式LSM-Tree的存储架构将数据分为基线数据与增量数据,通过合并操作将不同分区的增量数据与基线数据不断融合,并存储在磁盘,从而减少内存压力。但合并会占用大量系统资源,严重影响系统可用性。因此提出了一种基于LSM-Tree架构的异步融合机制,通过细分合并流程,将数据融合异步化,有效地缩短了单次数据合并的时间。实验表明,提出的异步融合机制可显著缩短数据合并时间,提高系统在高频写入场景下的鲁棒性和可用性。 |
| 关键词:
分布式数据库
LSM-Tree
数据合并
异步融合
数据分区
|
|
| Research and implementation of asynchronous compaction mechanism of distributed database based on LSM-Tree |
|
| DU Yide, LIU Wenjie |
|
| School of Computer Science, Northwestern Polytechnical University, Xi'an 710072, China |
| Abstract: |
| With the continuous development of information technology, distributed database has become a research hotspot. Due to the limit support for SQL and defects in transaction processing and consistency of distributed databases based on NoSQL architecture, NewSQL databases based on LSM-Tree become gradually the mainstream of applications, such as TiDB and OceanBase. The distributed LSM-Tree storage architecture divides the data into baseline data and incremental data. Through the compaction operation, the incremental data of different partitions and the baseline data are continuously merged and stored on the disk, thereby reducing memory pressure. However, compaction will occupy a large amount of system resources and seriously affect system availability. This paper proposes an asynchronous compaction mechanism based on LSM-Tree architecture. By subdividing the compaction process, the data merging is asynchronous, which effectively shortens the time for a single compaction operation. Experiments show that the asynchronous compaction mechanism proposed in this paper can significantly shorten the data merging time and improve the robustness and usability of the system in high-frequency writing scenarios. |
| Key words:
distributed database
LSM-Tree
data merging
asynchronous compaction
data partitioning
|
|
| 收稿日期: 2023-04-06
修回日期:
|
| DOI: 10.1051/jnwpu/20244220303 |
| 基金项目: 国家自然科学基金重点项目(61732014)与华为合作研究项目(D5204220342)资助 |
| 通讯作者: 刘文洁(1976—),副教授 e-mail:liuwenjie@nwpu.edu.cn
Email：liuwenjie@nwpu.edu.cn |
| 作者简介: 杜轶德(1999—),硕士研究生
|
|
|
|
|
|
|
|
参考文献: |
|
|
[1] LU W, ZHAO Z H, WANG X Y, et al. A lightweight and efficient temporal database management system in TDSQL[J]. Proceedings of the VLDB Endowment, 2019, 12(12): 2035-2046
[2] O' NEIL P, CHENG E, GAWLICK D, et al. The log-structured merge-tree(LSM-tree)[J]. Acta Informatica, 1996, 33: 351-385
[3] HUANG D, LIU Q, CUI Q, et al. TiDB: a raft-based HTAP database[J]. Proceedings of the VLDB Endowment, 2020, 13(12): 3072-3084
[4] YANG Z, YANG C, HAN F, et al. OceanBase: a 707 million tpmC distributed relational database system[J]. Proceedings of the VLDB Endowment, 2022, 15(12): 3385-3397
[5] LUO C, CAREY M J. LSM-based storage techniques: a survey[J]. The International Journal on Very Large Data Bases, 2020, 29(1): 393-418
[6] YAN B, CHENG X, JIANG B, et al. Revisiting the design of LSM-tree based OLTP storage engine with persistent memory[J]. Proceedings of the VLDB Endowment, 2021, 14(10): 1872-1885
[7] SARKAR S, STARATZIS D, ZHU Z, et al. Constructing and analyzing the LSM compaction design space[J]. Proceedings of the VLDB Endowment, 2021, 14(11): 2216-2229
[8] BINDSCHAEDLER L, GOEL A, ZWAENEPOEL W. Hailstorm: disaggregated compute and storage for distributed LSM-based databases[C]//Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems, 2020: 301-316
[9] HUANG H, GHANDEHARIZADEH S. Nova-LSM: a distributed, component-based LSM-tree key-value store[C]//Proceedings of the 2021 International Conference on Management of Data, 2021
[10] DAYAN N, WEISS T, DASHEVSKY S, et al. Spooky: granulating LSM-tree compactions correctly[J]. Proceedings of the VLDB Endowment, 2022, 15(11): 3071-3084
[11] 张晨煜, 刘文洁, 庞天泽, 等. 基于分布式数据库的相关子查询优化[J]. 西北工业大学学报, 2021, 39(4): 909-918 ZHANG Chenyu, LIU Wenjie, PANG Tianze, et al. Optimization of correlate subquery based on distributed database[J]. Journal of Northwestern Polytechnical University, 2021, 39(4): 909-918 (in Chinese)
[12] 景苌弘, 刘文洁, 高锦涛, 等. 面向分布式数据库的HTAP研究与实现[J]. 西北工业大学学报, 2021, 39(2): 430-438 JING Changhong, LIU Wenjie, GAO Jintao, et al. Research and implementation of HTAP for distributed database[J]. Journal of Northwestern Polytechnical University, 2021, 39(2): 430-438 (in Chinese)
[13] 刘文洁, 李戬勃, 李战怀, 等. 一种面向金融应用的海量分布式关系数据库[J]. 华中科技大学学报, 2019, 47(2): 121-126 LIU Wenjie, LI Jianbo, LI Zhanhuai, et al. A massire distribnted relational database for financial application[J]. Journal of Huazhong University of Science and Technology, 2019, 47(2): 121-126 (in Chinese)
[14] DAYAN N, IDREOS S. Dostoevsky: better space-time trade-offs for LSM-tree based key-value stores via adaptive removal of superfluous merging[C]//Proceedings of the 2018 International Conference on Management of Data, 2018
[15] RAJU P, KADEKODI R, CHIDAMBARAM V, et al. Pebblesdb: building key-value stores using fragmented log-structured merge trees[C]//Proceedings of the 26th Symposium on Operating Systems Principles, 2017 |
|
|
|
|
|
|
|
