|
|
论文:2024,Vol:42,Issue(3):453-459 |
|
|
引用本文： |
|
|
刘文洁, 吕靖超. 面向分布式数据库的算子并行优化策略[J]. 西北工业大学学报 |
|
|
LIU Wenjie, LYU Jingchao. Operator parallel optimization strategy for distributed databases[J]. Journal of Northwestern Polytechnical University |
|
|
|
|
|
|
|
| 面向分布式数据库的算子并行优化策略 |
|
| 刘文洁, 吕靖超 |
|
| 西北工业大学 计算机学院, 陕西 西安 710072 |
| 摘要: |
| 随着网络技术的不断发展,数据规模呈现爆发式增长,使得传统的单机数据库逐步被分布式数据库所取代。分布式数据库采用节点协同工作方式解决了大规模数据存储问题,但由于增加了节点间通信开销,查询效率却不如单机数据库。分布式架构下,存储节点的数据仅用作多备份的冗余,为系统故障时提供数据恢复,并未被利用起来改善查询效率。针对上述问题,提出了一种面向分布式数据库的算子并行优化策略,通过对关键物理算子进行拆分,将拆分后的子请求均匀分配到存储层多个节点,由多个节点并行处理,从而减少查询响应时间。上述策略已经在分布式数据库CBase上进行了应用,实验表明,提出的并行优化策略可显著缩短SQL请求查询时间,并提高系统资源利用率。 |
| 关键词:
分布式数据库
并行查询
查询优化
负载均衡
数据分区
|
|
| Operator parallel optimization strategy for distributed databases |
|
| LIU Wenjie, LYU Jingchao |
|
| School of Computer Science, Northwestern Polytechnical University, Xi'an 710072, China |
| Abstract: |
| With the continuous development of network technology, the scale of data has shown explosive growth, which leads gradually to replacing traditional single machine databases with distributed databases. Distributed databases solve large-scale data storage problems through collaborative work among nodes, but due to increased communication costs between nodes, its query efficiency is not as good as a standalone database. In a distributed architecture, the data of storage nodes is only used as redundancy for multiple backups, providing data recovery in case of system failure, and it is not utilized to improve query efficiency. In response to the above issues, this article proposes an operator parallel optimization strategy for distributed databases. By splitting key physical operators, the split sub requests are evenly distributed to multiple nodes in the storage layer, which are processed in parallel by multiple nodes, thereby reducing query response time. The above strategy has been applied on distributed database CBase, and experiments have shown that the parallel optimization strategy proposed in this paper can significantly shorten SQL request query time and improve system resource utilization. |
| Key words:
distributed database
parallel query
query optimization
load balancing
data partitioning
|
|
| 收稿日期: 2023-05-30
修回日期:
|
| DOI: 10.1051/jnwpu/20244230453 |
| 基金项目: 国家自然科学基金(61732014)与华为合作研究项目(D5204220342)资助 |
| 通讯作者: 刘文洁(1976—) e-mail:liuwenjie@nwpu.edu.cn
Email：liuwenjie@nwpu.edu.cn |
| 作者简介: 刘文洁(1976—),副教授
|
|
|
|
|
|
|
|
参考文献: |
|
|
[1] LU W, ZHAO Z, WANG X, et al. A lightweight and efficient temporal database management system in TDSQL[J]. Proceedings of the VLDB Endowment, 2018, 11(12): 2035-2046
[2] HUANG D, LIU Q, CUI Q, et al. TiDB: a Raft-based HTAP database[J]. Proceedings of the VLDB Endowment 2020, 13(12): 3072-3084
[3] CHARAPKO A, AILIJIANG A, DEMIRBAS M. PigPaxos: devouring the communication bottlenecks in distributed consensus[C]//Proceedings of the ACM SIGMOD International Conference on Management of Data, 2021: 235-247
[4] FUNKE H, TEUBNER J. Data-parallel query processing on non-uniform data[J]. Proceedings of the VLDB Endowment, 2020, 13(6): 884-897
[5] ROMANOUS B, WINDH S, ABSALYAMOV I, et al. Efficient local locking for massively multithreaded in-memory hash-based operators[J]. VLDB Journal, 2021, 30(3): 333-359
[6] FENTPHILIPP, NEUMANNTHOMAS. A practical approach to groupjoin and nested aggregates[J]. Proceedings of the VLDB Endowment, 2021, 14(11): 2383-2394
[7] FEGARAS L, NOOR H. Translation of array-based loops to distributed data-parallel programs[J]. Proceedings of the VLDB Endowment, 2020, 13(8): 1248-1260
[8] CONRAD A. Database of the year: postgres[J]. IEEE Software, 2021, 38(5): 130-132
[9] YANG Z, YANG C, HAN F, et al. OceanBase: a 707 million tpmC distributed relational database system[J]. Proceedings of the VLDB Endowment, 2022, 15(12): 3385-3397
[10] 张晨煜, 刘文洁, 庞天泽, 等. 基于分布式数据库的相关子查询优化[J]. 西北工业大学学报, 2021, 39(4): 909-919 ZHANG Chenyu, LIU Wenjie, PANG Tianze, et al. Optimization of correlate subquery based on distributed database[J]. Journal of Northwestern Polytechnical University, 2021, 39(4): 909-919(in Chinese)
[11] 景苌弘, 刘文洁, 高锦涛, 等. 面向分布式数据库的HTAP研究与实现[J]. 西北工业大学学报, 2021, 39(2): 430-438 JING Changhong, LIU Wenjie, GAO Jintao, et al. Research and implementation of HTAP for distributed database[J]. Journal of Northwestern Polytechnical University, 2021, 39(2): 430-438(in Chinese)
[12] 刘文洁, 李戬勃, 李战怀, 等. 一种面向金融应用的海量分布式关系数据库[J]. 华中科技大学学报, 2019, 47(2): 121-126 LIU Wenjie, LI Jianbo, LI Zhanhuai, et al. A massire distribnted relational database for financial application[J]. Journal of Huazhong University of Science and Technology, 2019, 47(2): 121-126(in Chinese) |
|
|
|
相关文献: |
|
|
1．杜轶德, 刘文洁.基于LSM-Tree的分布式数据库异步融合机制研究与实现[J]. 西北工业大学学报, 2024,42(2): 303-309 |
|
|
2．张晨煜, 刘文洁, 庞天泽, 岳艳涛.基于分布式数据库的相关子查询优化[J]. 西北工业大学学报, 2021,39(4): 909-918 |
|
|
|
|
|
|
