MongoDB scalability for astronomical time series: The POEMAS solar radio telescope evaluation without HPC
MongoDB 在天文时间序列中的可扩展性:POEMAS 太阳射电望远镜在无 HPC 条件下的评估
The increasing temporal resolution and structural diversity of modern solar instruments place growing demands on database systems used in observational astronomy. At the Center for Radio Astronomy and Astrophysics Mackenzie (CRAAM), this challenge is amplified by the need to consolidate heterogeneous data streams from multiple telescopes within a single virtual machine. With only 32GB of RAM available (16GB allocated to the database), a central design question emerged: when restricted to a single physical host, can a virtualized sharded cluster offer practical scalability advantages over a standalone deployment? To investigate this, we conducted an empirical evaluation of MongoDB using 10ms observations from the POEMAS radiotelescope, tested at volumes of 15M, 150M, and 500M documents. Results show that, although sharding introduces coordination overhead for selective queries, it provides substantial gains for global aggregations, achieving speedups above 600<mml:math><mml:mo>×</mml:mo></mml:math> while maintaining compression ratios near 85%. The analysis identifies an operational threshold of roughly 150 million documents per collection to sustain stable performance under the available resources. Based on these findings, the same single-node configuration used in the benchmarks was employed to process the full historical POEMAS dataset, totaling 3.3 billion records and producing approximately 50GB of consolidated FITS products. These products and their associated metadata are made available to the community through a cloud-hosted portal with reduced operational cost. This work documents practical scalability boundaries for astronomical time-series in resource-constrained environments and supports the deployment currently operating at CRAAM.
展开 ▾在仅 32GB 内存、16GB 分配给数据库的单机虚拟化条件下,证明分片 MongoDB 比独立部署更适合全局聚合查询,加速超过 600 倍且压缩率约 85%;识别出约 1.5 亿文档/集合的稳定运行阈值,并实际完成 33 亿条 POEMAS 历史记录的处理与产品开放。
以往天文时间序列数据系统研究多侧重 HPC 或大规模分布式集群,本文承接这一方向,转向在单机虚拟化、资源严重受限的条件下检验 NoSQL 分片方案的实用边界。实验表明,MongoDB 分片虽然给选择性查询带来协调开销,但在全局聚合类任务中能获得数量级提升,并保持高压缩率,这为中小型天文台的低成本数据基础设施提供了参考。随着太阳观测时间分辨率与数据量继续增长,此类轻量级分片部署有望进一步结合智能分片键、查询优化与云存储,在不依赖 HPC 的前提下扩展至更多科学数据产品。
刊出 2025-12-27 · 收录 2026-08-20