AstroFlow: A customizable workflow management system for astronomical data production and case study of EP-WXT
AstroFlow:天文数据生产可定制工作流管理系统及 EP-WXT 案例研究
In the era of time-domain astronomy, automated and efficient data processing is a crucial factor in ensuring astronomical scientific output. With the international collaboration trend in large-scale sky survey projects, data processing algorithms developed through multi-institutional collaboration face challenges in system integration, scheduling capability, extensibility, and traceability. In this work, we employ containerized execution modules to resolve dependency issues in multi-institutional software integration, implement complex workflow orchestration through message queue control modules, including cyclic dependencies, dynamic parallel relationships, and resource/priority-based scheduling. The system also non-intrusively implements the International Virtual Observatory Alliance (IVOA) provenance data model through aspect injection mechanisms, ensuring traceability and reproducibility of data products. In Einstein Probe (EP) data processing, the system has processed over 20,000 general observations, 170,000 FXT observations, and 21,000 VHF and Beidou alerts, supported over 300 agile algorithm updates, and detected more than 17,000 sources and nearly 200 transient events. The system demonstrates excellent generality and portability, smoothly migrating from the single-module LEIA to the 12-module EP, and supporting data fusion for new payload FXT and multiple data transmission links. This workflow management system provides a flexible, extensible, and standardized data processing framework for future large-scale astronomical observation projects, effectively addressing key issues such as multi-institutional collaboration, complex workflow orchestration, and scientific data reproducibility.
展开 ▾通过容器化执行模块和消息队列控制模块解决多机构软件集成、循环/动态并行调度与资源优先级调度问题,并以切面注入非侵入式实现 IVOA 溯源;在爱因斯坦探针中处理逾 2 万常规观测、17 万 FXT 观测和 2.1 万警报,发现 1.7 万余个源、近 200 个暂现事件。
该工作承接时域天文学与虚拟天文台对自动化数据处理、溯源和标准化的需求,将容器化和消息队列引入天文数据流水线,使多机构算法集成不再受依赖冲突制约。相比传统单体或定制流水线,AstroFlow 通过切面注入实现 IVOA 溯源模型而无侵入,兼顾产品可追溯与复现。在爱因斯坦探针从 LEIA 到 EP 的迁移中,系统展示从单模块扩展到 12 模块、并接入 FXT 及多条数传链路的通用性。未来这类工作流框架有望成为大型巡天和时域项目的通用底座,进一步结合云原生调度、事件驱动分析和自动化暂现源筛选,提升多载荷、多信使数据融合的科学产出效率。
刊出 2026-02-04 · 收录 2026-08-20