File I/O for Big data
“Innovating Big Data Assimilation technology for revolutionizing very-short-range severe weather prediction”
An innovative 30-second super-rapid update numerical weather prediction system for 30-minute/1-hour severe weather forecasting will be developed, aiding disaster prevention and mitigation, as well as bringing a scientific breakthrough in meteorology.
File I/O patterns in Ensemble simulation and data assimilation
Approach: I/O Arbitrator
- Keeping the file I/O API’s (netCDF), data is transferred from the ensemble simulation jobs to the data assimilation job without storing/loading data to/from the file system
- Application programs are not modified
Note: the current prototype system extends netCDF