Work Description

***************************************
VISTA TEC Database
Usage Notes
***************************************

Authors: Hu Sun, Jiaen Ren, Yang Chen, Shasha Zou
Contact info: for questions related to the dataset, please contact us:
- Hu Sun: husun at umich dot edu
- Yang Chen: ychenang at umich dot edu
- Shasha Zou: shashaz at Umich dot edu

Also, you can explore our algorithm and database interactively here: https://vista-tec.shinyapps.io/VISTA-Dashboard/

*** IMPORTANT NOTICE ***

Please register using this Google form (url: https://docs.google.com/forms/d/e/1FAIpQLSdv3E9Ej9AIz-_MTQ84B8heplsw1JfC6fQeNdgvERLCYjiZig/viewform?usp=sf_link) to leave your email, name and institution information to access and get the latest updates of the VISTA database!

************************************** General Description ***************************************

This dataset contains the fully-imputed (no missing-value) total electron content (TEC) map based on the original Madrigal TEC map, covering the period from 2005 to 2020. The dataset is generated via a video imputation algorithm called VISTA. VISTA stands for Video Imputation with SoftImpute, Temporal-Smoothing and Auxiliary data. The data features high imputation accuracy and solid spatial-temporal smoothness.

The generated TEC map is of size 181-by-361, based on a 1-latitude, 4-min local time grid and is of 5-min cadence. Data is organized on a daily basis, and grouped in folders labelled by year. All data files are of .hdf5 format.

************************************** Code Availability *****************************************

Code for VISTA, auxiliary data generation (Spherical Harmonics) can be found at the GitHub repo:
https://github.com/husun0822/TEC_impute

**************************************** Data Structure ******************************************

Each .hdf5 file contains several data channels structured as follows:

- File
- data
- SH # fitted spherical harmonics map-series (181*361*288)
- latitude # latitude grid (181*361)
- local_time # local time grid (181*361)
- VISTA # fitted VISTA map-series (181*361*288)
- VISTA_smooth # fitted, boundary-smoothed VISTA map-series (181*361*288)

The .hdf5 file also contains several attributes:

1. SH_order: order of spherical harmonics fitting (default is 8)
2. SH_mu: tuning parameter for the Tikhonov regularization in spherical harmonics fitting (default is 0.1)
3. Rank: maximum rank set for the VISTA fitted map (default is 181, the maximum possible rank, only relevant for the algorithm, not the final rank of the fitted matrices)
4. lambda_1: tuning parameter for the L2-norm of the factor matrices of the VISTA algorithm
5. lambda_2: tuning parameter for the temporal smoothness of the VISTA algorithm
6. lambda_3: tuning parameter for the auxiliary data penalty of the VISTA algorithm

Details of these parameters as well as the data channels and visualizations can be found in our VISTA database paper (link below in the reference section).

**************************************** References **********************************************

(VISTA algorithm paper) Sun, H., Hua, Z., Ren, J., Zou, S., Sun, Y., & Chen, Y. (2020). Matrix Completion Methods for the Total Electron Content Video Reconstruction. arXiv preprint arXiv:2012.01618.

(An example of application of VISTA database) Zou, S., Ren, J., Wang, Z., Sun, H., & Chen, Y. (2021). Impact of Storm-Enhanced Density (SED) on Ion Upflow Fluxes During Geomagnetic Storm. Frontiers in Astronomy and Space Sciences, 162.

(VISTA database paper) In preparation, coming soon.