***************************************
	   		               VISTA TEC Database
	      		                  Usage Notes
		             ***************************************
WARNING 2023-12-01: 19-Jan-03 has badly fitted values. Please avoid using it. All other days' files are ready to use.


Contact info: for questions related to the dataset, please contact us:
  - Hu Sun: husun at umich dot edu
  - Yang Chen: ychenang at umich dot edu
  - Shasha Zou: shashaz at umich dot edu

Notes on other resources: 
	1. You can explore our algorithm and database via our database dashboard: https://vista-tec.shinyapps.io/VISTA-Dashboard/
	2. You can find a coding example on using the data as well as additional technical details in the User_Manual.pdf file.
	3. Please register using this Google form (url: https://docs.google.com/forms/d/e/1FAIpQLSdv3E9Ej9AIz-_MTQ84B8heplsw1JfC6fQeNdgvERLCYjiZig/viewform?usp=sf_link) to leave your email, name and institution information to access and get the latest updates of the VISTA database.


**************************************  General Description   ***************************************

This dataset contains the fully-imputed total electron content (TEC) map based on the original Madrigal TEC map, covering the period from 2005 to 2020. The dataset is generated via a video imputation algorithm called VISTA (Video Imputation with SoftImpute, Temporal-Smoothing and Auxiliary data). The generated TEC map is of size 181-by-361, based on a 1-latitude, 4-min local time grid and is of 5-min temporal cadence. Data is organized on a daily basis, and grouped in tarred files labelled by year-month. All data files are of .hdf5 format.


**************************************  Code Availability   *****************************************

Code for VISTA, auxiliary data generation (Spherical Harmonics) can be found at the GitHub repo:
https://github.com/husun0822/TEC_impute


****************************************  Data Structure   *****************************************

Each .hdf5 file contains three data channels structured as follows:

- File
      - data
	    - SH                     # fitted spherical harmonics map-series (size = 181*361*288)
	    - VISTA                  # fitted VISTA map-series (size = 181*361*288)
	    - VISTA_smooth           # fitted, boundary-smoothed VISTA map-series (size = 181*361*12)

Note: Only 12 frames per day are stored in the "VISTA_smooth" channel, containing the first 6 and last 6 frames of the day, and the remaining frames (i.e. the 7th to the 283rd frames) are identical to the "VISTA" channel.


The .hdf5 file also contains several attributes:

1. SH_order: order of spherical harmonics fitting
2. SH_mu: tuning parameter for the Tikhonov regularization in spherical harmonics fitting
3. r: maximum rank set for the VISTA fitted map (default is 181, the maximum possible rank, only relevant for the algorithm, not the final rank of the fitted matrices)
4. lambda_1: tuning parameter for the L2-norm penalty of the factor matrices of the VISTA algorithm
5. lambda_2: tuning parameter for the temporal smoothness penalty of the VISTA algorithm
6. lambda_3: tuning parameter for the auxiliary data penalty of the VISTA algorithm

Details of these parameter choices as well as the data channels and visualizations can be found in our VISTA database paper (see below in the reference section). 


An additional file named "grid.hdf5" contains channels about the spatial grid of the data:
- File
      - data
	    - latitude                    # latitude grid (181*361)
	    - local_time                  # local time grid (181*361)


****************************************  References   **********************************************

(VISTA algorithm paper) Sun, H., Hua, Z., Ren, J., Zou, S., Sun, Y., & Chen, Y. (2022). Matrix completion methods for the total electron content video reconstruction. The Annals of Applied Statistics, 16(3), 1333-1358.

(VISTA database paper) Sun, H., Chen, Y., Zou, S., Ren, J., Chang, Y., Wang, Z., & Coster, A. (2023). Complete Global Total Electron Content Map Dataset based on a Video Imputation Algorithm VISTA. Scientific Data, in press.

(VISTA database application paper) Zou, S., Ren, J., Wang, Z., Sun, H., & Chen, Y. (2021). Impact of storm-enhanced density (SED) on ion upflow fluxes during geomagnetic storm. Frontiers in Astronomy and Space Sciences, 8, 746429.