The SDO/JSOC team is happy to announce that data products for HMI and AIA after 2023.12.23 are available for export. Incoming data are being fully processed as they arrive. Less recent data will become available over time as we continue to make progress recovering from the disastrous flooding of the computer facility on 26 November 2024.
The request queue may be long as the community collects products that are now available, so please be mindful of how you download data. We ask that you limit your data requests initially by using no more than one account and by waiting for completion of a pending request before submitting a new one. Please keep your request under the 49 GB limit.
About 5% of the data before 2023.12.23 will not be available from JSOC until the data have been restored from tapes. You may search the earlier data and keywords using the lookdata tool. If the data are online, lookdata will provide ‘as-is’ links to FITS files and report as unavailable the missing segments. BE WARNED, these ‘as-is’ files do not include any of the usual metadata keywords. You may request export of online files with metadata, but any export request that includes data segments that are offline will fail.
The complete set of AIA synoptic data are available at: http://jsoc.stanford.edu/data/aia/synoptic. The AIA synoptic data are calibrated Level 1.5 images with reduced 1k x 1k resolution at a regular 2-minute cadence.
A number of archival HMI data series for the entire mission are available thanks to the Max Planck Institute for Solar System Research (MPS). The available series include all of the Doppler velocity, continuum intensity, and line-of-sight magnetic field data files at both 45s and 720s cadence. MPS also hosts many helioseismology data series and some other partial series. A catalog can be found at the German Data Center for SDO. The MPS-hosted data must be accessed using a different interface at jsoc1.stanford.edu/export/dataview. No “on-export” processing is available for these series, and other direct access methods will not work. BE WARNED: the dataview and export tools are in beta release and may be finicky; we are working to improve the documentation and performance.
The team is working to expand the catalog of complete data series available at JSOC as quickly as tapes can be read. We are prioritizing these most popular data series in the coming weeks: HMI.SHARPs (standard and CEA), AIA Level 1, and the HMI vector magnetic field.
We are grateful to our user community for your support and patience.
The JSOC team continues its work to recover the JSOC systems from the flood damage of 26 November 2024. Significant progress has been made since the previous update.
Restoring public access to the JSOC export system is a major priority for the JSOC team, and they are optimistic that this can be accomplished at least in limited form soon. Full access to all HMI and AIA data through the JSOC will not be possible until the data recovery is complete.
The JSOC team is restoring the SDO data analysis pipeline as quickly as possible and has made significant progress in the past two weeks. We are also working to restore external access to the SDO data archive, but that capability remains unavailable. Some current observations are now available at sdo.gsfc.nasa.gov, suntoday.lmsal.com, and jsoc1.stanford.edu/hmi_latest.html
We have begun processing most near-real-time (nrt) SDO data products. The primary exception is the SHARPs, which require sequential processing. We are in the process of back-filling so that SHARP processing can resume. We have been producing definitive Level 0 and Level 1 data for HMI and AIA as the calibrations become available. We are also beginning to make definitive observables (Level 1.5 science images) for both HMI and AIA. These data are being processed from the time of the flooding incident going forward.
The database server is functioning adequately for the purpose of the pipeline, and "warm" back-up database servers have been enabled. However, we are expecting a few-week delay before bringing that part of the system up to full functionality as we wait for replacement hardware. Until then, we will not be able to provide access to the archived SDO data. We are working on alternative data-access methods; watch for updates. Replacements for the damaged data servers are expected to arrive soon, which will allow us to more fully evaluate storage disks and restore the online data sets.
All routine IRIS data pipeline products from JSOC are being produced regularly. A few intervals are awaiting the arrival of housekeeping data that was not processed at the time.
Some progress to report. As of 19 December 2024, the team was able to restart the first stage of the processing pipeline for incoming data from the AIA and HMI instruments and the IRIS mission. This stage creates and archives Level 0 images (uncalibrated raw images). Level 1 processing may be able to resume when the backlog of Level 0 processing is cleared, but the JSOC data system is operating at significantly reduced capacity while we await the delivery of replacement hardware.
Replacement hardware has been ordered, and a few components have arrived, but delivery dates for major components such as the database servers and disk servers are uncertain over the winter break.
We do not yet have a timeline for resumption of higher level processing and export to the community. In parallel with our recovery efforts we are working on alternative options for data access, but unfortunately do not yet have any solutions in place.
The instruments are healthy and continue to operate normally; no science data has been lost. We continue to be thankful for your patience and understanding.
As of 9 December 2024 the assessment of the damage to the JSOC due to the 26 November flood is continuing and replacements of equipment are commencing. Restoration of the functioning system will come after repairs to the database hardware are complete.
First some good news.
Now for some not so good news.
Plans for the coming weeks
Restoration of the database servers that manage the storage of the billions of files in the JSOC repository is our first and foremost challenge. This is our highest priority, since other elements of the recovery depend on having a fully functional and secure data base management system. Even though we do have a partially functioning system that helps assess the degree of damage and identify what is missing, the restoration will take time to repopulate both the data base and the data archives. The JSOC is not presently able to produce calibrated images or higher level data products. Restarting the data processing pipeline depends on having a functioning database management system to access ancillary calibration and housekeeping information and to organize new data products as they are made.
We are grateful for your patience as our small team works to restore the JSOC functionality. Continue to check our JSOC Emergency Resources page for alternative ways to access some HMI and AIA data in the mean time.
On Tuesday, November 26, 2024 a 4-inch chilled-water pipe in the Solar Dynamics Observatory (SDO) Joint Science Operations Center (JSOC) server room broke. This caused major flooding in the building and extensive water damage in the lab that houses the machines that process and distribute data from the Helioseismic and Magnetic Imager (HMI) and Atmospheric Imaging Array (AIA) instruments and from the IRIS spacecraft. The Stanford JSOC team is working to assess the extent of the damage, but it is severe. Science data processing for HMI, AIA, and IRIS will be down for an extended length of time, as will access to the archived data at JSOC.
The data capture systems for all three instruments remain functional, so ultimately incoming data are not expected to be lost. Instrument health and safety monitoring is being handled at the Instrument Operations Center at the Lockheed Martin Solar & Astrophysics Lab (LMSAL). The instruments continue to function normally.
The team will provide updates as more information becomes available. We deeply regret any inconvenience and appreciate your patience as our team prioritizes the repair and recovery of the affected systems.