As of 9 December 2024 the assessment of the damage to the JSOC due to the 26 November flood is continuing and replacements of equipment are commencing. Restoration of the functioning system will come after repairs to the database hardware are complete.

First some good news.

• The JSOC facility is now dry.
• The data capture systems for HMI, AIA, and IRIS continue to operate, and we have implemented a temporary additional offsite back-up of the incoming telemetry packets at the GSFC SDAC, so we do not expect that any data will be lost.
• The tape drive library system appears undamaged, as do the on-site tape archives that hold the rest of the JSOC data. This will make recovery of the historic data much faster.
• We are now able to produce level 0 image data suitable for monitoring the basic performance of the instrument cameras.
• We are in the process of ordering replacements for damaged equipment. We do not yet have a schedule for delivery. The Stanford University administration has been very helpful about minimizing the delays in ordering replacements.

Now for some not so good news.

• Water damage was concentrated in the area where the leaking water pipe sprayed water floor to ceiling. The rest of the JSOC floor was under a few inches of water, but most equipment was raised above the floor level. Other laboratories in the building suffered less impact, though water flowed into the halls, under doors, and down to the subbasement and damaged the elevator.
• Several (roughly 20%) of the database servers and data servers were damaged. Several large data partitions that are on disk do not function. Quite a number of disk drives were damaged; however, assessment of how much we expect to be able to recover from those is ongoing. Several UPS power supplies were damaged, along with a substantial inventory of unused tapes. A number of other computer systems, including our web servers and the WSO data reduction computer, were also damaged.

Plans for the coming weeks

Restoration of the database servers that manage the storage of the billions of files in the JSOC repository is our first and foremost challenge. This is our highest priority, since other elements of the recovery depend on having a fully functional and secure data base management system. Even though we do have a partially functioning system that helps assess the degree of damage and identify what is missing, the restoration will take time to repopulate both the data base and the data archives. The JSOC is not presently able to produce calibrated images or higher level data products. Restarting the data processing pipeline depends on having a functioning database management system to access ancillary calibration and housekeeping information and to organize new data products as they are made.

We are grateful for your patience as our small team works to restore the JSOC functionality. Continue to check our JSOC Emergency Resources page for alternative ways to access some HMI and AIA data in the mean time.