28 October 2019 – Updated Frequently Asked Questions
General Closure Information
Q: Is Berkeley Lab open?
Berkeley Lab’s main hill site will remain closed through Wednesday, October 30. Personnel working at all Berkeley Lab locations other than the main hill site should report to work as usual.
On Monday, PG&E notified us that the Lab’s main hill site is within an area that will be subject to another Public Safety Power Shutoff (PSPS) beginning as early as Tuesday morning, October 29, due to strong winds forecast to occur from Tuesday morning through midday Wednesday. At this time, we do not know if the Lab’s offsite locations will also be affected by the new PSPS expected for Tuesday and Wednesday.
JGI personnel should look for communications from JGI management for details. For work locations on the UC Berkeley campus, refer to instructions from https://news.berkeley.edu. We will keep you updated as we learn more.
Q: Who is permitted on the main hill site?
Currently, power remains out at Berkeley Lab’s main hill site and fire danger is high. The site is in a safe and stable condition. We have shifted to minimum staffing in the Emergency Operations Center (EOC) and are further restricting access to the main hill site.
Until further notice, the main hill site is restricted to all personnel except the following: individuals approved to be in the EOC, Plant Maintenance Technicians, Protective Force personnel, and a select list of others required to maintain safe operations. What has changed is as follows: until power is restored, security personnel will not grant access to individuals on the emergency-essential personnel list that had been in use since the Lab closure. If you need special access to the site, request access from the representative listed below for your area prior to coming to the site.
Operations staff: Call the EOC at 510-495-2947
Biosciences Area: Nikki Humphreys, 510-289-7435
Computing Sciences Area: Helen Cademartori, 510-207-3144
Energy & Environmental Sciences Area: Pater Lau, 510-409-4898
Energy Sciences Area: Mike Wisherop, 510-289-1553
Energy Technologies Area: Jerri Carmo, 510-486-7655
Physical Sciences: Asmita Patel, 510-207-9451
Q: When did PG&E turn the power off at Berkeley Lab?
A: PG&E shut off power to Berkeley Lab’s main hill site at 10:01 p.m. Saturday, Oct. 26, as part of the current public safety power shutdown (PSPS). Power to the Joint Genome Institute (JGI) in Walnut Creek was not shut off as expected.
Other Berkeley Lab offsite locations are currently not in PG&E’s current shutoff zone and remain open. Check for updates at status.lbl.gov, at 1-800-445-5830, and on a new Twitter account for Lab staff, users, and others: @LBNLstatus.
Q: Has UC Berkeley also experienced a power outage?
All or some of the UC Berkeley campus has been affected by the public safety power shutoff (PSPS), including Donner Hall. Please check https://news.berkeley.edu for status updates on UC Berkeley operations and closures, and https://vcresearch.berkeley.edu/campus-power-outage-october-25-2019 for RESEARCH-specific information.
Q: When will the closed Lab sites reopen?
There is no current estimate of when PG&E will restore power to the Lab’s main hill site. Sustained high winds are expected for at least the next 24 hours. PG&E has announced that its teams will need to inspect power lines and repair damaged equipment before restoring power. Once power is restored, the Lab’s main hill site will remain closed until Lab teams have safely restarted its facilities and operations, which will require a minimum of 24 hours.
Q: Why am I not allowed on the affected sites during the closure?
A: We want you and our emergency operations personnel to be safe. During a power shutdown, most Lab power-reliant services, such as building lighting and ventilation, will not be available. In addition, normal Lab safety services may not be available, and the absence of power may additionally cause unexpected, potentially unsafe conditions. To keep everyone safe, the Lab needs to minimize the number of people on-site, and know who is on-site and what they are doing.
Q: How will mission-critical equipment be protected?
A: Some equipment has been moved over to backup generators, which will be refueled as needed. Other equipment has been shut down in a prescribed order based on equipment-specific procedures. Operations teams are working in coordination with mission area leadership, through mission area representatives in the EOC called Mission Support Officers, to shut down equipment to safe and stable status. Similar procedures will be followed after power is restored when it is time to restart the equipment.
Q: What mission elements are shut down and which are still operating?
A: Mission functions for which a loss of power would be detrimental have been placed on backup generator power. All other buildings and functions on the main hill site are being placed in a safe and stable condition in anticipation of a power shutdown. See below for more information about user facilities.
Q: Where can I go to get the latest information on the power shutdown?
A: The latest information will be updated on status.lbl.gov, at 1-800-445-5830, and on a new Twitter account for Lab staff, users, and others: @LBNLstatus. You should check these sources frequently. Since you may not have functioning power to reach a website, also keep in touch with your supervisor.
Business Systems
Q: What IT systems will be operational during this PSPS event?
A: For the most up to date information, visit it-status.lbl.gov. IT anticipates that system availability will be the same as it was during the previous PSPS outage. Email and other cloud systems are anticipated to be operational. Core web-based business systems such as FMS and HRIS are anticipated to be operational. Science systems including SVM and HPC have already been shut down.
Networks inside buildings may not be operational once powered down. This includes wi-fi. Remote access to voicemail should be operational and inbound calls to landlines should function normally.
IT system availability during an outage is dynamic. Anticipated availability is based on the best information available at the time this was written.
Q: Will I still be able to authenticate using my cell phone? With my key?
A: Yes, Lab Identity Management Systems are anticipated to be operational during the PSPS outage. For Operations employees, logging in to Windows workstations using Yubikey or Google Authenticator is possible regardless of whether Identity Management services are available (workstations only, not other services).
Q: If I am going to work remotely using a PC, will I be able to access my Google files? What about with a mobile device?
A: Yes, Lab Identity Management Systems are anticipated to be operational during the PSPS outage so your Google files will be available. Otherwise, the answer depends on whether you have an “active session” with Google. See below.
Q: Once the shutdown is over, how long will it be before IT services are available.
A: Networking, identity management, and telephones are our first priority and should come back quickly, in minutes to hours. Business systems are the next priority (during working hours), provided that cooling is operational in the building. Scientific systems, especially clusters, take a significant amount of time to restart and require full cooling capability in the building. It typically takes one to three days to restore full cluster operations.
Q: How do I recover lost work/data as a result of the shut-off?
A: Work with your local IT support to identify how your information is backed up and restored. For staff using Druva for backups, restoring data is self-service once power and networking are restored. Information on restoration is available here. The IT Helpdesk is available to assist with restorations, as well as to assist with system and data restoration.
Q: What is the current and expected status of Lab IT systems?
A: Visit it-status.lbl.gov for information.
User Facilities
Q: What’s the status at NERSC, and who can NERSC users contact for information?
A: The Cori system and major file systems are down. Users having urgent jobs or workflows on Cori that will be impacted, please contact NERSC via https://help.nersc.gov and we can discuss options for getting these jobs run before the outage or at another ASCR HPC Facility. NERSC users may refer to the regular status websites: https://www.nersc.gov/live-status/motd/ or https://nersc-status.lbl.gov. We expect these websites to remain operational through the power outage and will be updated frequently. You may also contact NERSC Operations at operator@nersc.gov or (800) 666-3772 for questions, but please do not call to ask when the outage will be over. The situation is fluid, and our power utility has not yet provided specific guidance.
Q: What’s the status at JGI, and who can JGI users contact for information?
A: JGI’s facility in Walnut Creek has full power. All staff are to follow guidance disseminated by JGI Management.
However, because many of JGI’s data systems (Genome Portal, MycoCosm, PhycoCosm, IMG/M) are dependent on NERSC, they may not be accessible. DNA sequencing activities (which archive data through these same data repositories), and JGI’s other technical capabilities (DNA synthesis, metabolomics) have been suspended, thus interrupting user support. JGI’s Project Management Office has already been in touch with JGI’s user base and notices have been posted on the various portal landing pages. The JGI proposal submission site (WIP) is down. Who to contact: “Public” inquiries: David Gilbert, degilbert@lbl.gov; 925-296-5643. Technical computing issues/status: Kjiersten Fagnan, JGI CIO, 510-220-5444; DNA Sequencing infrastructure: Chris Daum, Genomic Technologies, 925-296-5623.
Q: What’s the status at the ALS, and who can ALS users contact for information?
A: The ALS has been safely shut down and will remain off for the duration of the power outage. Our user portal, ALSHub, will also be down. The ALS ended user operations at 6 a.m. on October 26 and notified affected users that beamtime was canceled for the remainder of the weekend. The next scheduled user beamtime begins Thursday, October 31 at 8 a.m. At this time, we cannot predict the length of the power outage or the impact to our equipment. Once power is restored, multiple checks will need to be made; we expect the ALS startup to take at least 8 hours depending on many variables. The ALS will closely monitor the power outage situation and notify users that have beamtime scheduled later this week well in advance of any potential impacts to operations. Contact: Andreas Scholl, a_scholl@lbl.gov
Q: What’s the status at ESnet, and who can ESnet users contact for information?
A: Energy Sciences Network (ESnet) will be operational during this outage. At this time, ESnet connectivity (including Internet access) to NERSC and LBNL is not expected to be impacted. For all issues related to ESnet, please contact the Network Operations Center via email, trouble@es.net or phone (510) 486-7607. ESnet provides live statistics for network traffic from its sites at https://my.es.net/sites/list
Q: What’s the status at the Molecular Foundry, and who can Foundry users contact for information?
A: The Molecular Foundry will remain shut down for the duration of the outage. Once Berkeley Lab is reopened, we anticipate our general spaces to be available to users. Individual laboratories and instruments should come online within a few business hours, depending on the inherent challenges and limitations of the systems involved. Please note that some Foundry staff will be working remotely during the shutdown, including being available by email to users, however, some may also be impacted by power outages in their homes that may limit their responsiveness. Contact: Michael Wisherop, Facilities and EHS Manager, mpwisherop@lbl.gov, (510) 486-7407
Transportation
Q: Will the shuttles be running?
A: There will be no shuttle service to and from the Lab main site while the site is closed. The Potter Street shuttle route will run between Potter Street, JBEI, and the downtown Berkeley BART station, but it will not continue up Hearst to the main hill site. Shuttle drivers will attempt to stay on the normal schedule, but delays are possible. Utilize NextBus for the most accurate route timing.
Reopening the Lab
Q: As soon as power to the Lab is restored will the Lab be opened to all?
A: The Lab will need to reopen in an orderly fashion. Designated operations personnel will return first to conduct a safety assessment of all buildings. Once completed, the staff will be contacted and provided information on returning to work. You should stay in touch with your supervisor to get more return-to-work information.
Supervisor Information
Q: What role does a supervisor play in an evacuation/non-opening of the Lab?
A: Supervisors should be communicating with their employees about remote work options.
Q: How should supervisors stay in touch with employees?
A: Supervisors should have contact information such as home telephone numbers and cell phone numbers for their team members. Remember that employees may not have power at home as well and it may be difficult to reach during a widespread power outage.