Remote Facility Policy

Version 1.0
December 1, 2020

I) LBTO Remote Observing Facilities Science Operations Policy

Draft Version 1.1

1. Introduction:

The Large Binocular Telescope Observatory (LBTO) is in the process of establishing a remote observing mode for observers operating from dedicated on-site “satellite” facilities located at partner universities or research institutions. This document outlines the operations policies governing this operational mode including I) procedures for notifying the observatory of the intent to remote observe II) guidelines for network access, facility maintenance, and staffing and III) declarations regarding risk and recommendations for risk management.

2. Notification of Intent:

• Partner Coordinators or a Responsible Party must inform the observatory of their intention to remote observe 10 days before the partner science block in question
• Partner Coordinators or a Responsible Party must provide the names and contact information of all observers participating in remote observing
• Partner Coordinators or a Responsible Party must inform the observatory of the physical location of their remote observing site and provide a phone number to the site and contact information in case of emergency.

3. Network Access and Use Policy:

All remote observing participants must fill out the Partner Institute Remote Observing Request Form to ensure correct access is granted to the LBT network and to verify that all users have read and agreed to the UA/LBTO Acceptable Use Policy.

4. Facility:

• LBTO highly recommends that observers use a remote observing site that meets the recommendations provided in the LBTO Remote Observing Facilities Technical Recommendations document.
• For safety, the Lead Observer must stay in continuous contact with the mountain-based OSA at all times. A zoom, skype, polycom connection must thus be available and running throughout the night.
• All setup, configuration, maintenance, and troubleshooting needed at the remote site is the sole responsibility of the partner remote observing site.

5. Personnel:

• LBT Staff: An LBT Service Observer (SO) or Instrument Support Astronomer (ISA) will be available during the first night of each remote partner science block to help ease observers into their run. After the first night, either an SO or ISA will be on-call to help with instrument troubleshooting. An Observatory Support Associate (OSA) will be available on the summit every night.
• Lead Partner Remote Observer: At least one experienced partner observer must be present at the remote observing site to supervise and execute observation from the remote observing location. The partner observer must be: adept at operating all facility instruments in all modes that they intend to use, capable of producing and editing scripts on-the-fly, able to communicate clearly with the OSA on duty, particularly if instrument on-call support is needed, be willing to perform basic troubleshooting if instructed.If necessary, the Partner Lead Observer may rotate amongst an observing team if necessary, but this information must be passed clearly to the observatory in the notification process and the Lead must be identified each night to the OSA and ISA/SO on call.
• Back-up Partner Observer: LBT highly recommends a back-up observer, preferably at (or with access to) a secondary remote observing site. The back-up observer should ideally also be expert and able to run all facility instruments in all modes that the partner intends to use. He/She/They should be included in the Notification of Intent.

6. Assumption of Risk:

• Remote Observing is a convenient mode of observing, but it comes with additional risks. These include, but are not limited to: network outages or failures on the mountain, in Tucson, or at remote observing sites, power outages, local site emergencies. By requesting remote observing mode, partners accept these risks. LBTO staff (see below) will be willing to help assess the issues and troubleshoot with alacrity, similar to any partner science run, however, partners acknowledge that observing time may be lost by agreeing to use remote partner sites.
• While OSAs, ISAs, and SOs will be either on-call or available on the summit to support all troubleshooting, they will not execute observations in remote observing mode. This will remain true even if a failure occurs. We strongly recommend a back-up observer to prevent loss of telescope time.

II) LBTO Remote Observing Facilities Technical Recommendations

Version 1.1

1. Introduction:

LBTO is in the process of establishing a remote observing mode for observers operating from dedicated facilities located on-site at one of the LBTO participating universities or research institutions. This document specifies the computer hardware recommended for these dedicated facilities, the software that should be installed, and the intended use of the systems. These guidelines represent a set of “best practices” developed after months of internal iteration and testing by our operations and IT staff.

We recognize that financial and space considerations or alternate strategies that disperse observing responsibility simultaneously over multiple remote facilities at different institutes in the same partner, may necessitate departures from the recommendations outlined in this document. Since, as stated in the Remote Observing Science Operations Policy, partners assume the risks by requesting remote observing mode, the observatory suggests partners plan their remote facilities in accordance with their own internal policies and goals.

2. Caveats:

This document does not address remote observing policies, which are covered in the LBTO Remote Observing Science Operations Policy. Nor does it address either policy or technical requirements for from-home observing.

3. Recommended General Specifications:

Two workstations configured identically as follows:

• 6x 3.2GHz core CPU
• 64GB RAM
• 250GB SSD
• 4TB HDD
• GPU
• Audio speakers
• 2x 4K HD display monitors

Additionally, the following hardware to support video conferencing and a read-only environmental display:

• 2x low-end PC
• 2x monitor with resolution 1920×1400 or greater.
• 1 high quality microphone
• 1 high quality sound system
• 1 wide angle video camera[1]

Lastly, the following UPS and networking hardware:

• UPS capable of providing 3.5 hours of uninterrupted power to the workstations and VC equipment.
• Gigabit Ethernet connection[2]

The components listed above are shown schematically in Figure 1 below.

Any hardware that meets these general specifications is acceptable, and in some cases a partner institution will be able to use existing hardware for some or all of the recommended systems.

4. A Specific Example:

The “shopping list” given below is only an example and is provided for an institution that is less concerned about the cost or the reuse of existing hardware and would rather quickly purchase hardware that is guaranteed to meet LBTO recommendations. The cost for a completely new complement of hardware meeting LBTO recommendations is estimated to be approximately $12K.

However, any complement of hardware that meets the general specifications given in Section 3 is acceptable.

5. Software:

The observing PCs shall have the following operating systems and software installed and configured:

• Fedora Linux version [3]
• AnyDesk
• Cisco AnyConnnect (available from https://vpn.lbto.org)
• Zoom
• Chrome

6. Intended Use of the Specified Computer Systems:

The components of the remote facility are shown in Figure 1 below.

The primary workstation will be used by the main investigator to connect to the LBTO computers in the control room in Tucson (room 507) and mirror the LBTO support staff display.

The auxiliary workstation will be used by another observer when working as a team. The auxiliary workstation will be used for local software; e.g., to run the Observing Tool (OT). The auxiliary workstation will also serve as a backup if the primary workstation fails, however, at any moment, only one workstation shall connect for remote control to the Tucson facility. Terminal based (ssh, scp) or web based (https://*.lbto.org) connections can, however, be initiated in parallel to instrument control sessions.

Figure 1 – The components required for a remote facility (see Sections 3 & 4), along with their intended use (see Section 6) are shown in a schematic layout.

The environment display will provide information about weather, telescope status, etc. This station is used in a similar way as LBTO’s “big display” in room 507 and on the summit, displaying FACSUM/Weather/StatServ or http://info.lbto.org.

Other workstations and laptops can be used by the observing team, however, the level of OSA, ISA, and IT support in the use of those will be limited.  Observers expecting full support should work with the computer systems as specified here.

Investigators shall use their own personal VPN authentication. Users who do not have a VPN credential shall make a request to their contact LBTO staff astronomer at least one week before their run. The LBTO IT group will provide a user-name and password that can be reused later. For security matters, the LBTO IT will only issue credentials to individual users, and will not allow generic or shared VPN accounts.

7. Technical Specifications Not Related to Computer Systems:

A permanent, dedicated, secure room shall be provided to observers at the partner institution containing the computer facilities described above. The room will include a telephone system that allows calls to and from the LBT offices in Tucson, and to the LBT on Mount Graham, including international calls for European partners, to the direct numbers required to reach LBTO staff members.

For additional information about remote observing at LBTO, please contact the Remote Observing Working Group

---

_{1 Wide enough to include the persons sitting at both work stations from the selected mount point.}

_{2 To safeguard against lost observing time due to prolonged network dropout, we recommend, when practical, that IT groups from remote observing facilities set up a 4G internet connection (also known as a “cell phone hot spot.”) Although it is unlikely that the bandwidth available from this backup network would allow the users without internet to continue their primary program, it would provide a slow connection that could be used to help a team member working from another site to take over (e.g., to receive an OB or a finding chart).}

_{3 LBTO recommends partners to use the same OS as being used by the Tucson and Mountain workstations. This will help LBTO for IT&SW support, and allow software distribution (RPMs from our LBTO dnf repository e.g., FACSUM, weather display, etc.). Future developments at LBT are expected to leverage modern web technologies, to provide platform agnostic software and lessen requirements on OS.}