Troubleshooting

Running lbckill/lbcstart

Many times, when the LBC User interface hangs or HK drops out, it is necessary to bounce the LBC software. To do this, ask the OSA or your support astronomer to login to the lbccontrol computer (the CMU) and run lbckill followed by lbcstart. Upon restarting, you’ll see a popup window on the LBC User Interface that says that the LBC has been restarted after a crash. The LBCs should return to the state they were in when the user interface hung, i.e. if only HK was running, then only HK will reinitialize, but if all Other Systems (rotator, filters, trackers and camera+shutter) were running, then HK and these will all reinitialize automatically. Go to the Power Control page to watch and if after waiting a minute or two, nothing is happening, click the “Turn ON” button to start HK or all systems yourself.

The LBC User Interface hangs or is unresponsive

• First try to toggle to and from the Log Analyzer page. This has helped in the past, when the OB Execution page has frozen, and is a quick and easy first step.
• If that doesn’t work, stop and restart the LBC software by running lbckill/lbcstart.

HK drops out:

• It is not uncommon that one of the channels (blue or red vacuum or thermal link temperature) monitored by housekeeping drops out. When this happens an email is sent to the lbc-alerts email group.
  • In this state, the LBCs cannot be turned on (a safety measure since they don’t have all the vacuum and temperature readings). It is necessary to run lbckill/lbstart to recover the dropped channel reading.
  • If this happens while observing with the LBCs, you can proceed, however at a convenient stopping time, ask the OSA or support astronomer to run lbckill/lbcstart to recover the dropped channel.

RPC errors:

• • RPC errors are communication errors between the LBC software running on the CMU and the CCD controller software running on the CCD Controller PC. 
  • The first thing to try is to bounce the LBC software by issuing lbckill/lbcstart.
  • If that doesn’t work, then you may need to power cycle the CCD Controller and the PC:
    1. Turn OFF  and back ON “Other Systems”.  This will power cycle all 4 PCs and, in the process, restart the CCD Controller software.
    2. Cycling power on the PCs takes about 5-6 minutes. It can be faster to restart just the CCD controller software, but there is not an automated way to do that, so an ISA or LBTO staff member should do that, after which they would need to run lbckill/lbcstart to reestablish the connection between the LBC and CCD controller software.

• Please report this in a note to IT 2397.

One channel inactive – Rotator error

While running an OB, you notice “warning” instead of “preset/expose/save” under one channel, while the other channel continues to take data.

• • Go to the Log Analyzer Page, search for all Warnings and Errors and note these. If you are logged into the CMU, then you can also run egrep “W R ROT | E R ROT” lbc.log, for example, to display all warnings and errors related to the LBCR Rotator system.
  • One channel in “warning” status and idle is a symptom of a rotator software or hardware error
    • Stop the OB, reload it and replay it.
    • If that doesn’t work, ask the OSA or support astronomer to run lbckill/lbcstart.
    • If neither of the above help, turn OFF and back ON “Other Systems”.
    • If the problem persists, then there may be a hardware error that requires daytime troubleshooting (e.g. IT 8295).

IRAF, IDL and display tool errors:

RB_Science hangs:

• Stop RB_Science following any of these three methods below and the restart it by typing RB_Science at the prompt.
  • Issue a control-C from the terminal window from which you launched RB_Science.
  • Click the “x” in the upper right corner of the ds9 window launched by RB_Science.

RB_Science crashes at startup:

RB_Science uses DMS to find the next image, and one reason it may crash at startup is if DMS is not running. The errors will be something like:

File “/usr/lib/python3.7/site-packages/requests/adapters.py”, line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host=’dms.mountain.lbto.org’, port=443): Max retries exceeded with url: /api//dbquery (Caused by NewConnectionError(‘: Failed to establish a new connection: [Errno 111] Connection refused’))

Run RB_Science with the nonewdataapi option, e.g. RB_Science --nonewdataapi. This should get around the problem with DMS being unavailable.

General problems with IRAF tasks:

• If there is a problem with an IRAF task, you may try to:
  •   >> unlearn task. Then reenter your specific parameters (epar task) and try again.
• If that does not help, exit iraf and restart it:
  • to exit, type  >> logout at the iraf prompt.
  • to re-enter IRAF:
    • first MAKE SURE YOU ARE IN THE RIGHT DIRECTORY (~/iraf);
    •  then type cl again. When in doubt >> cd  and then >> cd iraf
  • Once back in iraf, don’t forget to move back into your working directory: >> cd /scratch/UTDATE
• Remember to load all the correct packages before continuing (LBTtools, Observe)

Problems with IRAF – ds9 interaction:

• Your IRAF session will interact with the last ds9 window opened. This can cause unexpected problems with tasks that require image analysis, e.g. the LBTtools.Observe.lbcrangebal task used to correct pointing and copointing and also the imexam task used often to estimate seeing.
• During LBC observing, the ds9 window opened by RB_Science should be the last ds9 window opened. If you have opened another ds9 since launching RB_Science, stop and restart RB_Science as indicated above.
• lbcrangebal should display the blue and red copointing images to the 2 frames in the RB_Science ds9, but sometimes it crashes with an error about ‘parameter xcur not found’ because it cannot get the cursor position from ds9.When this happens, it is usually necessary to kill RB_Science and to exit IRAF:
  • kill RB_Science
  • logout from your IRAF session
  • restart your IRAF session
  • restart RB_SciencesentT

IRAF imexam, lbcrangebal cannot write a log:

• Are you running these tasks from a directory to which you have write access, and which has subdirectories called LOGs and Misc? /newdata and /Repository/ are both readonly. We recommend running these tasks from a subdirectory which you may create under the local directory /scratch and in which you have run LBTtools.prepdir (which creates the LOGs and Misc subdirectories).
  • Follow these directions for setting up LBTtools package in IRAF

LBC image problems:

Too few/many downloaded pixels

A readout or data transfer problem sometimes arises where the resulting LBCB or LBCR image will have black rectangles at the same place in all 4 chips and the data in these rows are shifted and wrap-around. This problem usually occurs on the top of the hour and the OSA will usually receive an email alert about “too few downloaded pixels” or “too many downloaded pixels”. Examples of images with “too few downloaded pixels”, displayed with the “ds9 -mosaicimage wcs” command, similar to what RB_Science uses, are below. This display command strips the pre- and overscan regions to display the field as it appears on the sky.

Figure 1: (a) Above. Examples of LBCR (left) and LBCB (right) images with “too few downloaded pixels”, displayed with the “ds9 -mosaicimage wcs” command, similar to what RB_Science uses, are below. This display command strips the pre- and overscan regions to display the field as it appears on the sky. (b) Bottom right. The central chip of the LBCB image is displayed to show the pre- and overscan regions in addition to the science data.

There is nothing that can be done about this intermittent problem. This occurs at the top of the hour and the next image will be fine. These images used not to be saved, but they are saved now in case anything can be salvaged. The black rectangles do not have 0 counts but instead have the width of the prescan (50 columns, [1:50,*]) plus the overscan (206 columns, [*,2099:2304]) and the bias level.

Rotational trailing: