Science Operations

Troubleshooting

This page aims to provide descriptions and resolutions of all errors which may be encountered when observing with the LBCs: errors in script execution; errors in running the IRAF and IDL tasks required to copoint and collimate; problems with data flow; and problems with collimation.

The instructions here aim to help visiting astronomers and nighttime staff quickly recover from problems, but it there are any troubles or questions, please contact the instrument support astronomer.

Errors turning on or executing scripts with the LBCs:

The LBC User Interface hangs or is unresponsive:

  • First try to toggle to and from the Log Analyzer page. This has helped in the past, when the OB Execution page has frozen, and is a quick and easy first step.
  • If that doesn’t work, stop and restart the LBC software by running lbckill.sh and lbcstart.sh from the CMU. Ask the OSA or ISA to do this. Upon restarting, you’ll see a popup window on the LBC User Interface that says that the LBC has been restarted after a crash. Go to the Power Control page to watch. You may see HK reinitialize and then all Other Systems (rotator, filters, trackers and camera+shutter) reinitialize automatically, depending on the state that LBC was in when it hung. Let it proceed and if it pauses for a minute before reinitializing Other Systems, then click the “Turn ON” button to initialize these yourself.

HK drops out:

  • It is not uncommon that one of the channels (blue or red vacuum or thermal link temperature) monitored by housekeeping drops out. When this happens an email is sent to the LBTO Science Operations group.
    • In this state, the LBCs cannot be turned on (a safety measure since they don’t have all the vacuum and temperature readings). The Instrument Specialiast, ISA or OSA needs to run lbckill and lbcstart on the CMU.
    • If this happens during the night, you can proceed with observing, however if for some reason you need to turn off/on the LBCs, then you will need to run lbckill/lbcstart to recover the dropped HK channel.  At an appropriate stopping time, ask the OSA or ISA to run lbckill/lbcstart.

RPC errors:

  • RPC errors are communication errors between the LBC software running on the CMU and the CCD controller software running on the CCD Controller PC. 
  • The first thing to try is to bounce the LBC software by issuing lbckill and lbcstart from the CMU.
  • If that doesn’t work, then you may need to power cycle the CCD Controller and the PC:
    1. Turn OFF  and back ON “Other Systems”.  This will power cycle the CCD Controller PC and its corresponding PC (and in the process restart the CCD Controller software).
    2. Cycling power on the PCs takes about 5-6 minutes. It can be faster to restart just the CCD controller software, but there is not an automated way to do that, so an ISA or LBTO staff member should do that, after which they would need to run lbckill/lbcstart to reestablish the connection between the LBC and CCD controller software.
  • Please report this in a note to IT 2397.

IRAF, IDL and display tool errors:

RB_Science hangs:

  • Stop RB_Science following any of these three methods below and the restart it by typing RB_Science at the prompt.
    • Issue a control-C from the terminal window from which you launched RB_Science.
    • Click the “x” in the upper right corner of the ds9 window launched by RB_Science.

General problems with IRAF tasks:

  • If there is a problem with an IRAF task, you may try to:
    •   >> unlearn task. Then reenter your specific parameters (epar task) and try again.
  • If that does not help, exit iraf and restart it:
    • to exit, type  >> logout at the iraf prompt.
    • to re-enter IRAF:
      • first MAKE SURE YOU ARE IN THE RIGHT DIRECTORY (~/iraf);
      •  then type cl again. When in doubt >> cd  and then >> cd iraf
    • Once back in iraf, don’t forget to move back into your working directory: >> cd /scratch/UTDATE
  • Remember to load all the correct packages before continuing (LBTtools, Observe)

Problems with IRAF – ds9 interaction:

  • Your IRAF session will interact with the last ds9 window opened. This can cause unexpected problems with tasks that require image analysis, e.g. the LBTtools.Observe.lbcrangebal task used to correct pointing and copointing and also the imexam task used often to estimate seeing. 
  • During LBC observing, the ds9 window opened by RB_Science should be the last ds9 window opened. If you have opened another ds9 since launching RB_Science, stop and restart RB_Science as indicated above.
  • lbcrangebal should display the blue and red copointing images to the 2 frames in the RB_Science ds9, but sometimes it crashes with an error about ‘parameter xcur not found’ because it cannot get the cursor position from ds9. When this happens, it is usually necessary to kill RB_Science and to exit IRAF:
    • kill RB_Science
    • logout from your IRAF session
    • restart your IRAF session
    • restart RB_SciencesentT

IRAF imexam, lbcrangebal cannot write a log:

  • Are you running these tasks from a directory to which you have write access, and which has subdirectories called LOGs and Misc? /newdata and /Repository/ are both readonly. We recommend running these tasks from a subdirectory which you may create under the local directory /scratch and in which you have run LBTtools.prepdir (which creates the LOGs and Misc subdirectories).

On robs, lbcrangebal does not send corrections (sendT = 0):

  • sendT=0:
    • If it happens that, after typing “Y” to send the pointing & mirror corrections, you find that lbcrangebal reports that sendT = 0 and no corrections were sent, then:
      • verify lbcrangebal has sendtel set to yes (this is usually the case and the problem lies elsewhere)
      • reload LBTtools and Observe packages
      • execute lbcrangebal again.
    • corrections not sent:
      • This doesn’t thrown an error message, but lbcrangebal will run through very quickly and not send corrections to the mirrors.
      • Check that prepdir has been run. If it hasn’t and there is consequently no subdirectory called “Misc”, then the mirror corrections will not be sent. Insure that you have followed these directions for setting up LBTtools package in IRAF.

LBC image problems:

Rotational trailing:

Sometimes the Blue or Red, or both, images show signs of rotational trailing. The stars are elongated into arcs of circles which appear to have a common center. When guiding is working, this center is off to the right of chip 1, on the tech chip used for guiding. The image below shows an example of this rotational trailing.


When rotational trailing is seen in one or both of the images, some things to check are: The rate of rotation of the LBC derotator may be in error: This rate is equal to the rate of change of the parallactic angle, and this increases towards zenith and towards the meridian. The rate is generated by the TCS-supplied telescope position and the computer time.

    1. Check that the time on the lbccontrol computer (CMU) and on the windows PCs is correct. Login to the CMU and type ntpdate -q 10.144.0.211 . The delay between the CMU and actual time will be output. Call SW support if this needs to be updated. Ask the support astronomer to log into the Windows PCs to check the times on these.
    2. Check that the pointing and co-pointing is good. If there is reason to doubt, slew to a nearby pointing star, acquire data and run lbcrangebal to adjust pointing and co-pointing.
      1. If you track a field from low to high elevation, then you should run lbcrangebal not only at the beginning of the observation, but also at least once again before it reaches high elevation, >~ 80 deg.

Uniform trailing or elongation across the field:

Uniform elongation across the field of view may arise from errors in telescope tracking, guiding or focus and collimation. Telescope “jumps” have sometimes been seen. Usually these leave a non-uniform trail. You can overlay the AZ-EL compass on an image displayed in ds9 using the WCS (File -> “Open Other” -> “Open Mosaic WCS”) by selecting “Multiple WCS” -> “WCS a” from the WCS menu. Telescope jumps and tracking errors are likely to be along only one axis. Ask the OSA to check the axis along which the jump or trail occurred.
Guiding errors may produce elongated images. The guide star on the thumbnail image displayed on the User Interface should appear stable; suspect a guiding problem if the star appears to dance. If you suspect a guiding problem, check:
    • the LBC log for message about guiding offsets and for Z2 (P01), Z3 (P02) updates, the TCS logs, Focus and collimation errors.
    • How long since the last dofpia run?
    • Is there a large difference between the mirror and ambient temperatures?
    • Is the temperature changing rapidly?
    • Are focus corrections from the tech chip being sent to the TCS?

Problems with Collimation

Some commonly seen problems and corrective actions are listed below. The first two deal with warnings or failures that might be seen immediately; the remaining ones document problems with convergence.

  1. Warning. Data files did not appear in /newdata, Please double check and then press: Q to exit or any other key to continue.
    • This occurs when the pupil image did not get written into the /newdata directory within the timeout period (99 sec?).
    • Check:
      • list the contents of /newdata: ls -ltr /newdata will order contents by the time of creation, with the last file written at the end of the list.
      • If the file is not in /newdata, even after waiting 1-2 more minutes, there may be a bigger problem with the data flow, and the OSA and ISA should be contacted.
      • Note that if you hit any key and the new file is still not there, the old pair of pupil images will be analyzed, and the corrections sent, again. BackOut only works to remove the initial Z4 and Z11 corrections, it does not undo any other set of corrections.
  2. No Good Pupils Found No pupils were found above the brightness threshold.
    • Are there clouds or is it still twilight? You may need to run dofpia or dohybrid in darker conditions or use the /X2 option to take longer exposures. Run dofpia, /backout or dohybrid, /backout to remove the initial Z4 and Z11 corrections sent.
    • Do the pupil images show a lot of coma? (see below).
  3. Collimation: Focus (z4) is oscillating wildly.
    • The pupil illumination pattern can sometimes develop a ring near the center, and when this is bright enough, FPIA fits this ring and treats it as the outer diameter upon which the focus (z4) correction is based. FPIA thinks the pupil is too small and sends a negative z4 correction which makes the pupil larger. But in the next image with these larger pupils, the intensity of this ring is diluted and the outer diameter is correctly fit and indicates that the pupils should be smaller. But then the ring is brighter, and so it goes on…

The development of this ring, a sign of higher order spherical aberration most likely due to the mirrors not being in thermal equilibrium, is the main problem.

  • How far apart are the glass and ambient temperatures? If greater than 1 degree, consider switching instruments.
  • Examine the z11 and z22 corrections leading up to this, using the [s|d]xHiZ (z22) or [s|d]xLoZ (z11) tabs in LBTplot.
  • Consider switching to another instrument (MODS or LUCI) if the mirrors are not in equilibrium, but if you must go on, then:
    • clear active optics and try again;
    • if that doesn’t work, exit dofpia and ask the OSA to add z22 or z11 before restarting. The by-eye active optics sheet can help with z11 and the model pupils shown here with z22.
Figure 1: A dofpia series during which a bright inner ring built up, to the point where the ring itself was fit as the outer diameter and FPIA entered into a focus oscillation.

Strong coma: dohybrid ought to remove strong beginning-of-the night coma. But in case it does not, you may need to ask the OSA to send a coma correction, using this by-eye Active Optics cartoon as a guide. The series of pupil images below arose from this situation where dohybrid did not correct coma sufficiently so that dofpia (which starts with the image 015408) sees only the bright spot and sends a large focus correction. In the hugely out of focus image (015542), you can that there is strong coma. When the coma is strong enough that dofpia cannot handle it, the corrections required are ~1500-2000 nm.

Figure 2: A dohybrid series during which the first part of the dohybrid series (WRS) did not correct the strong initial coma and the second part (FPIA) mistakenly fit the bright arc as the outer pupil diameter, launching then into a series of focus oscillations.