Science Operations

Troubleshooting

Instrument configuration times out

Sometimes the script hangs with an error message that INSTCONFIG has timed out and a prompt to retry/abort/ignore. The “instconfig” command configures the instrument, moving a number of mechanisms in parallel, and, in addition, implicitly runs the “imcslock” command. The “instconfig” command will time out if any of the mechanism moves times out, either because the component failed to go to the correct position, or because feedback on its position was not received. (Timeouts of IMCSlock are discussed below).

The corrective action if INSTCONFIG times out is:

  1. do not respond to the abort/retry/ignore prompt immediately;
  2. but, first, on the MODS gui, press the UPDATE button -or- type the command “refresh instconfig” in the MODS Dashboard command window;
  3. then in the script window enter “r” to retry.
  4. If you get a message “ERROR: INSTCONFIG instrument configuration already in progress – command disallowed”, then go back to steps 1-3 (i.e. you may not have waited long enough before sending the “r”).

INSTCONFIG ends in an error

There have been instances where the instconfig ends in an error (meaning one or more mechanisms did not get set), but the script proceeds. Note the error and examine the GUI Dashboard and Communication Log for any signs of an error: When a mechanism is moving or not in position, the text in the box indicating its position is amber/orange instead of black. When a mechanism move ends in an error, the background of the box turns red. Warnings and errors appear in red in the Communications log.

If you notice that the instconfig has completed with a fault, then control-C the script that is running and repeat it. Usually on the second try the instconfig will finish successfully.

IMCS fails to Lock

If IMCSlock fails, the IMCSlock button on the MODS Dashboard will turn red and the script will pause with some messages about the IMCS lock fail and a query to either “a” (abort), “r” (retry) or “i” (ignore). Before typing any response:

Check the laser power by typing

irlaser

in the command window of the MODS Dashboard, or

isisCmd –mods1 m1.ie irlaser

in a terminal window on one of the mountain workstations (obs2, obs3 or obs4). For MODS1, replace the two instances of “1” in the above command by a “2”.

The irlaser should be “enabled” and power output “1.0mW” for MODS1 and MODS2. The output of the above command will contain information on the following parameters:

      • IRLASER=ON/OFF is the AC power state to the laser control box, it does not turn on the laser proper!
      • IRBEAM = ENABLED/DISABLED indicates if the laser output is enabled
      • IRPSET = IR power set-point (what is requested) in mW
      • IRPOUT = measured IR laser power in mW

If the laser is not “enabled”, check whether the mods1Wake.pro or mods2Wake.pro script was run. If the laser is “enabled” and power output is reading the correct output value (1.0mW) hit “r” to retry. Sometimes the IMCS is slow to lock after a change of configuration and at certain elevation/rotator angles and exceeds the 150-sec timeout threshold. Hitting “r” to retry should cause it to lock in a few seconds.

If a retry does not work, then you should involve the ISA or OSA in further troubleshooting:

  • If the laser is on, enabled and the power out (irpout) is 1.0, yet the laser still does not lock after a retry (“r”), there may be another problem:
    • One problem we’ve had lately are moths or some debris blocking the laser. Slewing the telescope may help.
  • If the laser is on and “enabled”, but the power output is not reading the correct output value, the laser or laserbox may need power cycling.
    • First, the ISA/OSA may try to adjust the power output (for MODS2, “1”->”2″). Power 1.0 will set the power to 1.0 mW. Issue the “irlaser status” command to see whether this has helped; it may be necessary to disable/enable the laser.

isisCmd –mods1 m1.ie irlaser power 1.0

  • As a last resort, a full power cycle of the IR laser and the lamp and laser box may be needed.

isisCmd –mods1 m1.ie irlaser status
isisCmd –mods1 m1.ie irlaser disable
isisCmd –mods1 m1.ie irlaser off
— wait 10 seconds —
isisCmd –mods1 m1.ie irlaser on
— wait several seconds —
isisCmd –mods1 m1.ie irlaser enable
— wait 10 seconds (a built-in safety between enabling the beam and the beam coming on) —
isisCmd –mods1 m1.ie irlaser <status>(query the status again)

Calibration Lamp does not Light

Occasionally, a calibration lamp does not light. We have seen this occur with the VFLAT lamp in MODS2 and with the Ne lamps in both MODS. The Ne lamp is typically used with Hg in the dual grating comparison lamp calibrations, and on its own in the red-only grating and prism comparison lamp calibrations. The dual grating red channel spectra below illustrate the difference between when the Ne and Hg[Ar] lamps are both on (top) and when the Ne lamp is not working (bottom).

NeLamp_not_working_example

If you notice that a lamp has not come on, try to turn it on again. The quickest way to check this is to use the buttons on the MODS User Interface to turn on the lamp (click “Ne”) and take an exposure (Click “GO”), after verifying the exposure time is appropriate. If this doesn’t work after 2 or 3 tries even, then it may be necessary to wait until the day crew has a chance to investigate. Either way, please let the operator or support astronomer know, so that the frequency of these failures can be noted.

New Images Not Displaying in modsDisp

  1. Are the images in /lbt/data/new? In a terminal type ls -ltr /lbt/data/new. This will list the files in /lbt/data/new with the latest files at the end. On the MODS dashboard, check the filenames of the “Last File” and the “Next File” for the relevant channel. Check if the index number of the “Next File” greater than that of the “Last File” by more than 1.
  2. If the images are not in /lbt/data/new and the index number of the “Next File” is more than 1 above the “Last File”, the file transfer between the CCD control PC and the mods1data computer has stalled. You need to run fitsflush for the appropriate channel. Do not try to transfer images while the detector will be reading out, so type this either during a long exposure (count on a few seconds to transfer each image) or when the exposure is paused or the script is done, but do not let the number of files not yet transferred build up to more than just a few. To run, in the command line of the MODS dashboard, type
    red fitsflush
    or
    blue fitsflush
  3. After giving the fitsflush command, you should see the “Last File” index number incrementing as the new images are displayed by modsDisp. You may note that once all the files are transferred, the “Last Image” index number returns to the value it had before running fitsflush and that image is again displayed. Don’t worry – this is normal. You can confirm that all the images have been successfully transferred with ls -ltr /lbt/data/new.

Exposure State Flags on UI out of sync

Sometimes the exposure status on the MODS GUI does not match what the exposure control is really doing, e.g. the shutter is shown to be closed, but the exposure countdown is in progress.

The state machine, the code in the MODS GUI that mediates the exposure control states and the IMCS, can get out-of-whack. If this happens, and a channel is hung, the first level of intervention is to issue the command “[red|blue] expdone”, which has been described below. If that doesn’t work, or only temporarily solves the problem, then you may need to exit and restart the MODS GUI.

Finally, the ultimate corrective is to reboot the 2 CCD control computers (mN.rc and mN.bc) and the data server (modsNdata). Note that the order in which the machines are brought up matters, since each CCD Control computer shares 2 SCSI disks with the data server. The data server should be up before the CCD Control computers are booted, however a more foolproof sequence would have the data server powered up but its boot process paused while the CCD Control computers are brought up. Once these are both fully booted, then resume the boot on the data server. This sequence is complicated and, pending documentation, feel free to call the instrument scientist if reboots are needed.

As a preventative measure, it is recommended to reboot, for each MODS, the two CCD Control computers and the data server before observing.

Exposure hangs upon completion

If the exposure hangs on “Exposure Done, Cleaning up…”, most likely there was a communication glitch. The “Exposure Done” signal was not successfully passed from the CCD control PC to the mods computer. Check whether the image on which it hung is in /lbt/data/new (type ls -ltr /lbt/data/new in a terminal). If it is, then in the command window of the MODS Dashboard, type (depending on which channel hung) either:
red expdone
or
blue expdone
This should cause the exposure control to become active again, and, subsequent exposures, if indicated in the script, will be taken. The “expdone” command was written for this specific error condition, where the GUI reads “Exposure Done, Cleaning up…” and should work best for it, but please report any ensuing problems with the image counter.

Exposure hangs at “Starting exposure”

If the exposure control is hung at “Starting exposure…”, it is most likely that there has been a communications glitch. Check that the last exposure has been written to disk (type ls -ltr /lbt/data/new in a terminal and verify that the image appears in mods(2)Disp). If so, then in the command window of the MODS Dashboard, type:
red expdone
or
blue expdone

This was not the intended use of expdone, and there may be problems with image counters and race conditions. Specifically, this command may trigger a new exposure while the current one is ongoing and cause at least one image of the intended sequence to be skipped.

Exposure ends but it is not reading out

Sometimes, especially for MODS1 Red, the exposure control seems to hang after the exposure is finished and before reading out. Each channel intermittently shows delays between the close-of-shutter and the beginning of the readout.  For MODS1 Red, these delays are excessively long, about 100 seconds. There is no action to be taken, but just wait and count to 100…

The frequency and characteristic delay for each channel can be measured by comparing the header values for EXPTIME and DARKTIME. For MODS1 Blue and MODS2 Blue and Red, the delays are about 20-sec or less; it is only for MODS1 Red that it is about 100-sec). Work is being done on this problem (IT 6500), but the cause is not yet well-understood.

Exposure countdown fails

If, during the exposure, the countdown timer is not working, this is a symptom of a communication glitch, where the CCD controller probably failed to tell the GUI that it had started the exposure. The recovery is to:

  1. Abort the exposure, or wait until it finishes;
  2. Refresh the GUI (Click the “Update” button at the bottom left of the Dashboard.) This clears all exposure and configuration state flags, then asks the instrument to report its current state; and
  3. Type

    red reset

    and/or

    blue reset

    in the MODS Dashboard GUI Command window, depending on which channel has the problem.

Strange File Names

Occasionally the image readout will not have the expected filename mods#x.YYYYMMDD.NNNN.fits, but instead something like 020101M8.08q.fits (which we call a “unique name”). This happens when the expected name will clash with the name of a file already in the raw data directory on the mods data computer but also for other reasons. Files on the mods data computer are copied automatically to /lbt/data/new, and will be in /lbt/data/new or in the subdirectory of /Repository which corresponds to the UTdate on which they were taken.

Common mistakes which result in “unique names” are

  1. to forget to update the date information on the Monocular Observing and Binocular Observing pages
    or
  2. to reset the counter to 1 without realizing that some technical images had been taken earlier in the afternoon.

If you notice a “unique name” file:

  1. Determine what the file should be called using the command gethead to list the value of the FITS header keyword, FILENAME:

    gethead uniquenamefile.fits filename

  2. Check whether any files with the expected filename are already in /lbt/data/new or /lbt/data/repository/UTdate.
  3. If the date is not correct, stop the script and edit the setup information to avoid the filename conflict.
  4. If the date is correct, and especially if there is a series of unique named images, there may be a deeper problem and contacting the ISA is in order.
  5. After a potential name-conflict, please notify the ISA by email, so they can insure both sets of images will be written to the archive with reasonable names.

modsAlign problems

The new modsAlign has gotten into a “race condition” where the ds9 window is launched and a popup error box appears with the message: An internal error has been detected. Invalid command name ” ” and, in the launching terminal window you may see something like:

ValueError: XPA$ERROR invalid command name “” (DS9:mods1Align c0a83a21:54342)

This has been seen only on the first instance of running modsAlign each night; when the ds9 window is up, new images are displayed to it without issue.

A delay was added to the script (March 2017) to try to prevent this race condition, but if you do still see this error, the corrective is to OK the popup error box, leave the ds9 window up, and repeat the modsAlign command.

There may also be a popup about ds9 preferences, but that often disappears before you can even click “OK” to acknowledge and accept these.

Terminal Commands Not Recognized

Check that the users are running under tcsh. At least one partner account used to come up with bash terminals by default, but a “tcsh” at the command line makes all the necessary connections.

Guide Star Fails to Acquire

  1. Check what the current conditions are.
  2. Confirm the GS is a point source.
  3. Does the guide star have R<~16-16.5

If the conditions are good and the magnitude criteria met, then the guide star should be easily detected. Check:

  1. What filter is the AGw guiding with. The default should be clear. Through B-Bessel, sensitivity is reduced by about 1 magnitude.
  2. Was the guide probe homed at the start of the night.

If neither of these suggestions helps, consult the OSA and ISA. It may be necessary to shut down GCS, cycle power to the guider or WFS controller, and then restart GCS.

Mask Selection Error (to be updated 08-Mar-22 )

At certain telescope orientations (particularly low elevations <40-45 degrees), custom masks located in slots 13-16 of the MODS1 slitmask cassette have occasionally returned a mask selection error. If a mask selection error occurs, you will see that, on the MODS Dashboard, the selected mask will be blank and both mask IN and OUT buttons will be orange, not grey. The script-running window will display errors like:

*** ERROR: MSELECT MSELECT=FAULT Mechanism out-of-position, in-position sensor not asserted. Reset MSELECT to recover.

Errors in the log may look like:

2012-11-14T10:02:57.162316 M1.IE>MC1 STATUS: MSELECT Selecting mask 16
2012-11-14T10:02:57.165169 MC1>ACQ STATUS: MSELECT Selecting mask 16
2012-11-14T10:03:19.457404 M1.IE>MC1 ERROR: MSELECT MSELECT=0 Move Fault, position at end of move 0 but requested position 16.000000
2012-11-14T10:03:19.462097 MC1>ACQ ERROR: MSELECT MSELECT=0 Move Fault, position at end of move 0 but requested position 16.000000

If you encounter this error:

  1. Please consult the OSA and possibly also the ISA.
  2. The OSA should slew to zenith, and, at zenith, you should select the desired mask using the mask drop-down menu on the MODS Dashboard.
  3. The background of the mask button will turn from grey to orange while the cassette is moving, and, once the mask is selected, the button will turn black-on-grey again.
  4. Once the mask is selected (it doesn’t matter whether you choose to put it IN the focal plane or OUT), you can run the acquisition script again to slew, configure the instrument and take the sequence of acquisition images. This should now work since the mask selection, which is where the problem lies, has been completed.

modsWake hangs on sending commands to the HEBs

The modsWake script hung on the command blue tedpower on, with the usual options to abort, retry or ignore. Eventually, I tried ignore, but then the script hung on the next command, blue ledpower off. All of the MODS services were running and nothing seemed awry. Finally, I stopped the ICs and all MODS services, and restarted these, following the Instrument Control Software Startup instructions in the white notebook and within the MODSStartUp page. After that, modsWake completed successfully.

These 6 commands in modsWake: 3 to the blue HEB and 3 to the red HEB do the following, using blue as an example:

  • blue tedpower on -> make sure the blue HEB internal thermal cooler is on
  • blue ledpower off -> make sure the blue HEB “lab” LED display is turned off
  • blue igpower on -> make sure the blue HEB vacuum ionization gauge is on

In this instance, either the blue IC may not have been responding, or the command completed, but the “DONE:” response was not sent, one of the more common IC-related communication faults.

Instrument Authorization Fails with GCS error

At the beginning of the night or after reconfiguring to MODS, the instrument authorization may sometimes fail with a GCS error. In this case,
the OSA and/or support astronomer will need to assist. The MODS troubleshooting page on the wiki has some tips.