Science Operations

Troubleshooting

Instrument configuration times out

Sometimes the script stops with an error message that INSTCONFIG has timed out. The “instconfig” command configures the instrument, moving a number of mechanisms in parallel, and, in addition, implicitly runs the “imcslock” command. If any of these mechanism moves times out, either because the component failed to go to the correct position, or because feedback on its position was not received, or if the IMCS IR laser fails to achieve lock within the prescribed time (120 seconds), then the “instconfig” command can time out.

The corrective action if INSTCONFIG times out is: do nothing on the script window yet (don’t answer the abort/retry/ignore yet) on the MODS gui, press the UPDATE button -or- type the command “refresh instconfig” in the MODS Dashboard command window. If this clears any INSTCONFIG-in-progress flags on the script, tell it R to retry.

If this does not work, please call the support astronomer.

Calibration Lamp does not Light

Occasionally, a calibration lamp does not light. We have seen this occur with the VFLAT lamp in MODS2 and with the Ne lamps in both MODS. The Ne lamp is typically used with Hg in the dual grating comparison lamp calibrations, and on its own in the red-only grating and prism comparison lamp calibrations. The dual grating red channel spectra below illustrate the difference between when the Ne and Hg[Ar] lamps are both on (top) and when the Ne lamp is not working (bottom).

NeLamp_not_working_example

If you notice that a lamp has not come on, try to turn it on again. The quickest way to check this is to use the buttons on the MODS User Interface to turn on the lamp (click “Ne”) and take an exposure (Click “GO”), after verifying the exposure time is appropriate. If this doesn’t work after 2 or 3 tries even, then it may be necessary to wait until the day crew has a chance to investigate. Either way, please let the operator or support astronomer know, so that the frequency of these failures can be noted.

New Images Not Displaying in modsDisp

  1. Are the images in /newdata? In a terminal type ls -ltr /newdata. This will list the files in /newdata with the latest files at the end. On the MODS dashboard, check the filenames of the “Last File” and the “Next File” for the relevant channel. Check if the index number of the “Next File” greater than that of the “Last File” by more than 1.
  2. If the images are not in /newdata and the index number of the “Next File” is more than 1 above the “Last File”, the file transfer between the CCD control PC and the mods1data computer has stalled. You need to run fitsflush for the appropriate channel. Do not try to transfer images while the detector will be reading out, so type this either during a long exposure (count on a few seconds to transfer each image) or when the exposure is paused or the script is done, but do not let the number of files not yet transferred build up to more than just a few. To run, in the command line of the MODS dashboard, type
    red fitsflush
    or
    blue fitsflush
  3. After giving the fitsflush command, you should see the “Last File” index number incrementing as the new images are displayed by modsDisp. You may note that once all the files are transferred, the “Last Image” index number returns to the value it had before running fitsflush and that image is again displayed. Don’t worry – this is normal. You can confirm that all the images have been successfully transferred with ls -ltr /newdata.

Exposure State Flags on UI out of sync

Sometimes the exposure status on the MODS GUI does not match what the exposure control is really doing, e.g. the shutter is shown to be closed, but the exposure countdown is in progress.

The state machine, the code in the MODS GUI that mediates the exposure control states and the IMCS, can get out-of-whack. If this happens, and a channel is hung, the first level of intervention is to issue the command “[red|blue] expdone”, which has been described below. If that doesn’t work, or only temporarily solves the problem, then you may need to exit and restart the MODS GUI.

Finally, the ultimate corrective is to reboot the 2 CCD control computers (mN.rc and mN.bc) and the data server (modsNdata). Note that the order in which the machines are brought up matters, since each CCD Control computer shares 2 SCSI disks with the data server. The data server should be up before the CCD Control computers are booted, however a more foolproof sequence would have the data server powered up but its boot process paused while the CCD Control computers are brought up. Once these are both fully booted, then resume the boot on the data server. This sequence is complicated and, pending documentation, feel free to call the instrument scientist if reboots are needed.

As a preventative measure, it is recommended to reboot, for each MODS, the two CCD Control computers and the data server before observing.

Exposure hangs upon completion

If the exposure hangs on “Exposure Done, Cleaning up…”, most likely there was a communication glitch. The “Exposure Done” signal was not successfully passed from the CCD control PC to the mods computer. Check whether the image on which it hung is in /newdata (type ls -ltr /newdata in a terminal). If it is, then in the command window of the MODS Dashboard, type (depending on which channel hung) either:
red expdone
or
blue expdone
This should cause the exposure control to become active again, and, subsequent exposures, if indicated in the script, will be taken.

Exposure hangs at “Starting exposure”

If the exposure control is hung at “Starting exposure…”, it is most likely that there has been a communications glitch. Check that the last exposure has been written to disk (type ls -ltr /newdata in a terminal and verify that the image appears in mods(2)Disp). If so, then in the command window of the MODS Dashboard, type:
red expdone
or
blue expdone

Exposure ends but it is not reading out

Sometimes, especially for MODS1 Red, the exposure control seems to hang after the exposure is finished and before reading out. Each channel intermittently shows delays between the close-of-shutter and the beginning of the readout.  For MODS1 Red, these delays are excessively long, about 100 seconds. There is no action to be taken, but just wait and count to 100…

The frequency and characteristic delay for each channel can be measured by comparing the header values for EXPTIME and DARKTIME. For MODS1 Blue and MODS2 Blue and Red, the delays are about 20-sec or less; it is only for MODS1 Red that it is about 100-sec). Work is being done on this problem (IT 6500), but the cause is not yet well-understood.

Exposure countdown fails

If, during the exposure, the countdown timer is not working, this is a symptom of a communication glitch, where the CCD controller probably failed to tell the GUI that it had started the exposure. The recovery is to:

  1. Abort the exposure, or wait until it finishes;
  2. Refresh the GUI (Click the “Update” button at the bottom left of the Dashboard.) This clears all exposure and configuration state flags, then asks the instrument to report its current state; and
  3. Type

    red reset

    and/or

    blue reset

    in the MODS Dashboard GUI Command window, depending on which channel has the problem.

Strange File Names

Occasionally the image readout will not have the expected filename mods#x.YYYYMMDD.NNNN.fits, but instead something like 020101M8.08q.fits (which we call a “unique name”). This happens when the expected name will clash with the name of a file already in the raw data directory on the mods data computer but also for other reasons. Files on the mods data computer are copied automatically to /newdata, and will be in /newdata or in the subdirectory of /Repository which corresponds to the UTdate on which they were taken.

Common mistakes which result in “unique names” are

  1. to forget to update the date information on the Monocular Observing and Binocular Observing pages
    or
  2. to reset the counter to 1 without realizing that some technical images had been taken earlier in the afternoon.

If you notice a “unique name” file:

  1. Determine what the file should be called using the command gethead to list the value of the FITS header keyword, FILENAME:

    gethead uniquenamefile.fits filename

  2. Check whether any files with the expected filename are already in /newdata or /Repository/UTdate.
  3. If the date is not correct, stop the script and edit the setup information to avoid the filename conflict.
  4. If the date is correct, and especially if there is a series of unique named images, there may be a deeper problem and contacting the ISA is in order.
  5. After a potential name-conflict, please notify the ISA by email, so they can insure both sets of images will be written to the archive with reasonable names.

modsAlign problems

The new modsAlign has gotten into a “race condition” where the ds9 window is launched and a popup error box appears with the message: An internal error has been detected. Invalid command name ” ” and, in the launching terminal window you may see something like:

ValueError: XPA$ERROR invalid command name “” (DS9:mods1Align c0a83a21:54342)

This has been seen only on the first instance of running modsAlign each night; when the ds9 window is up, new images are displayed to it without issue.

A delay was added to the script (March 2017) to try to prevent this race condition, but if you do still see this error, the corrective is to OK the popup error box, leave the ds9 window up, and repeat the modsAlign command.

There may also be a popup about ds9 preferences, but that often disappears before you can even click “OK” to acknowledge and accept these.

Terminal Commands Not Recognized

Check that the users are running under tcsh. At least one partner account used to come up with bash terminals by default, but a “tcsh” at the command line makes all the necessary connections.

IMCS fails to Lock

If IMCSlock fails, the IMCSlock button on the MODS Dashboard will turn red and the script will pause with some messages about the IMCS lock fail and a query to either “a” (abort), “r” (retry) or “i” (ignore). Before typing any response:

Check the laser power by typing

irlaser

in the command window of the MODS Dashboard, or

isisCmd –mods1 m1.ie irlaser

in a terminal window on one of the mountain workstations (obs2, obs3 or obs4). For MODS1, replace the two instances of “1” in the above command by a “2”.

The irlaser should be “enabled” and power output “1.0mW” for MODS1 and MODS2. The output of the above command will contain information on the following parameters:

      • IRLASER=ON/OFF is the AC power state to the laser control box, it does not turn on the laser proper!
      • IRBEAM = ENABLED/DISABLED indicates if the laser output is enabled
      • IRPSET = IR power set-point (what is requested) in mW
      • IRPOUT = measured IR laser power in mW

If the laser is not “enabled”, check whether the mods1Wake.pro or mods2Wake.pro script was run. If the laser is “enabled” and power output is reading the correct output value (1.0mW) hit “r” to retry. Sometimes the IMCS is slow to lock after a change of configuration and at certain elevation/rotator angles and exceeds the 150-sec timeout threshold. Hitting “r” to retry should cause it to lock in a few seconds.

If a retry does not work, then you should involve the ISA or OSA in further troubleshooting:

  • If the laser is on, enabled and the power out (irpout) is 1.0, yet the laser still does not lock after a retry (“r”), there may be another problem:
    • One problem we’ve had lately are moths or some debris blocking the laser. Slewing the telescope may help.
  • If the laser is on and “enabled”, but the power output is not reading the correct output value, the laser or laserbox may need power cycling.
    • First, the ISA/OSA may try to adjust the power output (for MODS2, “1”->”2″). Power 1.0 will set the power to 1.0 mW. Issue the “irlaser status” command to see whether this has helped; it may be necessary to disable/enable the laser.

isisCmd –mods1 m1.ie irlaser power 1.0

  • As a last resort, a full power cycle of the IR laser and the lamp and laser box may be needed.

isisCmd –mods1 m1.ie irlaser status
isisCmd –mods1 m1.ie irlaser disable
isisCmd –mods1 m1.ie irlaser off
— wait 10 seconds —
isisCmd –mods1 m1.ie irlaser on
— wait several seconds —
isisCmd –mods1 m1.ie irlaser enable
— wait 10 seconds (a built-in safety between enabling the beam and the beam coming on) —
isisCmd –mods1 m1.ie irlaser <status>(query the status again)

Guide Star Fails to Acquire

  1. Check what the current conditions are.
  2. Confirm the GS is a point source.
  3. Does the guide star have R<~16-16.5

If the conditions are good and the magnitude criteria met, then the guide star should be easily detected. Check:

  1. What filter is the AGw guiding with. The default should be clear. Through B-Bessel, sensitivity is reduced by about 1 magnitude.
  2. Was the guide probe homed at the start of the night.

If neither of these suggestions helps, consult the OSA and ISA. It may be necessary to shut down GCS, cycle power to the guider or WFS controller, and then restart GCS.

Mask Selection Error (to be updated 08-Mar-22 )

At certain telescope orientations (particularly low elevations <40-45 degrees), custom masks located in slots 13-16 of the MODS1 slitmask cassette have occasionally returned a mask selection error. If a mask selection error occurs, you will see that, on the MODS Dashboard, the selected mask will be blank and both mask IN and OUT buttons will be orange, not grey. The script-running window will display errors like:

*** ERROR: MSELECT MSELECT=FAULT Mechanism out-of-position, in-position sensor not asserted. Reset MSELECT to recover.

Errors in the log may look like:

2012-11-14T10:02:57.162316 M1.IE>MC1 STATUS: MSELECT Selecting mask 16
2012-11-14T10:02:57.165169 MC1>ACQ STATUS: MSELECT Selecting mask 16
2012-11-14T10:03:19.457404 M1.IE>MC1 ERROR: MSELECT MSELECT=0 Move Fault, position at end of move 0 but requested position 16.000000
2012-11-14T10:03:19.462097 MC1>ACQ ERROR: MSELECT MSELECT=0 Move Fault, position at end of move 0 but requested position 16.000000

If you encounter this error:

  1. Please consult the OSA and possibly also the ISA.
  2. The OSA should slew to zenith, and, at zenith, you should select the desired mask using the mask drop-down menu on the MODS Dashboard.
  3. The background of the mask button will turn from grey to orange while the cassette is moving, and, once the mask is selected, the button will turn black-on-grey again.
  4. Once the mask is selected (it doesn’t matter whether you choose to put it IN the focal plane or OUT), you can run the acquisition script again to slew, configure the instrument and take the sequence of acquisition images. This should now work since the mask selection, which is where the problem lies, has been completed.

modsWake hangs on sending commands to the HEBs

The modsWake script hung on the command blue tedpower on, with the usual options to abort, retry or ignore. Eventually, I tried ignore, but then the script hung on the next command, blue ledpower off. All of the MODS services were running and nothing seemed awry. Finally, I stopped the ICs and all MODS services, and restarted these, following the Instrument Control Software Startup instructions in the white notebook and within the MODSStartUp page. After that, modsWake completed successfully.

These 6 commands in modsWake: 3 to the blue HEB and 3 to the red HEB do the following, using blue as an example:

  • blue tedpower on -> make sure the blue HEB internal thermal cooler is on
  • blue ledpower off -> make sure the blue HEB “lab” LED display is turned off
  • blue igpower on -> make sure the blue HEB vacuum ionization gauge is on

In this instance, either the blue IC may not have been responding, or the command completed, but the “DONE:” response was not sent, one of the more common IC-related communication faults.