VAMPIRES:Observing:Standard modes

Most information in this doc applies to polarimetric and single-camera imaging modes.

😎 Indicates items specific to polarisation mode

📷 Indicates items specific to single-camera mode

• Observing procedure overview
• Setting up vControl
• Acquiring data
• Finishing observing

Important: check before you start:

• 😎 Insert HWP into AO188 when using polarisation mode (and enable ADI sync if desired)
• Ensure g2cam is running (to allow archiving)
• 😎 Ensure the polarised beamsplitter is in - do vampires_beamsplitter 2 in a VAMPIRES terminal, or
• 📷 Remove the beamsplitter for single-camera mode - do vampires_beamsplitter 1 in a VAMPIRES terminal (positions 3, 5, and 7 also work)

⛔️ adding or removing the beamsplitter causes ~4 mm of focus shift, so make sure to refocus accordingly when changing modes.

• Ensure no differential filters or lenses are in do vampires_diffwheel 2
• 😎 Ensure Enable HWP is checked
• 😎 If syncing HWP with CHARIS or FPDI - see here.

Introduction

The most common mode for VAMPIRES operation is the differential polarisation mode. Most steps are common between NRM and full-pupil mode; simply set the NRM mask wheel to EmptySlot or EmptySlot2 for full-pupil mode. Other options are the same.

To acquire data, vControl sends the chosen timing settings via ethernet to a separate Raspberry Pi computer, operating in near-realtime, which in turn sends near-realtime control signals to an Arduino-based microcontroller. The microcontroller runs in true realtime, sending precise timing signals (~5 microsecond precision) to the camera triggers and FLC modulator.

However when viewing live video in realtime in vControl, all triggering is taking place internally within the cameras and the FLC is not used. This means that integration time settings set for live video preview are not necessarily the same as being used for acquisition - so be careful!.

This section will go through the observing procedure step-by-step.

Observing procedure overview

Once you've read this all and are a pro, here is a quick reminder:

• Set up session and launch vControl (as described below)
• Enable archiving to gen2 if desired by checking the ARCHIVE TO Gen2 checkbox
• Set camera subwindow and integration times. Some suggested settings for masking:
  • H width & V width: 256
  • Exp time (in Camera Control): 0.01 s
  • int Time (in Acquisition settings): 10000 us
  • Num Frms: 1600
• Insert desired mask (if needed) and filter
• Start live video and, with help of Peak Flux readout or line profile, adjust exposure time and EM Gain.

⚠️ Be sure to put the decided integration time in the Acquisition Settings panel, in microseconds!

• Enter a filename prefix
  • Naming convention is targetname_setnumber, e.g. alfOri_01, to correspond to entries in the log. Timestamps and metadata are appended automatically.
• Put desired number of loops in Loop in the Acquisition panel, and hit Execute Series!*
• 😎 Make sure QWPs move when filter is changed and the HWP moves successfully (i.e. look for successful messages in the console)
• Make sure frame counts progress as expected, and make sure via GINGA that archiving is happening.
• To finish acquiring data before specified no. of loops have finished - just press the Abort! button and wait patiently. It will save out the current cube and set various things back to nominal, then once the GUI returns to the usual grey background you can do the next task.

*📷No. of cubes taken = no. of loops * 4. A single cube may be taken by pressing Execute Acquisition instead of desired.

Setting up observing session

If not already set up, there are a few things you may need to do before starting:

Setting up VAMPIRES computer session

VNC session

Note: IP addresses and passwords have been redacted. Please contact the SCExAO team directly for this information.

All operation should be conducted via VNC (not SSH) to ensure stable operation even if the remote connection drops out. The main VNC screen (5901) should already be running, so simply use your favorite VNC client (I use RealVNC Viewer) to connect to REDACTED:5901

Important: Please use -shared (or whichever appropriate setting for your VNC viewer) to avoid kicking people out.

If this doesn't work, it could be that the VNC server isn't running (e.g. due to a restart or careless user). You can start it via ssh:

ssh lestat@vampires cc-vncspawn

Open vControl

Most operations can be performed through the vControl GUI, written in Matlab. Before launching it, is a good idea to do

ps aux | grep -i matlab

to see if any other unexpected Matlab processes are running (e.g. if a previous user left one open in another session). If so, you should terminate them in case they are trying to access the same hardware. You can do this with

kill -9 <pid>

Now you can open Matlab. Note: Matlab MUST be launched from within the vControl directory (due to a Matlab / Java permissions quirk). So from the shell do

cd ~/code/vControl
tmux new -s vControl # not strictly necessary, but convenient
matlab

then from within the Matlab Command Window simply type vControl. If all goes well you'll see a bunch of devices initialise in the command window, and then the GUI will open. If you get an error there are some troubleshooting steps here.

Note: Do not try and launch multiple instances of vControl (e.g., in a separate VNC server). This kills the beast.

Running g2cam to enable archiving

For data to be archived to gen2, the local g2cam process must be running on the VAMPIRES computer, via the run_gen2 script. This should usually be left running in a terminal window, but it is important to confirm this before observing, or frames will not be archived.

To confirm it is running

• Find the open terminal window which has the label lestat@sonne: ~/code/vControl/run_gen2
• Perform a test acquisition by ensuring the ARCHIVE TO Gen2 box is checked in the vControl GUI and pressing Execute Acquisition.
• You should see the archiving requests, including the filename and current time, show up in the termnal window.

If archiving appears not to be working

• If you can't find it, double check it's not running in some other window by doing ps aux | grep gen2
• If you have found it but is stalled, terminate the currently running g2cam process by hitting ctrl+c in its terminal window
• Restart the g2cam process by opening a new terminal window, and doing

cd ~/code/vControl/run_gen2
./run_g2cam.sh

Set up other programs and windows

There are some other things you should have open and visible in addition to vControl.

• Firstly, make sure the Matlab Command Window is always open and visible, as this is where important warnings and errors will be displayed!
• When using NRM masks or the Lyot stop, the pupil viewing camera should be used to view the pupil and ensure mask alignment.
• In addition to the local shell, a terminal window connected to scexao2 should be kept open (as various commands will be typed here. Just do ssh -Y scexao@REDACTED. No password is required.
• If you want to observe the main instrument control (i.e., dm_displee, chuckcam, ...) connect to the scexao2 VNC server at REDACTED:5901 (remember to use -shared!).

Synchronising HWP with CHARIS / FPDI 😎

Make sure total cycle time of VAMPIRES is less than that of CHARIS. VAMPIRES cycle time is the time taken to acquire a cube, rotate the HWP and write the data to disk. An estimate of the data acquisition time is just t_int * n_frames but you'll need to add extra time to allow for file saving etc - so test to be sure.

Then check the Wait for CHARIS checkbox in the GUI. In this mode, it will:

1. Acquire a cube when Execute Series is pressed
2. Stop and wait to receive a signal from scexao2. scexao2 middlemans between CHARIS and Vampires to ensure FPDI is synchronized as well.
3. Rotate the HWP and acquire the next cube.

You must make sure that an instance of the scexao2 HWP daemon is running.

• Locate (and then kill) an existing PDI daemon by running pgrep -lf hwpsync on scexao2.
• Start one with the appropriate options (either -cv or -cfv; see shortened doc below) by calling hwpsync <options>:

    hwpsync <num> [-v] [-c] [-f] [-p <p>] [-d]
Options:
    <num>       Number of CHARIS frames per position
    -v          Vampires PDI
    -c          CHARIS PDI
    -f          Fast PDI

In regards to <num>: the HWP can be rotated only every n-th CHARIS frame, which increases the Vampires and FPDI duty cycle. Keep it below 1 min though.

Setting up vControl

Once you're in vControl, check the cameras are working by hitting the Start Video button and make sure something (noise / light) appears in the two camera windows. If so, press Stop Video and you're good to go. First, go through and adjust the various settings:

Camera Control panel

Note you can only change camera settings when the camera is stopped.

• Readout Mode - choose between Non-Frame Transfer and Frame Transfer modes - see Andor documentation for gory details. In frame transfer mode the shortest exposure time is 1/(frame rate), this is not an issue in NFT. Data acquisition is always in NFT so this does not affect that. But NFT can be useful for very short integration times in live preview mode.

• Exp Time - this sets the exposure time (in seconds) for live video mode only. It does not change the exposure time used for data acquisition series. This allows you to set a much shorter integration time using NFT (non frame-transfer) mode than is possible with external triggering. Typically you would adjust the exposure time here to get the desired counts, and then use that to set the integration time in Acquisition settings (see below).

• EM Gain - sets the EM Gain; a number between 0 (EM Gain off) and 300. Actually, it can be set higher than 300 but you must only do this if you are truly in the single photon per pixel regime, otherwise it will damage the camera. For all common uses you should never need to go above 300. Adjust so that the peak counts are a few thousand (but no greater than ~20 000). This affects both live-video and data-acquisition.

• Everything else - leave at defaults for now.

TODO: explain advanced camera settings, cyc time and effect of nframes (acquire nCubes).

Acquisition settings

This controls settings related to actual data-acquisition series (i.e. it programs the Arduino timing controller). You can hit Test Timing Controller to make sure it's all working - if so, you should see a few lines ending with OK - acquisition complete at the console.

• Num Frms - Sets the number of frame in a given image cube. Since the HWP is rotated between cubes, this should be a number that gives you less than a minute of acquisition time. 800 frames is traditional for 512x512 (56 fps), or 1600 for 256x256. The FLC switches between each frame, so 800 frames would give 400 frames of polarisation state A, and 400 frames of state B.

• int Time - the actual integration time in microseconds for each frame. The fastest possible for 512x512 frames is 18000 us, and the fastest in 256x256 is 10000 us. The longest possible integration time is 1000000 us (1 s), limited by the FLC itself.

• FLC Offset - offset (in us) between start of camera trigger and FLC switch. Leave at 0 for now.

• Start delay - delay (in us) between pulling the FLC gate high and the first frame. Leave at 100000 us for now.

Setting a Line Profile

To evaluate the exposure times and EM Gain settings described above, it is useful to use the line profile tool. To set the line profile, follow the following steps:

1. Make sure the live video is running (this won't work if it's stopped)
2. Click in the video window (left or right) that you want to place the line in
3. Press the Set Line Profile button
4. Click the start and end points in the chosen window, and then press Enter
5. To change the maximum Y-axis value in the graph box, change the value in the Prof. max box. To hide or show the overplotted line, use the Show line checkbox.

The line profile uses a built-in Matlab graphics function which means it is rather picky about the order you do things. Just refer to the above steps...

TODO - fix the line profile behaviour at other subwindow sizes, or at least document the problem here!

Image settings & Crop mode

Control binning and subwindowing. Normally binning is not used, so H bin and V bin should be 1. H width and V width set the subwindow size. 512 is maximum, though usually 256 is good. Very high speeds can be achieved by setting the window smaller - see Andor documentation for details. H cent and V cent give the centre of the subwindow - these are almost always left at 256.

Crop mode should be set to Norm for normal operation. Iso mode is used with an external focal-plane mask to achieve very fast speeds. But this mode is in development and not yet documented.

(TODO: describe subwindow and crop speeds and settings)

Mask Wheel

Use in combination with the Steer Im/Pup window to adjust the aperture mask settings. Positions are stored in the Preset File (as specified in the Preset File field). This can be be edited if desired, e.g. to update new mask positions - it's in ~/code/vControl/WheelPresets.

Choose a mask from the Mask drop-down menu and press Move Wheel to move to the new position. This will adjust the wheel rotation position and the X and Y position. The wheel angle can be manually set (using the Set Degrees) and Nudged if desired. Various masks are listed, but for science only the following should be used:

• EmptySlot
• 7 hole
• 9 hole
• 18HoleNudged
• AnnulusNudged
• Mirror (convenient for pupil-viewing or blocking light)
• LyotStop
• EmptySlot2 (convenient when using Lyot stop for quick switches)

When moving the mask wheel (especially for the first time for a given mask on an observing run) you must look at Bramcam to make sure the mask is aligned properly. You may need to adjust Bramcam's exposure time / gain / stretch as per it's keyboard shortcuts (press F1 for help in Bramcam). If needed you can adjust the X and Y position using the Steer Im/Pup window as described subsequently. Make sure you keep the same positions for the whole run or it won't calibrate properly!

Steer Im/Pup

Hit the purple Steer Im/Pup button to launch the separate steering window. The Up/Down and Left/Right rows allow the NRM masks to be adjusted (whilst viewing Bramcam). Click the respective button to nudge it in that direction, or directly type a position (in Zaber steps) into the text boxes.

After an automatic mask move (e.g. when a new mask selected) press Refresh to update positions

The top Focus panel allows focusing of the two main cameras. Vampires has a global, absolute focus adjustment (the Abs+ / Abs- row) which moves the camera-lens assembly, and a differential focus (the Diff+ / Diff- row) which moves one camera with respect to the other for co-focusing. These focuses may be nudged with the buttons or an absolute value entered into the text box. The Abs focus is in units of mm, while the Diff focus is in Zaber steps.

Co-focusing is performed via a separate automated procedure and should already be correct - you shouldn't change this. Only change the Abs focus if a focus adjustment is required (e.g. when going between ver different wavelengths).

If something goes wrong and you restart vControl, make sure you press the Steer Im/Pup button again to launch the window. A stale copy of the window may still be open!

Other settings

• To change the filter, simply select it from the drop-down list in the Filter Wheel panel and press Move Filter Wheel.
• Camera cooling must be turned on, and should be set to -90 degrees. The temperature will be in red while it's cooling and turn green once it's settled. If it can't maintain -90, try -85.
• FLC temperature should be set to 45 degrees
• The quarter-wave-plate pair can be manually adjusted using the Quarter-wave Plates panel, but in regular use this is automatically adjusted based on the selected filter.
• The waveplate type in the Waveplates panel should be set to AO188_HWP.
• Soft Acq is not yet documented (TODO)

Remember - the exposure time setting for live video preview is not automatically the same as being used for data acquisition - double-check this!

Acquiring data

Once everything is set up to your satisfaction, you can go ahead and acquire data. Enter a filename in the filename box, usually the name of the target and a set number, e.g. alfOri_02. Date/time stamps and settings metadata are appended to the filename automatically. Data is saved to /data/YYYYMMDD/. Then enter the desired number of 'loops' in the Loop box. A loop consists of the 4 HWP positions. E.g. if you enter 10 in the `Loop' box, you will get 40 files (each a cube of the number of frames defined in Acquisition settings.

Then press Execute Series and data acquisition will begin!

Note that you want to press Execute Series to take data. The Execute Acquisition button only takes a single cube.

Once data acquisition has begun you must keep an eye on it, especially at the start. If not already there, you should see the quarter-wave plates rotate to their calibrated position for the chosen filter. In the console you will see the responses from AO188 confirming that the HWP has rotated - you should see it say things like OKAY hwp move 22.50. Keep an eye out for errors. You should see ongoing reports of acquisitions completing and writing to files. In the status bar of the vControl GUI you will see it counting up the frames for each cube.

Keep an eye on the console ('Command Window') and vControl status bar, looking out for errors! Make sure happy messages confirming completed acquisitions and successful HWP moves keep showing up. If they stop, either something has gone terribly wrong or your acquisition has finished.

The vControl background turns blue while acquisition is ongoing, and will return grey when it's done. But things can go wrong...

Acquisition errors / crash

Usually things go ok, but occasionally acquisition freezes mid-cube, believed to be due to an Andor buffer-overflow problem. This is currently being investigated, but for now there is a kludge (or two)...

An internal watchdog timer checks that new frames arrive in a reasonable time. If not, an error is thrown and the vControl background should turn a warning blood-red, and then an auto-restart sequence begins. This is very much a kludge, and it appends __RESTARTED__ to the filenames of the resumed series, so you know it happened.

However other things can possibly go wrong, including network problems or other failures which mean you won't see the error messages. A separate watchdog script can be run locally, described here.

In more serious errors, the cameras may stop working or not even initialise - see here for the solution.

Aborting an acquisition series

Due to the fact that the acquisition is largely at the mercy of the Arduino’s timing signals, aborting can be tricky. But it usually works. Press the red Abort! button, and wait. The abort flag stays set, and as each loop ends it will pass this flag up. Thus once it completes the current waveplate angle, etc, it should cleanly abort, and the grey background and 'Idle' status will reappear.

Once you’ve aborted, things may be unstable (the Andor camera gets quite upset at these things). If you're worried, restart vControl to be safe, before continuing observations. Restart using the Exit button on the lower-right corner.

Finishing observing

When you've finished, there's not much to do, except take a dark frame.

Dark Frames

A script is provided to check for unique combinations of exposure times and EM gain settings, which can be run on many files to determine which darks to take.

$ cd /mnt/data/20220224
$ find_dark_settings ABAur*.fits HD32297*.fits
Exp. time:  0.250 [s] /   250000 [us] EM Gain: 300
Exp. time:  0.500 [s] /   500000 [us] EM Gain: 300

To take darks, the simplest method is to put the Mirror into position in the pupil mask wheel to block all light, and then take a single series with the same integration times and EM Gains as used throughout the night.

Exiting vControl

When done, exit vControl by pressing the Exit button at the bottom of the vControl window. This will also close the camera's shutters, protecting them from bright daytime engineering.

Always exit vControl by pressing the `Exit' button at the bottom of the GUI window. Otherwise it will not shut down the cameras properly and could lead to damage.

Nominal State

There is a script to return VAMPIRES to its nominal state (as hard-coded into the script) that can be run at the end of the night or when things get weird.

$ vampires_nominal -h
Returns VAMPIRES to its nominal bench state.

1. Returns beamsplitter to polarizing cube
2. Returns differential wheel to Open/Open
3. Moves pupil wheel to "EmptySlot"
4. Moves focus to 16 mm
5. Removes any focal plane masks

Usage:
    vampires_nominal [-h | --help]

Options:
    -h --help      Display this help message