SHARC II Data Reduction Workshop

November 8, 2004

 

09:05   Colin Borys: Intoruduction

 

Welcome.  Agenda. 

Goals:

1.              Transfer expertise from Caltech to community

2.              Transfer expertise fro community to Caltech

3.              Learn about difficulties with hardware/software

4.              Improve data acquisition techniques

 

09:10   Darren Dowell: Brief update

 

1/3 of array died. Fixing it now,  Will be done soon. Nasymth move worked.

 

09:10   Colin Borys: Caltech success stories

 

SLUGS sources: SHARC @ CSO good at 350µm.  Sources at z<0.05; are bright enough to do as a backup project.  Takes 0.5h in tau~0.06, 1.0 in ta=0.08; sources are between 0.5 and 3 Jy.

High redshift Spitzer HLIRGs being observed (SoiferÕs Bootes field).  First one if 226±45mJy.

Min Yang looking at Stanford 0.1<z<1 galaxies; seen 20 of them so far.  Typical time of ~2h in best weather; <100mJy typ. Flux.

Darren Dowell has beautiful map of Fomalhaut disk; 150mJy/beam, 1.2Jy total.  Observed with tau=0.03 for 3h!  Publication in prep (Ken Karsh).  Used Ôcrush –deepÕ reduction.

Orion: 4h observation over almost 1¡ strip.  RMS of 300mJy to 1Jy/beam.  Observed with Box scans and mosaiced.  SCUBA took 18h to get similar map at 450µm.  Published by Martin Houde.

Low-z interacting galaxy survey: Antennae & Arp 299. Observed 14 of 42 sources; work by Jonathan Bird & Darren Dowell.

Deep field toward Abell 370 observed in Chopped mode with slow sweep.  10h in good weather (tau=0.044).  Two sources at ~80mJy detected, which were known from SCUBA.

Darek Lis is working on Galactic molecular clouds.  SgrB2, MonR2, W51, S68, HH24, Box scans for 0.5h in good weather; 50Jy sources.

Attila Kovacs: high-z SCUBA sources. Detected Lockman Hole sources with 3-4h at tay-0.06 to yield rms=5Mjy.  LE12, LE21, LE31, LE2 all detected.

Other things: Moon image, Saturn, Titan, E-Eridani.  Titan is a poor calibrator!  SgrA*, Horsehead. Nebulae.

 

09:25   Colin Borys: Software and Requirements

 

CRUSH by Attila Kovacs – supported

SHARCSolve by Darren Dowell – trying to phase out

Ancillary stuff by Colin Borys. – Some being phased out for upcoming ImageTool by Attila Kovacs.

            Ancillary items:

                                                Header_update: alters header keywords

            Boxscan: calculates box_scan parameters

            SHARCCal:                applies cal to reduced map; can take out offsets.

            SHARCGap:               tests file for timing gaps.

            SHARCSmooth:         Performs a PSF fit to reduced SHARC2 map.

            SHARCLog:               gets info out of headers

            SHARCStat:               computes statistics on reduced maps.

            SHARCTau:               Uses Jon BirdÕs tau fits to estimate tau; requires taufit file

Presentation: IDL, GRAPHIC, GAIA, DS9.

Software installation: set up CRUSH directory; install utilities anywhere.

For convenience with CRUSH, use the Ô-outpath=Õ argument to move output files; have script move into CRUSHÕs directory.

For maintaining log files, use Ò>!Ó to pipe file out.  CRUSH can now do ranges (e.g., 10417-10426).

Publishable images: IDL AstroLib routines.  Colin Borys has a program to do SHARC plotting.

GRAPHIC is available for many platforms; also produces publication-quality images.

 

09:40   Attila Kovacs: CRUSH reduction

 

Overview: how data reduction should work.  How CRUSH works internally; CRUSH options.  How to handle data.

Why submm is so difficult: Sky us around 100,000Jy; sky noise is 100-1000Jy; faint things are only 0.01Jy.  Dynamic range is thus 107 – analogous to 16th mag star in the day!

Atmospheric power variation is very 1/f, turning over at 10Hz or so!

Signals must be differenced; Lissajous sweep or chopping.  Simulated chopped noise is 2-3 times higher than for Lissajous, without deconvolution noise!  Chopped data is not fully understood; actual measurements seem to match to within 30% (according to Darren).

Sweep speed: want optimal interconnections to calibrate pixels against each other; calibrate as many as possible.  Should not be periodic.

Reduction by Singular Value Decomposition, a mathematically rigorous maximum entropy solution.

SVD is difficult because itÕs computationally costly, isnÕt good with nonlinearities, has degeneracis/singularities; time=dependent noise

Parallel effort at GSFC produced some results.

CRUSH approach: SVD by iteration.  At each stage, assume one thing dominates; make an estimate by maximum likelihood; remove it; go to next thing.  It does converge, and has some advantages: itÕs intuitive in approach; is fast (linear with data size); can deal with nonlinearities; is easy to manipulate/change.

Example: Pixel offsets fitted, then Source model,then row drifts, then detector 1/f, then acceleration response.

Models in order of decreasing typical brightness: pixel offsets, correlated noise, gain model, pixel weights, chopping residuals, temperature graditents, row drifts, 1/f drifts, time weights, residual spikes, regional correlations, acceleration, temporal, spectral.

Preliminary simulations showed that this iterative approach does converge.

Recent simulations by Tom Tyranowksi showed that cleaning would work to recover very faint sources.  However, it was difficult to clean ripples that are large compared to array size.  This comes from certain parameters: temperature gradients, row drifts, and 1/f drifts; gain modeling affects them too.

Gain fitting and correlated noise are difficult.  Linear and nonlinear responses.  Overall gain drift is around 30% over full range.

Weighting by both pixel-only and time-only.

Anomalous behavior for pixels is important; can have strange noise, pickup.  Identified in time-stream and FFT space; looking for unreasonable fits.

Deep images of Lockman Hole show it is possible to get good performance.  Map residual histogram is very Gaussian up to about 10sigma (may be source flux).  However, noise is 2x wider than expected from statistically independent pixel noise.

Looking at pixel-pixel correlations, the covariance shows that pixels within around a 35ÕÕ FWHM Gaussian are correlated.

 

10:10   Attila Kovacs: CRUSH options

 

Three categories: reduction options (change pipeline), scan options, and model options.

All options in Òcrush –helpÓ or on web.

Principal reduction options:

            Brightness-related:      -faint and –deep

            Size-related:                 -compact

Scan-specific options:

            Average:                      downsamples data

            Chopped:                     if you chopped

            Scale:                           multiplies scans by some factor (e.g. Ò=scale=1.23 13852 13853)

            Pointing                       FAZO/FZAO adjustments

            Tau:                             using network server is optimal; can be overridden (e.g., Òtau=225GHz:0.046 15224Ó)

Generic options:

            Activation iteration      turns on model at some iteration

            Time constant              what time period applies to many models.  Large is robust, while small is aggressive.

Source model options:

            Fidelity                        when the fainter-than-source models are allowed

            Extended                     For large scale structures, it tried not to remove models that could look like extended structure

            Exposure                     clipping edges

            Convolve                     Makes reasonable pixel size.

Gain fitting and adjustment:

            GainRounds:               number of iterations

            GainGoal:                    convergence criterion.  For faint sources, needs better convergence.

            Opacity adjustment:     real-time tau fitting.

Weighting/flagging:

            Weight Time:               should be smaller than beam crossing time – donÕt want real structure!

            Degrees of Freedom:   if not enough DOF, pixel will be bad.

Output options:

            Outpath, name:            file locations

            Precess:                       moves epoch

            Resolution:                  map pixellization.  Default is 1.62ÕÕ.

Tips:

            Reduce all data together

            Reduce data at different scan directions

            If reduction is weird:

                                                Use different brightness

                                                Change gain convergence

                                                Use –compact

                                                Adjust timescales

                                                Try to identify problematic scans

            Trade extended flux for flatter baselines

            Check pointing

            Use Mai-Tau

Observing tips:

            Use proper scanning pattern

            Point often!

            Calibrate often!

            Use DSOS

            Keep logs!

 

10:30   Attila Kovacs:             Data Structure

 

What formulae you want to use.

            Using FV to look at FITS files will help.  Investigate scans to see contents:

                                                Primary image

                                                Coverage

                                                RMS

                                                Signal/Noise

                                                Single scan data; histograms etc.

Primary image: flux value I at each point is the ÒMeasurement FluxÓ as would be seen by detector with no atmosphere.  Natural units are nV.  Calibrate by response to known source to produce pseudo-flux units such as Jy/beam.

Aperture fluxes:

            Flux = Apix/(4.85Ó*4.77Ó)*sum(pixels) [Jy/I]

            Flux=1/9 *sum(pixels) [Jy/I] with default pixellization

Peak fluxes: good is faint

            Flux=Abolo/Abeam*I [Jy/I]

                                                =4.85*4.77/2¹/s^2      s=8.5ÕÕ*FWHM/2.35

Smoothing with beam:

            Typical smoothing is Gaussian with beam.

RMS images:

            Measurement uncertainty in map.  For default pixels & Gaussian, just simple equation: sigma in map is 1.83*FWHM*sigma

            Flux uncertainty in an aperture is 1/9 * sum of sigma^2.

            Excess noise: factor is (1+C) higher than expected.  Is sum of correlated pixel noise, and C~1 typically.

CRUSH suite of utilities:

            ImageTool:                  image view/manipulation

            Show:                          image view portion of ImageTool

            CoAdd:                       adds maps

            Jiggle:                          Allows you to shift-on-the-fly

            CoVarSee:                   Visualization for covariance matrices

            Histogram:                  useful for determining excess noise

            DeConvolve:               superresolution

 

(Break)

 

11:15   Jon Bird: Calibration

 

Tau measured by tippers at 225GHz and 350µm every 10 minutes.  ThereÕs a lot of noise in the measurements.  Least-squares fitting smooths out the tau measurements; available online.

Fit is in UT fraction of day; typically between 2h and 20h UT, and covers most observations.

Fits are NOT always reliable: e.g., 2004-09-25 shows difference between 350µm and 225GHz.  Need to choose portion of UT day when you believe tau fits

Mai-Tau server parses fit table & produces fitted tau value.  Uses 350µm by default.

Calibration: ÒknownÓ fluxes from HORIZONS database; want to convert CRUSH nV output into Jy.

Aperture is FWHM+20ÕÕ radius.  Aperture size not important, but should be the same for all cases (same for all calibrators & science target).

Known sources: Mars, Uranus, Neptune, Callisto, Ceres, Vesta, Pallas, Ganymede.

In 2004 Aug/Sep run, calibration came to nearly the same value (21.3% for all sources over time).

Tau fitting is good and getting better – gets to 20% absolute!

 

11:35   Colin Borys: Calibrators

 

Availability of sources is challenging.  Blazars canÕt be used.  Solar system objects are only good ones, but are variable in a predictable way.

Primary calibrators are bootstrapped the secondary calibrators.

List of many calibrators is available on the web.

ThereÕs a UIP catalog for use when observing with SHARC2: please use these!

PSF calibration: PSF photometry is most commonly used for point source extraction.  Equivalent to Òconvolving with the beamÓ.  SHARCSmooth does this.  Answers the question: Òwhat is the best fit amplitude of a Gaussian at the center pixel?Ó

Aperture photometry: different from Attila.  Use a radius for source, and an annulus for sky estimation. Important caveat: when convolving, RMS will be lower than itÕs supposed to be!

What do you use: point sources are best with aperture photometry; extended sources use PSF photometry.  For deep integrations, correlated sky signal is still in the map!

How-to of calibrating data: get good estimates of atmosphere, a decent collection of calibrators, and a choice of flux extraction.

Example: ULIRG reduction

            MaiTau shows good tau fit to the time of observation

            Use CRUSH with faint and compact with no convolving

            Then SHARCSmooth to produce a good smoothing.

            Use a calibrator in the same way.

            Use GAIA to find Signal and RMS maps.  Brightest pixel is 1.80 ± 0.02 Jy.

Calibration uncertainty is probably 10%.

 

11:50   Darren Dowell: Tweaks

 

Pointing Correction: -FAZO and –FZAO can adjust prior to reduction

Header_Update utility can be used afterwards, e.g. before CoAdding.

Pointing residuals (T-Terms) are at about 2.1Ó FAZO.  Can be adjusted per night.

Hiroko has been studying temperature variations: a whole beam for different FAZO vs. ZA.

DarrenÕs crop tweak uses pushd and imagetool and popd.  Use –out option to avoid overwriting file.

MosaicsÉ leave for later.

 

(Lunch)

 

1:10     Colin Borys: user questions

 

Velu: working with resolution for Spitzer.  Has done Orion, Fomalhaut.  Deconvolution is good!  Tested Lucy algorithm with modification (by IPAC for IRAS).  Takes advantage of oversampling, using the final output of CRUSH and the PSF to deconvolve.  Runs as a standalone.

Ken: Can it be done as a part of CRUSH?  Need to know the PSF well.

Attila: PSF is small, but variable.  Depends on weather, elevation.

Dominic: We get 8.6±0.45ÕÕ for a set of calibrations.

Ken: Fomalhaut PSF is estimated from CRUSHÉ

Attila: Smoothing is only 4ÕÕ by default, more in deep mode.  Probably want to turn off convolution.

 

Jingwen: CRUSH reduction of a galaxy (RXJ094144).  SHARCSolve finds the source, CRUSH does not!

Attila: Apologies. V1.34 has a lot of time constant options tweaked for certain maps, and turned out to be too long for certain other projects.  For Lockman Hole, reducing the time constants – especially drift to ~100 – helped.  When you donÕt see a source you know is there, there must be a model absorbing it, such as 2D gradients, rows, and drifts.  HavenÕt done much chopping data, so CRUSH doesnÕt fully replace SHARCSolve.

 

Alexandre Beelen: We integrated for 7h on a faint object, but it can no longer be reduced by CRUSH without out-of-memory problems.  ItÕs 2GB of data.  With early versions, there was a big field curvature.  In 1.3b2, it was flatter but fainter?

 

Eric Murphy:  Variety of map sizes made; can CRUSH reduce without edge effects?

Attila: Yes, but there was a problem in CRUSH that forgot to downweight edges properly.

Colin: Lots of SINGS and similar sources. 

Attila: Always a tradeoff: flat maps or large-scale variations?

Darek Lis: Is that intrinsic to data acqiusition, or something else?

Attila: Limiting factor is speed of sky variation.  How fast can you sweep in 1/f time of sky, which is 100ms.

 

Dominic: We have gotten good images on extended sources, with some negatives.

Attile: ThatÕs because of poor gain fitting.

Dominic: Weird wiggles (worms).  Where from?

Colin: Try one pass with convolve=-1.  This traces out the one really bad pixel.  Want to find & remove.

Eric: Can you mark up a file with bad pixels?

Colin: Yes; CRUSH can write out pixel gains file & use iteratively.

Attila: Ò-gainsÓ option.

Colin: we need to make a tool for this or change in new versions.

Dominic: M51 has major baseline problems

Attila: Try compact flag for some things.

Darek: Should try the Ring source simulation with that.

Colin: Could we release the Simulator to the public?

Attila: ItÕs really a good idea to find a single characteristically-bad scan and fix that one, then go through full reduction.

Dominic: Nonconvergence problems later in iterations.

Attila: ThereÕs a divide-by-zero problem that produces infinities.  ThereÕs an option in 1.33+ with a –debug option that checks models for infinities.  These are software bugs.

Colin: Things that seem to fail to converge are when sky is highly variable.

Attila: ThereÕs a time constant issue as well.  Sky is not stable for an hour, so different scans may be very different.

Colin: For SLUGS, would remove one scan at a time until it works.

Attila: Reduce a basket of representative scans to see what models correlate with background noise, which is the biggest thing to fit.  It is sky variability that really screws things up.

Dominic: JohannesÕ question about reducing SgrA* with highly varying taus.  SHARCSolve doesnÕt handle well, and CRUSH removes extended structures.

Attila: Try using a version after 1.3 – extended sources are more preserved.

Darren: Need to understand how the data is reduced & mosaiced.

 

Kristen: SXDF map has several negative and positive bumps.  SCUBA tells you where to look, and you donÕt get the sources in the right places.

Attila: Previous CRUSH versions have had trouble with this.  New version better,

Colin: Try reducing individual scans, and throw away really bad ones.

 

 

(Break)

 

03:20   Alexander van Engelen

 

Working on SCUBA II algorithms, specifically for scan maps.

Atmosphere is hardest part to work with; simulations being developed using a model based on SHARC2 data.

Model is Kolmogorov spectrum fluid dynamics model.  Emission is 2D screen at 800m altitude, Gaussian random screen blowing past observatory at constant wind speed.  Effective speed is 5000ÕÕ/sec = 15m/s.

SCUBA II raster scan at an angle to fill in array (450µm array is undersampled).

Reprojected source+atmosphere gives a source with big streaks from atmosphere.

Questions remain about assumed structure, power spectrum, motion.  Is array FOV too big to assume constant atmosphere?

SHARC2 data on MS0451 field, using Lissajous scan.  Data is overwhelmingly common-mode.  Fit linear gains only on common-mode signal.

Model predicted a Kolmogorov Pµ1+(w0/w)(8/3) spectrum.  SHARC2 data is pretty close.

Animation on uncalibrated data with modeled screen.  Animated gif made by nifty C code.

After removing common-mode signal, thereÕs a lot of structure thatÕs mostly instrumental.  However, thereÕs still a strong correlation with nearby pixels, which appears only when sky is not opaque.

Look at derivatives in signal to find slope of fit plane across the array; positive correlation would indicate a confirmation of Kolmogorov spectrum.  Unfortunately, SHARC2 drift overhwlems – fitting is too difficult.

Since data is so common-mode, a simple mean should be sufficient even for SCUBA-II. Some small-amplitude correlated structure remains.

 

03:35   More user concerns

 

Attila: histogram of points in Lockman Hole field.  Scaled to sigma, it makes sense.  You can trust anything about 3.5sigma (for small fields) to be real, not statistical occurrence of faux sources.

Attila: NGC891 scan with new reduction improves mosaic, but lissajous scan pattern was nonoptimal for this source.

 

03:50   Colin: summary of issues

 

Alexandre BeelenÕs high-z quasar issue: new version of CRUSH (1.34) runs out of memory.  Also, different versions of CRUSH changes negative bowling around sources.

            Answers: break data in half, reduce, coadd.

EricÕs galaxy: observing mode was poor, and coadding was problematic.  Maps look good with new CRUSH/coadd.

Kristen: high-z galaxy survey with false negative/positive bumps.  Posited that this is from bad pixels.  ColinÕs solution: use Ò-convolve=-1Ó to look at individual scans to find bad pixels.  New version may fix.

Failure to converge / NaN being worked.  One bad scan usually the culprit.  Attila wants Ò-debugÓ output for bad scans.

Jingwen: chopped observation works in SHARCSolve but not CRUSH.  Attila working this out.  L1014 has 4 sets of scans, each of which can be reduced individually but with drifting fluxes.  Tau changed during observation, so using taufits helps.

Dominic: baseline fitting?

Attila: removed from CRUSH, can be put in imagetool.

Neal Evans: Is it known where this comes from?

Attila: Combination of gain problems.  Measure illumination vs. beam efficiency improperly, so residual bright sources remain.  Proposed test to figure this out: scan a point source across the array at different elevations to figure out how its brightness correlates with sky response.

 

04:05   Darren: Prosaic Things

 

Lessons Learned:

            DonÕt use extended sources for focusing (e.g., NGC2071, IRAS16293-2422, Jupiter, Venus).  Use small things (Mars, Uranus, Neptune, Callisto, Ganymede, Ceres, Vesta, Pallas, Crl618, CRL2688, IRC10216, OH231.8, Arp220).

            DonÕt point on NGC2071 (use CRL618, HLTau, OH231.8 instead)

            DonÕt calibrate on NGC2071, blazers.

            DonÕt use Titan since Saturn is in sidelobes.

            For big fields, map full area in as short a time as possible.  Mosaics tend to have stitches.

Difficult projects:

            High dynamic range observations (negative artifacts and sidelobes are both at 4% level)

            Faint, widely extended emission

            Integrating total emission.  Must use same procedure for source & calibrator, using sky aperture.

Tips:

            Use DSOS; check that itÕs working (monitor screen); allow settling time for ZA slews; turn off at end of night.

            Check source catalogs before going to CSO.

            Use CAL_* sources for Extrasolar calibrators.

            Check pointing/focus every 45 minutes until 8pm; then check pointing every hour and focus every 2 hours thereafter.

            Report every problem, even minor ones.

New things (Colin):

            DSOS is tricky to use, but very useful (especially at high elevations)

            Now on Nasmyth.  Practically, means that SHARC will be always nearly-on and can be started up quickly.  Thus, SHARC/Bolocam will be flexibly scheduled.

            Scanning a bit more complicated (Attila); array rotates on sky.  Therefore, can accidentally scan along the array when close to zenith & scanning in AltAz.

Dominic: what is subscription rate like for SHARC?  Large projects?

            Colin: last semester was low.  Looks high now.  About 2/3 got time.  SHADES is a big project, but is only 8 nights.  Bolocam asked for 80 nights.  May be possible that more observing time in the future will be available, and other groups may collaborate.

Colin: CRUSH exploder may be formed for SHARC data reduction purposes.

Colin: Thanks for coming!