2.12-b1 (2012 May 15) Overview ===================================================================== This release brings CRUSH up-to-date with GISMO's 2012 April run, with new features, improvements and a current configuration for the finalized state of the instrument. There are also fixes and improvements for the function of other instruments, and to CRUSH in general. This is still a test (beta) release. As such, some bugs may exist, and this and that can change. Please report any issues and bugs that you may encounter. Thanks -- A. ===================================================================== [NEW] {since-a1} Added some new color schemes to 'show' (hint: you can cycle through the colors by pressing 'k' or 'K' (for reverse cycle) in the 'show' display', while pressing 'h' brings up a help screen with all the functions!). [NEW] {since-a1} {GISMO} Ability to set options based on IRAM scan IDs. These work just like the 'serial' options, but use the IRAM specific format of scan IDs that include the date ('YYYY-MM-DD.nnn'). E.g. id.[2012-04-11.132:2012-04-13.26] jansky 18.6 sets the conversion factor (counts/Jy) to 18.6 starting scan #136 on 2012-04-11 until (but not including!) scan #26 on 2012-04-13. [NEW] {since a1} Gain changes are immediately applied to all signals that rely on the same set of gains. This adds a refinement on the removal of gradients (all arrays), the LABOCA twisting mode or the GISMO wiring-trace 'flip' mode. [NEW] Pointing reductions now measure the source shape in a more sophisticated way, providing a measure of the x and y asymmetry of the source (and lobes!). This, in principle allows focusing on a single image of a point source, provided that it is characterized how the source shape changes with focus settings. The conversion of shape parameters (asymmetry and elongation) into focus changes is controlled by the new configuration keys 'focus.xcoeff', 'focus.ycoeff' and 'focus.zcoeff'. The definition of these is still in progress. [NEW] Pipeline elements can be activated/deactivated scan-by-scan, and conditionally induced by date, MJD or serial number. As before, global options should precede the listing of scans. [NEW] {GISMO} Two new pipeline features have been added, to help deal with the observed jumps in detector levels (probably due to E-M pickup on the SQUID muxes). The first of these is the addition of a 'flips' modalility, that can be decorrelated. These model signals on muxes with alternating gains on pairs of pixels. The model is preliminary and awaits confirmation from Dominic Benford who designed the wiring traces that give rise to these current flips. The second new feature is 'dejump', which looks for sudden transient changes in the readout levels (somewhat similarly to how 'despike' works). The critical level can be set by the 'level' subkey, as a relative noise threshold (>1.0) while the length of jumps that are shorter than the 'minlength' setting (in seconds) will be flagged, whereas longer jumps will be re-levelled and kept. [NEW] {GISMO} Bias-dependent settings are now allowed via conditionals under the 'bias' key. E.g. to define different conversion values (from counts to jansky): bias.[8000] jansky 36.6 bias.[10000] jansky 24.67 [NEW] New logging entries for source elongation and angle, as well as projected elongation in the telescope x direction. See README for details. [NEW] 'drifts' can now take the value 'max', which fits just a single offset value for the timestream of a detector channel. As such the setting is identical in functionality to 'offsets'. [CHANGE] {since a1} Changes to keystrokes in 'show': 'K' now works with 'k' to toggle colorschemes back and forth, while convolving to beam is performed by 'Q' (paired with devoncolution 'q'). [CHANGE] Naming of focus quantities for logging has been changed. See the logging section of the README for details. [CHANGE] Map redundancy check moved from scan to compound map. [FIX] {GISMO} All pointing information in FITS is now correctly interpreted (or so I hope). Looks like Nasmyth offsets in CRUSH model are inverted w.r.t. the IRAM constants. it works, so probably leave it as is... [FIX] All HDUs in the output image FITS now contain the full header information as expected. [FIX] Found reason why filters did not work together with time weighting. The problem was that MultiFilter did not propagate the number of data points to its children, which ended up with infinite frame dependents as a result. The fix includes checking to make sure that dependents are valid, and propagating the number of data points to children. [FIX] {GISMO} Focus and Nasmyth offsets were not parsed correctly. (Was looking for these values in the wrong HDU.) [FIX] Filter levelling was not taking frame weights into account. [FIX] {GISMO} Fixed treating horizontal and equatorial (projection) offsets. [FIX] {GISMO} Suggested Nasmyth pointing offsets sign fixed and confirmed. [FIX] Fixed and cleaned up rendering transformations and zooming of various image classes used by CRUSH to make parts work that weren't really used before. These will eventually provide the imaging for the upcoming native crush-2 'show' tool. (We are still using the CRUSH-1.xx 'show' and other tools). [FIX] PNG centering: previously, PNG images were centered on the nominal tracking center, not the image center, sometimes resulting in images that were shifted to one side and part of it cut off on the other side. From now, the PNG images always fill the allotted image area. [FIX] The results of pointing reductions (via 'point') were ever so slightly incorrect in reporting. (The fix does not change reductions, only the way the pointing fit to the source is performed/analyzed.) [FIX] PolarVector2D power op was wrongly defined. However, since this feature was never used in CRUSH, there was no damage done by it. [TWEAK] {since-a1} {GISMO} Refined the way de-jumping counts its parameters dependecies (used for obtaining proper noise weights). [TWEAK] {since-a1} De-jumping produced unintended time-weighting residues. Now, at the end of the de-jumping algorithm, the previous time weights are reinstated, except in the re-levelled regions, where uniform weights are presumed (until more correct weights are assigned). [TWEAK] {since-a1} Decorrelation of sky gradients now properly includes the measured common-mode gains. (Essentially a Spinoff of the common mode). [TWEAK] {since-a1} 'flips' gains linked to 'mux' gains and are applied on top of those, without independent fitting. [TWEAK] Time-weighting renormalization adjusted. [TWEAK] Uncertainty calculation of elongation measure improved. [CONFIG] {since-a1} {GISMO} Recalibrated pre-2012 configs for 17.5" FWHM beam (previous 16.7" was assumed but was quite certainly too low!). The new calibration is simply a rescaled version of the old calibration for the different beam size. [CONFIG] {since-a1} Reenabled MUX gain modelling. It is necessary to reach white-noise limit. [CONFIG] Finalized pointing model, and pointing log (for incremental corrections) for the 2012 April run. The model is good to ~3" rms in both directions for the entire run, while the incremental corrections should further reduce the pointing uncertainty to ~1--2" rms. [CONFIG] {GISMO} Updated 'deep' configuration to run for fewer (12) iterations. Nevertheless, the best deep maps are usually obtained with 25--30 iterations. Set 'rounds' manually if more iterations are desired. [CONFIG] {GISMO} Gradients re-enabled in 'deep' mode. LSS filter cannot completely replace gradients because of 'edge' effects... [CONFIG] {GISMO} tau(225GHz) to PWV conversion changed to match that on IRAM's weather page (ATM07). [CONFIG] Updated 'extended' configuration based on AtlasGal. [CONFIG] {GISMO} Added pointing model for the main part of the April 2012 run. [CONFIG] {GISMO} Finalized RCP and pixel data for April 2012 run. [CONFIG] {GISMO} Finalized conversion factors (for bias settings 8000, 10000, and 12000) based on Mars and Neptune. Good to ~1% or better. [CONFIG] {GISMO} Finalized pointing models for the 2012 April run. The model gives a blind pointing to 3" rms accuracy in both directions for the entire duration of the run. On top of that incremental pointing corrections are applied based on nearby pointing sources, s.t. the final pointing accuracy of CRUSH for the 2012 April run ought to be around 1--2" rms or better. [CODE] {since-a1} Cleaned up access to associated instrument properties in source model. [CODE] {since-a1} Added equals() for Mode to go with its hasCode(). [CODE] {since-a1} Modes that share gains with other correlated modes are now Spinoff modes of the parent gain provider mode. Also, new code allows for more flexibility in specifying mode gains. [CODE] Units completely revamped, with smarter parsing & compounding. Better integration with Data2D and derived classes. [CODE] Data2D and children now use Compound units and dynamically evaluated 'jansky', 'beam', and 'pixel' units. This makes life a whole lot easier when doing operations like smoothing/regridding... [CODE] Gismo.copy() updated to correctly deal with fields. [CODE] Many classes are updated to hide private fields, and access them through getters/setters only. The change reduces the propensity for nasty bugs. [CODE] More work on imaging classes. The day of native CRUSH-2 image tools is getting ever closer... 2.11-2 (16 March 2012) Overview ====================================================================== This is a maintenance update of CRUSH 2.11, fixing the broken 'beammap' function, and fixing many smaller issues of the previous release. Some improvements (esp. for GISMO) are included also. ====================================================================== [BUG] {beammap} Beammap function did not work. Now fixed. [NEW] The 'poll' command now recognises and expands aliases. [NEW] 'write.png.plane' can be used to select which image plane is dispalyed in the PNG image. The argument can be one of 'flux', 'noise' 'weight', 'time', or 's2n'. The default value is 'flux'. [NEW] 'write.png.bg' now accepts 'transparent' as an argument. [NEW] 'write.png' now enabled by default, s.t. a PNG preview image is generated with every FITS file. This makes it easier to 'browse' crush images. The default size is 500x500 pixels, the color scheme is 'colorful' and background is 'transparent'. These settings can be overwritten by the various sub-options of 'write.png'. Also, to provide a high quality preview, the preview images are always smoothed by 1/2 beam (resulting in just 12% degradation in resolution), even if the FITS image itself is not smoothed, and are presented as S/N, to de-emphasise the noisy edges. [NEW] {GISMO} Now handling scans with exotic coordinate basis (e.g. ecliptic, galactic or supergalactic frame). [NEW] {GIDMO} Now using PNTROW, and PNTCOL keywords to determine the pointing center on the array, allowing, in principle, to center the source anywhere on the array. The changing of the pointing center does not affect the pointing model, which is referenced to the geometric array center (but this needs to be tested, though!!!). [NEW] {GISMO} Improved conformance to IRAM FITS files: (1) support for raw scanning offsets (not 'True' angles); (2) parsing of static scanning offsets, and the nasmyth offset (set by user in PAKO); (3) Now using telescope location from FITS (having had confirmed that these agree with Juan's values). (4) Nasmyth offset incorporated into pointing (but it does not actually center source on array!) [CHANGE] {GISMO} 'pointing.log' key renamed to 'pointing.table'. [CHANGE] {GISMO} 'dataname.end' now accepts a comma-separated list of filename endings, which are tried in order until a match is found. [FIX] {GISMO} 'rotation' key had no effect. Fixed. [FIX] Long values to full length (8 character) standard FITS keywords resulted in a exception. Now fixed. [FIX] Date-dependent configurations sometimes left an unconfigured day between connecting intervals. In principle, the end date of one interval should match exactly the starting date of the next interval since both are interpreted as 00:00:00 UT, when the time is not specified explicitly. E.g.: date.[2012.03.01--2012.04.01] march2012 date.[2012.04.01--2012.05.01] april2012 [FIX] {GISMO} Scans in equatorial coordinates could not be processed before because the rotation of scanning offsets into native horizontal offsets preceded the prerequisite calculation of the parallactic angle. Fixed. [FIX] {GISMO} Tracking errors were missing from scanning positions! (Also, in the old version these errors were assumed with the wrong sign. Untypically, the values are commanded minus actual.) [FIX] {GISMO} Version-specific conditionals were not parsed as advertised. Now fixed. [FIX] {GISMO} Automatic merge version detection failed with old FITS format (ver 1.6) unless explicitly set by the 'ver' option. [FIX] {APEX} Photometry formatting of negative fluxes always produced results in uJy/beam. Fixed. [TWEAK] 'write.png' now smoothes thumbnail image by half a beam (12% degradation of resolution) and presents S/N to give the preview image a higher quality visual appearance. [TWEAK] {GISMO} Pointing model expanded to 16 constants, with terms dependent on 2AZ, as well as refraction correction. [TWEAK] Slight speedup of scan reading, by skipping processing of FITS rows with no telescope data (or if invalid otherwise). [TWEAK] {GISMO} The time window for high quality incremental pointing correction has been expanded to 1 hour. This mitigates the effect of a single pointing datum with anomalous refraction. [TWEAK] {GISMO} Sign convention of GISMO Nasmyth offsets made to conform with IRAM's convention, both for the pointing model and the suggested offsets. Note, however, that Juan's poiting fit gives a V that has an opposite sign compared to P11. [TWEAK] {GISMO} If (for wahetever reason) the FITS merge version does not appear in the FITS file, assume the latest version possible... [TWEAK] 'faint' and 'deep' no longer switch to 'absolute' despiking after the first iteration. This is to prevent over-zelous despiking. [TWEAK] Source extraction via 'detect' now offers two types of false detection rates (one based on purely Gaussian statistics, and one with guesstimate corrections for actual distribution of sources/negatives). In its output it also offers some explanation as to what these values and confidence percentages mean. [TWEAK] Source extraction 'detect' now does a cleanup round, to ensure no additional sources can be found inside the filter bowls of the last successful extraction round. [TWEAK] {GISMO} Using a new synthetic RCP, based on the measured one on Neptune, with positions for all pixels, not only the ones detecting during the beammap. [CODE] Configurator.Entry made into a proper inner class. [UPDATE] {GISMO} Updated static pointing model for new FITS. merge. Should be good to 2.5" rms in both directions, for the April 2011 run. [UPDATE] {GISMO} Run 5 (starting 2012-03-03) configuration added. However, with the issues we were experiencing with M5 and M6, there is not much use for this configuration... 2.11-1 (16 Feb 2012) Overview =================================================================== This is a major update of CRUSH. It fixes a but on reporting pointing offsets for SHARC-2, provides much better support for PolKa, based on the Dec 2011 run. It also fixes various other issues, and offers improvements (esp. for GISMO). Under the hood, a lot of code was restructured for cleaner and faster operation. =================================================================== [BUG] {SHARC-2} suggested FZAO pointing offsets were wrongly calculated (sign error). Fixed. [BUG] Crossing times were not correctly calculated (returned size, not actual crossing time...). Fixed. [NEW] Reductions in moving frames (e.g. planets, Moons). The option 'planetary' can be used to specify moving frames, if these aren't automatically detected based on the scan headers. [NEW] Introducing a parallel processing framework for images. The goal is to speed up image processing functionality on modern multicore systems. Most image operations are now utilizing all available CPU power. This will result in a small gain for reduction purposes, but should make the upcoming new 'show', 'imagetool' (etc.) more responsive, especially when handling large images... [NEW] {GISMO} Implemented new merged data reading. The new FITS files (ver. 1.7) correspond more closely with the original IRAM data, and thus hopefully fix the pointing problems we experienced earlier. [NEW] {PolKa} New loggable quantities 'analyzer', 'wpdelay', 'wpdir', 'wpok'. See the logging section of the README for details. [NEW] {PolKa} A map of the polarization angles can now be produced also via the 'source.polar.angles' option or the 'spa' shorthand. [NEW] {PolKa} Now taking into account the non-sinusoidal modulation of polarized light resulting from the oblique incidence on the waveplate. The incidence angle can be set by 'waveplate.incidence', while the direction of the incidence is set by 'waveplate.incidence.phase'. [NEW] {PolKa} The angles (in the horizontal frame of the C-cabin) of the analyzer grids can now be set separately via 'analyzer.v' and 'analyzer.h'. [NEW] {PolKa} A workaround for missing waveplate data during the 2011 December run. Waveplate angles can be reconstructed using the total- power modulation of a specified channel ('waveplate.tpchannel') using a specified harmonic of the waveplate frequency ('waveplate.tpharmonic'), and using the phases 'analyzer.h.phase' and 'analyzer.v.phase' (which can be measured via 'waveplate.tpchar' when the waveplate data is there). [NEW] {PolKa} As of 2011 Dec, the time offset to the last available waveplate crossing is stored in one of the unused LABOCA channels. The option 'waveplate.tchannel' can be used to specify which channel that is, allowing a full reconctruction of the waveplate crossings. [NEW] {PolKa} There is still a problem with the accuracy of the waveplate timestamps, which impacts how accurately the waveplate angles are reconstructed. Luckily, the physical rotation of the waveplate seems extremely uniform. Thus, one can use 'waveplate.regulate' to use a linear fit to the timestamps rather than the inaccurate timestamps themselves. The quality of the fit can be improved by despiking via 'waveplate.despike'. [NEW] {PolKa} Occasionally, the waveplate restarts in the wrong direction. If this is properly diagnosed, the data can be reduced correctly with the 'waveplate.counter' option. [NEW] {PolKa} Pipeline step 'purify' can now be added to the ordering of possible operations to remove the total-power modulation at any stage of the reduction. Note, that whether or not this option is used the total-power modulation is always removed before constructing Q and U maps... [NEW] {PolKa} Reimplemented Fourier half-wave-plate (HWP) filter capability for removing total power modulation when the waveplate angle is both jittery and projected. The filter can be activated by 'filter.hwp'. As before, the jitter can be set by 'waveplate.jitter', while the number of harmonics filtered is set by 'filter.hwp.harmonics'. The filter may be useful for early PolKa data, where the stored waveplate angles were flaky. However, when the angles are stored correctly, 'purify' is the preferred method for removing the total-power modulation. [NEW] {PolKa} An image with the polarization angles (A) can be produced using the 'source.polar.angles' option, or with the shorthand 'spa'. [NEW] {PolKa} If analyzer is not in place (total-power mode) then polka reductions revert back to pseudo-laboca mode. This is similar but not the same as normal laboca reductions, since the configurations are modified by polka. The main purpose of the new feature is to allow batch reductions of scans (e.g. for logging) without prior knowledge of whether or not polarization or total-power mode was observed. [NEW] {PolKa} Check to make sure that polarized scans (with horizontal or vertical analyzer in place) are not reduced together with total- power scans (without analyzer). When both type of scans are submitted the first scan determines the reduction mode (polarization or total- power), and subsequent incompatible scans are discarded. [NEW] Ranges now accept '<' and '>' to specify open-ended ranges. E.g The following ways of specifying ranges are all acceptable: <456.0 >3.1415 * *:456.0 3.1415:* 3.1415:456.0 3.1415-456.0 (positive only!) [NEW] {GISMO} It is now possible to set options for GISMO based on the FITS merge version (currently 1.7), using the 'ver' key and a range specification in the conditional. E.g. ver.[<1.7] pointing.model=old.model ver.[1.6-1.7] forget=pointing.model.static You can also force specify the version via 'ver=X' option. When used CRUSH will ignore the version information stored in the FITS and use the explicitly defined value. [FIX] APEX scans in unusual coordinates systems (e.g. ecliptic, galactic did not set coordinates as expected. These were always (0,0). Fixed. [FIX] {SHARC-2} The gain compression under load is now correctly reflected in the NEFD values. [FIX] {b1} Undoing of prior point-source filtering was not using stored and applied value but a recalculating filtering. [FIX] Source model type now determined by first scan, allowing dynamic changes depending on the preprocessing of the scans. [FIX] Updated parsing of configuration entries to take into account bracketed expressions properly. Previously spaces, tabs and : and = characters inside conditional brackets did not work as expected. Now, they should... [FIX] {PolKa} Direction of waveplate rotation was wrongly assumed, resulting in wrong parallactic angle derotations. [FIX] Conditional by serial number were not activated by default. [FIX] {PolKa} Polarized power (P) is now more-or-less properly de-biased for the Rice distribution. The points where the polarization cannot be determined are flagged. More rigorous de-biasing is left for later, but this makes only a slight difference in the low-S/N (<4) regime only. [FIX] {PolKa} All Stokes parameters now correctly include the 50% loss due to the analyzer grid (as of 2.11-a6). [FIX] {PolKa} 'waveplate.jitter' overrides the value calculated from the stored frequencies. [FIX] Scan maps were zero-leveled in every iteration, whereas this is meant to happen only once at the beginning. [FIX] Counting source parameters, and source generations for each integration has been made more universal. Previously only scalar maps and subclassed took care of this. [FIX] {PolKa} Various fixes since 2.11-a1. If 'waveplate.frequency' is manually set, then 'waveplate.channel' and 'waveplate.fchannel' are ignored. Also fixed reduction error when waveplate channels are not defined. Previously, all channels werre flagged in the second iteration when this was the case. Finally, waveplate data is correctly extrapolated when waveplate data is not directly available. [FIX] {SCUBA-2} When both an SDF and a FITS file were present in the same directory for the same subscan, it was read twice in 2.11-a1. Now, the SDF will be ignored, and only the FITS will be read, as expected. [FIX] {a1} GISMO reductions were broken in first alpha (2.11-a1). [FIX] Multithreaded reductions on multicore systems could end up in unpredictable race conditions while procuring the composite source model. Luckily, the chances of this happening were tiny, so the fix should be unnoticeable for the most part... [FIX] {GISMO} 'pointing.model' and 'pointing.log' now accept path specifications with shell variables. (Prior to the fix only literal paths were processed). [FIX] {SCUBA-2} Fixed error when both an SDF and a FITS file are present for the same scan and in the same location. [FIX] {SCUBA-2} Updated the SCUBA-2 proprietary module to work with CRUSH v2.11 (module Version 4). [FIX] Fixed problem with Complex FFTs whereby the real and imaginary components got mixed up, as a result of reversed iteration order. The good news is that Complex FFTs were never used in CRUSH, so the bug was completely harmless. [FIX] If the path to the Java executable was too long, it could produce an error when trying to write that information to the FITS image. Now fixed. [FIX] Smoothing of fixed signals always used boxcar, not the specified window function. It's not a feature that was much used, and the fix should make little difference anyway. [FIX] The function of 'B' and 'b' keys (i.e. brightness control) in 'show' was not working as expected in sqrt scaling, due to a sign bug. Now fixed. [CHANGE] Some FITS keywords describing the Java environment running crush has been changed to more intuitive names. [TWEAK] Obsoleted key 'filter.whiten.neighbours' removed from configuration. [TWEAK] {PolKa} Q and U can be iterated without interference to total- power blanking. Also, the blanking mask includes significant portions of the Q/U maps. [TWEAK] {PolKa} Polarization angles and fractions are not calculated when the polarization fraction is estimated to be zero. [TWEAK] Various small performance tweaks: eliminating some temp arrays and declaring more variables final. [TWEAK] {SCUBA-2} Slight tweaks to the SCUBA-2 configuration files to enhance 'deep' mode reductions. [TWEAK] Calibration and tau tables are loaded as necessary and stay persistent. This speeds up the reading of sets of scans using the same data. [TWEAK] {GISMO} Pointing logs are loaded as necessary and persist, making the reading of multiple scans more efficient. [TWEAK] Motion filter frequency range restricted to <1Hz. [TWEAK] Slightly tweaked 'faint' despiking settings, reverting to more stringent despiking of faint sources. Even more so for GISMO. [TWEAK] {GISMO} 'deep' reductions to whiten more aggressively than usual. [COSMETIC] Reorganized FITS header key ordering to bring the most important keys up front. [CODE] Cleaned up old GISMO frame parsing... [CODE] {b1} Instrument.sourceGain now generalized to all instruments, not only SHARC-2. [CODE] Access to serial numbers, mjd and source name are through getters and setters only. It allows invoking conditionals automatically as soon as these values are defined. [CODE] Instrument name access through getter/setter only. [CODE] {PolKa} Polarization products are now prepared using Parallel. [CODE] ScalarMap masking overhauled. masks can be composited, e.g. from diffent submaps (e.g. I, Q, U). [CODE] {a1} Source model processing separated from synching. [CODE] {a1} Improved error handling in new Parallel processing architecture. In the first alpha, an IllegalMonitorState exception was thrown rather than the trace of the real error that was at the root of the problem. [CODE] Old Parallel architecture replaced by the new task-oriented hieracrchy in source model tasks. [CODE] HDU readers converted to new Parallel architecture. However, only single threaded reads are used until concurrency is properly debugged for these... [CODE] Avoid cloning of Threads (which is wrongly allowed in Java 5 & 6) and use Runnables instead for implementing parallel tasks... As a result of this 'fix' CRUSH now runs under Java 7 and with lesser distinction also under the GNU Java (libgcj/gij). [CODE] Added number formatting classes that omit trailing zeroes. These will be likely used in the new display function... [CODE] Made concurrent access (e.g. on multicore systems) to source models more safe by requiring synchronization of all methods that explicitly change the model data. [CODE] {GISMO} PointingTable.Entry made into proper inner class. [CODE] Made access to images more conventional by restrincting direct access to fields, and using convenience methods instead. [CODE] Generalized projections, grids and images using grids. [CODE] GridImage changed to GridImage. As a result, a whole lot of code has been moved around to generalize objects s.t. more of the crush tools can be used also on non-astronomical images (perhaps sometime in the future...). [CODE] Undid generic types on SourceModel. It did not really make sense to use them here (no specific reverse access to scans and instruments is necessary, and restriction to instrument subclasses is easily done through the constructor). At the same time, the lifting of the restriction makes for nicer code, and removes some annoying obstacles. [CODE] {2.11-a1} Various fixes in image/instrument initialization, and gaussian source insertion/removal. [CLEANUP] Removed some unused variables, spotted by Eclipse Indigo. [CLEANUP] Removed obsoleted key 'whiten.neighbours' from configuration. [UPDATE] {GISMO} Updated the Run 4 (April 2011) configuration files with the much improved pointing of the new merge. (Gone are the mysterious pointing drifts[!] which were probably the result of tilt-corrections not being properly included in the absolute AZ/EL coordinates.) [UPDATE] {APEX} Updated LABOCA and SABOCA configurations (opacities, calibration scalings and RCPs). LABOCA data is good thru 2012.11.08. SABOCA data is good thru 2011.10.25. 2.10-1 (15 Oct 2011) Overview ===================================================================== This is a major update of CRUSH. The main new feature is an innovative motion filter, which may be used to reject signals induced by the scanning telescope movement (esp. when it is roughly periodic). The new filter is especially useful for GISMO faint/deep reductions. Many big changes improvements to Fourier filtering in general, including some critical bug fixes to restore calibration stability of point source fluxes when Fourier filters are used (both spatially, or on the timestreams). Also, fixed polarimetry with PolKa. ===================================================================== [BUG] Large-scale structure (LSS) filtering via 'source.filter' was not applying filter corrections (or did it incorrectly) due to a malformed conditional statement. This affected flux calibration in 'deep' mode reductions, typically by 5%--10% (depending on the filtering setting for the particular instrument). For GISMO, the effect was larger (~30%--40%) due to the much heavier filtering used. [BUG] {a1} Fourier filter corrections were not reflected in the point- source flux corrections scheme or NEFD numbers in 2.10-a1. Now included. [BUG] {PolKa} Polarization maps were wrongly synced back to timestreams as total power, explaining why polarization reduction was broken when iterating on Q & U. [NEW] The biggest change is the addition of a new motion filter. The periodic motions of the scanning can excite vibrational modes in the telescope as well as the instrument. Since these are scan-synchronous signals, they will have a fixed imprint in the maps, showing up as artefacts. To counter this, CRUSH can now spectrally filter the timestreams to reject the frequencies where most of the motion power is. The feature is controlled by the 'filter.motion' key and its subkeys 's2n' (setting a minimum S/N for removed components) and 'above' (setting a minimum relative fraction to peak component for removal), and 'range' (setting the target frequency range). E.g. filter.motion.range 0.03--10.0 filter.motion.s2n 10.0 filter.motion.above 0.05 will identify and filter features of the scanning motion, between 30mHz and 10Hz, that are at least 10-sigma significant and have an amplitude at least 5% of the strongest feature. The filter itself is enabled by filter.motion (It is enabled by the default configuration, see 'default.cfg') The motion filter is part of the unified filter architecture also introduced in this version. [NEW] Introducing a unified filter architercture for frequency domain filtering of timestreams. The umbrella filter is controlled by the 'filter' key, and its components are specified by 'filter.ordering'. The available, and default, components are filter.ordering motion, kill, whiten corresponding to motion filtering (see above), killing specific frequencies and spectral bands altogether, and the adaptive whitening filter. The filtering architecure makes sure these filters do not conflict and that the source filtering and paramter accountings are always correct. Additionally, the umbrella filter optimizes for speed by requiring a single FFT pair per channel for all filters. [NEW] Another new component of the new filter structure is the kill filter, controlled by 'filter.kill'. The sub-option 'bands', can specify the frequency ranges, which should be totally discarded. This can be used to set up specific notch filters for instruments that have known bothersome resonances. E.g. filter.kill.bands 0.33--0.39, 5.74--6.71, 12.2:12.3 will set up the notch filter for three bands, each specified in Hz. The filter itself is enabled, as usual by filter.kill When used standalone, the filter uses DFT or FFT depending on which is faster. (Same is true for the motion filter). [NEW] The new key 'filter.whiten.below.max' gives more control over how much the whitening filter may boost spectral power when whitening frequencies with a deficit of signal relative to the white noise level. (i.e. when 'filter.whiten.below' is used). This is to make sure that the whitening does not insanely enhance certain frequencies. [NEW] {PolKa} The new key 'source.polarization.fraction' allows writing images of the polarized fraction (F) as well as the default products (N, Q, U, P, & I). You can clip away the noisy bits via the 'rmsclip' sub-option. For convenience, the shorthand 'spf' is defined for this option. E.g., you may write: > crush polka -spf -spf.rmsclip=0.01 [...] To enable the writing of polarized fraction showing only the parts with <1% rms. [NEW] 'pol' is now a shorthand for 'source.polarization' [CHANGE] For better conformance with the new filter-architecture, the old 'whiten' key and its subkeys have been moved under 'filter.whiten'. At the same time 'whiten' remains a valid alias for backward compatibility. [FIX] {APEX} Fixed error when observer names contain quotation marks. Normally they should not, but because the apecs syntax often requires strings to be enclosed in quotes, these can end up as extraneous quotes in the FITS headers. The fix includes a workaround by removing any quotes from observer names. Thanks Giorgio for helping to get to the end of this! [FIX] Setting PNG image background via 'write.png.bg' was ineffective. Now fixed. [FIX] {b1} 'whiten.level' was not used in 2.10-b1, resulting in whitening as if level was set to 1.0. That may have been a little too strong... [FIX] {a1} Filtering order fixed. In 2.10-a1, filters were applied in reverse order (i.e. default iteration order). [FIX] {a1} Nyquist response of multiple filters in 2.10-a1 fixed. [FIX] Removed superflous noise rescaling when applying LSS filter via 'source.filter'. [FIX] Some fixes to noise whitening. The zeroth component is now ignored, since that is the domain of the drifts filtering. The neighbour-based round is also adjusted to reflect the lack of a zeroth component. [TWEAK] {PolKa} The polarization products Q, U, P, I (and F) are not written if the 'source.polarization' option was never enabled during the reduction, allowing to solve for the necessary models. [TWEAK] Leveling and re-wrighting of output maps happens only when the map type supports it. As a result of this change polarization Q and U maps, and the derivatives P (polarized power) and F (polarized fraction) maps are no longer levelled. [TWEAK] {PolKa} Undo duplicate total-power-modulation removal. Instead do it just once, right before Q/U modulation modeling. This should be sufficient to prevent TP leakage into polarization signals. [TWEAK] {PolKa} Tweaked configuration: allowing downsampling and noise whitening for polar maps. [TWEAK] Relaxed whitening level. Should preserve source structures more effectively... [TWEAK] Suggestion of pointing offsets ('point' option) now smoothes by the optimal amount rather than the beam if and when 'smooth.optimal' is defined. [TWEAK] {a1} Motion filter enabled only for GISMO faint/deep modes, but not otherwise (as opposed to 2.10-a1 where it was always enabled), since it can interfere with the recovery of larger scales. [TWEAK] Amplitude noise, used for determining when a deviation from target noise level is significant, was wrongly calculated erring on the conservative side. The result of the fix is somewhat more aggressive detection of non-white noise, and therefore a stronger noise whitening filter. [TWEAK] Tweaked noise whitening, by eliminating some unnecessary FFTs, and by conforming to the common dependent accounting instead of its own measure of noise adjustment. [TWEAK] {GISMO} Commented superflous 'whiten.below', which could interfere with the 1st generation motion filter. [TWEAK] Noise whitening filter now fully ignores the DC component. [TWEAK] Further tweaks to noise whitening to fully account for killed frequency components. [TWEAK] Deprecated 'neighbour' based noise whitening. There is no real need for it, and it can really mess up when used with other filters. [TWEAK] Adjusted DataPoint decimal formatting s.t. the precision of the value matches that of the printed error. (E.g. '45.2 +- 1.1' vs the previous '45 +- 1.1'). This is simply more pleasant to read, without providing any real information with the increased precision. [TWEAK] {SHARC-2} Nominal beam sizes switched back to 8.0", which was the default value before 2.05. It seems, after all, that this is closer to the true diffraction-limited beam size than 8.5" would be... [TWEAK] Despiking relaxed in 'faint' and 'deep' modes to >10-sigma spikes (from >8-sigma before). [TWEAK] {SABOCA, SCUBA-2} 'accel' is now a shorthand for 'correlated.accel-mag'. Previously, 'correlated.accel-norm' was part of the pipeline but without an alias for SABOCA and SCUBA-2. [TWEAK] {APEX} Checking for invalid coordinates (-999 deg) is now inlined for greater efficiency. [CLEANUP] Cleaned up units and simplified coordinate axes. [CODE] Motion filter moved under the common filtering architecture. [UPDATE] {LABOCA, SABOCA} Configuration files (RCPs, opcaity and calibration lookup tables) have been updated thru 9 Oct 2011 for LABOCA and 5 Oct 2011 for SABOCA 2.05-2 Overview ===================================================================== This is a maintenance update on the last release. It fixes a bug that broke skydip reductions in the last version. It also contains minor fixes and tweaks, and some re-coding. ===================================================================== [BUG] {APEX} Skydip reductions were broken due to trying to validate equatorial coordinates, which are no longer parsed from the tables. [FIX] Telescope accelerations (for decorrelating and flagging) now include the cos(EL) term as expected. [FIX] Scanning velocities and telescope accelerations had a branching disconitnuity at 180 degrees (between +180 and -180). The fix should take care of this, and provide the expected behaviour at the 'dateline' boundary. [FIX] Fixed signals (e.g. telescope position/acceleration signals) can now be absolute (e.g. acceleration signals) or floating (e.g. position signals). Floating signals are automatically levelled to zero mean, while absolute signals are kept as they are. The separation should allow better decorrelation of their responses. Previously all signals were assumed floating. [FIX] Various small and fairly inconsequential arithmetic fixes. [FIX] One of the Complex FFT routines has not copied contents into a real array properly. However, this particular routine was never used in crush so the bug was inconsequential. [FIX] Signal.differenciate() fixed and improved. The functionality was not really used so far, but may be in the future... [TWEAK] {APEX} Changed smoothing of telescope position data to 48ms ('positions.smooth=0.048') matching the native telescope rate. [SPEED] Various small speed optimizations on iterating over arrays and image smoothing. As a result, reduction speeds may have improved by a few percent (i.e. not much). [CODE] The beginnings of a common FFT filter umbrella structure, which will be the base of noise whitening and profiled filters in the near future... [CODE] New 1D array resampling functionality, which can be used for dynamic resizing of filter profiles in the future... [CODE] 2D Complex FFT to rely on native 1D Complex FFT (rather than NR style array). [CODE] CoordinatePair default values changed to 0,0 from NaN,NaN [CODE] Vector classes replaced by faster Lists in Configurator and Pipeline. [UPDATE] {LABOCA, SABOCA} Zenith opacities and calibration lookup tables updated thru 28 Aug 2011. 2.05-1 (3 Aug 2011) Overview ===================================================================== This update fixes bugs affecting the pointing of the APEX cameras and GISMO. Also fixes the filter correction bug of CRUSH-1.xx tools (e.g. 'detect'. Includes a updated/improved 'detect' tool, and offers several fixes and improvements in general. The main new features are object- depended configuration capability, and better diagnostic of invalid scans/subscans, and handling of unusual APEX scans (e.g. in Ecliptic or Galactic coordinates). A lot of under-the-hood code changes to clean up and reorganize some old code, and bring the native CRUSH-2 tools one step closer (perhaps in the next release...) ===================================================================== [BUG] {detect} Major bug found with the filter correction algorithm of 'detect' tool. This could have resulted in somewhat wrong source fluxes when heavy spatial filtering was used (e.g. at 2--3 times the beam size). [BUG] {APEX} An issue was identified with applying pointing corrections for APEX scans, resulting from is an inconsistency in the raw data between the reported AZ,EL and RA,DEC coordinates. This did not affect reductions before version 2.05-b1, as long as the 'pointing' option was not used to try correct for pointing offsets (when 'pointing' was used, it produced unpredictable results). The first attempt at a fix (in version 2.05-b1) assumed that the AZ,EL values were correct. However, this seems not be the case. The latest fix uses BASLONG and BASLAT columns (usually in equatorial) exclusively, which seems to work well, and also results in simpler, cleaner code. [BUG] {GISMO} There was an oops when trying to use an absolute pointing model together with the incremental corrections based on a pointing log for GISMO. In essence, a differential pointing model was used where an absolute model was needed, resulting in hugely wrong corrections. Now fixed :-). [NEW] {APEX} New astrometry code can handle unusual basis coordinate systems in the raw data file (e.g. scans performed in Ecliptic, Galactic, or Super-Galactic coordinates). Thus, for the first time fully complying to the MBFITS specification on coordinates. [NEW] Implemented bicubic spline interpolation for inbetween pixels. This replaces the old piecewise quadratic interpolation, which could produce small discontinuities at patch boundaries. The interpolation is used for the fast convolution of large-scale-structure (LSS) filtering, via the 'source.filter' option. The default interpolation method can be changed via the new 'source.filter.interpolation' key, which can take the following values: nearest Nearest pixel value. linear Bilinear interpolation. cubic Bicubic spline method. [NEW] {detect} The source extraction tool 'detect' was always designed to work in a way that would accomodate non-Gaussian map noise, by getting some empirical measure of the high S/N tail statistics from the map itself. However, most instruments do produce very Gaussian noise and therefore, this adaptive method may be overly cautious and less accurate than the strict assumption of Gaussianity. For this reason, the latest update to 'detect' introduces the '-gaussian' command-line option to enforce the use of Gaussian statistics for calculating confidence levels and cumulative false detection rates. [NEW] {detect} New 'detect' option '-dynamic' can be used for re-estimating the noise distribution after each extraction round. This may be helpful if the presence of unextracted sources leads to a bias in the noise eastimate earlier on. Previously this was the default behaviour. However, the removal of noise peaks in the extraction can also lead to overly optimistic noise estimates, especially when extracting sources from a small map (i.e. at low S/N cutoff). For this reason, noise reestimaion is now disabled by default. [NEW] The option 'subscan.minlength', originally introduced for the SCUBA-2 module, is now extended to all instruments, allowing to specify the minimum length of an integration (subscan) that is processed. Integrations shorter than the specified length (in seconds) will be ignored by crush. [NEW] Conditionals under the 'object' key can be used to set object- specific conditions. E.g., the object-dependent 'bright' setting for Jupiter can be specified by the config statement: object.[Jupiter] bright As such, the 'bright.cat' method introduced in 2.04 is now deprecated. The condition check is case insensitive and is satisfied whenever the source name starts with the specified sequence. E.g. object.[PNT_] point specifies to derive pointing corrections for all objects whose names begin with the string 'PNT_'. Thus, in SHARC-2 similar lines replace the hardcoded automatic pointing for such sources, making this more transparent and user adjustable. [NEW] {GISMO} Introducing the new configuration keyword 'dataname.end' which specifies the naming convention of merged GISMO FITS files. E.g. for runs 1-4, the appropriate setting is: dataname.end GISMO-IRAM-condensed.fits The above line now appears in 'gismo/default.cfg' as part of the default configuation. This setting was hardcoded previously. With the new keyword, there is more flexibility for changing the naming convention in the future, if necessary... [NEW] New jackknifing option through 'jackknife.alternate' will invert every other scan on the list. This, more controlled, jackknifing has some advantages over the default random approach: (1) For an even number of scans, *exactly* half will be inverted, and (2) scans taken after one another (assuming chronological listing) will have opposite source signatures. Both of these result in a more perfect cancellation of source signals, especially for smaller datasets, than the random jackknife. [FIX] Improved checking for invalid integrations/subscans. Those with too little data (either too few channels, set by 'mappingpixels', or too few frames, set by 'subscan.minlength') will be ignored after producing a warning message. [FIX] {SCUBA-2} 'subscan.minlength' was not used by the SCUBA-2 module, which was expecting just 'minlength' instead. Now fixed. [FIX] {detect} Revised how confidence levels are calculated in the detection source list. The new method should take more properly into account the detection bias (the fact that any peak is more likely to be a source than noise after several sources have already been found in a map). The old method was crude, and also quite wrong... [FIX] Fixed a few outstanding issues in the way coordinate grids were calculated when a non-trivial matrix transformation was present. Luckily, such transformations were never used by crush, making these entirely harmless bloopers... :-) [TWEAK] {detect} Interpolation method between pixels has been changed from piecewise quadratic to bicubic spline in crush-1.xx tools (e.g. 'detect' or 'imagetool'). This may have a small effect (e.g. on fluxes reported by 'detect'). [TWEAK] Piecewise quadratic (i.e. old) interpolation inbetween image pixels has been improved to fall back to linear when lack of data (e.g. edges) does not allow for quadratic interpolation. This method is now deprecated in crush-1.xx tools (e.g. 'detect', see above) but can be selected for filtering in reductions via 'source.filter.interpolation' (see further above). [TWEAK] Added 'pnt.dX' and 'pnt.dY' to the list of loggable quanties. These are the pointing increments in the telescope's native frame. I.e. these are the same as 'pnt.dAZ' and 'pnt.dEL' for horizontal mounts, and 'pnt.dRA' and 'pnt.dDEC' or equatorial mounts. [TWEAK] {GISMO} Pointing reduction no longer suggests absolute pointing on screen. It was confusing, and was also inconsistent with the values written into the the log. Therefore, it's best to omit (for now, at least). [TWEAK] {GISMO} 'maximum-likelihood' estimation made default from start for GISMO. [TWEAK] {GISMO} Run 4 data erroneously invoked configuration from the end of Run 3 (2010 April). This wasn't harmful, since the settings were then overwritten by the proper Run 4 configuration. Nevertheless, the unintended behaviour has been eliminated. [TWEAK] {SHARC-2} Nominal beam sizes are changed slightly (e.g. 8.0" --> 8.5" at 350um), to more realistic values. As a result, the default conversions from volts to Janskys ('jansky' option) has been updated too to yield the same peak fluxes as before. [TWEAK] {GISMO} Updated 'deep' configuration based on the optimal parameters of the deep-field dataset. These are shorter 1/f stability timescale, more stringent LSS filter, and longer iterations. [CODE] New regrid algorithm implemented in CRUSH-2. This relies on antialias filtering (via an appropriate Gaussian filter) before straightforward resampling using the interpolation method of the image (bicubic spline is default). The new method is cleaner, more efficient, and has been tested. [CODE] Reorganized 2D image and grid classes for more hierarchical structure. This is in preparation for a more generic (not necessarily astronomical) image processing classes. [CODE] Migrated (and tested) CLEAN algorithm, and regrid functionality of CRUSH-1. These aren't used right now, but are in preparation for migrating the old CRUSH-1 tools into CRUSH-2. [CODE] Complete overhaul of imaging and spherical grid classes. The new gridding is cleaner, more generic and more efficient, while the new imaging classes have been designed with more powerful and generic plotting capabilities in mind -- bringing a native crush 'show' tool one big step closer... [CODE] {APEX} Cleaned up code handling the selectable basis system of APEX MBFITS files. [CODE] Changed regridding functions to rely on interpolation rather than the old slow Gaussian method. [DEPRECATE] 'autobright' option, introduced in 2.04-2, is deprecated. Use object dependent conditionals instead, which were introduced in this version. Check new features above for details. [UPDATE] LABOCA & SABOCA configuration data (RCPs, skydips and calibration tables) updated and good thru 28 Jul 2011. 2.04-2 (13 Jun 2011) Overview ===================================================================== This update of crush offer some important fixes to how timestamps are handled (esp. for the GISMO pointing logs), and other improvements to the GISMO pointing model. As usual there are many other smaller fixes and improvements also. ===================================================================== [BUG] Timestamp parsing/formatting was done in local time rather than UTC as it should have been. This was a major problem with the processing of GISMO timestamps and the application of incremental pointing corrections because MJD and UT were wrongly calculated. [NEW] {GISMO} Pointing model now contains absolute pointing constants by default (unless 'pointing.model.incremental' is set) and is used for calculating differential corrections to whatever pointing model was used during the observing. This allows changing the telescope pointing model during observations without penalty. [NEW] {GISMO} Added time variability to GISMO pointing constants. The time variablility terms can be ignored by setting the 'pointing.model.static' option. [NEW] {GISMO} Included the finalized pointing log for the April 2011 run. The table can be found inside the gismo/ sub-directory and can be used for applying incremental pointing corrections, via 'pointing.log', on top of the pointing model. [NEW] CRUSH now reports a sub-version string (both on console and in FITS files) that allow identification of intermediate development versions. [FIX] Some FITS image tools seem to have trouble understanding negative longitudes, even though the FITS standard (e.g. Hanisch 2001, Calabretta & Greisen 2002) do not prescribe any range, in which angles have to be represented. Nonetheless, to conform better to non-standard FITS utilities, the longitudes are now written in the 0--2Pi range. [FIX] FITS timestamps are now parsed and printed in UT rather than UTC, as they should be :-). [FIX] Conditional set by 'date' are now interpred as proper FITS dates. [TWEAK] {GISMO} Log quantities 'pnt.X', and 'pnt.Y' are now absolute pointing offsets including the corrections applied by the telescope pointing model (but not the dynamic corrections by the inclinometer). [TWEAK] When jackknifing options are used, the source name gets automatically appended with '-JK' to indicate it being a jackknife rather than an actual source map. [TWEAK] Logging now reports 'RAh' values in 0--24 range (rather than -12:12 as before). [TWEAK] {GISMO} Slightly adjusted default GISMO log format. [TWEAK] Added 'Tamb', 'pressure', 'humidity', 'windspeed', 'windpk', and 'winddir' as log quantities. Not all instruments/telescoped may provide all these values. [TWEAK] {GISMO} The pointing corrections from the telescope pointing model and the inclinometer can be logged via 'modelX', 'modelY', and 'tiltX', 'tiltY' labels. [TWEAK] {GISMO} Adjusted acceptable readout ranges. [TWEAK] {SHARC-2} Added 'Uranus' to 'sharc2/bright.cat', so it is reduced with the 'bright' setting by default. [TWEAK] Pointing/calibration reduction automatically invoked for GISMO and SHARC-2 when reducing single scans 3 minutes or shorter. [CODE] Improved accounting for leap seconds. The new code allows the loading of a standard NIST leap-seconds table from ftp://time.nist.gov/pub/ Currently, this table is not automatically loaded or updated. Such automatic leap second update might be implemented in the future... [CODE] Reporting of non-fatal exceptions during scan reading is fixed. [CODE] FITS image header parsing moved up: AstroMap -> AstroImage. [CODE] Some cleanup of Coordinate set() vs. setNative() methods. [UPDATE] {LABOCA, SABOCA} Updated LABOCA and SABOCA configuration data (RCPs, opacities and calibration tables) with the latest from the APEX pages. All data is good thru 25 May 2011. 2.04-1 (28 Apr 2011) Overview ===================================================================== This version of crush brings many cool new features, like PNG image writing and automatic scan type detection, as well as better photometry support by fixing bugs and adding new features to the relevant code. Many more innovations make this one of the most feature-packed releases to date.. Under the hood a lot of code changes to make the decorrelation more transparent, solid, and bug proof. Many other fixes correct various smaller issues. SHARC-2 configurations come with updated astrometric parameters, thanks to a recent discovery by Mike Dunham on the variability of these. Many improvements to GISMO reductions based on the Apr 2011 run. ===================================================================== [BUG] {APEX} Obsolete trimChopper() and markChopperEnds() functions removed. Bugs in these caused subscans to be needlessly dropped from the reduction, leading to increased photometry errors in the prior update (2.03-2). [BUG] Fixed bug in applying an a-priori source model via the 'source.model' key. The problem was with the coordinate transformations when a nontrivial transformation matrix was defined. [BUG] {PolKa} Instrument object name had a wrong case letter in the specification file in the 'instruments' sub-directory. Fixed. [BUG] {PolKa} Fatal error on instrument initialization fixed. It was due to a null options field to the source model. [BUG] {2.04-b4} A bug was carelessly introduced into 2.04-b4, which resulted in empty or mostly empty maps. Now fixed. [BUG] {beammap} Error in beammap reduction mode during initialization is now fixed. Same null pointer issue as the one that broke polarimetry reductions (further above). [BUG] {PolKa} Several bugs fixed that broke PolKa reductions. (1) waveplate angles were not calculated causing NaNs, and (2) spurious polarization signal filter was badly set up. As a result, de-biasing created NaN timestreams. [BUG] {SABOCA} Reductions broken due to invalid ADC range flagging. [NEW] Added new capability to create PNG thumbnail images for all FITS images. The PNG image writing is activated by 'write.png', the size can be set with 'write.png.size' (default is 300x300), and the colorscheme is determined by 'write.png.color'. You can also set the backgrounf via 'write.png.bg' [NEW] {APEX} Automatic pointing scan detection. Reduction will suggest pointing offsets for standalone pointing scans. However, no automatic pointing suggestion for multiscan datasets (you can still manually set 'point' to do it anyway). [NEW] {APEX} Automatic skydip detection. No need to manually set the 'skydip' option. :-). Also, skydips will be removed from mapping data sets, in case they are accidentally included in the list. [NEW] {GISMO} Automatic skydip detection. (see also above). [NEW] {SHARC-2} Automatic pointing/calibration reduction for standalone scans of pointing/calibration sources whose catalog names start with 'PNT_' and 'CAL_'. [NEW] More control over the decorrelation of phases (e.g. for chopped photometry data). The 'correlated..phasegains' keys and 'phasegains' can be used to derived gains from the correlated phases rather than using the correlated fast samples -- either per modality or as a global setting. The 'phases.estimator' key can specify what estimator to use for deriving phases, if it is different from the defalt estimator type set by 'estimator'. [NEW] The results of pointing reductions (via 'point') are now available for logging. The pointing information are accessible with keywords starting with 'pnt.' while the fluxes and geometry have keys that begin with 'src.'. See Section 2.7 of the README for details. [NEW] Additional instrument-dependent logging capabilities. Check Section 2.7 of the README for details on what instrument-specific qunatities are available. [NEW] Quantities 'UT' and 'UTh' have been added to the list of values that can be logged. [NEW] {GISMO} Pointing corrections can be calculated and applied automatically via an incremental pointing model using the 'pointing.model' option [NEW] {GISMO} Incremental pointing corrections can be automatically derived and applied based on the scan logs, using the 'pointing.log' option. (Tweak: avoid double counting nearest pointing.) [NEW] Additional jackkniving possibilities, allowing to randomly invert frames or channels via 'jackknife.frames' and 'jackknife.channels' respectively. Such jackknives give less realistic noise maps that the regular 'jackknife' option, which inverts entire integations/subscans, because frame and channel jackkniving may belie temporal or spatial correlations. However, for datasets with too few scans for a true jackknife, these give at least some alternative to estimate noise characteristics, even if they are not perfectly ideal. [NEW] 'autobright' option allows to automatically set 'bright' reduction mode for sources listed in the '/bright.cat' file (one source name per line). Currently, 'autobright' is enabled for 'gismo','laboca' and 'sharc2' with planets that typically require the 'bright' reduction mode. [NEW] {PolKa} Changed the algorithm for total-power modulation removal from FFT-based filtering to time-domain analysis. The new time-domain filter is both faster (linear with data size) and works well even if the waveplate rotation is very jittery. Thus, the new algorithm should yield more robust rejection of the unwanted power modulation. The new option 'waveplate.oversample' can be used to fine-tune the time-domain filter. [FIX] {PolKa} 'waveplate.frequency' option had no effect. Fixed. Also, now when the option is set, it is used to calculate waveplate angles with it. However, this means that the phase information of the polarization is lost, as the reference timings are lost. [FIX] {PolKa} No warning message when external timestamp data is not available. Such timestamps were used only in the initial commissioning phase of LABOCA, and are no longer needed for proper operation. [FIX] {PolKa} Power modulation filter was wrongly calculated. [FIX] {SHARC-2} Fixed triggered decorrelation issue. The first iteration did not detect the trigger. This meant that chopper response was decorrelated once even for non-chopped observations. Neither was 'gains.span' trigger aware, which meant that the chopper response was always decorrelated when the spanning option was set. [FIX] Fixed a multitude of problems with the derivation of gains from the phases. As a result, chopped photometry reductions can now derive sky-noise gains properly using phases. Therefore, the configuration has been updated to use this feature rather than relying on the accuracy of pre-supplied fixed gains. [FIX] The option 'rcp.gains' was ineffective since 'pixeldata' overwrote those gains afterwards. By switching the order of RCP and pixeldata reads, the function of 'rcp.gains' should be restored. [FIX] {PolKa} 'I' is now proper Stokes I, rather than the unpolarized power before. The unpolarized power is written as 'N', and is more sensitive than 'I' = 'N' + 'P'. For low polarization fractions, therefore it is a good idea too use 'N' as a proxy for the real 'I'. [FIX] {PolKa} More proper noise calculation for polarized power 'P'. [FIX] {PolKa} Non-destructive writing plus small optimizations. As such polarization images can work with 'source.intermediates', if desired... [FIX] {PolKa} Tau relations were not defined for 'polka'. Fixed by referring to the 'laboca' relation. [FIX] Small fix for dependent accounting of decorrelation. Now all channels in a correlated mode get their dependents cleared upon the update of the correlated signal. This is the correct behavior. [FIX] Empty maps are not written but skipped as intended. [FIX] Fixed log formatting with 's0'...'s9' formats (human readable format with 0...9 significant figures. [FIX] Log entries supplied by first integration not working. Fixed. [FIX] It turns out that old PolKa data already contained the POLTYP values, making it impossible to reduce old polarization data, where it was recorded as 'N' always. The new keyword 'analyzer' can be used to override the stored setting. Valid values to 'analyzer' are 'H' or 'V'. [FIX] Jackkniving was not functional. This is now fixed. [FIX] Highpass filtering thru 'drifts.method=fft' was badly broken. [TWEAK] The invocation of 'bright.cfg', 'faint.cfg' and 'deep.cfg' configuration files with the corresponding options is not longer hard- coded. Rather, these happen via the conditional settings established in 'default.cfg'. [TWEAK] {APEX} Adjusted APEX photometry configuration to solve for sky-noise gains and apply these as the source gains also. The change decreases the reliance on an up-to-date pixel data file for accurate photometry. [TWEAK] {APEX} Disabled noise whitening for photometry. It has little or no benefit, and if anything it may harm reliability... [TWEAK] Adjusted initial blanking and clipping when a source model is supplied via 'source.model'. [TWEAK] {APEX} photometry reductions now check to make sure enough good pixels remain in the reduction (see 'mappingpixels' and 'mappingfraction') before proceeding to obtain photometry from the subscans. [TWEAK] Chop phase data are updated whenever a change could result (after time weighting, despiking or whitening) to make sure all correlated signals end up in the right places (phases vs samples). [TWEAK] Pointing applies its own exposureclip to discard the under- exposed parts of the map, thereby increasing the reliability of the pointing measurement. [TWEAK] {PolKa} improved filtering of total-power modulation, by better tailoring to the waveplate jitter, and application only before the derivation of Q and U maps. [TWEAK] Automatic veclocity clipping ('vclip=auto') adjusted s.t. low scanning speeds are more stringently rejected. [TWEAK] {LABOCA} Adjusted 'gainrange' option for 'cables' and 'boxes', to reject very low (<0.1) gains relative to average. As a result, the new configuration ought to be more reliable... [TWEAK] {PolKa} Disabled whitening for PolKa. The whitening filter may inadvertendtly remove polarization signals, so it's best to avoid it. [TWEAK] {SHARC-2} The typical Nasmyth plate-scale values are made default, making it easier to generate overriding options for current runs without the need for pre-pending date-specific conditionals... [TWEAK] {LABOCA} Not using resistors to decorrelate amplifier boxes and cables. While, the inclusion of channels wired with 1 MOhm resistors, instiead of bolometers, could help obtain more robust solutions, in practice the evidence is that they slightly degrade the reduction quality. Thus, the 'noresistors' option is now set by default. [TWEAK] {LABOCA} Switched the order of decorretaling amplifier boxes (now first) and gradients (now second). This reflects the prominance of signals in these two modes, leading to more robust solutions given the level of degeneracy between the two. [TWEAK] {GISMO} Switched the order of MUX decorrelation and gradients. See above entry for more details. [TWEAK] {GISMO} Adjusted default log format: 'RAh' with 1 decimal, and added 'NEFD' and 'scanspeed'. [CODE] Complete reorganization of the decorrelation code among classes that take part in the process. The new locations of the code make the behaviour more transparent. [CODE] Cleaned up some of the gain derivation code, with an eye on using more common code to get gains from phases or from samples. [CODE] Disabled gain renormalization, until it can be made perfectly safe... [CODE] Double precision weights of stored phases and phase signals. [CODE] 'uniform' now sets all gains uniform. [CODE] {LABOCA} Keep resistors properly flagged, so they can be used for decorrelating amplifier boxes and the flexible cryogenic band cables. [CODE] {APEX} Validating chopped-modulated offsets s.t. those without enough frames for estimating offsets are removed from the set. [CODE] {PolKa} Use Modulation rather than Mode as nomenclature for separating the polarization signals. [CODE] Slight changes to weighted median calculation methods. No change in functionality. [CODE] Decorrelation od signal skips over channels with zero gains or weights. [CODE] Added some assertions in places where NaNs or Infs can be produced. [CODE] Disabled old subscan logging code. Its function has been superceded by a much more versatile logging capability. [CODE] Log formats now accept C-style '\t' to signify a tab in the format specification string. [CODE] Chopper state is now copy safe (i.e. not sharing the same object after copy. [CODE] Removed copy() from Integrations. There should be no need to duplicate data, and the depndent field classes can be very troublesome for copying over properly. It's safest to not have this feature. [CODE] Log formats can now deal with properly escaped sequences. [CODE] Rotation of arrays was a little bit messy. This has been cleaned up, offering more consistency across various arrays (GISMO, SHARC-2, SCUBA-2). The applied rotation is also printed on the console allowing verification. [COSMETIC] {PolKa} The detailed information is shown only for the unpolarized power map, while other maps are written more silently. [COSMETIC] {LABOCA} Now reports line-of-sight tau and equivalent PWV when setting zenith tau (manually or from skydip table). [COSMETIC] {SHARC-2} Duplicate reporting of 'direct' tau values when falling back on MaiTau is fixed. [COSMETIC] Decorrelation steps that do not flag channels by gain do not display a census of active channels unnecessarily... [UPDATE] {LABOCA} Updated pixel data files to include resistors. [UPDATE] {SHARC-2} Updated configurations with the latest array astrometry from Darren Dowell, Mike Dunham and Attila Kovacs. The zoom of the SHARC-2 array is very sensitive to its distance in the optical path. Some 10% variation has been found in the astrometric parameters since Sep 2004. The exact date boundaries of these changes are still under debate, but the currect configuration presents a first best guess. We hope to refine the configuration in upcoming releases. [UPDATE] {GISMO} Pixel positions have been fitted to the Neptune beammap using a quadratic distortion model, yielding 0.6" rms in x and 0.9" rms in y (not bad for a 17" beam!). The fitted positions are now the default. 2.03-2 (3 Apr 2011) Overview ===================================================================== This update fixes an unfortunate configuration bug with the APEX bolometers. Due to a badly placed line of code, 'chopped' configuration settings were always activated, resulting in suboptimal pipeline settings for scans obtained in mapping mode. A few smaller issues were also fixed (especially in the photometry code). Photometry errors may have increased slightly, but these uncertainties are expected to be more accurate than before. ===================================================================== [BUG] {APEX} Conditional settings for photometry were always active for APEX scans due to a coding error. Should be fixed now... [FIX] Fixed GLS projection problem around RA~12h (and AZ~180 deg). (Also updated SphericalCoordinates.getNativeOffsetFrom() method.) [FIX] Lines containing only empty spaces in the configuration files resulted in errors when parsing. The fix checks for 'empty' lines and ignored these as expected. [FIX] Source gains were not used for photometry reductions as intended. This is now fixed. As such options like 'source.fixedgains' now work as expected. [TWEAK] Default file names are now constructed with an edited version of the catalog source name, with problematic character sequences (such as white spaces, quotes, *, ?, /, and \) replaced with underscores ('_'). This makes it easier to refer to the output files from the command line. Thanks to Giorgio for the suggestion. [CODE] Changed the markup of photometry phases. [CODE] Cleaned up phase update code to work properly on a selection of channels. (The old code was a mule, which only really worked when phases were updated for the full instrument. Luckily, that was the case...). [CODE] Dependent accounting fixed for phase update. The last sample in the phase has slipped through previously. [CODE] Removed extra phase decorrelation step from photometry code. [CODE] No automatic phase updates after weighting, whitening or despiking. However, phases are updated with the offsets. [CODE] Better handling of 'source.fixedgains'. [CODE] More shared code between regular Fourier filtering and noise whitening. [COSMETIC] Extra white space on the console output (from a skipped decorrelation step) removed. [UPDATE] {LABOCA} Updated opacities ('laboca/tau.dat') and calibration scaling data ('laboca/calibration.dat') up to and including 29.03.2011. 2.03-1 (21 Mar 2011) Overview ===================================================================== This is a major update of CRUSH. It fixes a critical bug that surfaced since 2.02-1, and affected releases before 2.03-b2. APEX configurations were not properly established when crush was run from outside of its distribution directory. The main new feature of this version is a powerful custom logging capability (read more about it in Section 2.7 of the README), and the ability to detect and flag channels at or near their saturation values. It also comes with updated photometry configuration for LABOCA (based on the patches released for the previous version). ===================================================================== [CRITICAL] {APEX} Common APEX configurations were not loaded when CRUSH was run from outside of its disctibution directory, because the invocation syntax did not follow the rules of the 'config' option (which did not take full pathnames, only relative ones!). Sorry for the inconvenience. [NEW] Introducing a powerful custom logging of scans (and reductions) via the 'log' and 'obslog' keys, and their various sub-options. You can specify what quantities are logged, and how they are formatted. Please have a look at the new Section 2.7 of the README for a detailed explanation of the logging capability. Over time, more quantities will be made available for these logs, so keep an eye out for changes and additions. [NEW] Pixels and data, which are at, or close to, the limits of the readout (ADC) range are now identified and flagged. You can use the 'range' keyword to specify the range of acceptable values, and 'range.flagfraction' to set the maximum fraction of affected samples before the pixel is discarded from the reduction. It is an expert-level setting. You probably do not need to play with this option unless you really know what you are doing... Note, that the range is always specified in the units of the raw data as it is stored in the data files (see 'dataunit'). LABOCA can set the range automatically, based on the backend gain setting, if the 'range.auto' option is set. [FIX] Photometry reductions had a small level (10--15%) of unaccounted flux filtering resulting from the decorrelation of phase offsets. As of now, the information in the on-source channel no longer contributes to the estimate of the correlated offset, and therefore there is no more undesired filtering of the source fluxes. [FIX] Decorrelation steps in photometry reductions did not report the number of unflagged channels correctly. However, this was simply a display issue, with no effect on the actual reduction. [FIX] Changed the way the 'config' option behaves. If the specified config file is not found in one of the standard locations for crush or for the instrument, CRUSH will attempt to interpret it as a regular pathname. This way users can actually store their custom configs anywhere on the file system, as long as the names do not clash with those distributed with crush. When using non-standard config files try avoid names like 'default.cfg', 'faint.cfg', 'deep.cfg' or 'bright.cfg' to minimize undesired behaviour... [FIX] Allows processing configuration values enclosed in quotes. E.g. crush -datapath="C:\My Data" Both single (') and double (") quotes are allowed (as long as they match at the beginning and end). [FIX] Slight issue with copying Instrument classes. If the instrument was already populated with channels, then the channel copies kept carrying their original Instrument object as their parent instead of its copy. Luckily the parental references were rarely used and in non-critical context only. This prevented the issue from being a much more serious bug... [TWEAK] Whitening restricted to rejection of excess noise (by disabling 'whiten.below') for chopped photometry reductions. Otherwise, with the fixed filter time-scale bug (see below), the whitening could go berzerk. Proper white noise is a nice property to have for maps, but it is not essential for photometry, where spatial information is not retained. Thus, the more restricted whitening has no negative impact whatsoever on the result :-). [TWEAK] Drifts removal updates filter timescales to the shorter of the previous filter time scale and the new filter time scale. This is the failsafe behaviour, even though the configurations were set up such that there should be no difference in behaviour as a result of this change. [TWEAK] Average gains and weights (for gain an weight flagging) now use robust means, with the 10% tails ignored. [TWEAK] {APEX} Chopped photometry reductions to use 'neighbours' method to despike, even in 'faint' and 'deep' modes. [TWEAK] {APEX} Photometry reductions to use more up-to-date pixel values. The pixel data table should be updated more regularly... [TWEAK] {APEX} Tweaked photometry configurations for better precision. These tweaks are used for the only photometry summary on the CRUSH pages. [CODE] Instruments are instantiated without adding channels. They are populated with channels only when reading the data. [CODE] Pixel offsets are now interpreted at the readout stage. [CODE] Overhauled scaling data to and from detector stage. [CODE] Statistics functionality moved to Statistics class. [CLEANUP] Fixed Debian package permissions providing global read access to all files of crush. 2.02-1 (7-Mar-2011) Overview ===================================================================== It is a significant update of CRUSH-2, bringing a new APEX photometry reduction mode (for chopped photometry), tau conversions for all instruments, and other new features. It also fixes a range of smaller issues from the prior versions, offers some tweaks, some code restructuring, and up-to-date configuration files for LABOCA and SABOCA. ===================================================================== [NEW] Preliminary support for APEX chopped photometry reductions. [NEW] Added 'sources' key, allowing to specify a catalog of test sources, which are inserted into the data. For an example catalog file, see 'example.mask' inside the crush distribution directory. By default 'source.fixedgains' is also set together with the 'sources' option to assure that test sources are extracted with the same pixel gains as the ones used for the insertion. [NEW] The new option 'correlated..phases' allows the decorrelating of chopper phases together with the usual decorrelation on the high- frequency samples. It is not recommended that you change the default settings for LABOCA (and SABOCA) unless you are really sure what you are doing. [NEW] The new option 'correlated..nofield' allows the decoupling of correlated noise gains from the field values stored under each channel. The field values are read and written via the 'pixeldata' option. The option is meant to provide more tweaking ability for true experts. [NEW] Dataset validation for APEX photometry scans. The validation discards all scans from the data set which are not compatible with the photometry of the first scan. Eventually, the validation will check for other types of inconsitencies (e.g. different filter bands, focus and skydip scans etc.) [NEW] Common default configuration for APEX instruments is moved under the new 'apex' subdirectory. [NEW] Tau scaling relations are now extended to all instruments not only for SHARC2. Normally, a set of useful relations should be defined for the instrument by default. Please contact Attila if you would want to add a new relation into the default configuration of the distribution. Once the scaling relations are established, you can set values with them. E.g., to specify opacities for GISMO using a PWV value of 0.5 mm, you would write (this time on the command line): > crush gismo [...] -tau.pwv=0.5 -tau=pwv [...] The first option ('tau.pwv') specifies the PWV value, while the second option ('tau') instructs to use the PWV for determining the appropriate GISMO in-band opacity. The following relations are defined as defaults: SCUBA-2: '186GHz', '225GHz', 'PWV' SHARC-2: '225GHz', '350um', 'PWV', 'direct' GISMO: '225GHz', 'PWV' LABOCA: 'PWV' SABOCA: 'PWV' The SHARC-2 'direct' value is not really a scaling relation, rather it is a calibrated method to convert total-power DC offsets into in-band line-of-sight opacities when operating at 350um. A PWV scaling relation is included in the configurations of LABOCA and SABOCA. [CHANGE] 'exposureclip' and 'noiseclip' now specify clipping values relative to the median exppsure/rms of the map, rather than the max/min values used before. This should make it more robust when a small part of the map is overexposed, having caused NaN images. [CHANGE] Decorrelation time constants (controlled by the options 'correlated..resolution') are now rounded to the nearest power of 2 frames to avoid awkward boundaries, esp. in relation with the 1/f 'drifts' filtering. [CHANGE] Repackaged utility classes for better organization. This does not affect the functionality of the bundled code, but will break prior versions of plugins (e.g. for SCUBA-2) which will have to be updated also. This change is one of the reasons for bumping the version number to 2.02... [FIX] A problem existed when adding Gaussian sources to images with FWHM larger than the beam size. In such cases the patch size used for the insertion was too small. Thanks Haukur, for finding this one. [FIX] There has been a recursion error when keys were removed from the configuration set, resulting in nested 'removed' branches being created. (This created some very long FITS keys, which caused errors.) A twofold solution was inplemented. 1. Checking for and disallowing nested branching of 'removed' keys, and 2. introducing a 'purge' function, which permanently removes branches from the configuration set, to be used by internally, e.g. when intersecting configurations. Many thanks to Haukur and Ciriaco for diagnosing this! [FIX] Check for very long HIERARCH fits keys, which cannot be written conventionally. Use abbreviated form when writing. [FIX] Small fix on parameter accounting of decorrelation steps. Now the dependencies are calculated only for the same unflagged channels as are used for the decorrelation itself. [FIX] Relax photometry position checking for moving objects, such as solar system bodies. [FIX] Fixed problem with 'config' option not accepting paths with variables and shorthands. [FIX] Fixed a small problem with the selection of channels used in normalizing gains of different correlated modes. [FIX] Fixed problem with 'correlated.<*>.nogains' not having any effect. [FIX] Invalid subscans no longer produce an error on validation. Instead, these are dropped from the reduction, as indicated by the corresponding warning message on the console. [FIX] Prevent error resulting from an empty "" command line argument. Although such empty arguments should never be produced, it never hurts to be safe... [FIX] An error was fixed in complex FFT function. Currently, this has no effect on CRUSH since it uses real transforms only. [FIX] {SHARC-2} 'tau=direct' did not take residual DC offsets into account. As such the tau values were not as accurate as they could have been when the instrument was not levelled at the beginning of the scan. Now, the issue is fixed, leading to more accurate 'direct' tau values at all times :-). [FIX] {SHARC-2} 'tau=225GHz' option was not working due to a hardcoded case-sensitive table lookup error. Now fixed. [FIX] {SHARC-2} The use of tau scaling relations has been generalized. It now allows the use of PWV values and user-specified relations. [FIX] {SHARC-2} Optimal smoothing ('smooth.optimal') is now set only for 350um reduction mode. At other wavelengths, the deep reductions default to beam smoothing. (Earlier, all wavelengths were wrongly using the optimal 350um value). [FIX] {SHARC-2, GISMO} Fixed a minor issue with the copying of pixels, with the pixel sizes having been shallow copies rather than the desired deep ones. [IMPROVE] {GISMO} Pointing information enhanced, by including cumulative horizontal offsets. [IMPROVE] Avoid double validation of subscans/integrations. [SPEED] {post-b2} parallelized source insertion via 'sources'. [SPEED] Various small performance tweaks relating to FFTs. Since FFTs are rarely used, and the tweaks are relatively minor, there is no noticeable boost to reduction speeds. [SPEED] Source extraction now using one source copy per thread, rather than per scan as before. Should result in a minor performance boost, epsecially for large maps and datasets. [TWEAK] {APEX} Exposure and noise clipping ('exposureclip' and 'noiseclip') are made more robust, by selecting the 5/95-percentile values as the reference, rather than the absolute max/min values as before. [TWEAK] {APEX} Smooth telescope positions only by 0.1s (used to be 0.25s), to be more sensitive to rapid motion changes. [TWEAK] {APEX} Checking for incompatible datasets (photometry vs mapping data, or photometry on different objects) is extended for checking the names of solar system (moving frame) objects also. [TWEAK] {SHARC-2} 'maitau.fallback' adjusted to use 'direct' tau only for 350um data, and 225GHz taus otherwise. (see sharc2/default.cfg). [CODE] Configurator.value now private, access only through get and set methods. [CODE] Use Parallel class to reimplement parallel operations on source (extraction, synching, boxing, and indexing). [CODE] Restructured pipeline code. Tasks are now performed scanwise rather than integrationwise. The new code is cleaner. [CODE] Restructured decorrelation code among Integration, Modality and Scan classes. The idea is to allow some operations (like gain estimation) to perform on scans rather than integrations. The effect is to have gains that span an entire scan and not just an integration within it. [CODE] Got rid of non-necessary exception handling when editing headers for scan data HDUs. [CODE] Instruments can now edit the FITS image headers, allowing for more structured editing. [CODE] The performing of tasks moved from Pipeline to Integration (and its subclasses). This allows for the definition of tasks that are specific to certain data types only... [UPDATE] {LABOCA, SABOCA} RCP, tau and calibration data up-to-date as of 07 Mar 2011. [UPDATE] {LABOCA, SABOCA} PWV to tau conversions refined for LABOCA and SABOCA based on all skydip data to date (4000+ for LABOCA and ~300 for SABOCA). 2.01-4 (19-Nov-2010) Overview ===================================================================== This is a maintenance update of CRUSH 2.01. The main change is a complete overhaul of the configuration parser engine, with the aim of overcoming some of its pathologies, especially with the way aliases and wildcards are processed. Apart from the parser, there has been an update of the FITS image code, to handle a wider variety of FITS images, in preparation for the new show/imagetool. Various smaller fixes and improvements are also part of this release. ===================================================================== [NEW] The command 'blacklist' without an argument now produces a list of all blacklisted settings on the console. [NEW] Introducing 'conditions' command, which allows checking on the currently active conditional statements. Used without an argument it lists all conditions, while an optional argument can be used to match only conditions that start with the specified pattern. [FIX] {parser} 'restore' commands was not recognised by the parser, which expected the deprecated 'replace' command instead. The GLOSSARY has been updated accordingly. The deprecated 'replace' command will be recognized as an alternative to 'restore' for some time to come... [FIX] {parser} Fixed parsing of wildcards '*' in 'forget', 'recall', 'remove', 'restore', and 'whitelist' statements. No wildcards are allowed for 'blacklist' (since these would be hard to enforce for arbitrary non-enxisting branches). [FIX] {parser} Fixes to how 'forget', 'remove', 'blacklist' etc. deal with aliased statements. [FIX] {parser} 'recall' now checks for pending conditionals. [FIX] {parser} Fixes to the way conditions are canonized. (removing leading white spaces, and replacing sequences of other spaces and '=' with a single '?'). [FIX] {FITS} Added precession FITS processing for Ecliptic Coordinates. [FIX] {FITS} Precessing coordinates now recognize 'FK4-NO-E' epoch, but process it just like the regular 'FK4'. [TWEAK] {GISMO} Added 'beam' size to default configuration. The same beam size was already assumed as the hardcoded default, so it changes nothing, except it makes the setting more transparent. [TWEAK] {GISMO} MUXes are decorrelated at the full time resolution of the downsampled data in 'faint' and 'deep' modes. [TWEAK] {deep} Source filtering over 5 beam widths is now default in 'deep' mode, instead of gradients. In case of some instruments, like SCUBA-2 and GISMO, the filter is made even more agressive to combat sky noise. [TWEAK] {parser} Configuration branches removed by the 'remove' command are now stored under the 'removed' key, for greater transparency. [TWEAK] Simplified the calculation of 1/f filter (drift) corrections. [TWEAK] 'faint' + 'extended' now enables the removal of gradients by default. It also enables row decorrelation for SHARC-2. This is more in line with the intended behaviour... [IMPROVE] {poll} Polling the configuration (via '-poll') now also prints the forgotten configuration keys with their values, such that the user may more easily decide if to 'recall' an old value, or set it freshly again. [IMPROVE] {FITS} Overhauled code for handling spherical projections and celestial coordinates, to conform more closely with the specification by Calabretta and Greisen (2002). Specifically, coordinate system transformations are now built in, and the projection classes recognize most of the designated FITS keywords. These changes will be used by the new 'show' and 'imagetool' utilities (coming soon!). [IMPROVE] {FITS} Overhauled code to allow images with non-square pixels, and to process 2x2 coordinate transformation matrices. Currently, neither of these features is used. Rather, it is an important preparatory step towards the new show/imagetool (coming soon!). [IMPROVE] {parser} Reduced costly string comparisons in parser. [SPEED] Noise whitening is made faster by using floating point precision instead of double precision arithmetic. [COSMETIC] {FITS} Conditionals are now properly bracketed in FITS headers. [UPDATE] Updated README on the configuration syntax/rules and on the explanation of the console output, and on checking the current configuration settings. [UPDATE] Bundled CRUSH-1.xx tools (e.g. 'show', 'imagetool', 'detect') have been upgraded to 1.63-15. [UPDATE] LABOCA and SABOCA configurations updated thru 17-Nov-2010. 2.01-3 (26-Sep-2010) Overview ===================================================================== This is a maintenance release of CRUSH-2.01. There have been a number of issues identified in the previous release(s). This update offers fixes to all issues identified thus far. There are also a number of smaller tweaks, improvements, and configuration updates. ===================================================================== [BUG] CRUSH tools (e.g. 'show', 'imagetool', and 'detect') indicated that the map fluxes were uncorrected, when in fact they were corrected for point-sources. This may have caused calibration issues when 1. 'source.filter' was used during reduction (default only for 'deep' mode SCUBA-2 reductions). AND 2. Additional filtering was performed via 'show' or 'imagetool' *after* reduction (via the '-extFilter' option), OR fluxes were extracted with the 'detect' tool. When both the above conditions were met, CRUSH applied duplicate point-source corrections. Part of the problem was traced back to a missing FITS header key/value pair. A second issue was an unexpected change in the FITS libraries, which defaulted even when the correct value was stored. Both issues are now fixed, and the correction scheme should now work as expected. [FIX] Rare divide by zero bug found and fixed when calculating the point-source filtering of decorrelation steps. [FIX] 'source.model' option worked only if the model was exactly the same size, and on the same grid, as the target map. This was fine if the model was produced from exactly the same set of scans, and if it was not cropped afterwards via 'imagetool'. The fix now allow models to be provided on any grid, and any size... [FIX] Beam units recalculated after regrid operation also, (not just after smoothing). [SPEED] Map indexing has been parallelized for better performance... [TWEAK] By accident 'median' estimators were made default in 2.01-2 for one iteration in 'faint' and 'deep' modes (but not in default mode). It is not a bad idea, in fact, to start with the more robust median estimators at first. Therefore, starting from this release, medians are made default for one iteration in all reduction modes... [TWEAK] {gismo} Configurations slightly tweaked on use of 'median' vs. Maximum-likelihood estimators... [TWEAK] Streamlined reading of data of all instruments, using a faster method of reading FITS binary tables (by columns instead of rows). [CONFIG] 'split' reductions should not be clipped by exposure before coadding. Instead, noise or exposure clipping can be performed after the coaddition, using 'imagetool'. [CONFIG] {laboca} 'faint' mode reductions now always decorrelate modes at the full time resolution of the data. This was already the default in non-extended mode, but now 'faint' together with 'extended' will also adopt these settings. [COSMETIC] Removed extra line-feed when 'source.filter' is used. [UPDATE] Expanded README, to include sections on 'Pixellization and smoothing' and on 'Image processing post-reduction'. Also reorganized its sections, and made small edits in parts. [UPDATE] {laboca} Calibration information ('laboca/tau.dat' and 'laboca/calibration.dat') updated until 26.09.2010, using data from the APEX pages [UPDATE] Bundled CRUSH 1.xx tools ('show', 'imagtool', 'detect' etc.) are upgraded to 1.63-14. The update fixes various issues with the handling of CRUSH-2 images. 2.01-2 [FIX] {java} The default JAVA runtime configuraton in the previous release was accidentally changed, during testing, to use 4GB of memory. This update reverts back to the usual 1GB requirement, and 32-bit memory model (since the -d64 flag is also non-standard accross virtual machines). Since these settings should normally be edited during installation, as descibed, the update is not critical. 2.01-1 Overview ===================================================================== This release fixes a critical bug, which strongly affected calibration in the 'faint' and 'deep' reduction modes. The bug had to do with the 'Jy/beam' unit normalization, when smoothing was used internally in the reduction. The main new feature of CRUSH is to support SCUBA-2 scan reductions. It also introduces a few other features, and provides some smaller fixes, tweaks, and cosmetic improvements over the prior release. ===================================================================== [CRITICAL] 'faint' and 'deep' mode reductions had incorrect normalization due to peak fluxes normalized to the instrument beam, instead of the proper image beam. For 'deep' mode reductions this skewed calibration by a full factor of 2. The images (S/N and structures are otherwise correct). The bug can be compensated manually by scaling images with (1 + smoothFWHM^2/instrumentFWHM^2) manually after reduction (e.g. by imagetool). Extremely sorry for this nasty bug!!! [BUG] {GISMO} Date-specific options were set too late for some of the settings to take effect in time. The date-specific configuration is now moved ahead for such settings to be loaded in time... [NEW] SCUBA-2 modules are available upon request. These modules are strictly private and proprietary, with restrictions on its distribution, use, and modification. If interested, please contact Attila for a personalized copy of the plugin. [NEW] {SABOCA} Automatic tau from skydips, and calibration corrections from observations of calibration sources are now included for SABOCA between 2009-09-13 and 2010-08-31, thanks to the information becoming available on the APEX pages. Only data with 'OK' quality designation is used. [NEW] 'source.type=null' can be used for sourceless reductions. When set, CRUSH will not even attempt to create a source model for data, but will process using the other pipeline steps. This is useful for reducing diagnostic or lab data (e.g. noise measurements), where modeling a source is meaningless. [FIX] {parser} Statements in which the key and value were identical strings were parsed incorrectly. Since such a scenario never occured in the supplied configurations, there was no maleffect. The parser has been adjusted to handle such cases. [FIX] {whiten} Make sure that there is a resonable frequency range for probing the white noise properties, even of the desired probing range falls outside of the available spectrum. In such cases use a minimum number of channels, set by the new key 'whiten.minchannels' closest to the desired range to determine the 'white' noise levels. [TWEAK] {faint} RMS weighting is made default in 'faint' mode reductions also for 'extended' sources. [TWEAK] {source.filter} Spatial filtering of source maps has been adjusted to act both on the scan maps (for deriving proper weights on these) and on the final composite (to properly reflect the filter blanking of the final output). [TWEAK] {GISMO} The GISMO reduction of 'faint' + 'extended' sources did not work as intended. Several tweaks have been applied to 'extended' mode reductions both in 'gismo/default.cfg' and in 'gismo/faint.cfg'. [TWEAK] {GISMO} The pointing center for run 2 has been changed back to the nominal array center, for better backwards compatibility with pointing in older scripts (e.g. CasA)... [COSMETIC] Extra line-feeds on console when 'pointing' (old 'center') option was used. Cleaned up... [COSMETIC] More cleanup on scan/subscan IDs, both on the reduction console and for the data output... [COSMETIC] Scans containing no valid integration are now dropped. (This should never occur, except for some SCUBA-2 scans, which may not have the antenna information in them). [COSMETIC] Use scan ID more consistently instead of scan number in various output file names and on the console. Also, the subscan ID is included where it is informative. There is no change in functionality. [COSMETIC] For multi-subscan reductions, the subscan ID was not separated by '|' from the scan ID on the console output. [COSMETIC] Scan reading/processing reporting made more structured, especially when processing multiple subscans in the same scan... [UPDATE] {copyright} E-mail address updated in the copyright statement appearing in the source files. [UPDATE] {LABOCA} Updated laboca configuration (RCPs, 'tau.dat' and 'calibration.dat') with the latest information from the APEX pages. Information is good thru 31 Aug 2010. Only data with 'OK' quality is used. [UPDATE] {SABOCA} RCP data update to latest from APEX pages. (Also added new tau and calibration data -- see above!) 2.00-1 Overview ====================================================================== The first official CRUSH-2 release at last :-)... It fixes a number of remaining issues from the final beta (b4) and provides a few small improvements over it. This version of CRUSH is now the official stable release for all instruments, replacing both CRUSH-1.xx and miniCRUSH as the default reduction package. The old versions will, however, coexists for the time-being, in case unexpected problems arise with the newer version. Users are strongly encouraged to convert to CRUSH-2 sooner than later. ====================================================================== [BUG] Disabled parallel scan reading. It appears that the FITS libraries aren't thread safe, leading to slight instabilities when processing FITS data in parallel. Fortunately, the effect was small enough s.t. it was not easily noticed. However, to be safe, crush is reverting to the original (and slower) serial processing of FITS data. [NEW] CRUSH is now available also as RPM or Debian packages, making installation a breeze under Linux (if you have root privileges). [NEW] Added man pages for crush and the various crush-1.xx image tools bundled with it. If you are installing CRUSH from an RPM or Debian package, then the man pages are fully accesible. You can also install the man pages (and the executables) from the archive (tarball or zip) distribution, by running 'install.sh' as root (or via 'sudo'). The man pages are also available online through the CRUSH homepage, under the 'Documentation' section. [NEW] {tar.gz|zip} Included 'install.sh' and 'remove.sh' scripts in non-package distributions to make system-wide access to the executables simpler and to install the newly added man pages (try 'man crush' after running 'install.sh' to verify the installation). Uninstalling is possible via the 'remove.sh' script. You need root privileges to run these scripts. [NEW] Added check for a suitable java virtual machine. If crush is configured to run on GNU Java (libgcj/gij), it will produce a warning message recommending users to change to something more reliable. [CHANGE] Scans are no longer sorted by serial number before reduction. Instead, the order, in which they were submitted, is retained. [CHANGE] {GISMO} The default GISMO output file names now contain the full IRAM scan IDs, i.e. YYYY-MM-DD., or their range when reducing multiple scans. This should make the origin of images traceable by name. (The same ID are also used on the console output during the reduction.) [CHANGE] Changed the way instruments are loaded, making them more plug and play modules (rather than the hardcoded list before). Instruments are now registered in the instruments subdirectory. A file, named the way you would invoke the instrument in crush, specifies the corresponding java class of the instrument. While, the change has no immediate effect to the user, it allows instrument packages added to or removed from crush installations at any time. (As instrument-related content grows, it makes sense to allow customized instrument installations). [FIX] {LABOCA} 'laboca/faint.cfg' contained old 'maxHe3rms' key. Corrected to 'H3.maxrms'. [FIX] {SABOCA,ASZCA,p-ArTeMiS} Created 'amps' alias for minicrush compatibility. [FIX] {pointing} Pointing corrections are properly accumulated. It is possible to both apply an initial pointing corrections (via 'pointing=x,y') and then make crush suggest incremental corrections (e.g. via 'point'). Previously, the suggestions were incremental to the initially supplied values, but it makes more sense to provide the user with the aggregated pointing correction at the end of the reduction. [FIX] {skydip} Skydip reduction was broken, because it was trying to find a group of channels by their old name, which is no longer used. The source code is now fixed with the correct channel group identification ('obs-channels'). [FIX] Windows .bat files were not working correctly when CRUSH was installed in a location whose path contained white spaces. The updated batch files should now work as expected. [FIX] Checking for updates on startup was suffering from incomplete version comparison, which should now be fixed. [TWEAK] {sharc2} Rehabilitated decorrelation on rows by default. The prior releases did this only for 'faint' reductions, which was fine in most cases, but occasionally the stability seems to have been worse, and some data really benefit from the row decorrelations. Nonetheless, it remains default to ignore the 'rows' in 'extended' reductions... [TWEAK] {point} The suggesting of pointing corrections is made more informative with integrated flux (in an adaptive aperture), S/N and FWHM uncertainty reporting, and providing elongation (major, minor axis and position angle on map). [TWEAK] {point} The FWHM estimation is made more adaptive, with a growing aperture until ~98% of the central source flux is captured. [TWEAK] {bright} Disabled all clipping of 'bright' source redcutions. [UPDATE] Updated README installation instructions to include system-wide install (executables and man pages), and Linux packages. [UPDATE] {README} Moved 'Recovery of Extended Structures' section further up in the README, thus giving it a more prominent place. Added exaples of pointing/calibration scan reduction, skydip and beammap reductions in the 'Quick Start' section. Small corrections elsewhere. [UPDATE] {crush.jar} Updated crush-1.xx image tools (e.g. 'imagetool' 'show', 'histogram') to 1.63-13. [UPDATE] {laboca} Updated LABOCA opacities until 2010-08-18. (calibration scaling factors not updated due to a broken www interface) Also added latest RCP data for Aug 2010. [UPDATE] {saboca} Added RCP for Aug 2010. 2.00-b4 Overview ====================================================================== Hopefully the last beta release before the finalized CRUSH-2.00 hits the streets. This update bring several fixes to the prior version and a brand new capability for reducing pointing scans to obtain suggested pointing corrections. ====================================================================== [NEW] Added preliminary pointing support to CRUSH-2. To obtain pointing corrections, reduce with the '-point' option. E.g. > crush laboca -point 11564 The pointing (and calibration information) is displayed at the end of the reduction. Currently, only the values are calculated, without displaying the result (as in the old crush) since much work is still needed to migrate the display capabilities to CRUSH-2. Note, the pointing corrections are calculated independently of the map orientation, thus the use of the 'altaz' option is not at all required. [NEW] The option 'pointing' is now used for both specifying pointing corrections at reduction time (as x,y), or can be used to calculate these automatically 'auto'. The old 'center' option is now deprecated, although it will be allowed for the foreseeable future. Thus, crush [...] -pointing=-3.4,5.0 [...] can be used to specify pointing offsets of -3.4" (in azymuth) and 5.0" (in elevation), and crush [...] -pointing=auto [...] is used for determining pointing offsets from appriate observations of pointing sources (which must be bright enough to see with enough S/N in single scans!). The shorthand 'point' stands for 'iteration.[last] pointing=auto", i.e. calculating pointing corrections in the last iteration only. The option 'pointing.method' controls the method of measuring positions and can be set to 'centroid' (default) or 'peak', at present. [FIX] Fixed issue with '~' and environment variables such as {$HOME} not being accepted in the 'datapath' specification. [FIX] {cosmetic} 'write.ascii' subscan numbering to start from 1 instead of 0. [FIX] 'write.spectra' malfunction if windowname was not explicitly specified. Now fixed to use 'Hamming' as default. [FIX] 'write.spectrum' PSD normalization is fixed. Output is now in 'Jy/sqrt(Hz)'. [FIX] {GISMO} FITS time-stamp was parsed wrongly after the latest changes to the FITS file during run 3. This cosmetic detail is now addressed. [CHANGE] Added header information to various ASCII output data (e.g. pixel data, RCP) to make it easier to trace the origin of such files [TWEAK] Improved centroid pointing algorithm. [UPDATE] Updated GISMO pixel position data (.rcp) using the improved centroid fitting algorithm. [UPDATE] Updated LABOCA tau and calibration data to 2010.06.30. [UPDATE] CRUSH-1.xx tools ('show', 'imagetool' etc.) are updated to the latest 1.63-12 release. 2.00-b3 Overview ====================================================================== This is a critical update. The previous release did not have the essential FITS libraries packaged. Additionally both prior beta releases suffered from a B1950 bug, which is now fixed. ====================================================================== [CRITICAL] Precession from Besselian epochs to Julian epochs (e.g. B1950 to J2000) was messed up, due to a typo in the MJD value for B1900.0. As such, the astrometry of all scans taken with B1950 coordinates is wrong, since the conversion to J2000 is automatic in crush. [FIX] FITS libraries (fits.jar) were not actually bundled in the previous update, which instead contained a dysfunctional symbolic link to it... [CHANGE] Overhauled code for defining regions on maps. The changes may only affect the 'beammap' functionality in minor ways... [UPDATE] Laboca 'tau.dat' and 'calibration.dat' files up to 6/23/2010. 2.00-b2 Overview ====================================================================== This is an update on the first beta release of CRUSH-2. A critical bug fix addresses the reduction dependence of calibration. It has always been a stated aim of CRUSH to make calibration completely independent of the reduction options used. Unfortunately, due to a number of bugs and features this was not fully accomplished. This new update hopefully lives up to the promise. Another major area of work has been a major revision to the way the noise whitening filter works. All instrument configurations have been updated to with new whitening parameters. Otherwise, the update also contains a number minor fixes and performace tweaks. It also includes updated LABOCA and SABOCA configurations up to June 2010. ====================================================================== [CRITICAL] {calibration,deep} Identified various problems with the filtering correction scheme. The idea is to make calibration entirely independent of reduction parameters. However, due to several coding bugs, the corrections were underestimated, resulting in overoptimistic noise levels, and reduced fluxes in 'deep' mode reductions. [BUG] {GISMO} Beam map data was erroneously handled. See details below. As a result all GISMO beam maps have been updated. [BUG] {beammap} Major bug with beammap algorithm. There was a botch-up in the pixel indexing scheme as well as what quantities were calculated and written. [BUG] {parser} There was a problem parsing statements, in which '=' was used after a condition, such as "-iteration.[2]forget=clip". Such statements were effectively ignored. This is now corrected. [CHANGE] {whitening} Major changes to the noise whitening filter behaviour. The new keyword 'whiten.below' allows the whitening filter to behave not only as a rejection filter (suppressing excessive spectral components) but also as an enhancing filter, whereby components below the white noise level are scaled up in order to arrive at a truly white noise spectrum (which is necessary for obtainig maps without intrinsic spatial correlations). This new mode of whitening is made default for all instruments. Two more new keywords provide more control over the whitening filter. 'whiten.neighbours' enables a whitening based on neighboruring spectral regions (at different spectral resolutions). Such a behaviour was already implicitly present. Finally, a the key 'whiten.proberange' can be used for defining a spectral range (Hz) in which the white-noise level is to be measured. All instruments are configured to use the truly flat part of their typical pixel spectra for white noise level estimation. The changes of the noise whitening may impact the results of reductions significantly. You can restore the old behaviour via the whiten.proberange *:* forget whiten.below configuration settings. [CHANGE] New release checking will pause for 5 seconds before continuing with the reduction, when a new release is available, to bring attention to the update. Users should always upgrade to the latest (stable) version. [CHANGE] Move map filtering step (see 'source.filter') up to the scan level. This way each scan map is filtered independently, before weighting and coadding. This should provide cleaner results when source filters are used. Especially the edges of scan-coverages should be better behaved... [NEW] New option 'write.scandata.details' can be used to add more inforation into the data tables of scan HDUs, such as channel gains, weights, offsets and filter profiles, noise spectra (and more...) [FIX] {spectrum} The 'spectrum' key resulted in spectra with a messed-up Nyquist component due to a coding error. Fixed. [FIX] {correction} Filtering corrections of decorrelation steps and the 1/f drifts filtering were not applied, because of a coding error. The corrections applied only to the noise whitening filter. As a result, deep mode fluxes were slightly (or in some cases) significantly below expected. [FIX] {correction} Filter corrections of drifts would have been applied repeatedly, if not for the above bug, which resulted in not being used at all. [FIX] {correction} Calibration corrections did not take smoothing into account. Smoothing increases the filtering effect because the negative bowls around point sources are blended into the positive peaks further reducing them. With the changes in place, the independence of the calibration on reduction parameters should be fully restored. [FIX] {deep} Deep reductions were not beam smoothed as intended. This is now corrected. [FIX] {scripts} Updated shell wrapper scripts to correctly deal with spaces in command line arguments (when quoted or escaped). This is achieved by replacing $* with ${1+"$@"}, which results in the correct expansion of arguments. [FIX] {decorrelation} Problem with the hipass filtering of correlated signals. Integer math was used where floating point was necessary, resulting (at times) in incorrect filter frame counts. This is now corrected. [FIX] {decorrelation} time-scales should never exceed the 1/f filter time scales. This is also true for any other time-scale in the reduction. Accordingly, a maximum timescale is now enforced for all things... [FIX] {Minor} Arithmetic error in calculating the noise in single-plane images. This did not affect any reduction, since CRUSH never uses single-plane images internally... [FIX] {regrid} Fixed regrid signal and noise normalizations. [FIX] {show} Changed 'show' tool to disable unit changes when displaying the S/N image. This produces the expected behaviour. [FIX] {Minor} A small fix for the interpolation of image values between pixel positions to return NaN when the nearest position is flagged. [FIX] {Minor} Fast convolution method to ensure flagging of any NaNs. [FIX] {p-ArTeMiS} Reinstated 'median' estimators as default for p-ArTeMiS. Using maximum-likelihood caused unstable solutions for some scans. [SPEED] Parallelized FITS scan reading. This should result in some performance enhancement on multicore systems, although the gain may not scale with the number of cores simply because the underlying FITS library is not thread safe, thus only the processing of steps are apeeded up. [TWEAK] {faint,deep} Disable 'weighting.frames' by default. The time weighting can lead to a lot of instabilities in the reduction. At the same time, there is no evidence for non-stationary noise for any of the instruments. It is better therefore to turn this feature off in general, and enable it only when truly necessary... [TWEAK] {Minor} Performance enhancement of image operations. [TWEAK] {Minor} Performance enhancement of some iterators. [TWEAK] {Minor} Use floats instead of doubles for medians where possible. [TWEAK] {Minor} Simplified initialization of some commonly used classes, by eliminating duplicate field inits. [TWEAK] {Minor} Simplified constructor of EquatorialCoordinates and EclipticCoordinates for the default (J2000) epoch. [TWEAK] {Minor} Converted much of the precession algorithm to faster floating point math, which is sufficient given that the precession constants aren't provided to higher accuracy. [TWEAK] {show} Turned on antialiasing for text in 'show' tool. It should remove the JVM dependence of the font antialiasing. [UPDATE] Updated LABOCA RCP, calibration and tau data, which is now available up to 06/20/2010. Thanks to Andreas Lundgren for bringing attention to the information. [UPDATE] Updated SABOCA RCP information, with the most current data from the APEX SABOCA pages. [UPDATE] Updated GISMO configurations (new RCP/coupling data and various tweaks to the configuration options. [UPDATE] All instrument configurations have been updated with new noise whitening parameters. [UPDATE] Updated nom.tam.fits libraries to 1.04. 2.00-b1 (24 May 2010) ###################################################################### First official beta release of CRUSH-2. :-) ######################################################################