1.05-6 (26-Sep-2010) Overview ====================================================================== This is a maintenace update of minicrush 1.05. The update fixes an issue with the handling of filtered FITS images (brought by the updated crush-1.xx tools bundled in this release), and provides up-to-date LABOCA/SABOCA configuration and calibration data. ====================================================================== [FIX] CRUSH tools (e.g. 'show', 'imagetool', and 'detect') indicated that the map fluxes were uncorrected, when in fact they were corrected for point-sources. This may have caused calibration issues when spatially filtering maps in CRUSH and using these tools for obtaining a flux measure. The problem was traced to an unexpected change in the FITS libraries. [UPDATE] Bundled CRUSH 1.xx tools ('show', 'imagetool', 'detect'...) have been updated to 1.63-14. [UPDATE] LABOCA and SABOCA configurations (RCPs, 'tau.dat', 'calibration.dat') have been updated thru 2010.09.26, using the data from the APEX pages. 1.05-5 [FIX] FITS libraries (fits.jar) were not actually bundled in the previous update, which instead contained a dysfunctional symbolic link to it... [UPDATE] Laboca 'tau.dat' and 'calibration.dat' files up to 6/23/2010. 1.05-4 Overview ====================================================================== This is a regular maintenance update with various small fixes and tweaks. LABOCA and SABOCA configurations have been updated with the most recent from APEX (RCPs, skydip and calibration data). ====================================================================== - [FIX] Updated shell wrapper scripts to correctly deal with spaces in command line arguments (when quoted or escaped). This is achieved by replacing $* with ${1+"$@"}, which results in the correct expansion of arguments. - [FIX] Fixed regrid signal and noise normalizations. - [FIX] Changed 'show' tool to disable unit changes when displaying the S/N image. This produces the expected behaviour. - [CHANGE] Renamed FITS image planes: Intensity -> Signal, RMS -> Noise. - [TWEAK] Turned on antialiasing for text in 'show' tool. It should remove the JVM dependence of the font antialiasing. - [UPDATE] Updated LABOCA RCP, calibration and tau data, which is now available up to 06/20/2010. Thanks to Andreas Lundgren for bringing attention to the information. :-) - [UPDATE] Updated SABOCA RCP data to June 2010. - [UPDATE] Updated nom.tam.fits libraries to 1.04. 1.05-3 Overview ===================================================================== This is an update to the prior release, containing fixes and improvements, but practically no new features (other than the new tool 'esorename' for renaming files from the ESO archive). The most major big fix is the catching of an embarrasing sqrt(2) change in the calibration factor if (and only if) downsampling was used. Other than that, all shell scripts now call bash, which causes less problems on some UNIX platforms. Also a fair bit of work went into the simulator. ===================================================================== - [NEW] Added tool 'esorename' to enable renaming ESO archive files back to their original naming convention used by minicrush (or BoA). - [BUG] ASZCA reductions inadvertently inherited the LABOCA frontend gains (where n in the FITS file specifies a gain of 2^n). However, ASZCA gains are undefined in the data file (a value of -999). This is now fixed and proper readout counts are used. As such calibration has been updated on 2007 Mars scan #5707. - [BUG] Downsampling inadvertently changed the calibration by sqrt(2). Many apologies if this caused you any trouble. The problem should be fixed as of now... - [FIX] Changed gain flagging to discard pixels with negative gains (except for SABOCA, where negative gains are sometimes allowed). Also the allowed sky-noise gain range is now restricted to between 0.3--3.0 times the median value. Hopefully, the changes will help flag out bad pixels in some of the more pathological cases, without causing problems for well-behaved data. - [FIX] Shell scripts have been changed to run under '/bin/bash' instead of '/bin/sh'. This is because on some Linux system 'sh' is not linked to 'bash'. - [FIX] Updated simulator code to work with single file ESO-archived data as well as the already supported telescope format where the scan data is spread over several FITS files in a directory. - [FIX] Fixed rounding issue of DMS and HMS angle and time formats. The problem affected only the human readable formatting of the angles but had no effect on any of the calculations as the angles themselves were accurate :-). - [IMPROVE] Updated simulator code to be more functional and easy to use. The white noise levels at the Nyquist frequency can be defined on the command line, with the decorrelation group, e.g.: > ./simulator laboca -decorrelate=1e-4 -cables=3e-6 [...] This sets the 1/f^2 noise level at the Nyquist frequency. The units are V/sqrt(s) at the detector stage. The simulator can also be run from outside of the minicrush directory, for convenience. - [IMPROVE] Simplified scan reading across instruments. All instruments now use the same scan reading code, with only the frontend-backend combination defined idependently. - [CHANGE] The NEFD number printed after the weighting step is now reflecting the proper point-source NEFD, as opposed to the raw noise measure before (the difference is that the new number takes into account the point-source also). - [CHANGE] Instriduced more stringent noise flagging for ASZCA, than what is used for the other instruments. See 'relative-noise-range' in 'aszca/default.cfg' for details. - [UPDATE] Accomodated SABOCA sign switch as of 09.04.2010. - [UPDATE] Updated bundled crush tools to 1.63-10. 1.05-2 - [NEW] Added licensing information (see license.dat). From here on minicrush is released under version 3 of the GNU General Public License (GPLv3). For details, refer to 'license.dat' or see - [NEW] Added platform and Java information into the FITS header. This may make it easier in the future to identify platform-specific problems when they occur... - [IMPROVE] Made various tweaks to deconvolution and regridding algorithms. The updated deconvolution method now works independent of the sign of the source flux. - [FIX] Point source extraction tool 'detect' did not clear masks if these were supplied, leading to the masked sources being listed as detections. - [FIX] Pressing 'u' in 'show' tool (for changing the image unit) did not update the unit label. This has now been fixed. - [SPEED] improved performance of some old FFT related code. MiniCRUSH benefits from these changes only when the 'spectrum' option is used. - [CLEANUP] Close input file streams after reading to free resources. 1.05-1 Overview ===================================================================== This update fixes a few bugs in the prior release, and adds a few cosmetic improvements (esp. in the 'show' displaying tool). It is also gearing up towards the eventual switch to CRUSH-2... ===================================================================== - [BUG] Saboca signals got inverted by accident. These are now back to having the correct signs. - [BUG] Downsampling had a small recursion bug, by which downsampled frames were re-used for downsampling further frames. However, given that downsampling was not used for instruments as default, the chances of corrupted reductions was unlikely... - [NEW] Made crush image utilities compatible with the next generation image files. Starting from CRUSH version 2.0, the RMS plane will no longer be the raw observing rms, but will be a direct measure of the noise in the signal plane, whether it is smoothed or not. This will make it much more convenient for users, who currently have to smooth the noise according to a recipe provided in the CRUSH document 'CRUSH-FITS-structure.pdf'. - [FIX] Cleaned up the FITS header information. - [FIX] Made conversion from pixelization to equivalent Gaussian smoothing consistent throughout. - [COSMETIC] 'show' tool has clickable components. You can change the colorscale, toggle between linear/log/sqrt scaling, or change units with just one click. You can drag the scale to adjust its range etc. The information panel provides you with hints. - [COSMETIC] Added 'mJy/beam' unit for 'show' tool. Units can be changed by pressing the 'u' key, in the program window (or by clicking the unit label on the right). - [COSMETIC] Updated 'show' utility to include coordinates, and used more human readable formatting. Also, it now has a cleaner look. - [CLEANUP] FFT routines have been further optimized for speed, with an ever so slight increase in performance... - [UPDATE] Bundled CRUSH utilities updated to version 1.63-8. 1.05 Overview ===================================================================== This is a maintenance update, which fixes occasional problems with ESO archived data, and issues relating to internationalization. A more major bugfix, which can affect behaviour, causes an increase in the versioning. However, only the most adventurous expert users may be affected. Most other users will see no difference in the reduction as compared to the previous version. ===================================================================== - [BUG] Behaviour was unpredictable is the default pipeline ordering was changed by moving 'sourcemap' from the end. Luckily, this would not have affected most users. Now, the source is properly synched to the data before the pipeline continues to further elements - [BUG] For ESO archived files, the LABOCA Monitor table was parsed unnecessarily, resulting in an error at times. This is now fixed. The problem was also the true culprit behind the fix published in the prior release (1.04-1). - [BUG] imagetool's '-origin' option resulted in botched y-coordinates. - [FIX] In locales where a decimal comma is used instead of a decimal point, the FITS headers became unreadable by other FITS tools. While this is strictly speaking a feature of the nom.tam.fits libraries used by CRUSH, a workaround has been implemented. - [UPDATE] FITS libraries are recompiled. 1.04-1 - [FIX] Made sure that even if 'he3' is accidentally defined for instruments other than LABOCA, it will not break the reduction... 1.04 Overview ===================================================================== Apart from the usual round of small fixes and improvements, the major change is added support for P-Artemis, and an update to the latest FITS libraries. - [NEW] Added preliminary support for p-Artemis. To reduce p-Artemis data run as: > ./minicrush p-artemis [...] An initial calibration factor was determined based on Jupiter scan 20097. - [BUG] Sign problem in recently updated SphericalCoordinates when dealing with offsets. (Fixed). - [BUG] Instantiation of projections from FITS ID returned the one and same projection object all the time. This mainly caused trouble with coadd, where this method was used repeatedly for each component map. - [FIX] Fixed frame flagging by invalid RA/DEC and AZ/EL coordinates in the FITS data. - [FIX] FITS libraries (nom.tam.fits) were not 1.5.0 compatible. They have been recompiled to restore 1.5 compatibility. - [FIX] RCP and pixel data file specifications can now include system variables in the usual way... - [CHANGE] Changed 'bright' configuration s.t. it no longer solves for pixel gains or for correlated instrumental signals. This much improves the reliability of reduction in the presence of very bright sources. - [UPDATE] Updated nom.tam.fits FITS library to latest version - [UPDATE] Updated bundled crush tools to 1.63-6. 1.03-4 - [FIX] Fixed horizontal <--> equatorial conversions. At last, it ought to be possible to reduce drift scans like they are meant to be. This will especially benefit ASZCA reductions... - [FIX] '-altaz' reductions were broken for earlier scans where native frame information was not stored in the SCAN header. - [FIX] Point source corrections under whitening now take smoothing into account. This should improve systematic calibration accuracy (especially for 'faint' and 'deep' mode reductions). - [FIX] Both 4 and 5 column RCPs are supported. - [FIX] Scanning speeds are always measured w.r.t. to the celestial frame. - [FIX] Fixed NaN filter time scale problem when scanning speed is zero (e.g. drift scans with ASZCA). - [FIX] Fixed NullpointerException problem with drift scans. (ASZCA) - [UPDATE] updated 'deep.cfg' to always disable clipping/blanking. This causes no change in the default behaviour of 'deep' mode reductions. - [UPDATE] updated bundled crush tools to 1.63-5. 1.03-3 - [FIX] Various fixes to calculating source filtering effects of noise whitening... - [FIX] Updated estimation of source-filtering by the various decorrelation steps. - [FIX] Refined the calculation of average source filtering for point-source flux correction. - [UPDATE] Updated SABOCA configuration for 2009 commissioning. Note, to reduce pre-comissioning data 2008 Sep -- 2009 Jan, please uncomment the necessary lines in saboca/default.cfg, or else specify the options '-jansky' and '-pixeldata' according to the instructions found therein... - [UPDATE] Updated configurations to use maximum-likelihood estimates mostly. They seem generally more trustable and certaintly a fair bit faster too... - [UPDATE] The preferred weighting method is a slightly modified 'differencial' (the modification sets the differential delay to the typical point source crossing times). For extended sources 'rms' weights are now preferred... - [UPDATE] Updated 'tau.dat' and 'scaling.dat' for LABOCA including Nov 2008 run. - [UPDATE] Updated 'pixel.dat' for LABOCA. 1.03-2 - [NEW] The default source size for extended mode reductions is now explicitly defined in the appropriate /default.cfg files. - [CHANGE] 'sourceweights' key is no longer used. - [FIX] Whitening range doubled to bridge gap with 'drifts' filtering. - [FIX] Gain normalization has been overhauled to be simpler/cleaner. The rescaling of correlated signals, which accompany gain changes, has been also made more correct. - [FIX] Calculation of pixel weights is now more safe. Also, the calculation is source weights has been improved. - [FIX] The automatic 1/f filter setting for 'extended' mode reductions could result in overfiltering. A more conservative algorithm is now used for determining filtering settings for extended mode. 1.03 Overview =================================================================== This release fixes a few critical bugs from the prior release, which should result in better images. Some new features added, like experimental support for photometric data reduction, especially from 'N-on' type observations. There are several other small improvements also... - [NEW] Run-anywhere implemented in minicrush. That is, it is no longer required to run minicrush from inside its own directory. - [NEW] Introducing key 'sourcemodel' which can be used to supply a crush map from a previous reduction as the prior source model. This is useful for getting rid of some negative ridges around sources when reducing large datasets in parts. A combined source model can be created by coadding the maps and applying the desired S/N clip (via 'imagetool') to create a prior source model for the next reduction without the negative features. - [NEW] Incremental gain changes are now fed back into the source model. This should result in better default reductions. - [BUG] It turns out that the factor in laboca/calibration.dat was used in the wrong way, based on what I was told. Sorry, but at least this time it was not my fault... - [BUG] Source removal gains should not include filtering corrections - [BUG] One of the arrays used in the noise whitening was inadvertently sorted but then used as if it were in the original order. This is now fixed. - [BUG] Parameter dependence accounting had a bug that caused it to accumulate and run away with iterations. Now fixed. - [BUG] A small bug in the renormalization of correlated noise after the recalculation of gains is now fixed. - [CHANGE] Overhauled coordinate handling, making it both more elegant and more functional. The new code can now properly handle drift scans, which was not the case before... - [CHANGE] High-pass filtering is now a proper pipeline step rather than a preconditioning step. It can be used instead of the 'drifts' modelling, although the hipass filtering is slower (due to FFT) and is probably not all that different from the faster 'drifts' modeling. Also, in the initial iteration the hipass filtering is discouraged since large DC offsets can 'confuse' it in the presence of flagged data. The syntax of 'hipass' is identical to that of 'drifts'. - [CHANGE] New chopped reduction. The solving for chopping offsets in the pipeline can be positioned by the 'chopoffsets' entry for the 'ordering' keyword. Normally, the solving for these offsets should precede any other pipeline step. This is the default behaviour. The model is automatically activated for chopped data. - [CHANGE] Deprecated 'chopped' option as it is now absorbed by 'photometry'. When reducing in 'photometry' mode, no map is produced instead a single flux is calculated. The reduction automatically detects wobbler use, and adjusts the reduction accordingly... The 'photometry' mode is still heavily under development and is an experimental feature for reducing both wobbled and Non observations... - [CHANGE] Minicrush now produces maps if at least a third of the working pixels are unflagged. (the criterion used to be half). - [CHANGE] NEFD calculation now uses effective on-source NEFDs, accounting for source gains and filtering effects... - [CHANGE] 'deep' reduction no longer assumes point sources if 'sourceSize' is explicitly specified... - [CHANGE] The correction for the filtering effects of decorrelating has been updated to a more correct scheme. Since this was typically a few percent correction only to begin with, the change should have minimal effect on the reductions... - [CHANGE] Option 'flag' can now take pixel ranges as well as individual pixels separated by commas. E.g. "flag 14,112-123,322" - [CHANGE] Small changes to velocity and acceleration clipping... - [CHANGE] Small change to noise whitening, in the way it tries to avoid overfiltering... - [CHANGE] The 'flag' option now takes pixel ranges as well as lists. E.g. 'flag 4,12-32,35,44,56-73'. - [CHANGE] Slightly changed the way the scanning offsets are read from the DATAPAR table, making use of the RA/DEC offsets added a while back. - [FIX] Improved, more user-friendly error reporting on data reading errors. - [FIX] Gain normalization could run amok, causing NaN values during the renormalization of the correlated noise. This has been fixed. - [FIX] 'extended' setting did not adjust 1/f filter properly if 'sourcesize' was not set. Now, the 'stability' setting (or 5 seconds correspondinf to 0.1 Hz) will be used as default for such cases. - [FIX] ASZCA configuration updated to flag channels 321-331, which are repeated readouts. - [FIX] Producing covariances could break for ASZCA reduction. Now fixed. - [FIX] Occasional NullPointerException in sky gradient modeling is now fixed... - [FIX] Occasional null MJD problem fixed. - [FIX] map.generation and Scan.sourceGeneration accounting cleaner up. - [FIX] 'drifts=auto' behaviour changed s.t. the maximum automatically determined 1/f filter time constant is set by 'stability'. - [FIX] Scan gains were not incorporated into source synching, possibly leaving residuals. However, since the can gains (i.e. 'scale') were normally close to 1, the maleffect was small. - [FIX] 1/f filter ('drifts') correction now accounts for filtering even in 'photometry' mode. This is important to get the noise right. - [FIX] Weight flagging now correctly takes the degrees of freedom into account. - [UPDATE] Updated laboca tau and calibration data -> Oct 2008. - [UPDATE] Updated ASZCA configs. - [UPDATE] Updated calibration. Yields ~8% scatter on both integrated and peak fluxes on 33 Uranus scans. - [TUNE] Less stringent initial clipping in default and 'faint' reduction modes. - [TUNE] Following the bug fixes, the configurations have been adjusted for optimal performance. Iterations in 'faint' and 'deep' mode have increased. - [TUNE] Maximum-likelihood estimators are rehabilitated for 'faint' and 'deep' reductions. - [CLEANUP] Removed several development classes that were no longer in use. 1.02-2 Summary ======= This is a bug-fix re-release of 1.02. Unfortunately all but the 'deep' mode reductions of the prior 1.02 release were broken. Sorry for that. - [BUG] Source removal had a nasty bug due to a typo. It was not functioning correctly at all, causing problems in all but 'deep' mode reductions. Very sorry for that. - [BUG] Calculating source weights could end up hanging in default and mode, or generally when 'sourcesize' was not expliticly specified. - [BUG] Erroneous behaviour when data file fell outside of the tau calibration scaling interpolator range. When this happened the calibration value used was going wild... - [FIX] Individual scanmaps were written without casting into the desired mapping unit (i.e. as raw uV). This is now corrected. - [TUNE] extended filter now automatically applies filter corrections for point sources. Also, the correction is made smart to avoid double corrections. 1.02 Overview ======== This release fixes a few critical bugs from prior versions. Most importantly, the noise whitening filter was not behaving as intended. Also, the He3 temperature correction for LABOCA was broken. This version builds more heavily on commonalities among instruments thus minimizing the possibility of some functions working for certain instruments but not for others. There are a number of new convenience features introduced in this release also such adding external calibration support (via a calibration data file), and automatic pixel-flags and RCP determination based on MJD (as defined in 'flagging.dat' and 'rcps.dat' inside the instrument subdirectories. The bundled 'crush.jar' file is updated to 1.63-1. All in all, this version brings many improvements, which at the end of the day should result in better images. Details ======= - [NEW] Added option 'sourceweights' allowing to calculate separate pixel weights for source modeling. If not specified, then the usual pixel weight will be used. The two are equivalent if the pixel noise is white. However, in all other cases, it is a good idea to calculate source weights separately. - [NEW] Added option 'relative-pixel-noise' to give user control how pixels are flagged based on their derived noise properties. The range specifies acceptable noise levels relative to the median pixel noise. A range is specified by the minimum and maximum allowed values separated by any '-', ':', ',', or white space characters. E.g., relative-noise-range 0.1--10 will allow channels with noise up to a decade more or less than the median pixel noise. - [NEW] Introducing 'time-weighting' in the pipeline. The time-weighting is performed on blocks of size equal to that used by the drift modeling (for now...) - [NEW] Added preliminary support for SABOCA. - [NEW] Added options 'spectrum' to write channel power spectra as an ascii table. Possible arguments are the name of the window function to use and the binning size (in powers of 2!), separated by a comma if both specified. The default window is 'Hamming' and the default binning is 512. - [NEW] Added option 'sources' to insert artificial sources in a real map. The argument is a filename to a standard CRUSH mask file. However, the peaks need to be specified in volts. - [NEW] Added key 'rcpgains', which when present forces MiniCRUSH to use the source gains directly from the RCP, rather than the gains that are calculated based on the correlated sky response. Don't forget to change the calibration factor to match BoA's (which is defined for exactly this case.) - [NEW] Added key 'maxHe3rms', which specifies (in mK) a maximum allowed base temperature fluctuation rms. If the scan exceeds this limit, it is removed from the dataset under reduction. This helps getting rid of scans where temperature fluctuation was excessive. - [NEW] Automatic cross-talker flagging based on MJD ranges defined in 'laboca/Laboca-cross.dat'. - [NEW] Added key 'uniform' to specify using uniform gains at start rather than the gains read from the pixel data file... - [NEW] Effective point source NEFD is added to console output for each scan as the last number in square brackets. As opposed to the round bracket values derived when weighting, this number take into account the point-source filtering corrections, thus providing the most reslistic average sensitivity estimate. - [BUG] Another noise whitening bug found and fixed: The time-series array that was used for the FFT calculation was not completely cleared from pixel-to-pixel, resulting in junk data remaining from one pixel to the next in the calculation. - [BUG] Weighted median was wrongly calculated. This is hopefully fixed now. Problems could have resulted when median estimators were used instead of the default maximum-likelihood ones. As a result of the fix, median estimators are now default for signal estimation except for 'deep' mode reductions... (but not for gains and weighting).. - [BUG] Whitening did not work as intended, as median estimation has destroyed the order of the spectral array, thus applying the filtering at the low frequency end of the spectrum. - [BUG] Pixels that had no position definition in the RCP were not 'placed far enough' to be flagged for impossible position later... - [BUG] He3 temperature correction was quite broken, in effect not doing anything (since the temperature gains were not read). There were several other issues as well, which are now fixed... - [CHANGE] Various changes to noise-whitening filter, to improve performace and calibration. - [CHANGE] Rehabilitated decorrelation on amplifier boxes. Some scans clearly show box-correlated signals, while this may not be so obvious with other scans. However, to get clean results, box decorrelation is now back... - [CHANGE] The individual scanmaps generated by the 'scanmaps' option now carry the MJD day as well as the scan number in their filenames to allow for duplicate scan numbers year after year... - [CHANGE] The output of "pixeldata=write" option now contains weights derived with point-source filtering correction. - [CHANGE] Changed behaviour of 'tau' configuration key. It can now be used either to specify a numerical zenith tau value, or it can specify a file to a tau table, from which values can be interpolated. - [CHANGE] Similarly to above, the parsing of the 'scale' key is also changed. It can either specify a numerical scaling value, or values can be looked up from appropriate calibration tables by the ISO timestamp. - [CHANGE] Switched to 'differential' weighting scheme. - [CHANGE] Automatic rcp-file lookup for LABOCA, based on MJD ranges defined in 'laboca/Laboca-rcps.dat' - [CHANGE] Gridding changed to ~1/5 beams. The finer grid results in better calibration. - [CHANGE] In 'deep' mode, now whitening first, before looking for time-stream spiky features. This makes better sense because spectral features are almost always present, and spikes tend to have a white spectrum anyway. Thus, the presense spikes does not distract the whitening, but the spectral features may appear as time-stream features... - [CHANGE] Whitening is now applied to all pixels not just 'good' ones. Also, the method now scales excess spectral power to the excess limit, rather than to the median spectral power. This results in less discontinuous specral shapes/filtering -- meaning less artifacts in the map. - [CHANGE] Reading pixel data files adds pixels flags to those already defined. Previously, it set flags ignoring prior values. - [CHANGE] More shared code among instruments. Helps to keep things synchronized with changes :-). - [TUNE] Turned off 'scanweighting'. Weight estimation shuold be good enough to arrive at the fait weights. However, scanweighting is performed before smoothing/filtering, and thus will not be representative of the final map. For this reason scan-weighting is now only enabled in 'faint', 'extended' mode. - [TUNE] Despiking by the 'features' method has been removed from the 'deep' mode configuration ('deep.cfg'). It could go 'crazy' with certain whitening settings and time-weighting. Instead, time-weighting and despiking should deal with the problem of wide spikes. This is the approach taken from now... - [TUNE] Recalibrated LABOCA conversion factor and taufactor. - [TUNE] Small change in setting the non-flagging limit for time-domain features. - [TUNE] Adjusted automatic filtering settings to try minimize overfiltering of timestreams. - [TUNE] Changed scan initialization order. - [TUNE] Raised clipping/blanking and scanmap despiking levels such as to minimize the unwanted scenario of noise peaks falling false casualty to these. The result is more reliable maps down to the detection limit. - [FIX] Slight pixel weight normalization in beta release 1.02-b1, due to misaccounting of the degrees of freedom... - [FIX] Whitening fixed s.t. it better cooperates with flagged data. This is achieved by the filter subtrating an excess noise signal rather than replacing signal with the filtered one, as before. The previous method overwrote all spikes and flagged data, thus the original information (e.g. source) were lost in the flagged bits. The new method preserved information in the flagged parts, only modifying it as needed. - [FIX] Small discrepancies in the dependency accounting (used for proper weight normalization) were fixed. - [FIX] 'gains median' did not actually switch to median gain calculation for group correlated models due to a coding bug. This is now corrected... - [FIX] Median estimators could cause problem when a single data point diminates the data set. In such cases the median would be too heavily dependent on that one point, causing it to be excessively cleaned and overweighted. Thus a safety pin was introduced that checks to make sure that the data set on which the median is evaluated is not too heavily dependent on a single point. When it is, the estimate reverts to the usual maximum-likelihood one. In short, this is to say that medians are good estimators when there is lots of data, not just a few points... - [FIX] Gain normalization is followed by an appropriate renormalization of the correlated noise estimate. The lack of this before was not necessarily a problem since initial gains were subject to the same normalization already, thus renormalizing factors very close to 1 resulted. Nonetheless, this is all more proper now. - [FIX] A little more consistency over pixel flagging. Blind channels are now retained and used in correlated signal estimation. They are not used for source mapping (for obvious reasons). - [FIX] Adjusted the use of weights for mapping to avoid double counting the filtering effects... - [FIX] The effect of whitening is now icorporated in the channel weight estimates. - [FIX] Fixed noise whitening normalization in light of Parseval's theorem... - [FIX] Filtering of extended structures had blanking at 3-sigma. Thus several noise peaks in a typical map would be blanked, making the filter biased lower, and thus elevating some positive noise peaks to above detection threshold. Now the filter blanking has been raised to 5-sigma, which should minimize such undesired behaviour... - [FIX] Some instrument data file names were hardcoded with the UNIX style file separators '/', making these unusable under Windows environments... 1.01 Overview ======== This update fixes many of the outstanding bugs in the prior (1.00) release, and adds some new features, such as support for the newly commissioned photometric observing modes for LABOCA. There has also been an overhaul of the scripting/configuration support, with slight changes to the syntax (see below and in the README). The README file has been updated, and the 'Quick start' section has been expanded into a more detailed starter's cookbook. Details ======= - [NEW] New option 'photometry' is introduced to allow reduction of data taken in the new photometry mode of LABOCA. Instead of making a map, a single point source flux is calculated and displayed at the end of the reduction process. A word of warning: this functionality is still in its experimental state. Expect it to evolve and finalize some time later... - [NEW] New option 'scanmaps' will write individual maps for each scan in the reduction as 'scan-.fits' in the usual output directory. - [NEW] New key 'stability' can be used for defining the 1/f stability time scale in seconds. This value is used for determining optimal filtering parameters for the reduction (e.g. for 'drifts' option), unless explicit values are specified. - [NEW] 'boards' is now defined for ASZCA as 10 horizontal stripes on the array. Decorrelating on these helps to arrive at cleaner timestreams. - [NEW] Added FITS keyword 'INTEGRTN' describing the total integration time (in seconds) for the map. - [NEW] The option 'spikewidth' can now take 'auto' as value, which will look for features up to the time scale determined by the 1/f filter setting... - [NEW] Rough ASZCA calibration on Mars (single scan at 30 deg). Peak 1.07uV for 65.57 Jy. Assuming 2mm water --> 58.3 Jy/uV. - [NEW] Multiple fixes and improvements in ASZCA reduction... - [BUG] Pixels in multiple subscans weren't correctly handled due to a cloning bug... - [BUG] 'offsets' was not working due to a typo. In any case, 'drifts' was the default and preferred way of obtaining residual DC offsets. - [BUG] As frames were slimmed to the minimum necessary size, the sample flags were erroneously copied. This did not have much maleffect in standard reductions... - [BUG] Noise whitening had an inverted scaling problem. Now fixed. - [BUG] ASZCA reduction was broken. Now fixed... - [FEATURE] LABOCA option '-he3=thermistor' was not effective due to a bug in reading the monitor table (Actually, the bug was in the FITS library). A workaround now corrects this. - [FEATURE] Median scanning speeds were off by a factor of 2, resulting in a bit of overfiltering at times... - [CHANGE] Slight changes to syntax. The new syntax is more coherent and less error-prone. The bigger changes are: * Key/value pairs can be separated by '=', ':' or white spaces or any combination of these in all places. Restrictions on the command line mean spaces cannot be used there. * the old 'iteration:XXX' keys are replaced with 'iteration.XXX'. * 'forget=iteration' is replaced with 'iterations=clear' for clearing iteration-based settings. * extendedopts -> extendedopt, and each settings compounds the previous one. The separator for shorthand listing is '|'. See README for complete details. - [CHANGE] Some renaming of the decorrelation groups, for better consistency between LABOCA and ASZCA. The following groups are now defined: (1) Detector groups wafers (ASZCA only -- was 'squidgroups') regions (ASZCA only -- 12 horizontal strips) (2) Electronic groups boxes (LABOCA & ASZCA) cables (LABOCA & ASZCA) amps (LABOCA & ASZCA) In case of ASZCA, cables and amps are effectively the same (1:1 mapping), so one should only use one of these, not both... - [CHANGE] The key 'whiten' for noise whitening now can take an optinal relative spectral amplitude (over the median) above which the whitening is performed. This is to accomodate settings other than the default value... - [CHANGE] LABOCA automatically loads the new RCP file for scans after the February 2008 alignment (from laboca/master-laboca-feb08.rcp). (Otherwise laboca/master-laboca-oct07.rcp is used). - [CHANGE] Disabled decorrelating on amplifier boxes for LABOCA by default. Instead decorrelating on 'cables' alone is encouraged. - [CHANGE] Cable gains were added to laboca pixel data file. These are now loaded as initial values, increasing the speed of convergence. - [CHANGE] Removing atmospheric 'sky' gradients is no longer default. There was little evidence for these at 850um for LABOCA. Therefore this reduction feature is now disabled. To re-enable just specify the option 'sky' either in the relevant configuration file or on the command line. - [CHANGE] Further changes to MEM. The target model used to be a beam-smoothed version of the map. This means that the MEM solution is subjected to very much the same artefacts (e.g. negative lobes) that affect the maximum-likelihood maps. From now, a more extremely smoothed map is used (with 5 beam widths), which should provide more realistic behaviour for low signal-to-noise signals both when a faint extended emission is present or with flat (zero) backgrounds. - [CHANGE] MEM correction is now applied after all other post-processing steps. All steps are performed, including smoothing. The MEM target model is generated from the beam-smoothed processed map by enforcing positivity. - [CHANGE] Revisited MEM algorithm. Changed entropy deifintion from H = - sum_i (1+A_i/sigma_i) log((A_i+sigma_i) / (T_i+sigma_i) - 1) to H = - sum_i sqrt(1 + A_i^2 / sigma_i^2) * log(X_i - 1) where X_i = sqrt(A_i^2 + sigma_i^2)/sqrt(T_i^2 + sigma_i^2) This definition has the following advantages over the original one: * Works for both positive and negative signals and models * lambda > 0 assures convergence towards the model * No need for explicit positivity regularization A a result of the last property, the previously introduced mem-clip, mem-absolute and mem-square options to sourcemap have been abandoned, and instead only 'mem' is retained. - [CHANGE] Behaviour of 'extended' mode under the default configuration has changed. Instead of iterating longer, it now suppresses decorrelation of band cables (for laboca) and squids/squidgroups (for aszca) until the third iteration. - [FIX] Bi-directional gain normalization was slightly off in the way the weighted average was calculated. This should have had imperceptible effect on prior reductions. :-) - [FIX] Source blanking stayed active in 'deep' mode. This is unnecessary as the benefits of blanking are only reaped when the source map is iterated. However, in 'deep' mode, only a single source map is produced. Thus blanking is now turned off for deep reductions. - [FIX] Filter blanking was uncannily tied to the regular source blanking setting, rather than using its own specification. This is now corrected... - [FIX] Syntax problem found in some configurations specifying 'extendedopts'. The configurations have been updated accordingly... - [FIX] Cross-talking channels weren't thrown out. They were flagged nevertheless, causing no harm... - [FIX] Changes to the usage of HIERARCH keys. - [FIX] Trimming after downsampling. Without it, the downsampled data could start or end with null frames, causing problems with some routines like logging. - [UPDATE] Updated 'laboca/pixel-12067.dat' and 'aszca/pixel-5700.dat' - [UPDATE] Updated configuration files. 1.00 - [NEW] Can now use non-default RCP files via the 'rcp=' - [CHANGE] 'pixeldata' option changed s.t. it can both read and write pixel defaults (gains, weight and flags). Syntax is either pixeldata=write Write the pixel data. pixeldata= read from - [NEW] Can now specify a list of pixels to be flagged explicitly via the 'flag=ch1,ch2,...chn' option, which simply takes a comma- separated list of backend channel numbers. - [NEW] Implemented MEM corrections to source map. The Maximum-Entropy mapping can be activated by selecting one of the options below for the 'sourcemap' keyword: mem-clip mem-absolute mem-squares These correspond to three approaches for ensuring positivity, itself a requirement of the MEM method. The first simply zeroes the negative parts of the map. The second uses absoulute values, while the third uses the squared-values in the MEM equation. The option 'mem' is identical to 'mem-clip'. Additionally, the "closeness" to the pure MEM solution can be chosen by lambda recommended to be in the range 0--1 (although higher values are admissible). The value zero applies essentially no MEM correction to the Maximum-Likelihood map, while higher numbers increase the importance of the Maximum-Entropy property relative to the Maximum- Likelihood property of the map. - [NEW] Subscans are merged for reduction. LABOCA should normally be operated in continuous mode, but sometimes that is not the case, leaving one with many subscans for the same scan. Now these are all merged as if they were part of one continuous data set... - [NEW] Implemented point source flux corrections for the various filtering effects of auxillary signal models. This is mainly affects reduction in 'deep' mode where only one source generation is produced. In other modes, iterating of the source map with clipping and blanking settings makes such corrections unnecessary for structures above the critical levels, but it may still help achieve faster convergence, thus its not completely without use then either... For the faint structures, below clipping and blanking, a different method can be used, which is a little more approximate and can be activated by the 'scanmap-correction' key. Neither of these methods applies corrections for the explicit filtering of extended structures via 'filter'. This can be done manually or via 'imagetool', which is filtering aware... - [NEW] New key 'nefd-range' allows to specify an acceptable range for the derived NEFD. Only when the average scan noise level, as measured from the data, falls within this range will it be used to contribute to the mapping. Syntax is either: nefd-range , or nefd-range The values are to be supplied in Jy sqrt(s) units. - [CHANGE] Changed laboca beam size from 18" to 19.5". This is more realistic given beams measured on point sources. - [CHANGE] Recalibrated LABOCA after the new filter corrections were implemented. The calibration has been separated into aperture calibration which is found to be 5.33 Jy/uV (in laboca/default.cfg) and points-source or peak-flux calibration, which is measured at 5.43 Jy/beam / uV (in laboca/deep.cfg). The aperture calibration is default for all but 'deep' reduction modes, while peak calibration is used for 'deep' reductions. Fluxes are generally expected to be accurate to 10%, without additional corrections. - [CHANGE] Filtering of extended structures now takes an optional second argument to specify the filter-blanking S/N level. When not explitictly specified, the regular 'blank' value is used. Thus the new syntax is: filter [blanking-S/N] - [BUG] Median calculation was a little buggy. It's not a huge deal because medians were rarely really used. But now they can be used!!! Indeed, medians may be preferable to maximum-likelihood estimators for the various auxillary models as it has little or no filtering effect on the source signals... - [FIX] In some cases unrealistically large pixel weights could result when decorrelation happened on pixel groups that had all but one member flagged. By restricting decorrelation and gain measurement to groups that have more than a single useful pixel, this is hopefully corrected. - [FIX] Small changes in the way file names are processed that should enable running minicrush under Windows platforms. This is not however tested. If anyone is willing to try and report problems to me at then I promise to fix those quickly :-) - [FIX] The FITS key 'V2JY' in unfortunately defined inversely for SHARC-2. For compatibility reasons, the minicrush output is made to comply. Thus the value actually measures voltage response to 1 Jy flux in a detector pixel (not the other way around!). - [FIX] Small bug in discovering pixel neighbours. Some invalid neightbours may have been added. This caused no problems whatsoever but it was not the intended behaviour... - [TWEAK] Tweaked 'faint.cfg' to be more tolerant to the potential presence of brighter features. - [TWEAK] Median correlated noise removal speeded up, by recycling temprorary storages. 1.00-rc2 Overview ======== Unfortunately, the prior release was extremely buggy, to the extent of being dysfunctional. Multi-threading especially was broken, affecting those running on multiple CPUs. After a thorough bug-fixing, minicrush was optimized for speed as well as for maximal image depths (esp. in deep modes). Further problems have been fixed with ASZCA reduction, which was also broken. Hopefully, this version is worthy to be called a release candidate (unlike the previous one). Details: ======== - [CHANGE] The option 'vclip' now takes one or two arguments in the form: vclip minv [maxv] where 'minv' and 'maxv' are minimum and maximum admissible scanning speeds in arcsec/s, and the second argument is optional. - [BUG] Option 'blacklist' was not cumulative thanks to an incomplete code change at some earlier point. Sorry. - [BUG] ASZCA-processing was dysfunctional in several prior releases due to an outdated initialization sequence, and a zenith tau that defaulted to NaN. - [BUG] More bugs in multi-threaded map-making. Some private variables that are meant to speed calculations, were shared among parallelized objects, causing them to 'crosstalk'. - [BUG] Frame spiky flags were not properly revised in successive despiking rounds. - [BUG] Sky gradients gains were botched, resulting in unpredictable behaviour. Gains are 'shifted' such as to keep the expected gradient over a constant field zero (as it ought to be). Same adjustment made for LABOCA cable twisting estimation. - [BUG] Despiking method 'features' did not fully clear sample flags from prior rounds of despiking. May have resulted in overflagging... - [BUG] When source modeling was turned off for some iterations, like in faint.cfg in 1.00-rc1, the source map got completely reset, causing the features modeled that far to be be flushed.... - [BUG] In prior version map sizes were wrongly calculated, often resulting in an ArrayIndexOutOfBoundsException being thrown when trying to actually make the map. Sorry. - [BUG] 'whiten once' directive caused misbehavior when multiple scans were reduced... - [FEATURE] 'gains maximum-likelihood' had no effect in prior release. Gains were always solved with median estimators. - [FEATURE] 'scanmap-redundancy' in 'default.cfg' was not well defined and thus reverted to the default value of 3... - [FEATURE] Channel flagging following despiking is now essentially the same no matter what despiking method was used, with only a correction for increasing the allowance if 'features' method was used AT ANY POINT PRIOR. In the previous release, channels that have been flagged for spiky features could have become unflagged by another despiking method under certain circumstances. Now spiky channel flags stick like they should. - [FEATURE] Spikes were not marked for data that is excluded from modeling. This is probably wrong, since the modeling flags can later be cleared, but the spike should stay active. It is another matter whether to count such as spike or not... - [FEATURE] 'gains median' had no effect except correlated noise model. This is now fixed. - [IMPROVEMENT] Source blanking has been combined with synching, making it considerably faster. As a result blanking will not take place under 'sourcemap nosync'. Another effect of the change is that blanking flags are cleared when blanking is disabled, which was not the case prior to this change. Clearly, this is the preferred behaviour. - [IMPROVEMENT] 'deep' configuration updated based on extensive testing with different settings. - [IMPROVEMENT] Speeded up map projection calculations by allowing compiler to inline most of the calls. - [NEW] Added 'Radio' (GLS) projection to the list of usable projections. This is mainly to comform with BoA's updated output. (Careful, it isn't really tested, but what could possibbly go wrong...) - [NEW] Average (median really) scan speed is calculated and used to determine optimal 1/f filtering for given source sizes. - [CODE] Cleaned up code and removed a lot of unused functions. - [UPDATE] Updated files for ASZCA (configurations and pixel.dat). - Small cosmetic changes... 1.00-rc1 - All configurations files have been created anew from the ground up. - [***BIG BUG***] In 1.00 beta versions, things went awfully wrong in all but 'deep' reduction mode. Basically, whenever the source map solution was iterated, garbage resulted. Extremely sorry if this affected you... - [BUG] In prior 1.00 versions the reduction crashed when the number of scans processed was smaller than the number of CPUs used. Sorry. - [NEW] Gave more control over what exatly the 'extended' flag does. Use the new key 'extendedopts' to specify a ;-separated list of options to set when 'extended' mode is selected. E.g., extendedopts drifts 10.0; blacklist sky will set the 1/f filtering timescale to 10 seconds and will never solve for sky-gradients. - [NEW] key 'taufactor' can be used to multiply tau values from skydips (or equivalent). Axel's research into the calibration revealed that skydip tau values are underestimated by a factor of 1.3 (+-0.1). Thus setting 'taufactor 1.3' makes the skydip information approximately correct. A possible explanation for the discrepancy is the optical loading dependence of bolometer gains, which can be effectively modeled as G ~ exp(-alpha * tau)... - [NEW] All configuration options, as effective during reduction, are written into the FITS header as HIERARCH keys + values. - [NEW] Can now specify a minimum redundancy each map pixel should have in a map generated from single scans via the 'scanmap-redundancy' keyword. This is useful to avoid using really underexposed mapping areas for each scan. - [NEW] 'weighting' can now take on options, specifying the method to be used. The following methods are available: rms Use the standard 1/rms*2 weighting robust Measure 'rms' using robust estimators (which are more resilient against spikes) differencial Measure 'rms' based on differences between two data points separated by some time in the time stream. This method is not biased by long term variations, but may not be 'fair' because it only measured noise in certain spectral locations rather than full band. - [DEPRECATED] Removed initial gain iterations, thus the corresponding 'gainrounds' and 'ordering:gains' configuration keywords no longer have any effect. - [FEATURE] default configuration was too strict, resulting in overflagging on bright sources. This has been relaxed. Now the default config ought to be good for up to a few hundred Jy... - [FEATURE] In prior 1.00 versions the console output sometimes indicated that source was solved during gain iterations even though it was not always true. This did not really affect the reduction, only the reporting (thus it was a feature not a bug...) - [IMPROVEMENT] default reduction now proceeds more gradually. Maps are clipped in relation to subsequent despiking levels, allowing fainter features to pass through additional despiking before recovered. - [IMPROVEMENT] Speeded up median offset removal routine, as well as allowed offset to be removed even if based on single datum. - [IMPROVEMENT] Further tweaks to despiking. 'neighbours' method now flags both suspect neighbouring frames as spikes. Critical spike fraction and count for channel flagging is accordingly adjusted to 2x the normally tolerated value. - [IMPROVEMENT] Spiky frames and spiky channels are flagged after each round of despiking. Thus the 'despike frames' option is no longer necessary and is therefore removed. The number of tolerated spikes per frame is set by the 'framespikes' option. - [IMPROVEMENT] Flagging spikes has been diversified to distinguish between three different spike flags, depending on the algorithm used for despiking. This way the different despiking method do not reset each others flags, which should make despiking a little more reliable... - [IMPROVEMENT] In 'deep' configuration, the default now is to iterate 10 times. This should result in somewhat better deep fields. - [IMPROVEMENT] Noise whitening algorithm smartened up. Will scale down excess power only, and renormalizes to keep median absolute spectral amplitudes unchanged. It works only on pixels that carry 'recoverable' flags, and skips over flagged or blanked data like other models. - [TWEAK] sky-gradients and LABOCA 'twisting' cables modeling tweaked to better complement correlated noise and correlated cable modeling respectively when not all cable pins are active (or not equally sensitive). - [TWEAK] Changed behaviour of 'gains' keyword. It now enables solving for gains for ALL signal models which have that possiblibity ( decorrelate, cables, boxes, boards, squids, squidgroups), unless these have the 'nogains' option set otherwise. Also, 'gains' can take an additional argument 'median' to specify that these should be estimated using medians rather than maximu-likelihood estimates. The 'median' method is more resilient against spikes, and it is therefore default. - [TWEAK] Gains are normalized over the average of pixels that 'see' and have not been previously flagged for abnormal gain behaviour. - [TWEAK] Source maps are not synched to data with incremental gain changes. This should result is slightly increased speed and more robust behaviour in the presence of spikes... - [SMALL BUG] in the way of accounting lost degrees of freedom during mapmaking. Used entire map for the estimate rather than scanmap... - [SMALL BUG] Scanmap leveling only in first map generation. - Source map initialization and source synching to timestreams are now also multithreaded making these operations faster when more than one CPUs are available for the calculation. - Updated the way the arrays are projected onyto the sky, accounting for the the fact the BoA array parameter files use the same offsets as the control system, which itself is not quite GLS... - Relaxed despiking in 'faint' mode to allow for up to 10 Jy sources to pass unflagged. This is useful when one is looking for faint structures around moderately bright centers (e.g. spirals around the brightish galactic nuclei). - Adjusted calibration to taufactor=1.3. Some 40 Uranus scans from the technical run yield blind calibration accurate to 6-7%!!!. - Updated external crush utils ('crush.jar') to v1.61-1. 1.00-b1 - Parallelized processing to use use multiple CPUs if available. In multi-processor machines, one can specify the number of CPUs that are to be 'left-alone' by crush, via the 'reservecpus' key. E.g. reservecpus 2 will instruct to use all but 2 CPUs available. Naturally, at least 1 CPU will always be utilized, even if too many are put on reserve... The number of parallel threads will be the smaller of the allowed number of CPUs and the number of scans processed. - [BUG] 'estimator median' instruction was not fully implemented earlier. This version corrects the omission. - [BUG] squid-groups decorrelation (ASZCA only!) was not active due to a typo in the code. Apologies if this affected you. - Despiking setting sticktened up again to provide better image depth. - [FEATURE] 'default.cfg' erroneously had 'featurewidth' key instead of 'spikewidth' specified. - [NEW] minicrush now checks for version updates and notifies when new versions are released (similarly to CRUSH). 0.99-5 - Changed noise whitening algorithm to whiten to median power in the spectral band between the 1/f filter knee and f_c/2. Ideally this part of the spectrum should be flat, as it is unaffected both the 1/f noise at low frequencies (provided the filter is well configured) and by the roll-off of any anti-alias filtering at the higher frequencies. Note, whitening is not enabled by default, and it is not recommended except as an experimental feature. - Changed map despiking algorithm to make it a little more robust against the presence of faint point sources. This should minimize the chance of accidental 'despiking' of point sources in 'deep' mode. - [BUG] 'tau' keyword, for specifying tau explicitly, was ignored when sky-dip data was also available. - [LABOCA] Disabled decorrelation on amplifier boards. The redundancy here is small (<= 5 pixels per board) making gain fitting unreliable. - [LABOCA] Fine-tuned configurations. Changed 'features' despiking parameters. - "scramble" key was not functioning. Fixed. - Updated external crush utilities to 1.61-b1. 0.99-4 - updated configuration files. Some more stringent despiking for deep fields... - updated crush.jar for external crush tools. - option 'featurewidth' renamed 'spikewidth' which was already the form used in the configuration file... - [FEATURE] It looks like ESO archive merges the several APEX fits files into a single fits per scan (which makes sense). However, minicrush was not trained to deal with these yet. This has been patched so should work now... - [BUG] UNIX shorthand '~' for home dirs and environment variables did now work for the 'outpath' option. Now should be fine. - [UPDATE] Refreshed spherical coordinate classes from tested crush-2 project (this should not affect the behaviour in minicrush). 0.99-3 - [BUG] In external tools (e.g. 'coadd' and 'imagetool'). Again, in last version introduced some new bugs with the projection update. This one was a miscalculated precession and coordinate epoch when precessing equatorial coordinates. All good now (hopefully)... - [BUG] writing equatorial maps using the external crush tools (e.g. 'coadd' or 'imagetool') had the wrong reference position in FITS (it was inverted by mistake). This bug was introduced only in the last beta relase with the projection update... - [BUG] imagetool's '-clip=' option was slightly misbehaving for equatorial maps following the projection update. This was only introduced in the 1.60-b1 release, and affects no earlier releases. 0.99-2 - [FIX] Updated 'faint.cfg' and 'deep.cfg'. Deep mode accidentally solved for source too early.