1.63-15 - [NEW] Reading of SMURF images is now possible, although not all SMURF image features are supported. Specifically, SMURF allows the creation of images with non-square pixels, or with arbirtrary transformations linking the images indeces to RA/DEC coordinates. Neither of these are currently supported by CRUSH, which will open these images nonetheless, after providing a 5 second warning, but will not be able to extract proper coordinate information or even aspect ratios. Thus, be very careful... - [NEW] New 'imagetool' option '-masks=extract' can be used to extract sources from a map based on a prior catalog (mask). The rms pointing error to assume during the extraction can be given as a second argument, separated by a colon. Thus: > imagetool [...] -masks=extract:2.5 [...] will extract the catalog of sources, assuming a 2.5" rms pointing accuracy. - [TWEAK] Searching for inter-pixel peaks is made more robust by adding sanity checks. Such searches are used by source extraction or by the "-masks=match" option. Prior to this change, the algorithm could occasinally produce unrealistic peak values, when used on region boundaries. 1.63-14 - [BUG] Due to a surprize change in behaviour in the FITS libraries, the flux correction of images was not recognized by CRUSH. This is now rectified. - [BUG] RMS image planes of CRUSH-2 images were wrongly normalized. In CRUSH 1.xx, the rms plane always carried the raw observing rms, while in CRUSH-2 the switch has been made to use the rms that corresponds directly to the smoothed image. While the necessary changes in the tools ('show', 'imagetool', etc.) were made a while back to read the newer images correctly, the implementation was lacking on the writing side, with the outputs from these tools carrying the RMS plane in the old fashioned way, but indicating, at the same time, that it was according to the new convention. 1.63-13 - [IMPROVE] Added image preprocessing capability to 'histogram' via 'imagetool', making it easier to generate histograms of images manipulated on the fly. - [COSMETIC] Updated the help screens of the crush tools (e.g. 'imagetool', 'show', 'detect', etc.). - [COSMETIC] Removed duplicate e-mail addresses from the executables. 1.63-12 - [FIX] Enforcing the default number formatting locale to print numbers with decimal points (never commas) into the FITS headers. This may resolve issues in non-English locations where decimal commas are commonly used... - [FIX] Cleaned up FITS headers for better imagetool compatibility with CRUSH-2. Added 'SMTHRMS' key to indicate whether Noise image is smoothed or not. Also, the keys 'ORIGIN', 'TELESCOP' and 'INSTRUME' now carry thier values over correctly. Finally, extraneous keys are not inherited from the input headers. - [TWEAK] 'F' in 'show' tool lists fluxes for only those regions, which can be extracted from map. 1.63-11 - [FIX] Fixed regrid signal and noise normalizations. - [FIX] Changed 'show' tool to disable unit changes when displaying the S/N image. This now produces the expected behaviour. - [CHANGE] FITS image utilities (e.g. 'imagetool', 'coadd' etc.) no longer carry over the scan HDUs. It should suffice to have this information in the original images produced by the reduction. - [CHANGE] Renamed FITS image planes: Flux -> Signal, RMS -> Noise. - [TWEAK] Turned on antialiasing for text in 'show' tool. It should remove the JVM dependence of the font antialiasing. - [UPDATE] Updated nom.tam.fits libraries to 1.04. 1.63-10 - [NEW] Option '-symmetric' added to 'extract' tool. By default the image noise is estimate only from the negative flux distribution. However, this is not very robust for small maps. Therefore, one has the option of the more robust (and more conservative) symmetric noise estimation. - [BUG] The option '-origin' was not behaving properly in 'imagetool'. - [FIX] Fixed rounding issue of DMS and HMS angle and time formats. The problem affected only the human readable formatting of the angles but had no effect on any of the calculations as the angles themselves were accurate :-). - [FIX] Shell scripts have been changed to run under '/bin/bash' instead of '/bin/sh'. This is because on some Linux system 'sh' is not linked to 'bash', causing errors when running the scripts. 1.63-9 - [NEW] Added platform and Java information into the FITS header. This may make it easier in the future to identify platform-specific problems when they occur... - [BUG] Changing which pieces of pixel data (gains, flags, weights and offsets) could not be unset. This caused problems when the values stored in the FITS were corrupted, as there was no way of overriding the behaviour. This is now fixed. - [IMPROVE] Made various tweaks to deconvolution and regridding algorithms. The updated deconvolution method now works independent of the sign of the source flux. - [FIX] Point source extraction tool 'detect' did not clear masks if these were supplied, leading to the masked sources being listed as detections. - [FIX] Pressing 'u' in 'show' tool (for changing the image unit) did not update the unit label. This has now been fixed. - [SPEED] improved performance of some old FFT related code. MiniCRUSH benefits from these changes only when the 'spectrum' option is used. 1.63-8 [NEW] Made crush image utilities compatible with the next generation image files. Starting from CRUSH version 2.0, the RMS plane will no longer be the raw observing rms, but will correspond directly to the noise in the signal plane, whether it is smoothed or not. This will make it much more convenient for users, who currently have to smooth the noise according to a recipe provided in the CRUSH document 'CRUSH-FITS-structure.pdf'. [FIX] Made conversion from pixelization to equivalent Gaussian smoothing consistent throughout. [COSMETIC] Minor changes to image information summary. [COSMETIC] 'show' tool has clickable components. You can change the colorscale, toggle between linear/log/sqrt scaling, or change units with just one click. You can drag the scale to adjust its range etc. The information panel provides you with hints. [COSMETIC] Added 'mJy/beam' unit for 'show' tool. Units can be changed by pressing the 'u' key, in the program window (or by clicking the the unit label on the right). [COSMETIC] Updated 'show' utility to include coordinates, and used more human readable formatting. Also, it now has a cleaner look. 1.63-7 - [BUG] imagetool's '-origin' option resulted in botched y-coordinates. - [FIX] In locales where a decimal comma is used instead of a decimal point, the FITS headers became unreadable by other FITS tools. While this is strictly speaking a feature of the nom.tam.fits libraries used by crush, a workaround has been implemented. - [UPDATE] FITS libraries are recompiled. 1.63-6 - [BUG] 'coadd' behaved as if all maps were centered on the same coordinate. This was due to an instantiation bug, by which all map projections ended up being the same object, rather than separate for each map. As a resust each map overwrite the projection information of the prior maps upon read. - [FIX] Removed unnecessary duplicate regridding step in 'coadd'. - [FIX] 'JACKKNFE' keyword in coadded was always set to true. Now, it is fixed to correctly indicate if the map was coadded via random jackkniving... - [FIX] FITS libraries (nom.tam.fits) were not 1.5.0 compatible. They have been recompiled to restore 1.5 compatibility. - [CHANGE] Improved source extraction algorithm, in the way false detection probabilities are calculated. Also, the output now indicates the effective false detection rate instead of the Gaussian false detection rate before. - [UPDATE] Updated nom.tam.fits library to 1.00. This also fixes the FITS reading errors that resulted in some disabled functionality in version 1.63-3. Now all functionality should be restored... 1.63-5 - [FIX] Restored lost backward compatibility to Java 1.5... 1.63-4 - [FIX] Fixed error when trying to manipulate images w/o scan information. - [FIX] fixed S/N image sign when negative scaling factor is applied. 1.63-3 - [UPDATE] Simplified FITS reading by making more use of common codelets... - [CHANGE] Reverted back to 2nd order polynomials for faint reductions - [CHANGE] Points source corrections are now applied to all crush maps by default. - [CHANGE] Disabled acceleration response modeling (was used in faint and deep reduction modes). There is no evidence that SHARC-2 acts as an accelerometer :-)... - [CHANGE] Simplified SHARC-2 flux correction scheme by converting filter correction of models to equivalent Gaussian FWHM. - [FIX] Adjusted the way the effective filter widths are calculated when map is filtered multiple times. Unfortunately, in such cases the filter profile is no longer Gaussian. To first order, the width of the composite filter is mainly detemined by that of the narrowest filter applied... - [FIX] re-writing of crush (SHARC2) images with imagetool was broken. It seems like the problem is with the FITS library itself... A workaround for now is that the scan info is discarded when the images are manipulated... - [FIX] Blanking seemed to have behavioural problems, causing funny flags appearing in the data. Besides, it did not even seem to do the job it was meant to. Thus, blanking is disabled for now by default. You can reenable it at your own risk... - [FIX] Pointing fits were broken. - [FIX] Several fixes to 'sourcecounts' functionality. - [FIX] Filter corrections weren't always correctly calculated... - [FIX] imagetool's '-masks=match' could result in an error of the specified region fell outside of the map. Now the routine will return a 'NaN' flux for such cases. 1.63-2 - [NEW] Added '-jackknife' option to 'coadd' allowing to generate randomly jackniffed maps. The FITS keyword 'JACKKNFE=T' is added to the image header to avoid confusion with properly coadded images. - [NEW] Imagetool has new feature '-growFlags' allowing to increase the flagged areas by a specified radius (or beam FWHM if no argument) is specified. This is useful to make sure that the areas surrounding a flagged pixels are also flagged. - [NEW] Added extra column into source extraction output, indicating the expected number of false detections above and including each source. - [NEW] Updated 'histogram' tool to allow it to generate other than S/N histograms. The option '-image' can be used to specify which image plane to use for histogram generation (I.e. 'flux', 's2n', 'rms' 'weight' or 'time'). Accordingly the value of the '-bin' option is interpreted to be in the default units for each image (I.e. the mapping unit for 'flux' and 'rms', its inverse square for 'weight', seconds for 'time' and unitless for 's2n'). - [BUG] Filter corrections were inconsistently applied when using extended structure filters (via FFT or convolution). Further inconsistencies between applying and undoing flux corrections. Both these problems are now fixed. - [CHANGE] 'K' in 'show' now always convolved to beam, whatever that may be... - [FIX] An old workaround to make FITS double header values be readable by GILDAS software has been causing problems when decimals are marked with comma, rather than point in the locale. Thanks Ricky Nilsson for finding this one :-). - [FIX] Restored compatibility to Java 1.5. In a few instances Arrays.copyOf function was used, which requires Java 1.6. 1.63-1 - [BUG] extended filter resulted in unwanted unit change... - [BUG] The regridding in the prior beta release (1.63-b1) was buggy. This could produce faulty 'coadd' behaviour also. Now fixed. - [CHANGE] The '-crop' option in imagetool will now crop images automatically at the edges if no argument is specified. - [CHANGE] Default FITS naming convention changed: instead of listing every individual scan that contributes to the map, only the first and last scan is listed when multiple scans are present. This results in much shorter, and human readable, default naming... - [CHANGE] Various improvements to coadd. When coadding directly (default) the maps are loaded one-by-one, decreasing the memory requirement. Regridding to common grid now happens only if the component maps do not already conform to one-another, and the now happens in a single go, rather than the two-stage solution used before (i.e. first regridding, then shifting). - [CHANGE] Gaussian source fitting (e.g. for pointing reductions) has been overhauled to make it faster and more reliable. - [FIX] Region.add(FWHM) behaved like Region.addPoint()... This was no big issue since the function was not used thus far... - [FIX] 'coadd' now uses noise-weights measured on individual maps unless '-noise=data' option is specified, in which case the time-stream derived weights are used... - [FIX] Smoothing by a beam smaller than a pixel caused NaN errors. This is now fixed. - [FIX] Fixed glitch with 'imagetool -masks=match' overwriting the image data. In fact, the catalog matching should not output any image at all, rather it should exit when the catalog match is completed. - [FIX] Fixed FFT display issues... - [TUNE] Cleaned-up some of the SkyMap code... 1.63-b1 - [NEW] imagetools's repertoir of operation extended to match sources to catalogs. The catalog can be loaded via the '-mask=' option, and the matching is performed via the '-masks=match' option. The output is the usual catalog/mask format, with 3 columns written into the comment field, in order: catalog S/N peak pixel S/N peak S/N The last two columns are the S/N levels that are found on the image for the given location inside a search area (which is calculated according to Kovacs 2006, assuming that the pointing rms <= the map pixelization.). - [NEW] Introduced the ability to blank out bright source signals during reduction, s.t. their interference with the other models is minimized. This should help get rid of negative bowls around bright sources. The corresponding configuration key is MAP_BLANKING_LEVEL, and command like option '-blank=', and it's value is a signal-to-noise level above which the blanking shoul be performed. Use together with -clip (MAP_CLIPPING_LEVEL), in similar way. Blanking values >4 are recommended, to minimize its activity on noise. - [NEW] Introduced DEFAULT_ESTIMATOR key and '-estimator=" option to specify what type estimator is default. The accepeted values are 'robust' and 'maximum-likelihood'. All else reverts to maximum-likelihood. - [BUG] Weighted median was (still) wrongly calculated. This is hopefully fixed now. Problems could have resulted when median estimators were used instead of the default maximum-likelihood ones. As a result of the fix, median estimators are now default for signal estimation (but not for gains and weighting). - [BUG] Multiplexer modeling median calculation was buggy. This was not normally used however, so should not impact the average user... - [BUG] The source extraction tool 'detect' has applied filter corrections twice when large-scale structure filtering was used (once at the time of filtering, and again at the extraction). Now any prior filtering flux corrections are undone first, to allow the extraction to operate on raw maps. The corrections are then reapplied to the residuals at the end of the extraction routine. - [BUG] Pixel sizes were not individual to scans, rather they were set globally by the last scan read. This could have caused a calibration problem of <10% when old Cassegrain mount scans were reduced together with Nasmyth mount scans. However, this caused no problem whatsoever with the reduction of like scans. - [CHANGE] Source extraction tool 'detect' now estimates zero levels based on medians only in the first rounds. Afterwards it reverts to finding the mode (i.e. peak) of the distribution, which has almost no dependence on the shape of the noise distribution at large deviations (thus it is not biased by the presence of sources or by the cleaning of noise). - [CHANGE] Source extraction tool 'detect' now estimates noise based on the negative side distribution alone. This should be more robust when a large number of sources are present in the map. Also more than one levelling/noise-restimation rounds are performed if necessary to uncover all sources. Improvements to internal flagging also. - [CHANGE] Reorganized source extraction algorithm, to select peaks in just a few rounds. This makes the routine perform at more or less constant speed irrespective of the number of sources (more likely it may be log(N)-type scaling...), rather than the old method, which was roughly linear with source count N. This is a huge advantage when using the method on a field with many sources. - [CHANGE] Added filtering information to displayed image info. - [CHANGE] Raw source maps produced with '-rounds=0' option will now at least zero-level timestreams. - [CHANGE] 'detect' no longer regrids input maps, since it has all the capabilities to work on coarse grids, which are also faster... - [CHANGE] Now supporting reading double-prescision BoA images. - [CHANGE] Slightly changed the quadratic interpolation method used for fine-tuning detection peaks in the source extraction. - [TUNE] Fixed gain iterations to 1 (instead of 'auto' convergent). - [TUNE] Reduced the number of iterations, as convergence is achieved faster typically. - [TUNE] '-faint' and '-deep' reductions now fit only sky gradients not 2nd order polynomials - [TUNE] Removed some unnecessary costly re-calculations of SkyMap images... - [TUNE] Lots of stream-lining - [FIX] In one place a Java feature sneaked in that is not avaiable before version 1.6.0, ruining backwards compatibility to 1.5.0 as advertized. The feature has now been rewritten to comply with the earlier API spec... - [FIX] Gain normalization is now always followed by an appropriate renormalization of the correlated signals. In the past this was effective only if direct gains were used, which was the default. - [FIX] The way the appropriate detection level was calculated for yielding the desired number of false detections (or confidence levels) had two issues: (1) it was an empirical formula based on random maps and (2) it was meant to be used with the old detection routine, which was pixelization dependent. The new routine has no pixelization dependence, and the number of independent points in a map can be determined (estimated) using Nyquist sampling theory. So this is the more correct approach for the current source-extraction method. - [FIX] Region flagging/unflagging was unnecessarily slow... - [FIX] When parsing masks files, lines that do not make valid catalog entries are simply ignored. This makes it easier that plug catalog outputs (e.g. from 'detect') straight back into other CRUSH utilities (e.g. 'show'). - [FIX] Improved source extraction after discriminative filtering: (a) by compensating for the effective blanking S/N change as a result of smoothing, and (b) by accounting for the filtering on the wings of the source, which fall below the filter blanking level. - [FIX] Speeded up source extraction by stream-lining region initialization. - [FIX] 'imagetool' and 'show' now both display image information after the manupulating steps. - [FIX] Various flagging fixes in source extraction. - [FIX] Regions cleared after stacking. Without this, 'show' would display the circular regions (and/or polygons) inappropriately for the stacked image, when stacking is performed under 'show'. - [FIX] BoA map peak fluxes were slightly misinterpreted. They were thought to have been normalized to instrument beams, instead they were correctly expressed to image beams. - [FIX] NaN problem fixed when when trying to smooth map to the FWHM it already has... - [CLEANUP] Cleaned up some old code.