A Guide to using MiniCRUSH. (A minimalistic version of CRUSH for the APEX bolometers) Last updated 30 October 2009 -- Attila Kovacs Table of Contents ================= 1. Getting Started 1.1 Installation 1.2 Quick start 1.3 The Tools of the CRUSH Suite 1.4 A Brief Description of What CRUSH Does and How 1.5 Command-Line Options and Scripting Support 2. Basic Configuration 2.1 Global configuration options 2.2 Pipeline configuration 2.3 Recovery of Extended Structures 2.4 Configuring the Source Map 2.5 Scan-specific configuration 3. Instrument Specific Configuration 3.1 Instrument Definition Files 3.2 Laboca specific options 3.3 Saboca specific options 3.4 ASZCA Specific options 3.5 P-Artemis specific options 4. Advanced Configuration 4.1 Advanced Pipeline Options 4.2 Advanced Source Map Configuration 4.3 Advanced Scan-Specific Options 4.4 Pipeline elements (Models) and their options 4.5 Further output options 5. Understanding the Console Output 6. Pointing Corrections 7. Examples 8. Further information ##################################################################### 1. Getting Started ##################################################################### 1.1. Installation ================= After unpacking the gzipped tarball, enter the 'minicrush' directory. If setting it up on a project account, edit 'minicrush.cfg' to specify the location of your data files, and the project id. You can also set the output directory. Example contents of minicrush.cfg: datapath ~/data outpath ~/maps project T-79.F-0002-2006 Check to see if your files are compressed. If your data files end with '.fits.gz' extension then you will want to also include a line: compressed in the configuration file. (Or you can specify this individually for scans on the command-line, if you do not want a global setting). Alternatively, you can create your user configuration outside of the minicrush/ directory, s.t. future updates to minicrush will keep your setting intact. To do this simply copy 'minicrush.cfg' into '.minicrush/' in your home: > mkdir ~/.minicrush > cp minicrush.cfg ~/.minicrush This is also a convenient way for users sharing a single minicrush installation to create personalized default configurations. You can create any number of user specific configuration files in ~/.minicrush. You can invoke these via either the 'config ' configuration entry or via the equivalent '-config=' command-line option at runtime. Java Configuration ------------------ Edit the 'setup.sh' script, and adjust the '-Xmx' option of JAVAOPTS to the maximum amount of memory you will allow the reduction to use. E.g. for 1GB, use: -Xmx1000M Note, that for 32-bit machines, the value of Xmx has to be below 2000. On 64-bit machines (with the appropriate 64-bit java VM installed) up to 4G or more (depending on the OS configuration) can be specified (the latter may require the use of the '-d64' option as well). You may also want to change the JAVA variable in 'setup.sh' to point to the version of java that you want to use, if this is not the default one. (The default is to use the latest version fron SUN in /usr/java/latest/bin) The minicrush software needs at least version 1.5.0, but 1.6 is recommended. The GNU java (default on RedHat and Fedora systems) is rather buggy and tends to cause trouble. Avoid using it if you can. To see what is your default java version, type: > java -version 1.2. Quick start ================ To reduce data, simply specify the instrument ('laboca', 'saboca', 'aszca' or 'p-artemis') and the scan numbers to minicrush. (You may have to add './' before minicrush if the current directory '.' is not in your path.) E.g., > ./minicrush laboca 10059 will reduce LABOCA scan 10059 with the default reduction parameters. You can specify additional options. Some of these apply to the reduction as a whole while others will affect the scan processing for those scan numbers that are listed *after* the option flag. If you are new to MiniCRUSH it is recommended that you read this document through Sections 1--2 (Getting Started & Basic Configuration). This should give you enough background to become a well-versed user already. :-) Once you get the hang of it, and feel like you need more tweaking ability, feel free to read on further to see what other fine tuning capabilities exist... Here're some quick tips: * Reduce as many scans together as you can. E.g. > ./minicrush laboca 10657-10663 10781 11233-11235 [...] * You can specify opacities, pointings, scaling etc, for each of the set of scans listed (See section for Scan-specific options). E.g., > ./minicrush laboca -tau=0.32 10657-10663 10781 \ -tau=0.18 11233-11235 [...] will use a zenith tau value of 0.32 for 10657-10663 and 10781, and 0.18 for the last set of scans. * If you suspect that you are missing extended structure (see section on the Recovery of Extended Structures), then you can specify the 'extended' option, which will better preserve large scale structures albeit at the price of more noise. E.g. > ./minicrush laboca -extended 10657-10663 * If your source is faint (meaning S/N < 10 in a single scan), then you may try using the 'faint' option. E.g. > ./minicrush laboca -faint 10657-10663 or, > ./minicrush laboca -faint -extended 10657-10663 to try preserve extended structures (see above). * For finer control of how much large scale structures are retained you can use the 'sourcesize' option (instead of 'extended'), giving a typical source scale (in arc-seconds) that you'd like to see preserved. Then CRUSH will optimize its settings to do the best it can to get clean maps while keeping structures up to the specified scales more or less intact. E.g. > ./minicrush [...] -sourcesize=45.0 10657-10663 Will optimize reduction for <~ 45 arcsec sources. * Finally, if your sources are not only faint, but also point-like, you should try the 'deep' option. This will filter large scale structures heavily to yield the cleanest looking maps possible. E.g., > ./minicrush laboca -deep 10657-10663 With just these few tips you should be able to achieve a decent job in getting the results you seek. There are a lot more fine-tuning possibilities for the curious mind. If interested, you can find all the documentation further below... 1.3. The Tools of the CRUSH Suite ================================= CRUSH provides a collection of useful tools. Here's a short description of what is there and what they are used for. Each tool, when run without options (or with the -help option) will provide you a list of available options on the console. coadd Add FITS images together. Use as a last resort tool as it is always better to reduced scans together. show A simple display program for the FITS files, with various useful functions for simple analysis and image manipulation After starting press 'h' for help. imagetool A tool for manipulating images. Can also deal with images produced by BoA (and to some degree other images also). calibrate A tool for deriving calibration information into an ascii file. histogram Provide a histogram of signal-to-noise distribution of an image. jiggle Align multiple images w.r.t. one-another. detect A source extraction tool for maps. You should make sure that the noise is close enuogh to Gaussian (e.g. with 'histogram') before relying on this. difference Allows to look at the difference between two supplied images. esorename Rename ESO archive files to their original names. 1.4. A Brief Description of What CRUSH Does and How... ====================================================== CRUSH is a pipeline reduction that is principally meant to remove correlated signals (correlated noise) in the time-streams to arrive at clean & independent bolometer signals, which are used to make a source map. As such it is not an interactive reduction software (e.g. as opposed to BoA). The term 'scripting' in CRUSH mainly means defining configuration options (in the command line or through configuration files) which are parsed in the order they are read. During the reduction CRUSH aims to arrive at a consistent set of solutions for various correlated signals, the corresponding gains and the pixel weights, as well as tries to identify and flag problematic data. This means a series of reduction steps, which are iterated a few times until the required self-consistent solutions are arrived at. To learn more about the details please refer to Kovacs, A., "CRUSH: fast and scalable data reduction for imaging arrays," Proc. SPIE 7020, 45, (2008). If that does not satisfy your curiousity, then you can find yet more explanation in Kovacs, A., PhD thesis, Caltech (2006). 1.5. Command-Line Options and Scripting Support =============================================== Configuration of minicrush is available either through command line options or via scripting. You have seen scripting already in the form of 'minicrush.cfg', which stores some user-defined values. But there is more to it. Both command-line options and scripting is organized in key/value pairs. The main difference is that on the command-line there are restrictions on syntax imposed by the shell and the command-line parser. E.g., no white spaces. Therefore key/value pairs may be separated either by '=', ':' or empty spaces, or any combination of these. Commant line options start with a dash '-' in front. Thus is, what may look like: key option1 option2=value in the configuration script, will end up as > ./minicrush [...] -key=option1|option2:value [...] in the command line. Otherwise, they two ways of configuring are generally identical to one-another. One exception is reading scans, which is done via the 'read' key in a script, whereas in command line, you simply list the scan number (or ranges or lists). I.e., read 10056 # in script > ./minicrush [...] 10056 # on the command line. In the next section you'll find a description of the scripting keys. Now that you know how to use them also as command line options, you can choose scripting or command-line, or mix-and-match them to your liking and convenience... Startup Configuration --------------------- At launching, minicrush will parse 'minicrush.cfg' and 'default.cfg' to establish a default configuration set. Both these configurations are first parsed from the files in the minicrush directory, then additional options or overriders are read also from the appropriate instrument subdirectories, if exist. Then, to allow for local (machine or user-dependent) configurations, minicrush will check, if the configuration files exists under the same name in ~/.minicrush directory of the user. If so, they will be parsed at this point. See more on this in the next section under the 'config' option. ##################################################################### 2. Basic Configuration ##################################################################### 2.1. Global configuration options ================================= Configuration options here are listed as scripting keys i.e., without a preceding dash. However, you can use the same options in the command line by adding the dash. (BTW, '=' can be replaced by a space in scripting...) Upon starting minicrush, first 'minicrush.cfg', then 'default.cfg' are parsed, establishing initial settings. (Remember, if these, or any of the invoked configuration files are found in '~/.minicrush/' then they will be read from there also after parsing these from the default location in the program execution directory. config= Load the configuration file filename. The file is looked for in the locations in the following order: 1. ./ 2. .// 3. ~/.minicrush/ 4. ~/.minicrush// Whenever a matching file is found its contents are parsed. Because of the ordering, it is convenient to create overriding configurations. Thus instrument specific settings can be used to override default settings, and user specific settings placed in ~/.minicrush/ can override shipping defaults. Whenever a configuration is parsed, there is a note of it on the console output so that one always knows which files were read and in what order. E.g. when using, > ./minicrush laboca -faint 12066 the following configuration files will be loaded in order (provided they exist): ./default.cfg ./laboca/default.cfg ~/.minicrush/default.cfg ~/.minicrush/laboca/default.cfg ./faint.cfg ./laboca/faint.cfg ~/.minicrush/faint.cfg ~/.minicrush/laboca/faint.cfg Each successively loaded file may override setting loaded before it. poll Whenever unsure what options are set at any poll=