News
Download
Documentation
Instruments
About
Author: Attila Kovacs
Last updated: 11 December 2018
Prerequisites
Create a new CRUSH instance
Options and input files
Reduction
CRUSH messages
Appendix A. Reporter API
This document describes how to call CRUSH from another Java program.
The CRUSH library mainly consists of two JAR archives:
crush.jar
jnum.jarYou will find both inside the CRUSH distribution directory under lib. As such you can simply add crush/lib/* to your classpath.
(jnum.jar contains generic Java numerical classes and utilities, which CRUSH heavily relies on, while the reduction pipeline is in crush.jar). Additionally, you will be needing the nom.tam.fits libraries (fits.jar) with version 1.15.2 or later (if and when available).
Beyond the Java code, CRUSH relies heavily on various configuration files and instrument meta-data. These are part of the CRUSH distribution. For CRUSH to work properly, it will need to know where to locate these configuration files. You have either specify the location of the CRUSH distribution directory via a environment variable CRUSH before you launch your program, e.g. in bash:
or, you can do it in Java after launch, using the static field of CRUSH:
before instantiating a new CRUSH object, as:
import crush.CRUSH;
// ...
// Set the location of the CRUSH distribution first...
CRUSH.home = "/home/johndoe/crush";
// Now, we can create a CRUSH instance...
CRUSH crush = new CRUSH("hirmes");You may want to tweak some of the Java VM options for startup, especially the -Xmx option that sets how much RAM your Java program may access. As a rule of thumb you’ll want Java/CRUSH have access to RAM about twice the combined size of all input scans you want to process, or more.
You will want to create a fresh CRUSH instance for each reduction process you want to launch. You can do that with the constructor:
The constructor will throw an exception if the instrument name is not recognised, or if the relevant class files for that instrument cannot be located, or if some other issue was encountered during initialization. (The exact type of the Exception may be dependent on the specific instrument class, which is why is in the most generic Exception form). The instrumentName argument is case-insensitive, but for consistency I recommend using lower case names, e.g. hirmes.
import crush.CRUSH;
// ...
// Create a new CRUSH instance for HIRMES...
// Note: an exception may be thrown...
CRUSH crush = new CRUSH("hirmes");Once you successfully created a CRUSH instance (without an exception), you can proceed by specifying any non-default reduction options, and list the scans to be reduced.
Add the extra reduction options via calls to:
where the option argument is a <key>[=][<value>] pair. They are just like the command-line arguments to CRUSH (but without the leading dash), or lines in the configuration files. E.g.:
crush.setOption("faint"); // Same as '-faint' on the command line
crush.setOption("datapath=/data/hirmes"); // raw data path
crush.setOption("outpath /reduced"); // output path
crush.setOption("name = Arp220.fits"); // output image file nameAs you can see above, key/value pairs are separated by white-space and/or = characters (any number and any combination of those).
The setOption() call will return true if the option was successfully set, or false if an existing lock on the option prevented changes. (The latter is normal behavior, it’s simply informing you that the option you requested was blocked).
CRUSH is optimized to take maximum advantage of the CPU resources on your system. This may be not so great when you are running CRUSH from another program, which may end up being CPU-starved as a result. Luckily, CRUSH has the options threads or idle. The former threads can be used to explicitly specify how many threads CRUSH may use. You can throttle CRUSH by setting threads to a value smaller than Runtime.getRuntime().availableProcessors(), e.g.:
Alternatively, you can specify the number, or percentage of processors to leave idle by CRUSH. For example, the following is equivalent to the above:
You can also specify a percentage of the CPU resources to leave unused, e.g.:
which will idle 1 CPU if you have 4, or 2 CPUs if you have 8 etc.
Tip: You can put all options into an appropriate configuration file for your program, and load them all with the single call:
This way you can avoid hard-coding options, and have for free a way to edit your configuration options on-the-fly without needing to restart your program even.
Once you have configured the options you want to use, you can load the scans to which tyou want the options applied, using:
public void read(String identifier) throws OutOfMemoryError,
FileNotFoundException, UnsupportedScanException, ExceptionHere identifier can be a file path/name, or a scan ID (e.g. 451.33 for flight 451, scan 33) or a scan number (e.g. 33), or a range (e.g. 33-36). Using file path/name is probably the most straightforward. It can be either a file path/name relative to the datapath option (if defined), or an absolute path. E.g.
The call will throw an OutOfMemoryError if the scan could not be loaded due to insufficient VM memory (If so, you might want to tweak the -Xmx option of the VM to allow Java to access more RAM). Or, it may throw an exception such as:
FileNotFoundException If no matching scan data file was found.
UnsupportedScanException If the scan does not match the instrument type.
Exception Any other exception/error...You can proceed setting more options and read more scans if desired, by successive calls to setOption() and read()…
Once you loaded all the desired input scans with the desired options, you can proceed to reducing the data with:
This will run the reduction pipeline, and write the requested data products before returning. It may take a whilem but if all goes well it returns normally without throwing an exception.
After the reduction is complete (or if it failed) you should finish by calling
to kill any background processes that may have been spawned by the the reduction process.
CRUSH comes with its own messaging architecture, defined in the jnum.reporting package. It is similar to Java’s built-in Logger class in functionality but with some different (better) features.
The basic functionality is defined by the abstract Reporter class.
To capture messages from CRUSH, you can create your own Reporter implementation that will deal with the different types of messages produced by CRUSH. Each message type is handled by a corresponding method in the Reporter class: info(), notify(), debug(), warning(), error(), trace(), status(), result(), detail(), values(), and suggest().
import jnum.reporter.*;
class MyReporter extends Reporter {
public MyReporter {
// Call super with a String ID
super("My-Reporter-ID");
}
public void info(Object owner, String message) {
// Deal with however you want with the message coming
// from owner. owner is either the specific object
// that sent the message, or else an enclosing class
// if the message was sent e.g. by a static call.
// ...
}
// ...
}(If you are a fan of the built-in Logger class, you might want to use the existing LogReporter class, which will conveniently forward CRUSH messages to the Logger object of your choice.)
CRUSH’s handles its messages through a Broadcaster, which is a subclass of Reporter that simply distributes incoming messages to the connected list of Reporter children.
By default, CRUSH.broadcaster has only a CRUSHConsoleReporter connected to it, which formats and prints messages to the console. If you rather not spam your console with CRUSH message, the you can call:
The above removes all existing consumers of messages, including the default CRUSHConsoleReporter instance. You can also do it more surgically, if you will, by:
(Where "crush-console" is the ID of CRUSHConsoleReporter).
Whether or not you want to keep the default console messaging, you can add any number of your own message consumers as:
You probably want to set up your Reporter before you instantiate CRUSH, which is why this is done via a static call. This way, any messages produced during instantiation will be captured by your class as well.
Easy-peasy.
To summarize the above, here is a sample code for integrating CRUSH into your Java program. First, a static initialization of CRUSH, once, and before instnatiation.
import crush.CRUSH;
// ...
// Specify where the CRUSH distribution resides...
CRUSH.home = "/home/johndoe/crush";
// Disable console messaging...
CRUSH.getBroascaster().clear();
// Add your own Reporter implementation
Reporter r = new MyReporter();
CRUSH.addReporter(r);After the global initialization above, for each reduction run you will need something like:
CRUSH crush = null;
try {
// Create a CRUSH instance for reducing HIRMES data
crush = new CRUSH("hirmes");
// Load options from your specific CRUSH config file
crush.readConfigFile("/home/myprogram/mycrushoptions.cfg");
// Set some more options, e.g. as input by the user, e.g. from a GUI
if(userSelectFaint) crush.setOption("faint");
// Read in the scan (or scans) with the options defined so far...
crush.read("/data/2016-12-03_HA_F354_009_CAL_unk_HAWD_HWPOpen_RAW.fits");
// ... more options, and more scans as you need...
// Now reduce...
// This will take a while, and will write data products as configured...
// You Reporter implementation (if configured) will be receiving lots of
// messages about what's going on, so that you can keep an eye on it...
crush.reduce();
}
catch(Exception e) {
// Deal with any exception that we may have encountered in the process...
}
if(crush != null) {
// Shut down any background processes that may have been spawned...
crush.shutdown():
}Here is some more information to help build your own Reporter implementation for capturing CRUSH messages.
First, your Reporter subclass will have to call Reporter’s only public constructor:
with a String ID of your choice. CRUSH’s broadcaster maintains a Hashtable of Reporters mapped to String IDs to which it sends messages. As such, each active Reporter must have a unique ID. Otherwise, adding aReporterwill replace a previously presentReporter` with the same ID.
Next, your Reporter class needs to implement the following methods (which are all declared abstract in Reporter itself):
public void info(Object owner, String message);
public void notify(Object owner, String message);
public void debug(Object owner, String message);
public void warning(Object owner, String message);
public void error(Object owner, String message);
public void status(Object owner, String message);
public void result(Object owner, String message);
public void detail(Object owner, String message);
public void values(Object owner, String message);
public void suggest(Object owner, String message);
public void trace(Throwable e);Most of the above takes an owner argument, which is either a specific object, or else a class, to which the message is associated with. I.e. owner may be a crush.Instrument instance or crush.Instrument.class. The owner is often the object or class that produced the message, but it does not have to be.
Finally, the last method trace() is for reporting stack traces, normally generated by exceptions and errors.
Addditionally, you may also override some of the Reporter methods that have default implementations. These (including their default implementations) are:
public void error(Object owner, Throwable e, boolean debug) {
` error(owner, e.getMessage());
if(debug) trace(e);
}
public void error(Object owner, Throwable e) {
error(owner, e, true);
}
public void warning(Object owner, Exception e, boolean debug) {
warning(owner, e.getMessage());
if(debug) trace(e);
}
public void warning(Object owner, Exception e) {
warning(owner, e, false);
}