Tuesday, February 17, 2009

Examples of Bag and MultiKeys

In this installment, I’m going to discuss Java usages tips and tricks for the Apache Commons Collection Bag interface. Honestly, the first time I saw the Apache Commons Collection Bags API, I thought "Well, that isn't very useful." The Bag interface states the intention of the class; it is to count occurrences of objects. Like all the products in the Apache Commons Collections, Predicates, Closures, and Transformers can be plugged into the bag at run time to determine the behavior. I ran some tests, which are at the end of the article. The cost of using the framework is negligible, at about 10 to 20 milliseconds. This of course is compared to a traditional method of sampling counting occurrences and stuffing into a map. You can be the judge of the test, let me know if you think the test is accurate.

So, what kind of problem justifies adding the complexity of Bags to a project? Joining the magic of a Transformer, MultiKey, and a decorator you can produce a dynamic solution for counting groups of occurrences of properties of beans in a collection. This would be suitable for a generic controller responsible for counting groups of properties. It would be very helpful in a reporting system. Let’s take a look at the sample and walk through the code.

The Code:

package com.blogspot.apachecommonstipsandtricks.bags;
import java.util.*;
import java.net.*;
import java.io.*;
import java.lang.reflect.*;
import org.apache.commons.collections.*;
import org.apache.commons.collections.comparators.*;
import org.apache.commons.collections.keyvalue.*;
import org.apache.commons.collections.bag.*;
import org.apache.commons.collections.bag.HashBag;
import org.apache.commons.lang.*;
import org.apache.commons.beanutils.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class MagicBagOfTricks
{
public static void main(String[] args) throws IOException, InvocationTargetException, NoSuchMethodException, IllegalAccessException
{
List<LaborForce> list = new ArrayList<LaborForce>();
URL url = new URL("http", "1796193846474123283-a-1802744773732722657-s-sites.googlegroups.com", 80, "/site/psenger/Home/data.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
if (((str = in.readLine()) != null))
{
do
{
String[] strings = str.split("\t");
strings = StringUtils.stripAll(strings);

State state = State.valueOf(StringUtils.trim(strings[0]));
Gender gender = "Male".equals(StringUtils.trim(strings[1])) ? Gender.Male : Gender.Female;
Integer year = Integer.valueOf(StringUtils.trim(strings[2]));

list.add(new LaborForce(state, gender, year));
} while ((str = in.readLine()) != null);
}
in.close();

Map map = PropertyUtils.describe(new LaborForce());
Set s = map.keySet();
s.remove("class");
System.out.println("All the possible properties on the LaborForce bean are [" + StringUtils.join(s.toArray(),",") + "]");

Bag masterBag = new HashBag();
Bag genderBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender" } ) );
Bag genderYearBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" } ) );
Bag genderYearStateBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" , "state" } ) );

genderBag.addAll( list );
genderYearBag.addAll( list );
genderYearStateBag.addAll( list );

Comparator comparator = ComparatorUtils.chainedComparator(new Comparator[]{new MultiKeyCompartor(0),new MultiKeyCompartor(1),new MultiKeyCompartor(2)});
Set<MultiKey> set = new TreeSet<MultiKey>(comparator);
set.addAll(masterBag.uniqueSet());
for (MultiKey multiKey : set)
{
System.out.println( "[" +StringUtils.join(multiKey.getKeys(),',') + "] = " + masterBag.getCount(multiKey));
}
}
private static class PropertiesMultiKeyTransformer implements Transformer
{
String[] methodNames;
private PropertiesMultiKeyTransformer(String[] methodNames)
{
this.methodNames = methodNames;
}
public Object transform(Object o)
{
List<Object> ooos = new ArrayList<Object>();
for (String methodName : methodNames)
{
try
{
ooos.add(PropertyUtils.getProperty(o, methodName));
}
catch (Exception e)
{
throw new FunctorException(e);
}
}
return new MultiKey(ooos.toArray(new Object[ooos.size()]));
}
}
private static class MultiKeyCompartor implements Comparator<MultiKey>
{
private int i;
private MultiKeyCompartor(int i)
{
this.i = i;
}
public int compare(MultiKey o1, MultiKey o2)
{
Object[] keys1 = o1.getKeys();
Object[] keys2 = o2.getKeys();
Object oo1 = null;
try
{
oo1 = keys1[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
Object oo2 = null;
try
{
oo2 = keys2[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
NullComparator nullComparator = new NullComparator(false);
return nullComparator.compare(oo1, oo2);
}
}
}


Between lines 18 and 36 of the MagicBagOfTricks class, we fetch the sample data. The data is in the format of StatetabGendertabyear. If you want to download the data, you can get it here.


List<LaborForce> list = new ArrayList<LaborForce>();
URL url = new URL("http", "1796193846474123283-a-1802744773732722657-s-sites.googlegroups.com", 80, "/site/psenger/Home/data.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
if (((str = in.readLine()) != null))
{
do
{
String[] strings = str.split("\t");
strings = StringUtils.stripAll(strings);

State state = State.valueOf(StringUtils.trim(strings[0]));
Gender gender = "Male".equals(StringUtils.trim(strings[1])) ? Gender.Male : Gender.Female;
Integer year = Integer.valueOf(StringUtils.trim(strings[2]));

list.add(new LaborForce(state, gender, year));
} while ((str = in.readLine()) != null);
}
in.close();


The purpose of code on line 38 through 41 is to just print the properties of the bean, LaborForce, minus the class method. It is important to see how the PropertyUtils describes the bean, because we will be accessing it later via the same API.


Map map = PropertyUtils.describe(new LaborForce());
Set s = map.keySet();
s.remove("class");
System.out.println("All the possible properties on the LaborForce bean are [" + StringUtils.join(s.toArray(),",") + "]");


On line 43, we create a single HashBag object. I will refer to this HashBag object as the masterBag. A HashBag is an implementation of Bag with the characteristics of a HashSet. When objects are stored in the HashBag, the inserted object’s hash code is assessed, just like a HashSet, and ensures the uniqueness of the object in the Bag as it is stored. You might think the count would be 1 no mater how many times you insert the object, but in fact, the bag counts the inserted and removed occurrences. Even more confusing is the fact that there is an iterator method that returns all multiple occurrences and a uniqueSet method that pulls a set of objects. Line 44 through 46, is where the magic happens. We create three decorated Bags, with TransformedBag.decorate. All of them with variations of the PropertiesMultiKeyTransformer and backed by a single HashBag, the masterBag from line 43. When an object is inserted into one of the decorated HashBags, it is transformed by PropertiesMultiKeyTransformer and the results are physically placed in the masterBag. PropertiesMultiKeyTransformer uses the PropertyUtils to pull all the given properties off the bean to create a MultiKey object. It’s important that the we always put the same property in each object index in the array of objects of the MultiKey. If we didn’t, we would get an exception. Furthermore, we can’t transform the object into an array of objects because HashCode doesn’t exist on an Array of Objects. This becomes a problem when we insert Objects into the bag. The MultiKey is a perfect fit for this problem and that is the reason for its use. This class is effectively a wrapper class.


Bag masterBag = new HashBag();
Bag genderBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender" } ) );
Bag genderYearBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" } ) );
Bag genderYearStateBag = TransformedBag.decorate(masterBag, new PropertiesMultiKeyTransformer( new String[]{ "gender", "year" , "state" } ) );


On line 48 through 50 we add the object to the decorated transformed bags, which are all backed by the masterBag.


genderBag.addAll( list );
genderYearBag.addAll( list );
genderYearStateBag.addAll( list );


On line 52 through 58 I ask the masterBag for a set, load the results in a TreeSet that has a chained Transformer. The results are a sorted set of MultiKeys with counts of the occurrences of the vectors in the list.


Comparator comparator = ComparatorUtils.chainedComparator(new Comparator[]{new MultiKeyCompartor(0),new MultiKeyCompartor(1),new MultiKeyCompartor(2)});
Set<MultiKey> set = new TreeSet<MultiKey>(comparator);
set.addAll(masterBag.uniqueSet());
for (MultiKey multiKey : set)
{
System.out.println( "[" +StringUtils.join(multiKey.getKeys(),',') + "] = " + masterBag.getCount(multiKey));
}


The inner class PropertiesMultiKeyTransformer is a Transformer constructed with an array of Strings that represent the method names to use with PropertyUtils.getProperty on the beans in the list. It dynamically pulls the given properties from the bean and loads them into the new MultiKey. There is a limit to the number of Objects a MultiKey can accept, but for this demo, we are ok. In addition, the Objects in the MultiKey have to be consistently in the same order for each comparator. We are using a ComparatorUtils.chainedComparator of MultiKeyCompartor which will throw a ClassCastException if the objects are the wrong order.


private static class PropertiesMultiKeyTransformer implements Transformer
{
String[] methodNames;
private PropertiesMultiKeyTransformer(String[] methodNames)
{
this.methodNames = methodNames;
}
public Object transform(Object o)
{
List<Object> ooos = new ArrayList<Object>();
for (String methodName : methodNames)
{
try
{
ooos.add(PropertyUtils.getProperty(o, methodName));
}
catch (Exception e)
{
throw new FunctorException(e);
}
}
return new MultiKey(ooos.toArray(new Object[ooos.size()]));
}
}
private static class MultiKeyCompartor implements Comparator<MultiKey>
{
private int i;
private MultiKeyCompartor(int i)
{
this.i = i;
}
public int compare(MultiKey o1, MultiKey o2)
{
Object[] keys1 = o1.getKeys();
Object[] keys2 = o2.getKeys();
Object oo1 = null;
try
{
oo1 = keys1[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
Object oo2 = null;
try
{
oo2 = keys2[i];
}
catch (ArrayIndexOutOfBoundsException e)
{
}
NullComparator nullComparator = new NullComparator(false);
return nullComparator.compare(oo1, oo2);
}
}


The code for the data transfer object used within this project is called the LaborForce object. See the following sample for the object. You can find State and Gender in some of the other projects used in this blog.

The Code:

package com.blogspot.apachecommonstipsandtricks.bags;
import com.blogspot.apachecommonstipsandtricks.*;
public class LaborForce
{
private State state;
private Gender gender;
private int year;
public LaborForce()
{
}
public LaborForce(State state, Gender gender, int year)
{
this.state = state;
this.gender = gender;
this.year = year;
}
public State getState()
{
return state;
}
public void setState(State state)
{
this.state = state;
}
public Gender getGender()
{
return gender;
}
public void setGender(Gender gender)
{
this.gender = gender;
}
public int getYear()
{
return year;
}
public void setYear(int year)
{
this.year = year;
}
}


The results:

All the possible properties on the LaborForce bean are [state,gender,year]
[Male] = 7842
[Male,1950] = 134
[Male,1950,AL] = 1
[Male,1950,AK] = 6
[Male,1950,AZ] = 2
[Male,1950,CA] = 1
[Male,1950,CO] = 4
[Male,1950,CT] = 3
[Male,1950,DE] = 3
[Male,1950,DC] = 1
[Male,1950,FL] = 2
[Male,1950,GA] = 2
[Male,1950,HI] = 4
..
..
[Female,2008,SD] = 3
[Female,2008,TN] = 2
[Female,2008,TX] = 6
[Female,2008,UT] = 2
[Female,2008,VT] = 1
[Female,2008,VA] = 2
[Female,2008,WA] = 3
[Female,2008,WI] = 3


Performance Test


The test I ran was very simple. I wrapped a timer around parts of the code that were unique to the process. And the results of using the Bag are small, compared to a traditional method.

The code:

package com.blogspot.apachecommonstipsandtricks.bags;
import java.util.*;
import java.text.*;
import java.net.*;
import java.io.*;
import org.apache.commons.collections.*;
import org.apache.commons.collections.bag.*;
import org.apache.commons.collections.bag.HashBag;
import org.apache.commons.lang.time.*;
import org.apache.commons.lang.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class BagStressTest
{
private static final int NumberOfTimesToRun = 1000;
private static final int MasterListSize = 10;

public static void main(String[] args) throws IOException
{
int i = 1;
SimpleDateFormat sdf = new SimpleDateFormat("ss.SSS");
StopWatch sw;
Bag bag = null;
Map<State, Integer> map = null;
long sumOfTime = 0;

List<LaborForce> list = new ArrayList<LaborForce>();
URL url = new URL("http", "1796193846474123283-a-1802744773732722657-s-sites.googlegroups.com", 80, "/site/psenger/Home/data.txt");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String str;
if (((str = in.readLine()) != null))
{
do
{
String[] strings = str.split("\t");
strings = StringUtils.stripAll(strings);

State state = State.valueOf(StringUtils.trim(strings[0]));
Gender gender = "Male".equals(StringUtils.trim(strings[1])) ? Gender.Male : Gender.Female;
Integer year = Integer.valueOf(StringUtils.trim(strings[2]));

list.add(new LaborForce(state, gender, year));
} while ((str = in.readLine()) != null);
}
in.close();

List<LaborForce> listOfSampleData = new ArrayList<LaborForce>();
for (int j = 0; j < MasterListSize; j++)
{
listOfSampleData.addAll( list );
}
System.out.println("The sample listOfSampleData length is " + listOfSampleData.size());
System.out.println(" ---------- Start Test ---------- ");
sumOfTime = 0;
sw = new StopWatch();
for (int j = 0; j < NumberOfTimesToRun; j++)
{
sw.start();
bag = TransformedBag.decorate(new HashBag(), new Transformer()
{
public Object transform(Object o)
{
LaborForce dto = (LaborForce) o;
return dto.getState();
}
});
bag.addAll(listOfSampleData);
sw.stop();
sumOfTime += sw.getTime();
sw.reset();
}
System.out.println("With a Bag, Average Time in seconds and milliseconds is " + sdf.format( new Date( sumOfTime / NumberOfTimesToRun ) ) );
System.out.println(" ---------- End Test ---------- ");

// Alternative Way.
System.out.println(" ---------- Start Test ---------- ");
sumOfTime = 0;
sw.reset();
for (int j = 0; j < NumberOfTimesToRun; j++)
{
sw.start();
map = new HashMap<State, Integer>();
for (LaborForce lbf : listOfSampleData)
{
Integer count = map.get(lbf.getState());
if (null == count)
{
count = 0;
}
count++;
map.put(lbf.getState(), count);
}
sw.stop();
sumOfTime += sw.getTime();
sw.reset();
}
System.out.println("Traditional counting, Average Time in seconds and milliseconds is " + sdf.format( new Date( sumOfTime / NumberOfTimesToRun ) ) );
System.out.println(" ---------- End Test ---------- ");
}
}


The results:

The sample listOfSampleData length is 156000
---------- Start Test ----------
With a Bag, Average Time in seconds and milliseconds is 00.032
---------- End Test ----------
---------- Start Test ----------
Traditional counting, Average Time in seconds and milliseconds is 00.026
---------- End Test ----------


Thanks for reading this article, I look forward to any questions or feedback.

Author: Philip A Senger

Friday, February 13, 2009

Examples of Set Theory in Java with Apache Commons Collections

Before I get started on Apache Commons Collections Unions, intersections, and sub collections let me clarify something. You should always push the activity of manipulating data to the data layer. In an N-Tier system, the term Tier implies the system has concerns separated into layers. Cluttering up a system with concerns all over the place will make a nightmare Brownfield System. However if you are unable and are forced to compose the object at your layer (which is not unusual), these techniques tend to be extensible. This type of scenario occurs if you are working with disparate systems. For example, half of the data may come from a SQL like data source and the other from a SOAP Service.

Proper Sub collections, Unions, and Intersections are terms used in a branch of mathematics referred to as Set Theory. It describes membership status of items within sets. Sets are identified generically by a letter. The term union is identified by the symbol ∪, Intersection ∩, sub sets and super sets are defined by ⊂ and ⊃ respectfully.

To illustrate a union consider the set A and B. When the two are union-ed they make the following new set called C. This is akin to the mathematical function plus.



Conversely when A intersects with B, C is created. As you can see, C is the difference of the two sets. This is very similar to the minus function.



A Sub Collection and Proper Sub Collection are a little more difficult to understand. While in this example A contains B making it a Proper Sub Collection of A. if B contained everything in A it would not be proper sub collection anymore, It would only be a sub collection.


Example : Unions in Java with CollectionUtils.union


Lets look at Unions and how this can be done in Java with the Apache Commons CollectionUtils.union.

The Code:

package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import java.util.*;
import org.apache.commons.lang.*;
public abstract class AbstractCollectionExampleUtils
{
protected static List<Integer> A = Arrays.asList(1, 2, 3, 4);
protected static List<Integer> B = Arrays.asList(3, 4);
protected static List<Integer> C = Arrays.asList(5, 6);
protected static List<Integer> D = Arrays.asList(1, 2, 3, 4, 5);
protected static List<Integer> E = Arrays.asList();
/**
* Intersection = cap
*/
protected static String intersection() { return "∩"; }
/**
* Union = cup
*/
protected static String union() { return "∪"; }
/**
* Subset of
*/
protected static String subsetOf(){ return "⊂"; }
/**
* Superset of
*/
protected static String supersetOf(){ return "⊃"; }
/**
* Not a subset of
*/
protected static String notASubsetOf(){ return "⊄"; }
/**
* Subset of or equal to
*/
protected static String subSetOfOrEqualTo(){ return "⊆"; }
/**
* Superset of or equal to
*/
protected static String superSetOfOrEqualTo(){ return "⊇"; }
/**
* Print Set
*/
protected static String set(Collection<Integer> S)
{
return " {" + StringUtils.join(S.iterator(), ",") + "} ";
}
}
package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import static org.apache.commons.lang.StringUtils.rightPad;
import java.util.*;
import org.apache.commons.collections.*;
public class Union extends AbstractCollectionExampleUtils
{
public static void main(String[] args)
{
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(B), 13) + "=" + union(A, B));
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(C), 13) + "=" + union(A, C));
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(D), 13) + "=" + union(A, D));
System.out.println("When" + rightPad(set(A), 13) + union() + rightPad(set(E), 13) + "=" + union(A, E));
}
private static String union(Collection s, Collection ss)
{
return set(CollectionUtils.union(s, ss));
}
}


The Results:

When {1,2,3,4} ∪ {3,4} = {1,2,3,4}
When {1,2,3,4} ∪ {5,6} = {1,2,3,4,5,6}
When {1,2,3,4} ∪ {1,2,3,4,5} = {1,2,3,4,5}
When {1,2,3,4} ∪ {} = {1,2,3,4}


Example : Intersection in Java with CollectionUtils.intersection


Now, lets look at intersections and how this can be done in Java with the Apache Commons CollectionUtils.intersection.

The Code:

package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import static org.apache.commons.lang.StringUtils.rightPad;
import java.util.*;
import org.apache.commons.collections.*;
public class Intersection extends AbstractCollectionExampleUtils
{
public static void main(String[] args)
{
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(B),13) + " " + intersection(A, B));
System.out.println("When" + rightPad(set(B),13) + intersection() + rightPad(set(A),13) + " " + intersection(B, A));
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(C),13) + " " + intersection(A, C));
System.out.println("When" + rightPad(set(C),13) + intersection() + rightPad(set(A),13) + " " + intersection(C, A));
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(D),13) + " " + intersection(A, D));
System.out.println("When" + rightPad(set(D),13) + intersection() + rightPad(set(A),13) + " " + intersection(D, A));
System.out.println("When" + rightPad(set(A),13) + intersection() + rightPad(set(E),13) + " " + intersection(A, E));
System.out.println("When" + rightPad(set(E),13) + intersection() + rightPad(set(A),13) + " " + intersection(E, A));
}
private static String intersection(Collection s, Collection ss)
{
return "CollectionUtils.intersection(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ") = " + rightPad(set(CollectionUtils.intersection(s, ss)),11);
}
}


The Results:

When {1,2,3,4} ∩ {3,4} CollectionUtils.intersection( {1,2,3,4} , {3,4} ) = {3,4}
When {3,4} ∩ {1,2,3,4} CollectionUtils.intersection( {3,4} , {1,2,3,4} ) = {3,4}
When {1,2,3,4} ∩ {5,6} CollectionUtils.intersection( {1,2,3,4} , {5,6} ) = {}
When {5,6} ∩ {1,2,3,4} CollectionUtils.intersection( {5,6} , {1,2,3,4} ) = {}
When {1,2,3,4} ∩ {1,2,3,4,5} CollectionUtils.intersection( {1,2,3,4} , {1,2,3,4,5} ) = {1,2,3,4}
When {1,2,3,4,5} ∩ {1,2,3,4} CollectionUtils.intersection( {1,2,3,4,5} , {1,2,3,4} ) = {1,2,3,4}
When {1,2,3,4} ∩ {} CollectionUtils.intersection( {1,2,3,4} , {} ) = {}
When {} ∩ {1,2,3,4} CollectionUtils.intersection( {} , {1,2,3,4} ) = {}


Example : Sub collections and Proper Sub collections in Java with CollectionUtils.containsAny, isProperSubCollection, and isSubCollection


The first two examples were simple enough. Let’s look at Sub collection versus Proper Sub Collection. We will use the containsAny, isProperSubCollection and isSubCollection static methods off the CollectionUtils class.


package com.blogspot.apachecommonstipsandtricks.examplesOfCollectionsAndSets;
import java.util.*;
import static org.apache.commons.lang.StringUtils.rightPad;
import org.apache.commons.collections.*;
public class SubCollections extends AbstractCollectionExampleUtils
{
public static void main(String[] args)
{
System.out.println("Intersection Tests : ");
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(B), 13) + " " + containsAny(A, B));
System.out.println("Is" + rightPad(set(B), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(B, A));
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(C), 13) + " " + containsAny(A, C));
System.out.println("Is" + rightPad(set(C), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(C, A));
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(D), 13) + " " + containsAny(A, D));
System.out.println("Is" + rightPad(set(D), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(D, A));
System.out.println("Is" + rightPad(set(A), 13) + intersection() + rightPad(set(E), 13) + " " + containsAny(A, E));
System.out.println("Is" + rightPad(set(E), 13) + intersection() + rightPad(set(A), 13) + " " + containsAny(E, A));
System.out.println("Is" + rightPad(set(E), 13) + intersection() + rightPad(set(E), 13) + " " + containsAny(E, E));
System.out.println("");
System.out.println("Subsets Tests: " );
System.out.println("B " + subsetOf() + " A indicates B is a subset of A, but are not equal. When B is equal to A it is usually denoted as B " + subSetOfOrEqualTo() + " A");
System.out.println("");
System.out.println("Is" + rightPad(set(A), 13) + subsetOf() + rightPad(set(A), 13) + " " + subsetOfOrEqualTo(A, A));
System.out.println("Is" + rightPad(set(B), 13) + subsetOf() + rightPad(set(A), 13) + " " + subsetOfOrEqualTo(B, A));
System.out.println("Is" + rightPad(set(C), 13) + subsetOf() + rightPad(set(A), 13) + " " + subsetOfOrEqualTo(C, A));
System.out.println("");
System.out.println("Is" + rightPad(set(A), 13) + subSetOfOrEqualTo() + rightPad(set(A), 13) + " " + superSetOfOrEqualTo(A, A));
System.out.println("Is" + rightPad(set(B), 13) + subSetOfOrEqualTo() + rightPad(set(A), 13) + " " + superSetOfOrEqualTo(B, A));
System.out.println("Is" + rightPad(set(C), 13) + subSetOfOrEqualTo() + rightPad(set(A), 13) + " " + superSetOfOrEqualTo(C, A));
}
protected static String containsAny(Collection s, Collection ss)
{
return rightPad(" CollectionUtils.containsAny(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ")", 67) + " = " + String.valueOf(CollectionUtils.containsAny(s, ss));
}
protected static String subsetOfOrEqualTo(Collection s, Collection ss)
{
return rightPad(" CollectionUtils.isProperSubCollection(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ")", 67) + " = " + String.valueOf(CollectionUtils.isProperSubCollection(s, ss));
}
private static String superSetOfOrEqualTo(Collection s, Collection ss)
{
return rightPad(" CollectionUtils.isSubCollection(" + rightPad(set(s), 13) + "," + rightPad(set(ss), 13) + ")", 67) + " = " + String.valueOf(CollectionUtils.isSubCollection(s,ss));
}
}


The Results:

Intersection Tests :
Is {1,2,3,4} ∩ {3,4} CollectionUtils.containsAny( {1,2,3,4} , {3,4} ) = true
Is {3,4} ∩ {1,2,3,4} CollectionUtils.containsAny( {3,4} , {1,2,3,4} ) = true
Is {1,2,3,4} ∩ {5,6} CollectionUtils.containsAny( {1,2,3,4} , {5,6} ) = false
Is {5,6} ∩ {1,2,3,4} CollectionUtils.containsAny( {5,6} , {1,2,3,4} ) = false
Is {1,2,3,4} ∩ {1,2,3,4,5} CollectionUtils.containsAny( {1,2,3,4} , {1,2,3,4,5} ) = true
Is {1,2,3,4,5} ∩ {1,2,3,4} CollectionUtils.containsAny( {1,2,3,4,5} , {1,2,3,4} ) = true
Is {1,2,3,4} ∩ {} CollectionUtils.containsAny( {1,2,3,4} , {} ) = false
Is {} ∩ {1,2,3,4} CollectionUtils.containsAny( {} , {1,2,3,4} ) = false
Is {} ∩ {} CollectionUtils.containsAny( {} , {} ) = false

Subsets Tests:
B ⊂ A indicates B is a subset of A, but are not equal. When B is equal to A it is usually denoted as B ⊆ A

Is {1,2,3,4} ⊂ {1,2,3,4} CollectionUtils.isProperSubCollection( {1,2,3,4} , {1,2,3,4} ) = false
Is {3,4} ⊂ {1,2,3,4} CollectionUtils.isProperSubCollection( {3,4} , {1,2,3,4} ) = true
Is {5,6} ⊂ {1,2,3,4} CollectionUtils.isProperSubCollection( {5,6} , {1,2,3,4} ) = false

Is {1,2,3,4} ⊆ {1,2,3,4} CollectionUtils.isSubCollection( {1,2,3,4} , {1,2,3,4} ) = true
Is {3,4} ⊆ {1,2,3,4} CollectionUtils.isSubCollection( {3,4} , {1,2,3,4} ) = true
Is {5,6} ⊆ {1,2,3,4} CollectionUtils.isSubCollection( {5,6} , {1,2,3,4} ) = false


The Apache Commons Collections has some enormous potential. I find I uses these static methods frequently. I hope this has helped you in solving your problems, please feel free to drop me a line if you have any questions.

Author: Philip A Senger

Saturday, February 7, 2009

Examples of Functors, Transformers, Predicates, and Closures in Java

One day, I found myself re-designing a procurement portal, and I kept re-writing the same for-loop over and over again (no pun intended). I had an epiphany; I could do better and I started using the Apache Commons Collection Utilities (Transformers, Predicates, and Closures). Now don’t think just because I started to use the Apache Commons Collection Utilities, the project was better. However, the result was a highly extensible framework…. Later it was dismantled by another team… but that is a different story (Grin).

Functors

I laugh every time I think of the word Functors, but that is because I’m immature, case in point, I still laugh at fart jokes. Anyway, Functors, or Function Objects, in the Apache or Jakarta Commons Collection Utilities are a set of interfaces designed specifically to be used against collections of objects. This framework embodies a balance between code reuse and behavioral specialization through composition as opposed to strict Object Oriented design. Composition is well suited for Creational patterns such as Factories, Structural patterns like Decorators, or Behavioral patterns like Strategies. The Apache Commons Collections framework defines three types of interfaces:
  • Closures are functions that can alter the object and get a reference to each object in the collection.

  • Transformers are responsible for transforming data from one format to another or from one object to another.

  • Predicates simply execute a conditional test against each item in a collection and return true or false for each item.
NOTE:
My examples sometimes use Anonymous Inner Classes and Inner Classes. Some developers have strong feelings about defining classes in this manner. There are times when doing this is appropriate and times when it is inappropriate. As with any programming solutions, this technique may or may not suit your needs or environment. So, let’s get over it and move on.

Closure

Here is a typical problem statement which maybe resolved with the use of Closures. I want to execute a specific method or change the state on every object in a collection. For example, I might want to execute the toString method. Please note that these examples do not pull out a value and transform into another collection of objects. They simply iterate over the collection and do something to it or with it. For the purpose of this first example, I will send the results of the toString to system.out. In the second example, I will alter the state of each bean and change the name of every object. Both examples use the utility method CollectionUtils.forAllDo.

Lets take some baby steps, and look at this example.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
import org.apache.commons.lang.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class SimpleClosure
{
public static void main(String[] args)
{
System.out.println("\nTest Number One Results :");
List<String> collectionOfWords = Arrays.asList("Java", "Example",
"Help", "Tips", "And",
"Tricks", "Apache",
"Commons", "Collections");
// Lets call toString on every object and print it out.
CollectionUtils.forAllDo(collectionOfWords, new Closure()
{
public void execute(Object o)
{
assert o != null;
System.out.print(o.toString() + " ");
}
});
System.out.println("\n\nTest Number Two Results :");
int i = 1;
List<DTO> collectionOfDTOs = Arrays.asList(new DTO(i++, "Java Tips and Tricks", Gender.Male, State.WI),
new DTO(i++, "Apache Commons" , Gender.Male, State.WI),
new DTO(i++, "Jakarta Commons" , Gender.Male, State.WI),
new DTO(i++, "Collections" , Gender.Male, State.WI),
new DTO(i++, "Closures" , Gender.Male, State.WI) );
CollectionUtils.forAllDo(collectionOfDTOs, new Closure()
{
public void execute(Object o)
{
DTO dto = (DTO) o;
assert dto != null;
String s = StringUtils.defaultIfEmpty(dto.getName(), "null");
dto.setName("Yoda says, " + s + " Rocks!");
}
});
CollectionUtils.forAllDo(collectionOfDTOs,PrintIt.getInstance());
}
}
The Results:
Test Number One Results :
Java Example Help Tips And Tricks Apache Commons Collections

Test Number Two Results :
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Yoda says, Java Tips and Tricks Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Yoda says, Apache Commons Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Yoda says, Jakarta Commons Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Yoda says, Collections Rocks!', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Yoda says, Closures Rocks!', gender=Male, state=WI}

You might have noticed I created a class called PrintIt and it has a static method called getInstance. This class is an implementation of the singleton design pattern. In a large system where Closures (Predicates and Transformers) are being created by the hundreds, it makes sense to keep the memory foot print to a minimum. This is accomplished by using a singleton pattern. Word of caution, there is only one instance of this class per Java Virtual Machine (jvm), so using static properties could cause big problems, if you don’t understand how they work.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import org.apache.commons.collections.*;
public class PrintIt implements Closure
{
// This class implements a Singleton Pattern
private static PrintIt ourInstance = new PrintIt();
/**
* Get a singleton instance of PrintIt
*/
public static PrintIt getInstance()
{
return ourInstance;
}
private PrintIt() // This is a singleton, dont change this!
{
}
public void execute(Object o)
{
System.out.println( o.toString() );
}
}
Once again, in the first example, all we did was iterate through a collection and call toString on every element in the collection. In the second example, we are actually modifying the state of the bean by rewriting the name of the DTO.. I love Yoda! Lets talk about transformers now.

Transformer

I’m a little older than the Transformer cartoons, but I cant help but think of the movie that was released not long ago. So, here is a problem statement that would best be resolved by transformers. You just have a whole bunch of string values from a Http Request object and you need to convert them to Integers.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class SimpleTransformer
{
public static void main(String[] args)
{
Collection<String> stringOfNumbers = Arrays.asList("1", "2", "3", "4");
Collection<Integer> intNums = CollectionUtils.collect(stringOfNumbers, new Transformer() {
public Object transform(Object o) {
return Integer.valueOf((String) o);
}
});
CollectionUtils.forAllDo(intNums, PrintIt.getInstance() );
}
}
The Results:
1
2
3
4
Again, I used the PrintIt class to print the results, but you get the idea with this example. I converted a collection of Strings to Integers. Of course there are many ways to skin a cat, this is just an example. You might have noticed by now, that the interface is public Object transform(Object o)… It is not a Java 1.5 Generics implementation. I’m not entirely sure why the Apache team hasn’t released a Java 1.5 Generics version of these utilities, but if you are in a pinch and you have to have it, someone posted a link to an adaptation of the library that has the Generics. http://larvalabs.com/collections

Here is a more practical problem statement. You have a collection of plain old java beans and in each bean a method that returns a String, part of the String represents the id into another system, environment or sub identity. For example, the String might be prefixed with three letters, for example "PAS". You might find some legacy ERP systems that do this to the PO number. Maybe the billing department puts the initials of the sales person or team at the front of the number. In this example we will transform an Array of Strings that have ids in them into an array of ids in the form of numbers only.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
import org.apache.commons.lang.*;
public class SimpleTransformer
{
public static void main(String[] args)
{
Collection<String> stringOfNumbers = Arrays.asList("ABC0001", "BCD0002", "CDF0003", "BFA0004");
Collection<Integer> intNums = CollectionUtils.collect(stringOfNumbers, new Transformer()
{
public Object transform(Object o)
{
String s = ((String) o);
return Integer.valueOf(s.substring(3, s.length()));
}
});
CollectionUtils.forAllDo(intNums, PrintIt.getInstance());
}
}
The Results:
1
2
3
4
Now let’s look at something a little more practical. As a problem statement, let’s say we have a billing object from the old system called OldBill, and we need to identify it in the new system, called NewBill. The ids in the old system started with “A” and a number. In the new system they will start with “Z” and the number from the old system plus 500. In the next example we will break apart the concerns of the transformers into two different transformers and glue them together with the utility class TransformerUtils.chainedTransformer. Creating a transformer in this manner allows us to plug in a different behavior or add and subtract behaviors, even on the fly.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class ChainedTransformer
{
public static void main(String[] args)
{
List<OldBill> aList = Arrays.asList(new OldBill("A1"), new OldBill("A2"),
new OldBill("A3"), new OldBill("A4"));
Transformer[] chainedTransformer = new Transformer[]{
new Transformer() {
public Object transform(Object o) {
return ((OldBill )o).getId().replace('A', 'Z');
}
},
new Transformer() {
public Object transform(Object o) {
char[] c = ((String) o).toCharArray();
int x = Integer.parseInt(String.valueOf(c[1])) + 500;
return new NewBill( String.valueOf(c[0]) + x );
}
}
};
System.out.println("The aList");
CollectionUtils.forAllDo(aList, PrintIt.getInstance());
List<NewBill> bList = (List<NewBill>) CollectionUtils.collect(aList, TransformerUtils.chainedTransformer(chainedTransformer));
System.out.println("\nThe bList");
CollectionUtils.forAllDo(bList, PrintIt.getInstance());
}
}
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
public class OldBill
{
private String id;
public OldBill(String id)
{
this.id = id;
}
public String getId()
{
return id;
}
public void setId(String id)
{
this.id = id;
}
@Override public String toString()
{
return "OldBill{id='" + id + "\'}";
}
}
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
public class NewBill
{
private String id;
public NewBill(String id)
{
this.id = id;
}
public String getId()
{
return id;
}
public void setId(String id)
{
this.id = id;
}
@Override public String toString()
{
return "NewBill{id='" + id + "\'}";
}
}
The Results:
The aList
OldBill{id='A1'}
OldBill{id='A2'}
OldBill{id='A3'}
OldBill{id='A4'}

The bList
NewBill{id='Z501'}
NewBill{id='Z502'}
NewBill{id='Z503'}
NewBill{id='Z504'}
Predicate

Predicates do one thing and one thing only, they return either true or false. As a problem statement, let’s say we have a collection of Strings and we want keep out values that can not be converted to numbers.

The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
public class SimplePredicate
{
public static void main(String[] args)
{
List<String> mixedup = Arrays.asList("A", "0", "B", "C", "1", "D", "F", "3");
Collection numbersOnlyList = CollectionUtils.predicatedCollection(new ArrayList(),
new Predicate() {
public boolean evaluate(Object o) {
try {
Integer.valueOf((String) o);
return true;
} catch (NumberFormatException e) {
return false;
}
}
});
for (String s : mixedup) {
try {
numbersOnlyList.add(s);
} catch (IllegalArgumentException e) {
System.out.println("I love CollectionUtils!");
}
}
System.out.println("\nResults of the predicatedCollection List:");
CollectionUtils.forAllDo(numbersOnlyList, PrintIt.getInstance() );
}
}
The Results:
I love CollectionUtils!
I love CollectionUtils!
I love CollectionUtils!
I love CollectionUtils!
I love CollectionUtils!

Results of the predicatedCollection List:
0
1
3
Ok here is something more useful. Have you ever had a need to do something like SQL but in Java? For example, you wanted to select beans from a collection based on some conditional blocks? Here are some great examples of SQL commands that can be used in Java and they behave just like SQL complex where clauses, distinct, like, and group by. In this example, we will be using the folloing utlitly classes and maps
The Code:
package com.blogspot.apachecommonstipsandtricks.transformersexamples;
import java.util.*;
import org.apache.commons.collections.*;
import org.apache.commons.collections.map.*;
import com.blogspot.apachecommonstipsandtricks.*;
public class PredicatesSQLSample
{
public static void main(String[] args)
{
List<DTO> list = Arrays.asList(new DTO(1,"Bob", Gender.Male, State.WI), new DTO(2,"Larry",Gender.Male, State.WI),
new DTO(3,"Bill", Gender.Male, State.WI), new DTO(4,"Sue", Gender.Female, State.AZ),
new DTO(3,"Bill", Gender.Male, State.WI), new DTO(4,"Sue", Gender.Female, State.AZ),
new DTO(5,"Joe", Gender.Male, State.AZ), new DTO(6,"Zoe", Gender.Female, State.MI));
Predicate sqlOrQueryPredicate = PredicateUtils.anyPredicate(new Predicate[]{
new Predicate()
{
public boolean evaluate(Object o)
{
return State.WI.equals(((DTO) o).getState());
}
}, new Predicate()
{
public boolean evaluate(Object o)
{
return Gender.Female.equals(((DTO) o).getGender());
}
}
});
Predicate sqlAndQueryPredicate = PredicateUtils.allPredicate(new Predicate[]{
new Predicate()
{
public boolean evaluate(Object o)
{
return State.AZ.equals(((DTO) o).getState());
}
}, new Predicate()
{
public boolean evaluate(Object o)
{
return Gender.Male.equals(((DTO) o).getGender());
}
}
});
Predicate likeNameStartsWithB = new Predicate(){
public boolean evaluate(Object o)
{
return ((DTO) o).getName().startsWith("B");
}
};

Collection aList = CollectionUtils.select(list, sqlOrQueryPredicate);
Collection bList = CollectionUtils.select(list, PredicateUtils.notPredicate( sqlOrQueryPredicate ));
Collection cList = CollectionUtils.select(list, sqlAndQueryPredicate);
Collection dList = CollectionUtils.select(list, PredicateUtils.allPredicate(new Predicate[]{PredicateUtils.uniquePredicate(), sqlOrQueryPredicate} ));
Collection eList = CollectionUtils.select(list, PredicateUtils.allPredicate(new Predicate[]{PredicateUtils.uniquePredicate() ,likeNameStartsWithB} ));
Collection fList = CollectionUtils.select(list, PredicateUtils.uniquePredicate() );

Map aGroupByStateMap = TransformedMap.decorate(new MultiValueMap(),new Transformer(){
public Object transform(Object o)
{
return ((DTO) o).getState();
}
}, TransformerUtils.nopTransformer() );
for (Object o : fList)
{
aGroupByStateMap.put( o, o );
}

System.out.println("\nAll the people :\nselect * from list");
CollectionUtils.forAllDo(list,PrintIt.getInstance());
System.out.println("\nAll the people in Wisconsin OR Female :\nselect * from list where ( state = WI or gender = female );");
CollectionUtils.forAllDo(aList, PrintIt.getInstance());
System.out.println("\nAll the people NOT ( Wisconsin OR Female ) :\nselect * from list where ! ( state = WI or gender = female );");
CollectionUtils.forAllDo(bList, PrintIt.getInstance());
System.out.println("\nAll the people in Arizona AND Male :\nselect * from list where ( state = AZ and gender = male );");
CollectionUtils.forAllDo(cList, PrintIt.getInstance());
System.out.println("\nAll the distinct people in Arizona AND Male :\nselect distinct * from list where ( state = WI or gender = female );");
CollectionUtils.forAllDo(dList, PrintIt.getInstance());
System.out.println("\nAll the distinc people with the name that starts with B :\nselect distinct * from list where name like \"B%\";");
CollectionUtils.forAllDo(eList, PrintIt.getInstance());
System.out.println("\nAll the distinct people grouped by state :\nselect distinct * from list group by state;");
Set states = aGroupByStateMap.keySet();
for (Object state : states)
{
System.out.println(((State)state).getFullyQualifiedName());
CollectionUtils.forAllDo((Collection) aGroupByStateMap.get(state), PrintIt.getInstance());
}
}
}
The Results:
All the people :
select * from list
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}

All the people in Wisconsin OR Female :
select * from list where ( state = WI or gender = female );
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}

All the people NOT ( Wisconsin OR Female ) :
select * from list where ! ( state = WI or gender = female );
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}

All the people in Arizona AND Male :
select * from list where ( state = AZ and gender = male );
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}

All the distinct people in Arizona AND Male :
select distinct * from list where ( state = WI or gender = female );
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}

All the distinc people with the name that starts with B :
select distinct * from list where name like "B%";
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}

All the distinct people grouped by state :
select distinct * from list group by state;
ARIZONA
com.blogspot.apachecommonstipsandtricks.DTO{id=4, name='Sue', gender=Female, state=AZ}
com.blogspot.apachecommonstipsandtricks.DTO{id=5, name='Joe', gender=Male, state=AZ}
MICHIGAN
com.blogspot.apachecommonstipsandtricks.DTO{id=6, name='Zoe', gender=Female, state=MI}
WISCONSIN
com.blogspot.apachecommonstipsandtricks.DTO{id=1, name='Bob', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=2, name='Larry', gender=Male, state=WI}
com.blogspot.apachecommonstipsandtricks.DTO{id=3, name='Bill', gender=Male, state=WI}

Well, that wraps it up. Next week I plan on showing you some more examples of this fantastic api called Apache Commons.

Author: Philip A Senger