Thursday, November 13, 2008

Managing a collection of data and running Query By Example with Coherence

Problem (1): Coherence NamedCache extends java.util.Map i.e, the data to be managed in the cache must be in a <key>, <value> pair. One of the challenges to manage just a Collection of keys or elements in a cache is to associate these keys with some values. If Values are not given we need to use a Serializable Object that can become it's values (or Nulls). So your collection {a, b, c ..., z} becomes [{a, <o>}, {b, <o>}, ... {z, <o>}] with o being:

public class o implements Serializable { }
Problem (2): Running Queries against the Keys. Coherence supports KeyExtractor that extracts attribute values from cache keys instead of on the Values. The Filter will look like filter = new EqualsFilter(new KeyExtractor("getAge"), value); that matches if the getAge () method of a cache key Object is equal to the value.
Problem (3): Query By Example. This is another challenge. Coherence cache supports Filter mechanism that can be run on existing value for its attributes. Out of the box there is no mechanism that can parse Query By Example. QBE means an Object with partial attribute set is passed to find out all or any cache keys that have same attribute values. Lets introduce a Discoverer class that has following:
protected Filter getFilter(String method, String value) {
Filter filter = null;
if (!(value.equals("*") || value.equals(""))) {
filter = new EqualsFilter(new KeyExtractor(method), value);
} else {
filter = new GreaterEqualsFilter(new KeyExtractor(method), "");
}
return filter;
}
Problem (4): Combining Key searches into one service to support multiple key types. First step is to make sure that each cache stores a specific type of Keys. And if one type of Key is very different from another type of key then create dedicated Discoverer classes that specializes in finding a specific type of Key. Cache configuration could look like:

<caching-scheme-mapping>
<cache-mapping>
<cache-name>Key1
<scheme-name>distributed-scheme
<init-params>
<init-param>
<param-type>string
<param-value>com.some.Key1
</init-param>
</init-params>
</cache-mapping>
<cache-mapping>
<cache-name>Key2
<scheme-name>distributed-scheme
<init-params>
<init-param>
<param-type>string
<param-value>com.some.Key2
</init-param>
</init-params>
</cache-mapping>
</caching-scheme-mapping>

Then maintain a local map that stores cache to Object type mapping as:
XmlElement elem = null;
NamedCache nCache = null;
XmlElement cSMap =
CacheFactory.getConfigurableCacheFactory().getConfig().getElement("caching-scheme-mapping");
Iterator mappings = cSMap.getElements("cache-mapping");
String dataType = null;
while (mappings.hasNext()) {
elem = (XmlElement) mappings.next();
nCache = CacheFactory.getCache(elem.getElement("cache-name").getString());
try {
dataType =
elem.getElement("init-params").getElement("init-param").getElement("param-value").getString();

objToMap.put(nCache.getCacheName(), dataType);
mapToObj.put(dataType, nCache);
} catch (Exception exp) {
// -- Log the Exception and skip adding Indexes
log(exp);
}
}
Once mapping is known, create a Discoverer as:
public Collection getKeys(Key filterKey) {
IDiscoverer disc;
try {
disc = (IDiscoverer) keyDiscoverer.get(filterKey);
if (disc == null) {
// -- Even though the cost of Reflection is high but to it takes
// -- a microsecond to initialize an Object. To reduce the
// -- reoccuring cost use a local Map to store the instance
String cName = filterKey.getClass().getName();
String discName =
cName.substring(cName.lastIndexOf(".") + 1) + "Discoverer";
disc =
(IDiscoverer) (this.getClass().getClassLoader().loadClass("com.some." +
discName)).newInstance();
keyDiscoverer.put(filterKey, disc);
}
return disc.getKeys(filterKey);
} catch (Exception exp) {
log(exp);
}
return null;
}
Once the getKeys () is delegated to right Discoverer, the Discoverer object can work on the key's attributes directly. One example being:

public Collection getKeys(Key key)
throws IllegalAccessException, InvocationTargetException {

if (!(key instanceof Key1)) {
return null;
}

String cName = key.getClass().getName();
NamedCache nCache =
CacheFactory.getCache(cName.substring(cName.lastIndexOf (".") + 1));
Key1 pKey = (Key1) key;
Filter filters[] = new Filter[2];
filters[0] = getFilter("getAttribute1", pKey.getAttribute1());
filters[1] = getFilter("getAttribute2", pKey.getAttribute2());
Filter composite = new AllFilter(filters);
return nCache.keySet(composite);
}
As you see the code uses Java Reflection where ever possible but avoids it in re-occurring calls. The Specialized classes like Key1Discoverer on the other hand works on class attributes directly.
This is a simple but quite efficient approach to manage a collection of data and run Templated Queries on it. Coherence cache configuration dictates Object type to be stored in a cache. Discoverer provides a mechanism to have a specialized way of finding keys and manipulate QBE and single service class using Java reflection allows to scale the key types and its discovery to any number of types. Would be glad to see any suggested improvements...

Friday, November 07, 2008

Transfer of Power pattern

This is a little Java trick that came out of the following Problem Statement: How to call a pre-defined method of an invoking class if it is defined in it? Or, invocation through precedence.
If I have three classes Invoker1, Invoker2 and MyUtility and both Invokers call a method of MyUtility, Is it possible for MyUtility to call a method of the Invoker if defined instead of it's own? I call it a transfer of power pattern - An Invoker class dictates what to call not by condition or heirarchy but by definition of its location.

// -- MyUtility.java
public class MyUtility {
private void preInvocation () {
...
}
public void doSomething () {
System.out.println ("I am doing something");
}
}
// -- Invoker1.java
public class Invoker1 {
public void invoke () {
new MyUtility ().doSomething ();
}

public void preInvocation () {
System.out.println ("Invoker1's preInvocation called");
}
}
// -- Invoker2.java
public class Invoker2 {
public void invoke () {
new MyUtility ().doSomething ();
}
}
The way this application works is if preInvocation method is defined in an invoking class, MyUtility should call it instead of it's own. A workflow is defined in a way that does not require a tight integration with an Object hierarchy.

How to do it? Define MyUtility in the following way:

// -- MyUtility.java
public class MyUtility {
private void preInvocation() {
System.out.println("Pre-Invocation of MyUtility");
}
public void doSomething() {
Throwable t = new Throwable();
StackTraceElement[] elements = t.getStackTrace();
String callerClassName = elements[1].getClassName();
Class clz;
try {
clz = Class.forName(callerClassName);
Method method =
clz.getDeclaredMethod("preInvocation", null);
method.invoke(clz.newInstance(), new Object[] { });
} catch (Exception exp) {
// -- Any Exception and call it's own.
preInvocation();
}
}
}
# Run Invoker1 => Invoker1's preInvocation called
# Run Invoker2 => Pre-Invocation of MyUtility
Have fun!

Saturday, November 01, 2008

I endorse McCain, Because...

@11/01/2008: After deep thinking and three days before Nov'4th I have decided to endorse McCain. And the reasons I will reveal after the result ;).

@11/04/2008: And the reason is that whoever I ever voted for has always won, and whoever I endorsed but did not vote has lost. And I do not have voting rights in America. Second, I did not have any goosebumps to endorse McCain. Lets not forget that he is a nice and decent man who gave his entire life to the service of this country. But, Sorry McCain this year we needed change. And a change from a Guy who is ground to earth and seems to think of us The Middle class America. We needed American troops to conclude its services in Iraq and concentrate on Osama Bin-Laden. We needed lower health care cost. And we needed a President of America and of entire free world to be at least a good Orator and able to connect to his audience. We needed someone who can bring back the respect once America enjoyed across the World. And this was an only opportunity to find a Candidate who is equally white as he is black - A nation that a future America represents.
Congratulations Obama and a Salute to McCain.

Monday, October 27, 2008

Nagle or not Nagle?

What is the Nagle algorithm in reference to TCP/IP?

Nagle's algorithm is a way of avoiding network congestion by limiting to at most one the number of "tinygrams" (that is, packets that are less than full size) an application can have outstanding at once. It does this by concatenating small segments until it has a full segment or it receives an acknowledgment of the outstanding small segment. The algorithm was proposed by John Nagle in RFC896. Its latest incarnation, which differs slightly from Nagle's original formulation, is described in RFC1122.

The Nagle algorithm can sometimes interact with the TCP's "delayed ACK" mechanism, another congestion avoidance strategy, in a way that increases network congestion and delay. For this reason, it is sometimes disabled, but this is appropriate only in very special circumstances. See the book Effective TCP/IP Programming or Rich Stevens' TCP/IP Illustrated, for complete details.

** Definition taken from searchnetworkingchannel.techtarget.com

This has been a problem with Windows OS (and probably with other OS as well) that TCP connection waits to send next packets unless an acknowledgment is received. One of the Clients reported a drastically reduced performance in Coherence's multi-cluster data replication strategy. When nothing was found even after reviewing the architecture, code reviewing a number of times and involvement of upper management, we had to bring in a Top Coherence expert and he tracked the problem in 15 minutes. It was Nagle.

Friday, October 24, 2008

At last found sometime to blog

Last two weeks have been non-busingly busy. First there were some anxiety to attend the first Coherence SIG in NY. Last time I met Cameron before this SIG was at a client's site and I had screwed it up then (Not Me really but Hertz never lost) in driving directions. It was also nice to meet Alex, Rob, Brian, Peter O., Parker, Craig, Phil, Peter U., Tom and Patrick. And then it was nice seeing some of my clients as well and discussing their current challenges along with introducing Boris, Vikas and Anusha to the Coherence team members that they interact by emails and phone but had never met in person. It was also nice to listen to Brian Oliver's talk on Asynchronous Order preserving WAN Replication strategy. Nice to see someone bringing components that I had been aware of and used before, together in a more logical and robust way. And also hear Steve outline his strategy in successfully managing 'N' number of teams and establishing a centralized Architecture team.
Then I found there are many takers of my weekend project (jdbc driver for Coherence) that Rob and Brian were kind enough to allow to be submitted to Coherence's Incubator project. In fact two of my other colleagues joined hands with me as well and one of who's has a much mature implementation than mine. While I still think my broad approach is better ;). But it is hard to find ample time to do something that requires off-the-route creativity, and with work burden piling up again I am not sure when I would and should donate anytime towards it. On top of this one of the client reported problems with an earlier implementation and even though understanding their frustrations it is almost impossible to devote anytime to debug it after any active engagement ends. Probably the problem is due to Nagle (TCP connection waiting to send next packets) but need more testing. Oracle has a support structure that is very good in tackling Customer issues and works better than contacting Consultants and Product Engineers directly. I have tried my best and engaged everyone I could have to support them and is good to work with some intelligent developers who are quick to tests things out. Big Guys are already on top of it so I need not worry about it. And then there are some very exciting and technologically challenging projects coming up. I am working simultaneously with at least 5 separate teams to do architecture reviews and coding. When too many things go on in parallel I easily become a stateless machine - do not remember anything. Even simple implementations take time as Context Switching becomes a challenge.
And with watching your investments at a mercy of people who you do not know and have little trust on it has not been very easy. And with so many things to do - Working with Customers, Pending Reading materials, Lagging ideas, traveling every week on a 4 hour+ flights twice and aching body it was nice to steal few minutes to blog.

Saturday, October 18, 2008

Election participation - Go Vote

With American elections just around the corner, it reminds me of elections in India. Average percentage of voting in India is around 40-45%. Out of the rest 55 a huge majority does not go vote because they are lazy, because they do not think their vote matters, because they think their candidate will win anyway and other miscellaneous reasons like health, weather and fear of anti-social elements just to find that a candidate has won who he/she did not want. So people living in a Democratic society it is our responsibility to go out and vote. Whoever you like but go out there and vote. My parents have always encouraged us to participate in elections. Not in rhetoric but in ink. I remember whenever I was home during elections in a place I was registered to vote I and my family always went and voted. This is a way to show off your freedom to the rest of Un-democratic world that do not enjoy what we do. And remember if you did not vote then you lose your moral right to whine about the laws, bills and candidates that hurt you. Its your country and its future is your responsibility. Shape it with your vote.

Wednesday, October 08, 2008

Is surge working or Terrorists already won?

A re-run of Bill Maher show today just shook me. After seven years of 9-11 Osama Bin Laden and his lieutenant is still alive. Work on World Trade Center is still not complete. American economy is seeing unprecedented downturn. The leader of the terrorists who attacked us might be laughing today. And what do we listen in the debate? "My friends I know where he is. I know how to take him out".. and no one asks that it has been a Republican government for last eight years, If someone does know how to take him out why was it not done? More than a Trillion dollars burnt on personal egos and revenge that could have been spent on Health care, Energy and other things. We see 4000+ lives lost with the main culprit still at large, that might have only taken a few hundred million well spent dollars to take him out and destroy his network. We see a Police officer in his uniform bashing his fellow American and planting doubt of him being a Muslim and words like "Kill Him" in Sarah Palin's political rallies and no one denouncing it.. "Who is Barack? - A terrorist!" and Republicans think this is the only way to win? May be Barack does not understand may be he does not, but at least he is clear what he intends to do.. "My friends I know how to do it" is still not making it clear how will it be done. The respect I had for McCain is badly hurt by what Sarah Palin and his other apparatus is doing. I guess they both promised to run a respectable campaign. But when the time is to get together and united we are demeaning personally our own fellow citizen. What has gone wrong with us? Did Terrorists hit us on our character and values?