बहुत दिन बाद फिर एक शेर

बदलती हैं रातें बदलते हैं दिन भी,
पर ये चंदा वहीं है और तारे वहीं ही।
बस सपनो में होंगी बदलती सी चाहें,
खड़ा था जहाँ मैं, खड़ा हूँ वहीं ही।

My favorite Indian restaurants in Sunnyvale/Santa Clara area

In general Bay Area is pretty good in Indian food and thanks to a tough competition you will find decent Indian food in most places. But following are some and their some recipes that stands different and awesome:

Bikaner Sweets, Sunnyvale - Paneer Tikka Masala
Bengali Sweets, Sunnyvale - Chicken Biryani, Chhole Bhature
Dosa Place, Santa Clara - Uttappam
Chaat and Curry, Sunnyvale - Kadai Lamb
Kwality Sweets, Santa Clara - Sarson ka Saag
Shaalimar, Sunnyvale - Seekh Kabab
Madras Cafe, Sunnyvale - Masala Dosa
Appakadai, Santa Clara - Aappam

Salute to the Shankaracharya of his religion - Cricket

Congratulations India on its Mars Mission

First my hearty congratulations to Indian scientific community specially ISRO to have achieved what many would have thought unimaginable just a few years ago. I would like to mention three very popular stigmas:

# Fix the Toilets Myth

• Many would ask with a country still with no proper toilets for everyone why Mars and why now? Answer is they are totally unrelated. India has enough money and resources to fix the toilets and provide everyone of its citizen better sanitary services. Money comes and money goes. It is the inefficiency and corruption in States that are unable to provide such basic facility to all of its residents not because India can not achieve this goal. 
• Second, like any Corporation you will find struggling divisions and the bread runners. You don't stop evolution and research in what you are totally good at just because some parts are non-functional and broken. Frankly ISRO has proven its capability to better utilize its resources and have championed what they are supposed to do. Broken toilet will and must not stop them.
• May be with its success we would be able to send those who were supposed to fix the Toilets out to Mars someday.

# A Country of Snake Charmers

• It is so surprising that even in 2013 you once in a while meet people who identify India a country of Elephants and Snake Charmers. Why would a country with such a backward outlook consider flying to Mars?
• There is a popular belief in Hindu mythology that Earth stands on top of a Snake head. Charming a snake means having an ability to know and control Earth's movement but also to understand and control the support structure of the Earth. The support structure of Earth is not a real snake but a sequence of planets and satellites moving in elliptical movement as a Cobra moves. Charming the snake is to champion our Galaxy. The snake charmers were not some dirty dhoti clad serpentine owners they were the scientists whose work mixed with religious theologies got lost in poor interpretations by half cooked intelligentsia.

# Triggering an Arms Race

• You can not create unless you have an ability to destroy - This what manifestation of Shiva says, a God of creation and destruction. You have to champion the tools and mechanism that can destroy to create an environment of peace.
• More than anything else what protected Americans and Russians during the cold war period was their ability to produce weapons of mass destruction. They did not engage in house to house close quarter combats. There were far less killed then than today's much one sided security engagements. Once you realize your strength and get an appreciation of your enemy for your strength is when you start to become friends. 
• Its not an armed race. It is a way to extend the ability of Humans to reach outer space and tell the world pity regional disputes are inconsequential. Collaboration will follow.

My historic Coherence Incubator association

BMW Ad

Only if everyone can respond like a BMW!

Elastic Charging Engine on Raspberry Pi

Fault Tolerance in Elastic Charging Engine

Joke: The pilot screamed - The plane is overloaded and we need four people to jump off the plane to save it. Unfortunately we only have three parachutes so one person has to make a sacrifice. The Doctor said - I save lives and I must be saved and he jumps with one parachute. The Lawyer said - I am the smartest person on Earth and I must be saved  and he jumps with a parachute. Seeing no one else was jumping, the priest said to the kid - Son, you have your entire life ahead of you, you take the last parachute and jump and I will sacrifice my life. Kid said - Don't worry Father, the smartest person on Earth just jumped off with my backpack.

Overloaded systems require drastic measures to be taken and different patterns have to be applied to save the system. What is important to understand is, once overloaded the system operates on the edge of the complete meltdown and patterns that are applied to mitigate and recovery are different from other types of fault recovery.
Elastic Charging Engine was built with fault tolerance in mind. With Oracle Coherence providing the core execution platform there are implicit fault tolerance in design. ECE extended its fault tolerance in its own design:

• Redundancy
• By the virtue of Oracle Coherence's design each cache entry in a partitioned scheme are backed up on another node preferably on another host. If the primary cache entry is lost the back up becomes the primary and in process creates a new backup on another host. Coherence implicitly is NODE SAFE. Any change in the state of the primary cache entry is by default applied to its backup synchronously before calling the process success (Pattern (I) - Data Redundancy).
•  After the provisioning exercise of finding the total number of nodes required for a given data set one additional host is typically added called a (n+1) configuration to make sure the cluster can sustain a machine failure without losing any data.

• Processing are co-located with Data

• As when/if nodes die the partitions move to new members and processing scheduled on those lost partitions are moved to new members. (Pattern (II) - Processing Failover).

• Need to provision the system to minimize the time taken for partition move. Faster the data moves to new owners, faster the processing moves. Minimize the Mean Time To Recovery.

• Requests destined for the just lost member are put in a suspense queue and are retried when the partitions get its new owner (Pattern (III) - Retry).

• Checkpoints
• ECE by default bundles requests together so that a batch of requests for Customers that are managed in partitions owned by a single cluster member can be processed together. This strategy is key to support higher through puts.
• If a batch is partially processed that some requests have been processed but not all before the node dies the same batch is submitted to a suspense queue to be retried. Problem is we do not want the same request already processed before to be reprocessed. 
• When a request is submitted ECE checks if the same request has been processed before (Pattern (IV) - Duplicate Detection).

With its second released version (11.2.0.0.0) also nick named Amazon, Elastic Charging Engine introduced a new feature "Overload Protection" extending its commitment to provide a true fault tolerant system.

Overload Protection:
ECE provides a layer of APIs that allows any application to interact with its system. This layer has APIs to construct messages that ECE can understand, data structures to send the request and receive responses in and policies around how when these requests are processed. Internally this layer called Batch Request Service (Or BRS) is an amalgam of the following behaviors:

• Identifying the Member where the submitted request be sent to for processing.
• Maintaining a batch of requests, its ripe policies and its responses.
• Tuning parameters to provision the ECE for ultra low latencies or excessive high throughput or something in the middle.
• Maintaining a suspense queue for the requests that needs to be retried.
• Managing a high degree of parallelization of threads.
• To make sure the entry point to the charging system can not be overloaded either by a natural increase in network traffic or denial of service attacks.

How does it work?
BRS has a pool of Threads dedicated to process events. Most of the time these threads wait for new batches to be sent but as these batches are sent to processing nodes of the cluster the CPU usage on those nodes increase. ECE is primarily a CPU bound application and the most precious element in its design are CPU cores. Though ECE has been benchmarked for extremely high throughput (>1000 requests/sec/core) but sending an extremely high number of requests still risks high CPU usage on processing nodes to spike up resulting in long queue build ups in BRS and eventually resulting in a meltdown.
ECE v11.2.0.0.0 introduced a new overload protection that guarantees if provisioned right will never accept requests that can not be processed with in a given SLA irrespective of how high the incoming throughput is. This is a key feature of the product when it comes to deploying a fault tolerant system at different levels. And the pattern used comes right out of the Joke in the beginning of this blog.

Pattern (V) - Shed Load:
BRS monitors the internal queue of the Thread Pool Executor and at any given time if the pending count is more than a configured value it rejects the submitted request. The rejected request has to be resubmitted by the network mediation based on certain policies typically suggested by ECE. As the system being in an overload situation is dynamic the pending queue size check is done for each submitted request. It is quite possible that one request has been rejected but immediately after requests are accepted for processing. Shedding Load is a legitimate fault mitigation policy specially more so in latency sensitive applications.
Question is how do we configure the acceptable pending count? It depends on the infrastructure's latency sensitivities. Wait adds to Latencies. If it is required to have zero latency impact on acceptable requests then the acceptable pending count should be set to zero. As the requests are submitted it would go through the provisioning flow immediately after. As the acceptable pending count is increased the average latencies would increase as well and between none and equal to the throughput will dictate what the latency per request would end up in.
Following are two snapshots taken from one of a smaller test setups. One snapshot plots throughput and the other latencies. In the following test the infrastructure is configured for around 40K throughput with acceptable latencies of around 50ms. The snapshot is taken from one of the instances of the event simulators configured at 20K with total of two such simulators running on one test host. Look at how the latency increases as the acceptable count is increased in an "overloaded" system gives an insight how this feature should be configured. At 120K which for this test set up was way more than what it was provisioned for, at zero acceptable count had no impact on the processing latencies at all even if the system was receiving way more number of requests than what the infrastructure is provisioned for. Acceptable pending count @20K increases the latency but still prevents a complete system meltdown. So a single tuning parameter allows you to prevent the system from fault to failure.

 














Enjoy!

My JavaONE 2013 session

And the recorded presentation:

मेरे कॉलेज के चार साल

बीते हुए उन सालों की जो याद मुझे अब आती है,
कभी पलकें गीली होती हैं कभी मुस्कां सी छा जाती है।

वो अल्ल्ढ़पन, वो बेफिक्री, वो दीवानों सा ढंग मेरा,
निकले बाहर उस गुलशन से बस याद वही रह जाती है।

वो चाय समोसों की शामें, अनबन सी होती कुछ बातें,
जनम जनम के बैर कहीं इक हंसी में जो घुलजाती है,
बीते हुए उन सालों की जो याद मुझे अब आती है।

वो पागलपन, वो शर्माना, वो धमकी वो सीना ताना,
वो आदत कुछ कुछ गन्दी सी, संकल्पों का खुद मिट जाना,
वो अनजान जो लगते थे अपने, बेहतर थे अपने वालों से,
राज़ जो थोड़े से अपने बतलाई उन्ही से जाती है,
बीते हुए उन सालों की जो याद मुझे अब आती है।

पलकें थोड़ी सी गीली है मुस्कान अच्छी सी आती है,
निकले हैं बाहर गुलशन से बस याद वही रह जाती है।

Sister certifying Electronic Voting Machines

It takes a lot more to conduct successful elections than turning up and casting your vote. I am happy to see my Sister contributing.

चलो एक और सुनो

कहाँ पूछूं खुद से परेशां क्यूँ हैं हम?
धुंधला लगे है आईना जो देखूं खुद को।

Colocating data in Coherence and NoSQL

Key/Value pair based distributed systems that partition data based on some hashing of keys pose a challenge of unnecessarily incurring additional network cost in storing and reading of strongly associated data with additional challenges of atomicity of operations across this data set. Fortunately both of Oracle's best of breed big distributed data source Coherence and NoSQL support co-locating associated entries and provide a rich set of APIs to create atomic operations. In both the systems association is dictated by the Map' keys, only differently.

Association in Coherence

Two entries will be co-located in the same partition if

1. The key of the associated entry implements a Coherence KeyAssociation interface (Or define a KeyAssociator and configure it in the cache config).
2. If these two entries are in two different NamedCaches then they must share the same cache service.
3. When the cache service calculates the partition Id for entries, it uses the associated key if the cache key implements the KeyAssociation interface. 
4. Coherence provides a set of APIs typically called TransactionLite framework to access associated entries from the BackingMap as a single unit of transaction. You can refer to the blog for an example.
5. Because Coherence keys can be complex objects it is much more flexible in terms of storing complex data structure.

Association in NoSQL

NoSQL has much simpler design but it is limited in a sense that it requires keys to be Strings.

1. NoSQL keys are structured as multiple major and minor keys.
2. If we are storing People in NoSQL and want everyone with the same last name to be in the same partition then, keys can be structured as:
1. /familylastname/-/myfirstname
2. /familylastname/-/wifesfirstname
3. /familylastname/-/kidsfirstname
3. Though these are three Map entries as they all share the same major key /familylastname will all end up in the same partition.
4. NoSQL then provides APIs like multiDelete() and multiGet() to delete or retrieve multiple keys as one atomic operation.

Working with Coherence and NoSQL together  

Oracle's Elastic Charging Engine is one such product that uses Coherence and NoSQL together with Coherence as a rating engine and NoSQL as its fast persistence store for the processed RatedEvents. That said aside, such integration is fast becoming very common and following pieces of codes demonstrate some of its use cases.

Code Samples

Lets start with an interface:

public interface NoSQLKeyAssociation extends KeyAssociation {
    String getRootKey();
}

public abstract class PartitionAwareNoSQLCacheStore extends AbstractCacheStore
 {
    private KVStore kvStore;

    public PartitionAwareNoSQLCacheStore(String kvServers) {
        KVStoreConfig kvStoreConfig = new KVStoreConfig("addressStore", kvServers);
        kvStoreConfig.setConsistency(Consistency.NONE_REQUIRED);
        kvStore = KVStoreFactory.getStore(kvStoreConfig);
    }

    @Override
    public void store(Object key, Object value) {
        storeAll(Collections.singletonMap(key, value));
    }

    @Override
    public void storeAll(Map map) {
        Map<NoSQLKeyAssociation, Object> transformedMap = transformer(map);

        Set<Map.Entry<NoSQLKeyAssociation,Object>> set;
        set = transformedMap.entrySet();
        ArrayList majorComponents;
        ArrayList minorComponents;

        for (Map.Entry entry : set) {
            majorComponents = new ArrayList();
            minorComponents = new ArrayList();
            NoSQLKeyAssociation associationKey = entry.getKey();
            majorComponents.add((String)associationKey.getAssociatedKey());
            minorComponents.add(associationKey.getRootKey());

            Key key = Key.createKey(majorComponents, minorComponents);
            byte [] bytes = ExternalizableHelper.toByteArray(entry.getValue())
            kvStore.put(key, Value.createValue(bytes));
        }

    }

    public abstract Map transformer(Map entries);
    
}

The method transformer( ) allows for the Cache Store's concrete class to either transform the Map entries in some form or return the Map as is if the keys are already of type NoSQLKeyAssociation. If not this method gives an opportunity to convert the key to a type of NoSQLKeyAssociation, for one.

In example where we need to store a Customer and all his Addresses a key similar to the following can be used, both to store these values in Coherence' NamedCache (CUSTOMER and ADDRESS caches) and also to NoSQL and in each system co-located together.

public class CustomerAddressAssociationKey implements NoSQLKeyAssociation {
    String customerId;
    String addressType;

    public CustomerAddressAssociationKey(String customerId, String addressType) {
        this.customerId = customerId;
        this.addressType = addressType;
    }

    @Override
    public Object getAssociatedKey() {
        return customerId;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        CustomerAddressAssociationKey that = (CustomerAddressAssociationKey) o;

        if (!addressType.equals(that.addressType)) return false;
        if (!customerId.equals(that.customerId)) return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result = customerId.hashCode();
        result = 31 * result + addressType.hashCode();
        return result;
    }

    @Override
    public String getRootKey() {
        return addressType;
    }
}

If two addresses HomeAddress (addressType = HOME) and WorkAddress (addressType = WORK) are created for a Customer with customerId = 1 then they will be stored as following:

In Coherence, Customer and his two addresses will be co-located as:
CUSTOMER Cache:

• <1, Customer>

ADDRESS Cache:

• <<customeraddressassociationkey<HOME>, HomeAddress>
• <<customeraddressassociationkey<WORK>, WorkAddress>

In NoSQL Customer's both address co-located as:
/1/-/HOME, byte[] for HomeAddress
/1/-/WORK, byte[] for WorkAddress

Enjoy!

Speech on assault on rave parties in India

Thanks for giving me time to share my thoughts. India is a free nation. It is a free democracy and it is a free society. As a citizen of India either by birth or by anyway other, we get certain constitutional rights. Rights that are derived from core human and our centuries old traditional values. And one of those rights is to choose how we live.



That said, unlike many western societies our social security is still families. Families are at the core of our fabric. Our democracy derives its energy from it. Mahatma's dreams of Panchayati Raj is a clear indication that in India democracy flows bottom up. If we need support - financial, psychological or emotional we revert to our families, our parents our siblings, our Pind, our community. And these societies over the years form what I call soft laws including certain behavioral guidelines. These guidelines are not written by a few on a piece of paper but are passed down to them by generations over centuries. Its the cost of social security they provide. Then there sits a big Gorilla called Indian constitution that supersedes all such laws and allows anyone who feels threatened by such social laws to seek protection under it. Such social laws can not be used as a justification to perpetrate violence or assault on people and law must take strong action against whoever does that.
I abhor such people and act, but I do support the very intent of such protest against the so called Rave parties. Some of the news that I have read about are nothing but disgusting. For pete's sake some of them were found with packets of condoms lying around with free flowing alcohol and nakedness. Would as a Dad, as a Parent as a family I approve my kid to go to such parties? Of course not and so am sure a bulk of this audience would not too. As a father I would do everything to register my protest against such events if I see that happen that I feel threatens my family and my values. I would not take a gun and shoot them up or break in or slap them around but I will make sure my and many such voices are strongly heard. Unlike many misconceptions Parents even in many western countries feel the same anguish and anger when they hear their kid is drinking, taking drugs or involved in sexual partnerships. Its a parental instinct and I strongly support my rights to do so.
When we were young 18, 19 or 21 we believed behavioral guidelines can not be thrust upon us and this is something that a Generation dictates. Let me say to you in '70s we saw a hippie culture. Where is that now? Are they relevant anymore? A lifestyle evolved in one generation has been disapproved by subsequent ones. So over a few generations the guidelines put forth by societies broadly remains by and large the same. And a sustainable change requires it being slow and not contextual. At the end it is us who make changes or safeguard our beliefs.

Thank you