Showing posts with label toplink. Show all posts
Showing posts with label toplink. Show all posts

Friday, August 21, 2009

Pushing changed data to Coherence

Problem Statement - How to capture data changed in an external data source and invalidate the Cache?
I attended a customer call where we discussed this same problem and I think I could be more explanatory in a blog than on a call. Before we consider any solution one thing that must be known is that Coherence is a data source. Not a relational data source, not like a directory interface but it is a data source - of a different kind. So the problem now becomes more generic - how do we synchronize two discrete data sources? The solution will revolve around similar solutions as if you synchronize an LDAP with a relational database. Have you done that?
Following are the four architectural line of attacks that can be taken. Depending on how stale-data can an application deal with, a few or none of these solutions may or may not work. So make your own judicious decision.

  • Invalidate the data with in.
  • Source of data change propagates the change.
  • Let an external mechanism do it.
  • Whoever changed the database should also change the Cache.
Invalidating the Cache Entries With In
  • [Time to live] Coherence Cache entries have a time to live attribute. TTL defines how long an Entry should live in the cache. Based on data access frequency and expected cache hits an appropriate expiration time can be set. If over an hour period 90% of cache hits are expected to take place in first 15 mins of entry being put and are less frequently later on then a ttl of 15mins gives you a good invalidation parameter. A typical use case is load the data, process it and invalidate it as soon as it is done. This can be helpful if the same entry is being accessed multiple times during the processing.
  • [Refresh Ahead Factor] This is based on a wonderful analogy of serving fries in McDonald's. If we ask for fries that are about to be over then we get the last pieces but it triggers an asynchronous load of fries from an oven. An appropriate refresh-ahead-factor can be set in the cache configuration that will trigger an asynchronous Cache Load operation (using a CacheLoader component - must have a read-through pattern) if data are accessed after the second half of this factor of expiration. So if data changed in an external data source it will be refreshed in the Cache and next access gets the latest. The refresh ahead assumes a continuous stream of data access while lesser database (or any external data sources) updates behind the scene.
Propagation by data source where changed occurred
  • If you are already on Oracle 11g a DCN mechanism can be used where applications can directly register with the database for change-events. After receiving the change-events the application can then propagate the changes to Coherence.
  • Responsibility of change event propagation lies with the owner where change occurred.
An External Agent
  • Oracle Data Integrator has a Changed Data Capture feature that can be used to push changes from a data base to Coherence.
  • Or a simple DB Adapter. An external application polls for data changes at a regular frequency, captures the changed data set and propagates the changes to Coherence cache.
  • Oracle's BPEL PM has an inbuilt DBAdapter that can propagate the change to Coherence using an embedded Java Activity.
  • Simple and could be light weight. Polling could be heavy and needs to be considered when provisioning database loads.
After all, you did it
  • Coherence supports three heterogenous platform - Java, C++ and .NET. It is very likely that application that changed the data in an external data source is running on one of these three platforms. Application that changed data in the database can be extended to propagate the same changes to Coherence as well. Transactional successes should be considered to avoid data being propagated in Cache but didn't succeed in database.
  • ORM like Toplink has SessionEventListener that can retrieve changed data set upon a database commit and this SessionEventListener can propagate the changes to Coherence cache(s).

Wednesday, February 25, 2009

Toplink MultiProjects with Cache Coordination

I would love to call it a Hub and Spoke architecture. Its pretty common to have enterprise applications grow in a way that numerous applications spring up that work and share only a subset of domain objects of a "Big Daddy" application. For example a central application that does Order management and creates Shipments. Other application only works on Shipment details and other may be doing an analytic on the Products sold. With Toplink it is fairly simple to do so and the trick is to partition the application in multiple independent units and map it with their respective data model accordingly. Toplink session allows to have multiple projects. Click on the Session and General tab, select Multiple Projects checkbox and provide the name of the primary toplink project (In our example that would be Order).

Select Multiple Projects tab and click the Add button. Select other components of the application, in our example it would be Shipment. Repeat the process for each component of the application.

Then create another Toplink session (Either in the same sessions.xml or multiple sessions_xxx.xml files) for each individual components. The following image shows the Shipment project:

And repeat for each individual applications. Deploy the Big application and then other "spoke" applications. One more important thing.. Make sure they all share the same Cache Coordination configuration. If using RMI same multicast address and port and their each individual Registry Naming Services.

And thats pretty much it you need. Toplink, Cache Coordination and judiciously partitioning the application to individual components can achieve a true hub-n-spoke enterprise application. For a sample project download it from: Sample Toplink Project

Saturday, February 21, 2009

Using Cache Coordination with Toplink

Most of my consulting engagements revolve around two products - Oracle Coherence and Oracle Toplink. While both have Caching component and caching in a distributed environment they are complementary and are built for two entire different purposes. Coherence is built for massive scalability for humongous amount of in memory data while Toplink is for persistence and can pretty much map any domain model with any data model. So the moment you find yourself in a situation where multiple applications need to share the same set of objects or need to be notified of changes in one application to another, a quick call with Oracle Professional services might help. One feature of Toplink that gets a little not so talked about is its ability to coordinate caches across multiple Toplink sessions. If your application is Toplink API driven and requires up to 500/600MB of in memory data some huge performance gains can be achieved using its Cache Coordination. During a recent project I got myself into some challenges while migrating a Toplink9g project and I hope this blog helps for future references.

  1. Cache Synchronization is now called Cache Coordination.
  2. Cache Coordination uses RCM and the old synchronization mechanism has been deprecated.
  3. Do not rely completely on Toplink10g workbench for creating sessions.xml
  4. Migrate to Toplink11g and it's workbench - It will make life much easier.
  5. If migrating 9g to 10g, 10g will work with the 9g's sessions.xml. 10g cache coordination configuration could be a challenge.
  6. Cache coordination is meant for mostly read-only and a low writes.
  7. If requirement is heavy object mutation across multiple instances or heterogeneous applications, consider Coherence.
Following explains how to test Cache coordination on a single machine (Done with 11g):
  • A sample setting for Cache Coordination using RMI
    • Type: RMI
    • Channel: ToplinkCommandChannel
    • Multicast Group Address: 237.14.12.12
    • Multicast Port: 25000
    • Packet Time to Live: 2
    • Announcement Delay: 1000
    • Select Remove Connection on Error
    • Select Synchronous
    • URL for Registry Naming Service: rmi://$HOST:1099
  • Starting rmiregistry
    • rmiregistry.exe -J-Djava.class.path="classes;toplink.jar" -J-Djava.rmi.server.logCalls=true -J-Dsun.rmi.server.logLevel=VERBOSE -J-Dsun.rmi.client.logCalls
  • Following is a simple sample sessions.xml
      <?xml version="1.0" encoding="UTF-8"?>
      <toplink-sessions version="11g (11.1.1.0.0)"
      xmlns:xsd="http://www.w3.org/2001/XMLSchema"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <session xsi:type="server-session">
      <name>Session</name>
      <remote-command>
      <commands>
      <cache-sync>true</cache-sync>
      </commands>
      <transport xsi:type="rmi-transport">
      <send-mode>Synchronous</send-mode>
      <discovery>
      <multicast-group-address>
      237.12.14.12
      </multicast-group-address>
      <multicast-port>25000</multicast-port>
      <announcement-delay>2000</announcement-delay>
      </discovery>
      <rmi-registry-naming-service>
      <url>rmi://$HOST:1099</url>
      </rmi-registry-naming-service>
      </transport>
      </remote-command>
      <event-listener-classes/>
      <profiler>toplink</profiler>
      <logging xsi:type="toplink-log">
      <log-level>all</log-level>
      </logging>
      <primary-project xsi:type="xml">ToplinkRI.xml</primary-project>
      <login xsi:type="database-login">
      <platform-class>
      oracle.toplink.platform.database.oracle.Oracle10Platform
      </platform-class>
      <user-name>username</user-name>
      <password>56F55BD62D7347611874F25952AA55F6</password>
      <sequencing>
      <default-sequence xsi:type="native-sequence">
      <name>Native</name>
      <preallocation-size>10</preallocation-size>
      </default-sequence>
      </sequencing>
      <driver-class>oracle.jdbc.driver.OracleDriver</driver-class>
      <connection-url>
      jdbc:oracle:thin:@localhost:1521:XE
      </connection-url>
      <cache-all-statements>true</cache-all-statements>
      <struct-converters/>
      </login>
      <connection-pools>
      <read-connection-pool>
      <name>ReadConnectionPool</name>
      </read-connection-pool>
      <write-connection-pool>
      <name>default</name>
      </write-connection-pool>
      </connection-pools>
      <connection-policy/>
      </session>
      </toplink-sessions>
And thats all you need to run multiple instances of an application on a single box with straight forward RMI driven cache coordination. Change made in one is visible in another JVM and access to that Object will be from a cache instead from an expensive database.

Saturday, November 22, 2008

Toplink10g to Eclipselink Migration

No brainer and Toplink team has done a great job to make it very simple and a matter of running a script. Even though the steps are clearly documented on http://wiki.eclipse.org/EclipseLink/UserManual but you may run into some gotchas. Some of the very advanced Toplink mappings use amendment methods and post insert, build events etc. These information are stored in Toplink's classes and descriptors XML files. The migration scripts of Eclipselink does not replace these classes even if the mwp file is successfully opened in the workbench. The solution is very simple.. Replace all classes starting with package name oracle.toplink to org.eclipse.persistence under the classes/ and descriptors/ directory of the Toplink workbench. And following script can be handy as well:

for i in `ls *`
do
sed "s/$1/$2/g" $i > t;
mv t $i;
done

Saturday, June 02, 2007

To find what data has changed in the database?

The problem statement is to find out what data has changed in the database if a transaction is committed. I am helping one of my Customers to build a similar application. Before I jump to the solution let me babble a bit.
We do work on a core application and then new requirements pop up time and again which do not disturb the core process but threatens to affect it. Such requirements are called crosscutting concerns. In one of my old blogs I have cursed such requirements. Things like Logging, Auditing, Profiling etc which does not restrict itself to a single component but affects the entire application. The solution to these problems is the concept of Interceptors. The code need not be changed but new Interceptors are written which can be added/hooked to the "project" which can provide the wanted solution. The concept of Interceptors is not new. Like HTTP plugins (like NSAPI and Apache plugins) Or ServletFilters which are added to intercept the HTTP requests and add/change the default behavior of the web server. Or even more powerful like EJB3 Interceptors or AOP.

So coming back to the original problem: In terms of databases there are two broad categories:
1. To implement it as a database trigger
2. To let the O-R mapper do it for you

Even though they both work, Option(2) is more elegant. It also provides the portability to the application with all the other goodies which O-R mapper gives. In Oracle's Toplink, it can be implemented as a SessionEventListener. The oracle.toplink.sessions.SessionEventListener is an interface which provides all the predefined events which take place in the lifecycle of a Session. If you do not want to implement or have empty methods, your class can extend SessionEventAdapter which does the same thing. The SessionEventListener is then hooked to the Toplink project with its' Toplink session. Here is how you do it:


# Client.java

public class Client {
...

public Session getSession () {
TestProject tP = new TestProject ();
DatabaseSession session = tP.createDatabaseSession ();
MySessionEventListener mL = new MySessionEventListener ();
session.getEventManager ().addListener (mL);
return session;
}
}

# MySessionEventListener.java

public class MySessionEventListener implements SessionEventListener {
...
public void postCommitUnitOfWork (SessionEvent event) {
Session session = event.getSession ();
UnitOfWork uow = (UnitOfWork) session;
UnitOfWorkChangeSet uCS = uow.getCurrentChanges ();
if (uCS != null) {
Enumeration enums = uCS.getAllChangeSets ().elements ();
// -- Traverse through the elements of what has changed.
}
}
}

Thats all you need... Run the application and Toplink will make sure that all the attached listeners keep an eye on all the events and in this case all what has changed.

Happy Toplinking...

Wednesday, May 23, 2007

Simple little things of Programming - How to Toplink?

Here is what I compiled a short high level overview on Toplink:

Introduction
Object persistence impedence mismatch is one of the most struggled areas of Software development. Using Object Oriented philosophy has become a norm to build scalable and complex systems. Pretty much all the current day applications use databases to store some type of data. The data are usually stored in relational model but the Objects are more hierarichal in structure. Due to the difference of technologies and the perception of the models it becomes harder to map the application with the data schema. The team ownership of the database schema is also usually different from the teams who design and build the application. That makes even harder to control how the internal components of an application are constructed. Using the jdbc drivers directly to retrieve and persist data makes the application harder to scale and manage. It also makes it more error prone and slower. To solve the problem O-R mappers come to rescue.

What is a domain model?
The domain model is a conceptual model of a system that describes the various entities involved in a system and their relationships. The domain model also documents the key concepts and the vacabulary of the system. The model displays the relationships among all major entities within the system and usually identifies their important methods and attributes. This means that the domain model provides a structural view of the system. In most cases the simple Java classes are used to create a domain model. These Java objects contain the attributes that together represent a state of the object. This is the state of the object that is persisted in the database.

What is a data model?
Data model defines how data are structured and used in the database. There are three model elements that a data model defines:


  • Structural:
  • A collection of data structures that are used to create databases representing the entities or objects modeled by the database.
  • Integrity:
  • A collection of rules governing the constraints placed on these data structures to ensure structural integrity.
  • Manipulation:
  • A collection of operators, which are applied to the data structures, to update and query the data contained in the database.

What is a Toplink?
Toplink is at core an O-R mapper, which stands for Object-Relational mapper. Oracle Toplink builds high-performance applications that store persistent object-oriented data in a relational database. It successfully transforms object-oriented data into either relational data or Extensible Markup Language (XML) elements. Using Toplink we can integrate peristence and object-transformation into our application, which staying focussed on our primary domain problem by taking advantage of an efficient, flexible and field proven solution.
Toplink is suitable for use with a wide range of Java 2 Enterprise Edition (J2EE) and Java application architectures. Use Toplink to design, implement, deploy and optimize an advanced, object-persistence and object-transformation layer that supports a variety of data sources and formats including the following:

  • Relational –
  • For transactional persistence of Java objects to a relational database accessed using Java Database Connectivity (JDBC) drivers.
  • Object-Relational –
  • For transactional persistence of Java objects to special purpose structured data source representations optimized for storage in object-relational databases such as Oracle Database.
  • Enterprise Information System (EIS) –
  • For transactional persistence of Java objects to a nonrelational data source accesses using a J2EE Connector Architecture (J2C) adapter, and any supported EIS recore type including indexed, mapped or XML.
  • XML –
  • For non-transactional, non-persistent (in-memory) conversion between Java Objects and XML Schema Document (XSD)-based XML documents using Java Architecture for XML Binding (JAXB).

Toplink includes support for Container-managed persistence (CMP) containers from a variety of vendors including Oracle OC4J, IBM WebSphere and BEA WebLogic Server and support for base classes that simplify the Bean-managed persistence (BMP). The Toplink runtime lets the application exploit the mapping metadata with a simple session façade that provides in-depth support for data-access, queries, transactions (both with or without external transaction controller) and caching.
The following diagram can illustrate the Toplink Runtime :

Toplink Key Features
Toplink provides an extensive and thorough set of features. Java developers can use these features to rapidly build high-performance enterprise applications that are both scalable and maintainable.
Some of the primary features of Toplink are the following:

  • Nonintrusive, flexible, metadata-based architecture.

  • Comprehensive visual Toplink Workbench

  • Advanced mapping support and flexibility (relational, object-relational, EIS and XML)

  • Object caching support

  • Query flexibility

  • Just-in-time reading

  • Caching

  • Object-level transaction support and integration

  • Locking

  • Multiple performance tuning options

  • Architectural flexibility



Getting Started
So, how to start with Toplink? What exactly does it do? How does it create the mappings? Where does the mapping reside? These are the right kind of questions to ask if you are starting or thinking of using Toplink. The goal of any O-R mapping is to create an object graph in memory that represents the Object hierarchy of the domain model and has all the definitions of its relationship to the data model. Oracle Toplink imports the database tables, Java classes and their relationships as a set of XML files. Once these files have been created, the object graph is built either by importing the project as an XML document or a Java class. The XML or the Toplink project Java class contains all the mappings to create the Object graph required to persist or query against the database and is one core artifact required to run a Toplink enabled application. The Toplink runtime uses the XML mapping or the Project Java class to create the Object graph in memory which is accessed by the Toplink session to provide the data access. Toplink makes the application agnostic to the details of the database as it is now managed in one single layer. Toplink also provides a set of Java interfaces (APIs) to access or manipulate its Objectgraph.

Understanding Toplink Metadata
The Toplink metadata is the bridge between the development of an application and its deployed run-time environment. Capture the metadata using Toplink Workbench (or JDeveloper) and pass the metadata to the run-time environment using deployment XML files (such as sessions.xml). We could also code these files using Java and the Toplink API but this approach is more labor-intensive. Toplink Workbench also provides an option to export the mapping as a Java class.
The metadata encapsulated in deployment XML files, lets you pass configuration information into the run-time environment. The run-time environment uses the information in conjunction with the persistent entities and the code written with the Toplink API to complete the application. Here is an example of a simple interface. The illustration shows how Toplink uses metadata (In XML format) to manage the data model, classes and the database table information.


Advantages of Toplink Metadata Architecture
The Toplink metadata architecture provides many important benefits, including the following:

  • Stores mapping information in XML descriptors and not in the domain model objects.

  • By using the metadata, Toplink does not intrude in the object model or the database schema.

  • Allows us to design the object model as needed without forcing any specific design.

  • Allows DBAs to design the database as needed without forcing any specific design.

  • Does not rely on code-generation (which can cause serious design, implementation and maintenance issue).

  • Is unintrusive: Adapts to the object model and database schema rather than requiring you to design their object model or database schema to suite Toplink.



Creating Project Metadata
A Toplink project contains the mapping metadata that the Toplink runtime uses to map objects to a data source. The project is the primary object used by the Toplink runtime. This section describes the principal contents of project metadata including Descriptors and mappings and Data source login information.
1. Descriptors and Mappings: The Toplink maps persistent entities to the database in the application using the descriptors and mappings we build with Toplink Workbench. Toplink workbench supports several approaches to project development including the following:

  • Importing classes and tables for mapping

  • Importing classes and generating tables and mappings

  • Importing tables and generating classes and mappings

  • Creating both classes and table definitions


Toplink Workbench supports all these options. The most common solution is to develop the persistent entities using a development tool such as an IDE like Oracle JDeveloper or a modeling tool and to develop a relational model through appropriate relational design tools. You then use Toplink Workbench to construct mappings that relate these two models. Although Toplink workbench does offer the ability to generate persistent entities or the relational model components for an application, these utilities are intended only to assist in rapid initial development strategies not to complete a round trip application development.
2. Amending Descriptors: An amendment method lets you implement a Toplink feature that is not currently supported by Toplink workbench. We would need to write a Java method to amend the descriptor after it is loaded and specify the method in the Toplink workbench for inclusion in the project metadata.

Data Sourch Login Information
For non-CMP projects we need to configure a session login in the session metadata that specifies the information required to access the data source. For CMP projects, the project contains a deployment login that specifies the information required to access the data source.
There are three ways to create Toplink mapping artificate: Toplink Workbench, JDeveloper or Using Toplink Java APIs to create the mapping directly. The two more frequently used IDEs are Toplink Workbench and the Integrated Toplink component for JDeveloper:
Toplink Workbench
Toplink Workbench is a standalone development environment that allows users to import Java classes or domain objects created externally in form of a jar or compiled classes in a directory. The workbench also allows creating and testing a Database connection using JDBC drivers. The workbench provides an option to use either of the development, testing or production database credentials for logging in to the database. The users can then import a set of database tables for which the mapping is required. Any classes that are in the workbench classpath can be imported into the workbench project and mapped to the database tables imported earlier. Once imported, the Workbench can recreate the database tables in a defined schema on any other database or schema. This provides a flexibility of code and mapping mobility. Once the mapping is completed, the project can be exported to an XML document or a Jav aclass whose package and path can be defined in the project properties.
Oracle JDeveloper
Oracle JDeveloper is an Integrated Development Environment that provides an ability to create multiple types of projects. The IDE also has a Toplink component that can be used to map the Java classes in the jDev project. The flexibility that the JDeveloper provides is that the toplink mapping can be done on the Java source itself anddoes not require the classes to be compiled. It provides one development interface for a complete application development.

Supporting Information

  • http://www.oracle.com/technology/products/ias/toplink/mwdemos/index.html

  • http://www.oracle.com/technology/products/ias/toplink/index.html

  • http://www.oracle.com/technology/products/ias/toplink/quicktour/viewlets/welcome_viewlet_swf.html