Dynamic Database Access Patterns

Author: James Long
Date: September 1, 1998
Email: long@fpi.com
Phone: 970-490-1408
Fax: 970-490-6021

Introduction:

The following patterns reflect some of the lessons I’ve learned as chief architect of Object Products' Organic Architecture (OA). The OA is a two level dynamic framework used to develop healthcare applications. The first level is a C++ whitebox framework that is used to develop dynamically accessible components and services. The second level is a black box framework for composing applications from domain models. Domain experts use the application framework to create models of domain objects, use cases, workflows, etc. These models are stored in an OODB, and are themselves the final application. The domain model also acts as the database schema for storing application instance data.

The most notable omission from these patterns is their lack of a Known Uses section, which is one of the reasons I didn’t submit them to PLoP. I’m hoping that through this workshop I can either complete these patterns, or invalidate them. My company has staked its life on evolving applications and the Organic Architecture. We could have used a pattern language for application evolution 6 years ago, but better late than never.

 

Dynamic Database Access

Context

You’re building an enterprise wide corporate information system based on a centralized database and distributed clients. The system will contain vast amounts of data, and is required to remain operational for a minimum of 10 years. Problem The business model as captured by the database schema must reflect the ever-changing nature of your business. But, technology is also changing, and we want to take advantage of advances as they become available. The rate of change of the business model is unrelated to the rate of change of technology. Forces Solution Decouple the client applications that provide access to the database from the business model that is stored in the database. Do this by dynamically accessing the database schema each time a client application is run. Design the client to operate based on the model it finds in the database. Do not hard code class, attribute, relationship or operation names directly in the client application. Rather, read the schema to see what is available, and present the application’s end user with access and control of the business model based on the structure and content of the model itself. Resulting Context You are now free to change the business model without having to re-write or re-compile code. Business experts can take control of the business model. New technical functionality can be added incrementally as it is developed without requiring changes to the underlying business model. The technology used to implement client applications is free to change independently of the business model. Rationale Stewart Brand, in his book How Buildings Learn, presents the idea that buildings are composed of various systems, and that these systems change at varying rates. Brand proposes that buildings which survive the test of time and continue to be beautiful and functional are composed in such a way that the various systems are free to change at their own rate. Much of this is obviously apparent. A brick house, where Structure and Skin systems are the same, is less apt to receive a facelift than would a house built of wood.

Software systems, like buildings, are composed of a series of systems, each with its own rate of change. Languages, operating systems, support libraries, tool kits, editors, legacy code, and local code under development are but a few examples of the various systems within a piece of software. For software systems to evolve gracefully we must learn to recognize the spheres of change within the system, and make a conscious effort to keep them decoupled.

 Related Patterns Logical Schema Stored As Data
Schema-Aware Proxies

Logical Schema Stored As Data

Context

You are implementing a large client/server database system in which the client application dynamically reflects the business model as it is represented in the database schema. Problem You want non-programming domain experts to build and maintain the business model in the database. The Database Management System (DBMS) you are using does not provide a very rich set of modeling constructs. You can define classes, create attributes and constrain those attributes to a few simple types. You can associate objects through relationships and assign those relationships limited cardinality constraints. But you have no way of expressing the more complex aspects of the model. The DBMS supports operations as callbacks to client-registered methods, so you can’t implement new triggers, constraints or more complex operations without resorting to writing code. You would like to be able to model construction and destruction constraints, to define edit and display characteristics of attributes, and to support complex relationship semantics such as Role or Shared Ownership, but these also require programming. Forces Solution Construct your own schema as instance data in the database. Rather than accessing the Physical Schema provided by the DBMS vendor you will access this Logical Schema that you have created in data. Design the Logical Schema to support your approach to domain modeling. Construct the Logical Schema schema such that the Logical Schema instances form a hierarchical name space with a single access point

Build tools to support the logical schema. There are several options here. You can build tools that automatically create the logical schema from the physical schema, and then let you customize the logical schema. This approach requires that you use DBMS schema tools for creating the physical schema. A better approach is to develop a Model Editor, which provides direct control over the logical schema for your business domain modelers. Have changes to the logical schema be automatically reflected in the physical schema.

 Resulting Context You now have a system that provides a rich modeling environment. The logical schema can be designed to provide the domain modelers with any and all modeling constructs needed to support the modeling process. The client applications are not limited to the information provided by the physical schema. The domain model contained in the logical schema is platform independent. Your logical schema can continue to evolve to embrace richer and more complete modeling constructs. Rationale In some ODBMS’s, such as MATISSE, you can actually modify the meta-schema. This allows you to add direct support for more complex modeling constructs. But, this is highly non-portable. In embracing change we look to minimize contact with external systems. Implementing a logical schema as data is very portable, and is preferred to meta-schema modification. Related Patterns

Concurrency Support Through Optimism

Context

You are implementing a large client/server database system in which the client application dynamically reflects the business model it finds in the database. The system provides enterprise-wide access to essential corporate information. This information is primarily accretive. The system does not require high transaction volumes, such as a reservation or online banking system. Problem In an enterprise-wide information system, such as either a business or clinical decision support system, you must provide uninterrupted access to system data. If you lock data for update you can possibly deny other users access to essential information. If the billing department is updating a patient’s insurance claims an ER physician could possibly be blocked from that patient’s medical information until the billing transaction was completed. There are ways to organize the schema to minimize this type of interaction, but with pessimistic locking it can not be completely avoided. Forces Solution Implement an optimistic transaction scheme for database updates. Always read data with no locks. If the DBMS  forces locking then simply read the data and then immediately release the locks. Instead of directly modifying this dirty data on the client you will maintain a change record. A change record can be implemented by storing both the new state and the old state, or as a parameterized operation against a specific object property.

With this style of optimism you perform a transaction by specifying a group of atomic change records to be applied. Only for the brief duration of the transaction are the various affected objects locked. Once an object is locked the property to be modified is compared to the dirty data in order to detect collisions. Collision detection can be ignored, can cause a transaction to fail, or can succeed by performing data merger rules. Transactions can be performed on the client, but are better handled on a server.

 Resulting Context Except during the brief moment in which a transaction is attempted all persistent data is always accessible to the client. The final system will be more complex in order to deal with unresolvable transaction collisions, but the important information maintained by the system is always available for access. Related Patterns

Transaction Processing On Server

Context

You are implementing a large client/server database system in which the client application dynamically reflects the business model as found in the database. You’ve enhanced your ability to properly model the business domain by extending the database meta-schema through the use of a Logical Schema. And you’ve improved data availability by adopting an optimistic transaction policy. Problem Client transaction commits require several complex steps. Locks must be established on data to be modified. Constraints must be checked and collisions detected. Permissions must be verified. And finally, triggers and operations may need to be executed both before and after transaction commit. These triggers and operations may involve some larger set of objects than were in the original transaction. These objects must be fetched, locked, constrained, etc. The time in which some set of data is unavailable to other system users must be minimized. Forces Solution Implement a Transaction Server application which can be run on the same machine as the database. This server will commit transactions on behalf of the clients. Clients submit change records to the server to request specific modifications to specific objects. The server locks objects, checks constraints, and performs operations and triggers.

Instead of connecting clients directly to the database through bindings provided by the DBMS vendor you will connect via the Transaction Server. The Transaction Server will connect to the database through the vendor provided bindings, and then will pass-through much of the database access API directly to the client applications. The server acts as a wrapper on the database, isolating the specific database binding from the client applications.

 Resulting Context You now have system in which the client applications are isolated from the DBMS. Requests to read and write objects are handled by the Transaction Server. Performing triggers and operations remotely has reduced network traffic. Slower clients perform better by offloading some of their work. Your system must now handle its own communication protocol, but this prevents you from being trapped by the limitations of any particular DBMS vendor. Rationale A Transaction Server does more than just speed up transactions. It helps to isolate vendor specific database API and makes your system more portable. It can be used to perform remote query processing. And, as a single point of access to the database it provides an ideal gating point for security and logging services.

Most important, it can help in maintaining the integrity of your data. The database contains the reality of the application domain. A transaction server maintains the integrity of that reality independent of the client applications. The rules for maintaining integrity are established by the Logical Schema, and are enforced by the Transaction Server. If the code for maintaining integrity is a part of the client application, then there’s always the chance that an outdated client can corrupt the database.

 Related Patterns

Schema-Aware Proxies

Context

You are implementing a large client/server database system in which the client application dynamically reflects the business model it finds in the database. You’ve enhanced your ability to properly model the business domain by extending the database meta-schema, either directly or through the use of a Logical Schema. Problem Your application code is primarily concerned with the display and manipulation of business object instances that it retrieves from and stores to the database. But, everyplace where your code references these dynamic persistent business objects you must add logic to consult the schema to deduce rules about the business model. As you access each property of a persistent object you must dynamically determine its type and structure. If the property is a relationship, you must determine this same information on every successor objects. Forces Solution Create a Proxy class on the client that has the ability to dynamically reflect any specific persistent object in the database. Provide an API on the Proxy to dynamically supply information about the object, its attributes, relationships and operations. The Proxy is responsible for maintaining the integrity of attributes and relationships. It enforces typing constraints, relationship cardinality, and marshalls arguments for object operations.

Wrap all database operations within the Proxy class.  Implement a lazy access policy for Proxy properties. Only fetch attributes and relationships that are actually accessed. Let Proxies manage the change records for modifications that are made to them. Provide a standard API on Proxy for object update, rollback, creation and deletion.

 Resulting Context Your client system now deals with all persistent objects as instances of a single client class. Database operations are hidden behind the Proxy wrapper API. The remainder of the client code deals with concrete instances and a known API. Related Patterns