Dynamic Database Access
Patterns
Author: James Long
Date: September 1, 1998
Email: long@fpi.com
Phone: 970-490-1408
Fax: 970-490-6021
Introduction:
The following patterns reflect some of the lessons I’ve learned as chief
architect of Object Products' Organic Architecture (OA). The OA is a two
level dynamic framework used to develop healthcare applications. The first
level is a C++ whitebox framework that is used to develop dynamically accessible
components and services. The second level is a black box framework for
composing applications from domain models. Domain experts use the application
framework to create models of domain objects, use cases, workflows, etc.
These models are stored in an OODB, and are themselves the final application.
The domain model also acts as the database schema for storing application
instance data.
The most notable omission from these patterns is their lack of a Known
Uses section, which is one of the reasons I didn’t submit them to PLoP.
I’m hoping that through this workshop I can either complete these patterns,
or invalidate them. My company has staked its life on evolving applications
and the Organic Architecture. We could have used a pattern language for
application evolution 6 years ago, but better late than never.
Dynamic Database Access
Context
You’re building an enterprise wide corporate information system based
on a centralized database and distributed clients. The system will contain
vast amounts of data, and is required to remain operational for a minimum
of 10 years.
Problem
The business model as captured by the database schema must reflect
the ever-changing nature of your business. But, technology is also changing,
and we want to take advantage of advances as they become available. The
rate of change of the business model is unrelated to the rate of change
of technology.
Forces
-
Microsoft completely upgrades its development systems every 18 months,
and provides forward compatibility and support for only a single upgrade
level. This means that within 3 years any system built using this technology
will be obsolete and unsupported.
-
Most commercial software customers require systems to be implemented using
Microsoft technology.
-
Your developers can upgrade your system every 6 months. To build a new
system based on new technology requires a minimum of 18 months.
-
Your business model changes monthly. Your developers don’t really understand
your business model, and they are not the best ones to change it.
-
The typical tenure of a CIO is 18 months to 3 years.
Solution
Decouple the client applications that provide access to the database
from the business model that is stored in the database. Do this by dynamically
accessing the database schema each time a client application is run. Design
the client to operate based on the model it finds in the database. Do not
hard code class, attribute, relationship or operation names directly in
the client application. Rather, read the schema to see what is available,
and present the application’s end user with access and control of the business
model based on the structure and content of the model itself.
Resulting Context
You are now free to change the business model without having to re-write
or re-compile code. Business experts can take control of the business model.
New technical functionality can be added incrementally as it is developed
without requiring changes to the underlying business model. The technology
used to implement client applications is free to change independently of
the business model.
Rationale
Stewart Brand, in his book How Buildings Learn, presents the
idea that buildings are composed of various systems, and that these
systems change at varying rates. Brand proposes that buildings which survive
the test of time and continue to be beautiful and functional are composed
in such a way that the various systems are free to change at their own
rate. Much of this is obviously apparent. A brick house, where Structure
and Skin systems are the same, is less apt to receive a facelift than would
a house built of wood.
Software systems, like buildings, are composed of a series of systems,
each with its own rate of change. Languages, operating systems, support
libraries, tool kits, editors, legacy code, and local code under development
are but a few examples of the various systems within a piece of software.
For software systems to evolve gracefully we must learn to recognize the
spheres of change within the system, and make a conscious effort to keep
them decoupled.
Related Patterns
Logical Schema Stored As Data
Schema-Aware Proxies
Logical Schema Stored As Data
Context
You are implementing a large client/server database system in which
the client application dynamically reflects the business model as it is
represented in the database schema.
Problem
You want non-programming domain experts to build and maintain the
business model in the database. The Database Management System (DBMS) you
are using does not provide a very rich set of modeling constructs. You
can define classes, create attributes and constrain those attributes to
a few simple types. You can associate objects through relationships and
assign those relationships limited cardinality constraints. But you have
no way of expressing the more complex aspects of the model. The DBMS supports
operations as callbacks to client-registered methods, so you can’t implement
new triggers, constraints or more complex operations without resorting
to writing code. You would like to be able to model construction and destruction
constraints, to define edit and display characteristics of attributes,
and to support complex relationship semantics such as Role or Shared Ownership,
but these also require programming.
Forces
-
Domain experts manage the Business Model, and they need to be able to fully
specify business constraints without writing code.
-
Your developers are busy with the next release. Having them also making
changes to the old release dilutes their efforts, and causes a configuration
management nightmare as the baseline system changes from beneath the new
release.
-
You are trying to maintain a level of database independence. For example,
you prefer to use an ODBMS, but you have political pressure to support
Oracle’s object model. The Oracle object model doesn’t support the more
esoteric capabilities of OO, such as polymorphism and inheritance.
That's a model you want to stay as decoupled from as possible!
Solution
Construct your own schema as instance data in the database. Rather
than accessing the Physical Schema provided by the DBMS vendor you
will access this Logical Schema that you have created in data. Design
the Logical Schema to support your approach to domain modeling. Construct
the Logical Schema schema such that the Logical Schema instances form a
hierarchical name space with a single access point
Build tools to support the logical schema. There are several options
here. You can build tools that automatically create the logical schema
from the physical schema, and then let you customize the logical schema.
This approach requires that you use DBMS schema tools for creating the
physical schema. A better approach is to develop a Model Editor, which
provides direct control over the logical schema for your business domain
modelers. Have changes to the logical schema be automatically reflected
in the physical schema.
Resulting Context
You now have a system that provides a rich modeling environment. The
logical schema can be designed to provide the domain modelers with any
and all modeling constructs needed to support the modeling process. The
client applications are not limited to the information provided by the
physical schema. The domain model contained in the logical schema is platform
independent. Your logical schema can continue to evolve to embrace richer
and more complete modeling constructs.
Rationale
In some ODBMS’s, such as MATISSE, you can actually modify the meta-schema.
This allows you to add direct support for more complex modeling constructs.
But, this is highly non-portable. In embracing change we look to minimize
contact with external systems. Implementing a logical schema as data is
very portable, and is preferred to meta-schema modification.
Related Patterns
-
Rule Server Maintains Integrity
-
Schema-Aware Proxies
Concurrency Support Through Optimism
Context
You are implementing a large client/server database system in which
the client application dynamically reflects the business model it finds
in the database. The system provides enterprise-wide access to essential
corporate information. This information is primarily accretive. The system
does not require high transaction volumes, such as a reservation or online
banking system.
Problem
In an enterprise-wide information system, such as either a business
or clinical decision support system, you must provide uninterrupted access
to system data. If you lock data for update you can possibly deny other
users access to essential information. If the billing department is updating
a patient’s insurance claims an ER physician could possibly be blocked
from that patient’s medical information until the billing transaction was
completed. There are ways to organize the schema to minimize this type
of interaction, but with pessimistic locking it can not be completely avoided.
Forces
-
A typical system user will perform many more reads than writes. And there
is no obvious indication as to which data is apt to be modified.
-
The inability to access system information in a timely fashion, such as
drug allergies or HIV information, could be the cause for serious legal
actions.
-
Most information is accretive. You most often create new information rather
than modifying old information. In a clinical information system you don’t
over-write an old clinical measurement, such as last weeks blood pressure.
Instead, you write new information to the system, and associate the new
information with other existing information through object relationships.
-
The order in which multiple updates occur to a single piece of data is
irrelevant. If two clinicians are accessing the same patient and one clinician
changes the patient’s last name to Jones and the other changes the patient’s
last name to Smith, it really doesn’t matter which name becomes persistent
last. (As long as we provide atomic transactions, we won’t end up with
Smones or Jith!)
Solution
Implement an optimistic transaction scheme for database updates. Always
read data with no locks. If the DBMS forces locking then simply read
the data and then immediately release the locks. Instead of directly modifying
this dirty data on the client you will maintain a change record.
A change record can be implemented by storing both the new state
and the old state, or as a parameterized operation against a specific object
property.
With this style of optimism you perform a transaction by specifying
a group of atomic change records to be applied. Only for the brief duration
of the transaction are the various affected objects locked. Once an object
is locked the property to be modified is compared to the dirty data in
order to detect collisions. Collision detection can be ignored, can cause
a transaction to fail, or can succeed by performing data merger rules.
Transactions can be performed on the client, but are better handled on
a server.
Resulting Context
Except during the brief moment in which a transaction is attempted
all persistent data is always accessible to the client. The final system
will be more complex in order to deal with unresolvable transaction collisions,
but the important information maintained by the system is always available
for access.
Related Patterns
-
Transaction Processing On Server.
-
Hot Updates Broadcast by Server.
-
Merger Logic Resolves Collisions.
Transaction Processing On Server
Context
You are implementing a large client/server database system in which
the client application dynamically reflects the business model as found
in the database. You’ve enhanced your ability to properly model the business
domain by extending the database meta-schema through the use of a Logical
Schema. And you’ve improved data availability by adopting an optimistic
transaction policy.
Problem
Client transaction commits require several complex steps. Locks must
be established on data to be modified. Constraints must be checked and
collisions detected. Permissions must be verified. And finally, triggers
and operations may need to be executed both before and after transaction
commit. These triggers and operations may involve some larger set of objects
than were in the original transaction. These objects must be fetched, locked,
constrained, etc. The time in which some set of data is unavailable to
other system users must be minimized.
Forces
-
There are thousands of distributed clients and possibly hundreds of active
clients at any one moment.
-
Some client machines are extremely under-powered and slow, while others
may be extremely fast. Transactions must run equally well on all machines.
-
A single transaction may have intermediate states which are invalid. Transaction
triggers may indirectly access objects which the user does not have permissions
to access directly.
-
The DBMS provides a Client Stub that you can link into your client
application to provide remote communication. It easier to just use this
stub they provide.
Solution
Implement a Transaction Server application which can be run on the
same machine as the database. This server will commit transactions on behalf
of the clients. Clients submit change records to the server to request
specific modifications to specific objects. The server locks objects, checks
constraints, and performs operations and triggers.
Instead of connecting clients directly to the database through bindings
provided by the DBMS vendor you will connect via the Transaction Server.
The Transaction Server will connect to the database through the vendor
provided bindings, and then will pass-through much of the database access
API directly to the client applications. The server acts as a wrapper on
the database, isolating the specific database binding from the client applications.
Resulting Context
You now have system in which the client applications are isolated
from the DBMS. Requests to read and write objects are handled by the Transaction
Server. Performing triggers and operations remotely has reduced network
traffic. Slower clients perform better by offloading some of their work.
Your system must now handle its own communication protocol, but this prevents
you from being trapped by the limitations of any particular DBMS vendor.
Rationale
A Transaction Server does more than just speed up transactions. It
helps to isolate vendor specific database API and makes your system more
portable. It can be used to perform remote query processing. And, as a
single point of access to the database it provides an ideal gating point
for security and logging services.
Most important, it can help in maintaining the integrity of your data.
The database contains the reality of the application domain. A transaction
server maintains the integrity of that reality independent of the client
applications. The rules for maintaining integrity are established by the
Logical Schema, and are enforced by the Transaction Server. If the code
for maintaining integrity is a part of the client application, then there’s
always the chance that an outdated client can corrupt the database.
Related Patterns
-
Server Enforces Security.
-
Stateless Database Connections.
-
Cache Remote Objects Locally.
Schema-Aware Proxies
Context
You are implementing a large client/server database system in which
the client application dynamically reflects the business model it finds
in the database. You’ve enhanced your ability to properly model the business
domain by extending the database meta-schema, either directly or through
the use of a Logical Schema.
Problem
Your application code is primarily concerned with the display and
manipulation of business object instances that it retrieves from and stores
to the database. But, everyplace where your code references these dynamic
persistent business objects you must add logic to consult the schema to
deduce rules about the business model. As you access each property of a
persistent object you must dynamically determine its type and structure.
If the property is a relationship, you must determine this same information
on every successor objects.
Forces
-
Dynamic code is hard to write. There is a lot of overhead in building a
system that examines itself in order to figure out what to do.
-
Accessing object relationships with high cardinality can often greatly
degrade system performance.
-
Different databases support different access strategies depending on how
attributes and relationships are physically stored. Exposing this sort
of detail breaks encapsulation, but hiding it can make a system unreasonably
slow.
Solution
Create a Proxy class on the client that has the ability to dynamically
reflect any specific persistent object in the database. Provide an API
on the Proxy to dynamically supply information about the object, its attributes,
relationships and operations. The Proxy is responsible for maintaining
the integrity of attributes and relationships. It enforces typing constraints,
relationship cardinality, and marshalls arguments for object operations.
Wrap all database operations within the Proxy class. Implement
a lazy access policy for Proxy properties. Only fetch attributes and relationships
that are actually accessed. Let Proxies manage the change records for modifications
that are made to them. Provide a standard API on Proxy for object update,
rollback, creation and deletion.
Resulting Context
Your client system now deals with all persistent objects as instances
of a single client class. Database operations are hidden behind the Proxy
wrapper API. The remainder of the client code deals with concrete instances
and a known API.
Related Patterns
-
Propagation of Local Changes.
-
Dynamic Type System.