This release fixes one serious bug and adds a new, important feature.

• CREATE TABLE had been reporting an error occasionally because it was unable to find the column type definitions.  This should no longer be an issue.
• INSERT and form status updates that use the AS NEW FORM clause now return resultsets when they complete successfully.  These resultsets contain the IDs of all of the newly created rows.  It’s the CodaServer version of MySQL’s last_insert_id() function, except it happens right away.  Like it should.

I would also like to take this opportunity to mention that CodaServer has a Google Group for users, and you can join it here.  Obviously we’re just starting out, but this is the easist way to get help with or become involved in the CodaServer community.

Til next time.

In his article Martin mentions a few alternative database products (he forgot to mention CodaServer, but I’ll let him off the hook this time) and reasoned that there may finally some legitimate alternatives to SQL-based RDBMS.  He also has some ideas about why RDBMS have been so popular for the last 30 years or so.

For many organizations today, the primary pattern for integration is Shared Database Integration - where multiple applications are integrated by all using a common database. When you have these IntegrationDatabases, it’s important that all these applications can easily get at this shared data - hence the all important role of SQL. The role of SQL as mostly-standard query language has been central to the dominance of databases.

He is absolutely correct about the Shared Database Integration pattern, and I may even classify that as a problem.

Integrated databases are static databases.

They are impossible to update as the business moves, as any changes must be approved and vetted across every team that integrates with them.  This means nothing happens until the last possible minute when a sufficiently huge “business driver” forces everyone to step in line.  Technically they are also a burden since every language and application framework has its own ideas about how database integration should happen, leading to the dreaded impedence mismatch, only with a database that none of the teams has authority to fix.  Everything is roll-your-own, and, while the situation is not impossible, it drains developer cycles rapidly and causes mysterious project delays down the road.

I think CodaServer could be well-positioned to be a part of the sea change Martin talks about.  It is sufficiently relational to be immediately familiar to people used to SQL, yet it is entirely web service based, using HTTP for all communication.  It also has more of an ApplicationDatabase philosophy, as each CodaServer application is treated as a unit (in fact schemas are called Applications) and all permission models and deployment tools reinforce this philosophy.  Like Python and its tab blocks, it makes it inordinately difficult to do things the wrong way.

I for one look forward to a world in which developers use the right tool for the job at hand.  Incidently, if you need form-based business applications, I have a suggestion…

Til next time.

Databases are not a valid integration point for applications.

I have seen this in virtually every place I’ve been and it’s a problem.  Everybody loves the database because we all understand it.  It’s easy.  It sits there with its port all open waiting for any little application to drop in and leave some data behind.  You don’t have to worry about the filesystem or sticking stuff on a particular server somewhere; files are dirty anyway.  The database is elegant, with relationships and tables and indexes.  You have drivers in every language, and all these pretty abstraction layers, ORM and such.  It even uses a nice querying language so you can get exactly what you want out of it.

What could be more perfect?

Unfortunately the database isn’t the cheap date she leads us to believe she is.  You’ll pay for all this beauty, yes sirree.  How, you ask?

1. Once you code to a schema from multiple applications, you can never effectively change it.  Sure, you can try, but it’ll take you 10 times longer than it should.  And since half of your apps are developed by programmers who are no longer with the company, you may not even know you needed that column until you drop it and all the sudden the payroll system crashes.  On second thought maybe you should leave it alone.
2. The database schema is designed with a ton of little rules in mind.  That absolutely_sure_flag column on the order_items table?  That is there to make sure that the guys in the warehouse were positive they stuck that item in the box, and if you don’t set it the accounting system doesn’t include that item in the rollup report.  You did remember to set it in your web service, right?
3. Databases are flat.  The elegance of everything being tables with columns is offset by the fact that meaning is implied through naming conventions and foreign key relationships alone.  This is fine if you designed the database and know what everything means.  It is less fine if the database was designed by a DB2 veteran who cut his teeth on a System/390 and felt that using more than 8 characters in a table name was a waste of “core.”
4. Database columns have a limited range of types, and all format-level validation is done in whatever application language is doing the inserts and updates.  Hopefully all of the applications will do the same validation, right?
5. Databases have a somewhat broken permission system.  This leads to audit properties and permissions being implemented at the application level, which is great if every application has the same login/permissions system but not so much otherwise.

Integrating at the database level gives you a brittle, inflexible system where all of your applications are tightly coupled.  After a few years your beautiful bride turns into a gluttonous beast consuming developer cycles like bonbons.

I think the reason for this is a simple misunderstanding.

Developers confuse the data model with the database.

The data model is what your company uses to do its work; it represents a model of the world in which they operate.  The database is just an artifact, a rough implementation of that data model.  Subject to change as the world comes into focus.

This is an admittedly fine line, but an important one.  Data models are tuned, databases are changed.  In the data model you can deal with abstractions.  Orders have Items on them which correspond to Products and are shipped to Customers.  Databases deal with specifics.  A row in the `order_items` table points to the `orders` and `products` tables, and the `orders` table has a `customer_person_id` that is keyed to the `people` table.  Or, if performance is an issue, maybe the customer details can be placed in columns directly on the `orders` table instead.  Maybe we can also drop a `product_name` into the `order_items` table as well, since that is all we need from the `products` table on those checkout screens that get loaded 1,000,456 times a day.

In any event, the data model should be all you need to know in your application code.  The database should be hidden behind an abstraction layer.

So how can we achieve this?  My personal preference is a business rules engine such as CodaServer.  (Did I mention it’s open source and free?)  It handles many of the above concerns:

1. Data integrity is enforced through an extensible workflow system with triggers that can be made to stop you from doing bad things with data.  If you make a trigger that halts an operation when a particular flag isn’t there, it will do so across all applications.
2. Tables have different relationships with each other.  They can be parents, children, or neither, plain tables or workflow-laden “forms.”  Since this is part of the schema, applications can figure it out and plan accordingly.
3. There is a regex-based type system so column values can be validated more stringently.
4. The user and permissions model is multi-faceted and a help rather than a hinderance.

Another solution is web services.  REST can provide a simple way of representing your data model in a database- and language-neutral way, and it is trivial to update to reflect database schema updates.  On the updating side, XML-RPC and SOAP can provide ways of parameterizing your data operations and presenting them across application languages.

Databases are wonderful technology, but like any tool, they are more appropriate for certain tasks than others.  Storing data and making it available for ad-hoc queries?  Great.  Providing a common API for applications written in different languages and for different purposes to share data?  Not good.

Til next time.

When I left off I was talking about a great new development methodology, called Interface-Driven Development.  Yes, with all capitals.  You can read all about the principles in the last article.

Team Organization

Many development teams are organized “horizontally.” That is to say, different team members have different specialties, and they gravitate towards working on those parts of the system.   This happens a lot in shops which use giant systems like Oracle, and in Waterfall organizations where the parts of the application are gone over in excruciating detail and doled out to developers at the end.

This causes several problems:

• Developers who specialize don’t learn how to do anything else other than their main skill.
• It is easy to shirk off responsibility to other people.  It’s a database problem!
• There is a lot of waiting around waiting for the other teams to finish their stuff.

Interface-Driven Development focuses on having “vertical” teams, with each developer solely responsible for every aspect of their part of the overall software system.  Craig handles the whole blog and nothing but the blog; every table is created by him, every model class, and every web page.  When it comes to the blog, he is the final arbiter.  Others can make suggestions, and he can ask for advice, but he owns it.  It becomes his Product, and it is his job to make sure it meets the needs of the other team members as well as the application as a whole.

This is madness, you say.  Won’t the schema get messy?  And what if he sucks as a web designer?  How can we be sure he knows what he is doing?

Read on for the answers.

Interfaces

Obviously, letting developers loose on a shared database can be unnerving.  This is why each Product gets its own schema on the shared database instance where the developers can go wild.  (Common functionality such as user logins and access controls are provided by another Product, or possibly through a business rules engine if one is handy)  They can create tables as they please, drop and add columns, use or not use stored procedures, whatever.

What gives them this flexibility are Interfaces.

Interfaces are the contracts between Products.  During the first phase of the project, when the Products are being given out to developers, all of their interactions are discussed.  Paul might say to Craig, “Hey, I need a way to get a listing of all your blog articles” or “Hey, I need a search function where I provide you a random text string and you return me a list of titles, article summaries, and links in order of relevance.”  Craig then defines an Interface using common web service protocols (REST, SOAP, XML-RPC) that meets all of the needs of the other Products.

It doesn’t matter what the blog schema looks like because no other developer sees it.  They interact with it solely through the Interfaces.

The Front End

At the start of the project, the developers also need to figure out the user interface for the application.  Walking through a few use cases, (how many doesn’t matter, you can iterate later) they develop the UI Palette for the application.  The UI Palette consists of the following things:

• Screen Chrome: Logos, Logout Buttons, My Account Links, Copyright Notices, etc.
• Primary Nav: Tabs
• Secondary Nav: Sub Tabs, Breadcrumbs, Section Splash Pages
• Forms:  Inputs, Textareas, Buttons, Form Tabs, Dropdowns, Calendar Popups, AJAX Controls, etc.
• Tabular Data:  Tables, Filters, Column Sorting, Pagination, Search Boxes, etc
• Content Rules:  Panels, List and Paragraphs Styles, Headings, Alert Boxes

Once the Palette is built, it is marked up quickly, CSS is put together, and an appropriate Javascript library is found or built that will handle any additional widgets.  If the team decides to use a view layer toolkit such as Smarty in PHP or tag libraries in Java, they can build out their various components there for an even deeper level of integration.

With the UI Palette in place, any developer can build web pages that look as pretty as those from the best designer on the team.  All of the components are standardized.  After the Products are released, the other team members can also make suggestions on how things might be better; since they are Customers it is logical that they would have feedback.

Responsibility and Authority

Since a developer owns their Product, they have a vested interest in making sure it is successful and thriving.  They have the responsibility to make sure it works for all of their Customers and the authority to do whatever is needed codewise to make it happen.

This is a powerful combination that isn’t often found together, and it tends to bring out the best in people.  I have found that most developers throw themselves into projects where they get to do everything themselves.  For most, it is an opportunity to “do it the right way” that they always wish they had during all the water cooler conversations.

It isn’t ideal for everyone though, and that is also a good thing.  Interface-Driven Development makes it apparent very quickly who can’t keep up with the rest of the team, and gives management a chance to reassign them elsewhere.

In my next article I will walk through a brief sample project to show how Interface-Driven Development works.

Til next time.

Most software developers work in the trenches, the corporate IT departments, writing code to help automate a company’s various processes.  While the Googles get the glory and Facebooks find the fame, our intrepid warriors make life better for one knowledge worker at a time.

One problem corporate developers have is that business processes, once in code, are difficult to change.  This is compounded when multiple languages or development platforms are in use, as the changes need to happen at the same time in multiple places.

The Old Way

Let’s start with an example taken from a simple warehousing operation.

• Pallets are delivered by UPS or Federal Express to the receiving department.
• They are then inspected for damage by the staff.
• If there is damage, they are rejected and sent back to the supplier.
• If they are good, they are “broken down” and the boxes on them are stocked in the warehouse.

Using a traditional SQL database, there may be a table for “pallets” with a column called “status” having each of the highlighted words above as possible values.  (There would be other stuff in there too, but I’m keeping it simple.)  There would also be a (say) PHP class called “Pallet” with methods corresponding to each of these actions, as well as business logic handling the validation of various fields and making sure that the specified pallet was in the right state to have the action performed.  Then there is a web tool that let users record this stuff online, and it would also do some form validation as the user tried to do various things.  There is also a Java-based reporting engine that runs against the database, and it needs to know about all of the statuses for some of its ad hoc querying capabilities.

Now, say management decided to change this.  They add a staging status between delivery and inspection, since the warehouse is receiving so many shipments and the staff can’t inspect them quickly enough.  The staging area will act as a short-term storage area, and the status will give management greater visibility into where their boxes are.   What needs to change?

Well, the database is simple.  “Staging” becomes a new status field value.  Just gotta make sure to add it to all of the environments.  The Pallet class gets a new method, and most of the other methods may need to change too to make sure they are checking for the new status.  The front end has some new conditional logic to take care of, and it needs a new button and status made available.  And the Java reporting system needs to know about staging.

The CodaServer Way

CodaServer handles “tables with status” using its FORM entity.  Instead of making status a data value in a table, it lets developers define the state diagram for the entity.  This is done by defining a series of “leads to” relationships:

• Delivered leads to Inspected.
• Inspected leads to Rejected or Broken.
• Rejected leads to Nothing.
• Broken leads to Nothing.

To add a new status to the list involves simply declaring the new status through an ALTER FORM command and rewiring Delivered to point to it via another ALTER FORM.

CodaServer lets you specify both an adjective and a verb for each status.

• Delivered/Deliver
• Inspected/Inspect
• Rejected/Reject
• Broken/Break

These verbs are what happens to the object to make it enter the adjective state.  Incidently, these verbs replace the traditional INSERT, UPDATE and DELETE operations in SQL databases, although you can still use UPDATE to modify an object without changing its state.

UPDATE pallets SET status = 'Inspected',

inspection_date = CURRENT_TIMESTAMP WHERE id = 4;

is replaced by

INSPECT pallets SET delivery_date = CURRENT_TIMESTAMP WHERE id = '4';

CodaServer handles all of the busy work of determining if Pallet 4 is in the Delivered status, without your having to do anything.  It also fires off the BEFORE INSPECT and AFTER INSPECT triggers if you have them specified to handle any validation or cleanup that needs to be done.

What’s more:

All of this is discoverable through metadata commands.

Applications can be written in such a way that they know about your business entities, their statuses, actions, and workflow without you needing to write any additional application code.  Think about this, because it’s a pretty big deal.  All application metadata is available through a cross-platform web services API, a business rules engine enforces your workflow and data integrity, and your workflow can change without you having to write more application code.

That’s developing smarter.

Til next time.

So as I was typing in my blog the other day, I tripped over the one true answer.  The development methodology to end them all.  Really.  Rather than just blurt it out, though, I’m going to try to milk it for a few blog articles.

Now, I won’t say that it is all new.  There are aspects of it that are currently part of agile development, and others that are part of programming by contract.  But those both are much more specific and operate on a lower level.  Maybe this new methodology is more of a meta-methodology.  Wouldn’t that be something?

Let’s get into it then.

Some Observations

I’ve been programming for a little while now and I’ve picked up on a couple things.

One, development tends not to scale too well as you add more people.  Fred Brooks had something to say about this when working at IBM, more specifically about how adding more programmers to a late project tended to make it later.  Having been involved from the beginning on several projects, I feel that his observation can be extended to pretty much any part of the development lifecycle.  There is always more friction with more team members.  Sometimes you disagree on technical issues, sometimes you have personality conflicts, sometimes you are dragging along dead weight.  There are always more meetings, and more overhead in general in integrating various viewpoints into a single codebase.

Two, programming is a “flow” activity that requires one to maintain a model of a complicated system in their head.  Joel mentioned the flow part and Paul talked about the mental part.  Each of these hints at another reason why development doesn’t scale across multiple brains so easily: Interactions with other people take you out of the flow state, and multiple developers mean that you need to figure out how other people do things and remember to switch to their mental model when working on different segments of the application.

Three, standardization is better than documentation.  I’ve been at many places that cling to documentation like a warm fuzzy blanket in February.  It absolutely has its place, I can’t deny that.  The bigger issue, though, is that unless you have standard components in code that you reuse throughout and don’t make everything a one-off, no amount of documentation will save you.  Developers need to understand what is going on simply though looking at the code itself, and nobody maintains documentation anyway.

Four, abstractions don’t mean s***.  I apologize to the sensitive people out there but it needs to be said.  Put another way, some developers tend to go into architecture mode and refuse to commit to a specific technical solution.  I remember a particularly humorous scenario where I was at a company who had purchased many $25,000 Oracle licenses and then tripped over themselves to make sure all their code used a database abstraction layer and dumbed-down SQL that didn’t use any of Oracle’s impressive features, like CONNECT BY.  Why?  So you can move to MySQL tomorrow?  “Sorry about the $200k, boss.  My bad.  Wah wuh.”  There is nothing wrong with committing to a particular platform, assuming it isn’t currently dead, dying, or a technical dead end.  Just be smart about it.  Besides, you’re eventually going to have to do it anyway.

Five, cross-platform is important.  Pretty much every development shop of any size has or will have multiple programming languages in use, at the same time, for developing software.  It’s inevitable and it’s real.  Application language frameworks are cool and handy, but they generally only work with one application language at a time and are therefore, by definition, only part of a solution.

Guiding Principles

Given these five things, the new methodology is organized along the following principles:

1. Minimize Developer Interactions
2. Encourage Developers to Do It Their Way
3. Value Standard Components Over Standard Documentation
   
4. Don’t Be Afraid of (Sensible) Commitments
5. Work Across Programming Languages/Environments

Now, there is that old “speed/quality/cost: choose two” thing that happens a lot with development teams.  While many methodologies (inaccurately) claim they can deliver all three, this methodology is honest.  It values speed and cost most, and tries to make quality predictable while not actively hurting it.  If you have a good team of coders who “do the right thing,” avoid hacks, and refactor when they need to, everything will be great.  Otherwise, well, you’re going to have low quality code anyway; the methodology just won’t stand in your way.

In the next installment I’ll dig into the details a little bit more.

Til next time.

Most business systems implement the concept of “users,” those crazy scamps who login and actually use the software.  They’re pretty important; each has different capabilities within the system and their access levels need to be made available to application code so that it can render a suitable interface.

The Old Way

Traditional database systems have very solid user management systems with one caveat: Nobody uses them.

This is one of the greatest antipatterns in enterprise development.  Due to the impedance mismatch between object-oriented application code and relational database tables, most application frameworks find it expedient to use a single database user with global permissions on the schema and implement a separate user management piece using database tables.  This is bad on several fronts.

• Database permissions are pretty bulletproof.  If you don’t have INSERT on a table, you can’t stick anything into it.
• Custom user management code is error-prone.  All of the thinking needs to be done by developers on a timeline.
• Custom user management code reinvents the wheel, poorly.  I have seen a number of half-baked permission schemes in my day, and I’m not really that old.  There is no consistancy from one company to the next and the inner workings/caveats of these systems tend to live in the head of one or two developers.

That said, developers have no choice but to do it this way.  The CRUD model used in SQL simply isn’t dynamic enough for a good permissions architecture.

Everything is an UPDATE.

Once you grant UPDATE on a table, you pretty much give a user access to do anything they want with it.

The CodaServer Way

CodaServer has a much different approach.  It is role-based, meaning that permissions are always assigned to roles rather than assigned to users directly.  Being the Business Rules Engine, it isn’t limited to having CRUD as a model.  You can create a process map that goes something this:

• Enter leads to Place or Cancel.
• Place leads to Ship or Cancel.
• Cancel doesn’t lead to anything.

Developers can then assign roles permissions on Enter, Place, Ship, and Cancel.  It also has an ACL-type permission scheme where developers can create adhoc permissions and grant them to roles, SQL-style permissions for table objects, and permissions for stored procedures.

This plurality of permission schemes is meant to obviate the need for roll-it-yourself user management code and remove a major headache from developement.

Another way that CodaServer differs from traditional SQL databases is that the role-based permissions system is actually part of the schema for each application.  It is deployed with the application as it moves between different environments and the roles themselves are managed by developers rather than database administrators.  (The administrators have their own domain of responsibility covered in an upcoming article)

The logic behind this is that saying which role has what access to the system is very much a part of the business rules.  Is there any reason why a shipping clerk should not be able receive a shipment in the development instance but should in production?  Probably not.  You may want to say that Charlie should be stripped of his shipping clerk role because things he signs for tend to disappear, but the role itself is still a valid entity and shouldn’t change.

A user in CodaServer can have any number of roles in an application environment, and their permissions in the system are a union of all the permissions of the roles they have been granted.  Metadata about a user’s roles and permissions can be gleaned through the SHOW ROLES and SHOW PERMISSIONS commands.  No more trolling around in the information_schema.
Finally, a user management system that works with you rather than against you.
Til next time.

They used a method called crosstraining.  Basically, developers would pair up with one another to go over parts of the codebase that one of them worked on, usually very challenging sections of the code full of exceptions and workarounds.  Everybody knew these sections were beastly, making it was easy to identify the areas where crosstraining is needed.  A knowledge transfer would occur and redundancy would happen shortly thereafter.  Only, it never seemed to work all the way.

Crosstraining gave the trainee an impression of what was going on in the code, but did not give them the understanding that can only come with the experience of actually trying to do something with that code.  Have you ever noticed that when you are a passenger in a car, you can think about all sorts of things and don’t pay attention to road signs or other cars or the lines in the road? Yet when you driving, you are keenly aware of everything happening around the vehicle?  It’s like that.

There is no substitute for actually getting your hands dirty in the code.

It’s a well established fact that it is much harder to read code than write it.  Instead of dropping the trainee in the deep end and letting them take the extra time needed to fix the bugs, the company looked for a reasonable-sounding shortcut.  When the person who wrote the code is with the yaks, though, that reading will still have to occur and the gut work will still need to be done.
ideally a company will have a method in place for building software that wouldn’t require crosstraining at all; all code follows the same patterns but with different data behind it.  If you already know the patterns everything else is obvious.

“Crosstraining” has become a euphamism of sorts for “our code is indecipherable.”  If that’s the case, a refactoring/rewriting session might be the only thing that will get it back to health.

Take that medicine; it will only taste bad for a minute.

Til next time.

Databases have always been a little awkward in the versioning department.

A database server is an environment all its own.  Different databases can have identical schemas but much different data in them.  Some of this data is very much part of the “structure” of the schema:  Tables that are used to populate important application drop down boxes, for instance, or metadata tables for various reporting functions.

This data is programmer data instead of user data, and the application can’t work without it.

How does one keep both schema and programmer data in sync across different environments?

The Old Way

SQL databases are specialists.  They do their thing handling queries and update statements flying in from all directions.  They maintain data integrity by following the rules of the schema.  The handle enormous loads.  But that’s about it.

Database servers are often known as “instances,” and for good reason:  They are entirely self-contained.  They generally don’t know about each other and don’t involve themselves in the rest of the software ecosystem.  If you want to set up a testing environment, you “fire up a new instance” and load your schema onto it.  As you roll out new versions of application code, you run the diffs of the schema against the new environment and backfill the data.

The developer (or system administrator) does the thinking while the database server just runs its queries and twiddles its hard drive platters.

We should be so lucky.

The CodaServer Way

CodaServer has a much different architecture.  One CodaServer manages your development, testing, and production environments.  If you need more capacity, you can fire up multiple CodaServer instances, point them at the same system datasource, and put a load balancer in front of them.  Since all the communication with CodaServer is over HTTP-based Web Services, it has an architecture very similar to that of the web, and can leverage all the same techniques used to scale general purpose web sites.

(For more detail on CodaServer’s architecture, you can look at the manual’s architecture page)

Having one server (or load balanced group of servers) manage all of your environments has another benefit:  Simple version control.

As you build up your CodaServer application in the DEV environment, CodaServer keeps track of all Data Definition Language commands (like CREATE TABLE) in its transaction log.  When you want to move your application to TEST, simply run the PROMOTE command.  When it’s ready for PROD, PROMOTE from test.  Your application is always constant in each environment because deployment always follows the same pattern.  There cannot physically be a schema in production that was not in development first.

But wait, there’s more.

CodaServer also has the concept of REF TABLES, tables that remain constant at all times between environments.  All schema changes and INSERTs, UPDATEs, and DELETEs to REF TABLES occur immediately in DEV, TEST, and PROD.  This is ideal for most types of programmer data.

Some might think it’s limiting that CodaServer doesn’t allow developers to create tables directly on different application instances.  This isn’t really as big a problem as it seems.

CodaServer’s only concern is protecting the data model of your applications.  Since the applications use standard SQL databases, you are free to add any additional tables you want for reporting, ad hoc denormalization, data transformations, etc.  The only caveat is that CodaServer based applications won’t know they are there, which is perfectly fine since they don’t affect the datamodel.

(It is highly recommended that you adopt a naming convention for these tables that is different than that of your CodaServer tables to prevent collisions.)

So there you have it:  A database that does the heavy lifting for you.

Til next time.