Tuesday, July 20, 2010

New Graft Feature: XSD Exports

Graft now has the ability to generate XSDs for any model. Visit the export tab of your chosen model.

THE DECISION

We have chosen to generate a ComplexType and an accompanying element of the same name for each class that doesn't represent an xsd primitive type. Each class association is represented within the ComplexType definition as either an element or an attribute depending on whether or not the association's type is one of the xsd primitive types. We optionally support a namespace, but at least at this stage further external namespaces will need to be added to the schema post generation.

THE DEBATE

XSD generation is a feature that has caused a fair amount of debate at Jodoro. This is because the XSD 1.1 standard provides many different ways to represent the same concepts.

For example, the Global Justice Data Model defines all of its entities and their associations in ComplexTypes and then defines an element of the same name to take each type. Each ComplexType tends to contain a ComplexContent which in turn may extend an appropriate ComplexType and/or define associations to elements of the defined ComplexType. This model provides maximum flexibility because a valid xml schema could contain any chosen subset of the defined elements. This is particularly useful when many different types of software systems need to consistently communicate with each other.

In contrast the CellML and FieldML schemas define very few top level elements (in fact one), and tend to fully define associations between ComplexTypes within the ComplexTypes themselves. This approach allows for a more formally structured approach to "valid" xml structures, which can be helpful in sharing information between very similar systems. Even these two very similarly structured xsds differ as FieldML does not use any namespacing and treats everything as a ComplexType while CellML utilizes the "cellml" namespace and defines both ComplexTypes and SimpleTypes.

The Schools Interoperability Framework (SIF) defines an element for each ComplexType, but demonstrates yet another structurally different way to build an xsd by defining very few named ComplexTypes and creating many nameless ComplexTypes in their definitions. This directs the consumer's focus to the named ComplexTypes, but comes at a cost of comprehension of the ComplexType definitions (and results in many nameless classes in the domain model). Like CellML, the SIF standard also defines SimpleTypes and attributes differentiating them from ComplexTypes and elements by whether or not they extend the xsd primitive types.

Beyond these structural differences we also needed to contemplate whether and how we support concepts such as enumerations. The issue for us is that this concept blurs the borders between meta-data and data. Technically each enumeration value is one of the valid instances of the enumeration type. It is tempting (and very common) to define enumerations within xsd schemas, particularly when the values are unlikely to change. However we would argue that even if the values won't change a better approach is to define the enumeration as a code of type string or integer, and to store and maintain the valid values outside of the schema. This provides a cleaner separation between structure and business rules.

We also discussed and debated many other commonly used xsd concepts such as "pattern", "maxlength", "union", "choice", "key", "any" and "all". At this stage we have chosen to leave all of these out as they are defining concepts that we are not currently explicitly representing within our domain modelling tool, and because we feel that, like enumerations, many of their uses are often business rules and arguably shouldn't be defined in the schema. If you have suggestions or issues with our current approach, please email myself or support@jodoro.com.

Doug - @douglasenglish .

Thursday, July 15, 2010

Graft Rails Model Inheritance

We've pushed out a new feature for Graft.

Up until now, each exported class has directly translated across to a Ruby on Rails Model, without regard for inheritance in the models.

Exported Rails Models will now include all the primitive of the Graft class, as well as any superclasses. So if you have BMW inherit from Car, and Car has a "license" of type "String" - both Car and BMW will have the license property. The same applies to relationships.

We've put up an example model that should help illustrate. You can export it to Rails using the instructions listed under the Export tab.

It's fairly new, so if you've got any queries or strike any issues, drop us a line via Twitter @jodoro or email support@jodoro.com.

Right now all classes are generated as Models, so you can consider this the superset. Later releases will allow for finer-grained control, such as flattening (e.g Single Table Inheritance).

Jon - @jonathannen.

Wednesday, July 14, 2010

Developing Business Process Models with Domain Models

Quite often Domain Models are developed around or in concert with Business Process Models. This often leads to a the question "how do they inter-relate?" (and subsequently, "who's in charge? what drives the definitions?").

It's important to relate Domain and Business Process Models, but the approach for doing this shouldn't be too onerous. At the most basic level, each Business Process step should have input and outputs that are driven off the Domain Model. If you are processing a shopping cart payment "Process Shopping Cart", the input may be "Shopping Cart" and the output "Invoice", both concepts in your Domain Model.

Generally this is applied at lower levels of a Business Process, L2 or L3, but there is nothing to stop you working top-down or bottom-up. For this type of exercise, I believe a top-down approach is the best fit. Bottom-up has a tendency to get bogged down in the detail (the wrong kind).

The obvious exceptions are cases where the low level processes are well understood. This can occur when the Business Process is already operating or established in another form. These existing processes often have associated pre-existing MI, data or other statistics - and often this information can be quite fine-grained. In these cases, bottom-up isn't a bad place to start.

Whatever the level, it's advantageous to elaborate on the details of the inputs and outputs. What elements of the Shopping Cart are required? Which are optional? For each domain concept, the key questions are "how is it used or applied?", "if it's optional, what are the rules?" and "what's the context?". Aim here should really be to reduce the inputs and outputs to what is absolutely necessary - this is for a few reasons:

  • It best informs your Domain Model.

  • It best translates out to other representations, such as a technology implementation - Clarity here will have numerous downstream benefits, particularly in the implementation.

  • The act of reduction/distilling itself is a good instrument to drive the exercise.

  • It's easier to see what's missing. Blanket terms can hide conflicting points of view and assumptions.

  • It greatly assists the construction and execution of tests and user-acceptance.

  • Finally, in the heat of delivering a project, it can sometimes be hard to argue "why?" against business requirements. This is an ideal opportunity to ask.



That's just a quick summary for now. We'll dive into some of the advantages and pitfalls of models led by Business Process in a later article.

Jon

Tuesday, July 13, 2010

Workshop your Domain Models

The value of a Domain Model is often overlooked. Done properly it defines the structural integrity of your software system in a language and an entity relationship structure that can be understood unilaterally by the business domain experts and the technical specialists. As such it significantly reduces upfront requirement errors, and better structures the solution to support future change requests.

But a domain model must be owned by both the domain and technical experts on the project for it to realize this value. I have utilized the following facilitation technique, borrowed from Feature Driven Development, to achieve this ownership on several of my projects. It was originally designed for Domain Modeling, but the techniques would equally add value to any form of workshop.


  1. Identify a facilitator. Yes, cliche, but also amazingly too often overlooked. Choose the facilitator wisely. They should not get dragged into the technical debates, and shouldn't be afraid to call time. Keeping the meetings moving is critically important.

  2. Identify a documenter. Encourage all of the participants to jot down points throughout the workshop - it's the documenters job to collate and distribute these at the end of each day. Focus on decisions and actions, backed up with justifications. Include lots of photographs - it is amazing how unifying their inclusion makes the process. The documentation should be released by close of business on each workshop day.

  3. Keep the numbers balanced and low. It's of course important to ensure the appropriate stakeholders are represented in the workshop, but the key word is represented. Too many active participants in a workshop will grind its productivity to a halt. I've found that between 4 and 8 people tends to work well. Keep in mind that the workshop participants should also include appropriate representation of the domain experts and the key technical implementers. As close to 50:50 as you can orchestrate the better.

  4. Choose your location carefully. Pick spacious, quiet and light rooms. Come prepared with appropriate equipment. Ideally choose a location that's outside the team's usual working location. You want to encourage full participation without distractions.

  5. Define team norms as a team. (And make sure "have fun" is on the list!). Don't knock this one until you've tried it. Asking the team to define their own norms is an excellent way to ensure all team members adhere to the ground rules. Punctuality? Mobile phones? When is lunch?

  6. Take lots of planned breaks, and finish early. Done properly such workshops are amazingly exhausting. On top of this, the team bonding experience of the breaks is perhaps almost as important as the workshops themselves. Make sure you finish the workshops early each day. I often finish them at lunch time. It's important to ensure the participants have time at the end of each day to take care of business as usual activities, and that the documenter has time to compile the days results. Morning are usually ideal for modelling - people seem to be fresher.

  7. Reach consensus on issues. Yes in an ideal world the entire project team would immediately and unanimously agree to each decision, but this isn't an ideal world and humans are good at arguing. It's a good idea to give one person, perhaps the project's solution architect, the ultimate veto on decisions. Coming to consensus is a commitment from the team that, even if they don't 100% agree with the selected approach, that they'll at least live with it. Once a consensus is reached it's final. The last thing the project needs is for the same resolutions to be requestioned in a corridor three days later by a subset of the participants.

  8. Foster diversity. A commonly used Feature Driven Development technique is to break the larger group into two or three subgroups (ensuring domain and technical representatives remain in each subgroup) and ask each to model the same problem space concurrently. After a set amount of time each subgroup resents and the group as a whole drives towards a common single solution. This may mean accepting one of the subgroup solutions entirely, or it may mean merging components of each.
  9. Keep a "Parking Lot". I've seen the same technique referred to by many names. Essentially it's a team maintained list of topics that need to be closed out by the end of the workshops. It provides a mechanism to capture ideas as they're generated by the team without interrupting the currently flow of activities.

  10. Pulse check progress. At the end of each day ask for feedback. What's working, what could work better? Is the room too hot? Were the brainstorming sections long enough? This feedback loop can massively improve the experience and end result.




Formalizing the domain model within a workshop process provides a very useful metric. For every week you spend in workshops defining the domain model you can expect a ramped up development team to spend three weeks building the content. So if it takes you four weeks to lock down a domain model for a project that needs to be finished the build phase within the next two months, you should probably take another look at the scope, or the time-line.


Doug

Thursday, July 8, 2010

Draw a Real Picture

In a previous article I mentioned five tips for Domain Modeling. In this post I'm going to drill down further on one of those:

1. Draw a "Real Picture".

I've used Real Pictures as a technique for a number of years; not only in Domain Modeling, but in Architecture, Design, Development and Planning. It's a pretty simple exercise, but yields a lot of interesting information. In this description I talk in the context of running a project, because that's the most common case -- but that is not universal.

A Real Picture is foremost a conversation piece. It lets the stakeholders involved discuss and explore the domain and it's context. Besides just capturing raw information, a Real Picture also serves as a gentle introduction for participants that might not be familiar with formal diagramming approaches. Often a good first step.

To start, get the participants together. You don't need to mass-invite everyone, but make sure all the key representatives and stakeholders are covered. So if it's a software project, you should have sponsors, end-users, designers, developers, testers and so forth. If you need to economise on people, focus on those that have most affected or impacted by the end result of the exercise, rather than on pure expertise.

There isn't much to developing the picture itself -- Simply draw the major concepts and how they inter-relate. In my experience, this is enough to start the conversation flowing; just draw and re-draw the diagram as the "oh, and then there is" comments evolve.

As a rule of thumb, I keep Real Pictures to one page - A3 or a White Board usually. If it gets really messy, redraw the picture. If necessary, drop in prompters to keep the dialogue flowing. Pick a concept and drill down - focus on quantities, qualities, time, costs, constraints:

- How many of these items are there? e.g. How many employees are at that location?
- Does it vary over the day, are there seasonal variations? e.g. Do you have a rush at Christmas, End of Financial Year?
- Will this change or move soon? e.g. Are you changing network providers? When did you last open a new outlet?
- What would invalidate this element or relationship? e.g. What security issues would make a location unviable?
- What do competitors or other equivalents do for this function or outcome? e.g. Does your competitor do this differently? better?
- Are there constraints on the way things should be? e.g. Does the government enforce strict reporting on certain items?

It's important that you don't constrain the actual process and format too much. The objective of this exercise is to get all the stakeholders together and to articulate a landscape that everyone understands. The end result should also be something that all stakeholders reasonably grasps and agree to.

At this stage, try not to abstract too much - early abstraction can lead to some necessary detail or important corner-case being lost. If a specific location, or server, or piece of software is mentioned, it may have special significance.

In one particular instance I found a number of stakeholders would refer repeatedly to a specific printer - i.e. the actual device itself and where it was located. Most of the project team didn't see any significance of this, myself included. The model and design assumed it was a standard office printer, the same any printer we used in our day to day. When we eventually dug deeper, it turns out this specific instance was special - it enabled secure printing. The way the device functions is such that a human operator never gets to see the contents. If you've ever received a bank PIN in an special envelope, then it's probably from a pretty similar device.

Not only was the fact that this device was special important - but also many of the project participants hasn't realised the significance of the information being send by this device. The communications that ended up at this printer required an extra level of security; it has a very special significance. Up until that point most of the team hasn't considered that special, but clearly it was. Similarly, the end-users thought that information was obvious. Suffice to say, staying concrete at this stage usually is the best default - Abstractions will flow later.

A Real Picture sounds trivial, in fact you might be over-the-top to call it a technique. Either way, it's a low impact exercise with a lot of benefits. At the end of the exercise you have:

  • A starter for the vocabulary that is in use. Perhaps more importantly, you've got a vocabulary that a variety participants are at least familiar with.

  • An outline of scope. With the Real Picture you should be able to draw a line around what's "in" and what's "out". If something's ambiguous, explore that some more. If scope has been defined as part of a project, this can be a useful validation point.

  • A fallback. If you hit a road-block or misunderstanding later, you might be down in some detail and a representation that some might not understand. If you have this, you can ask "but didn't we say these relate in that other diagram?". It's not the absolute source-of-truth, but it's a useful pivot.

  • An understanding of the various perspectives of the stakeholders. Usually the Real Picture will help illustrate the natural focus and bias of the stakeholders.

  • If the participants can't understand or arrive at a picture, then the composition or overall scope needs addressing. Every participant in the project or domain should at least be able to understand the A3 view of what you're trying to realise.



Jon

Wednesday, July 7, 2010

Customizing large industry standards

THE PROBLEM

Throughout our consulting engagements with corporates attempting to implement industry standards, two predominant issues keep recurring:
1. Customization of the standard means forking from the standard. Taking updates from this point is at best extremely labor intensive, and at worst impossible.
2. Very few organizations want to implement an entire standard from day one. A more typical usage is a phased approach with core services incrementally released over several years. Absorbing the entire standard up front results in service consumers being expected to understand and work with verbose and confusing generated artefacts (such as XSDs) filled with far more data fields than have actually been implemented. In fact I'd hazard a guess that there are very few cases where more than half of those data fields will ever be implemented.


WHY?


The integration of data and its structure is a major problem for almost all organizations - the bigger the organization the bigger the problem. We have seen many organizations look towards domain industry standards to provide a more structured blueprint for internal and external integration.

Business-to-business standards tend to be fairly effective because each organization involved has a strong financial incentive to adhere to the standard. An example is the AS2805 standard for Australian EFTPOS transactions, which itself was based on the ISO standard 8583.

Internal integration is, however, another story. If you've ever worked in the IT arm of a large organization you'll know what I'm talking about. Almost all software development is project based, and it's an all too common story to hear that the 'refactoring' was descoped to 'phase 2' which - surprise, surprise - never actually ends up being funded.

However, for those lucky enough (or perhaps unlucky enough) to participate in the rare sort of project that does attempt to implement the 'phase 2', you've probably also discovered that adhering to an industry standard is anything but easy. For starters there are few tools available for dealing with standards. Most open standards are available in XSD format accompanied by MicrosoftWord, and MicrosoftExcel documentation, and most proprietary standards are accompanied by equally proprietary vendor lock-in product sales. Beyond this, no standard is ever quite what you need, either missing large areas important to your business, or overly bloated due to a missing-the-point attempt to please all consumers all at once, and more commonly a little of both. Almost all approaches require the consumer to start with the entire standard and then customize for their needs.


SO WHAT?

As practitioners in this space, and still recoiling from the war wounds, these two problems were front of mind in the inception of http://graft.jodoro.com.

Customization

Every model in Graft is internally stored as tiny deltas. When you extend an existing model in Graft you simply add further deltas to the same base structures. Version control and management is baked into the framework of the application. As such, even if you completely rename an existing class, you can easily adopt any future associations to the base class within your extended model. When and what you adopt is controlled automatically via "releases" configured in the 'specify' application.

Templating

In Graft we provide users two distinct ways to extend a model:

1. The most commonly understood approach we call 'active' extension. This quite literally means you start with the entire base model in your scope. Graft then allows users to explicitly descope parts of the model they don't wish to use. Such descoped classes will remain in the default view, but grayed out. This allows users to see what they have removed, and provides the ability to reverse the removal at a later point. Although users can still see grayed out descoped items, such items will not be included in any exports. An example of an actively extended model can be found here: http://graft.jodoro.com/models/31881

2. We are particularly excited to introduce a new approach to extending that we call 'passive' extension. This allows users to begin with everything descoped, and to draw forward only what is of utility right now. This is the approach we strongly recommend be used when working with large industry standards. An example of a passively extended model can be found here: http://graft.jodoro.com/models/31893

Often organizations wish to borrow from more than one standard. Perhaps, for example, the street address structure is particularly weak in their chosen strategic industry standard. Graft allows any number and combination of active and passive extensions within the one custom model. This is again configured in the 'specify' application.

Feel free to extend these and any other public models within Graft. By default extensions of any models will remain private. You can however choose to make your extended models public too by visiting the admin tab.


Doug

Tuesday, July 6, 2010

5 Domain Modeling Tips

It's been a while since we've updated the blog. In the meantime, we've been working hard -- both consulting in the domain modeling space, as well as preparing the latest release of Graft. Given this, we're going to blog less about the technical implementation - and more about using Graft and the process of Domain Modeling.

To start we thought we'd discuss five key tips for building a domain model. We find these are generally applicable, but how and when you implement them will depend a lot on context. Key factors include the time at hand, pre-existing collateral, requirement maturity and the availability of Subject Matter Experts.

Here is a quick run down for now. We'll flesh these out over the coming weeks:

1. Draw a Real Picture
This has a lot of names - In this case it is just an informal sketch that every stakeholder will understand. Typically this is done on a white-board, but paper or online work just as well. Emphasis here isn't on building a refined model, it's about getting the landscape view, establishing an initial vocabulary and providing a basis of scope.

This picture doesn't have any formal semantics, it functions more as a conversation piece to illustrate and start grappling with the domain. Build this using whatever works for the participants. That said, if done right it can be an opportunity to gently start introducing some key modeling concepts and approaches.

2. Goal-Based Workshops
Many modeling exercises risk becoming esoteric. We encourage goal-based modeling that drives towards outcomes. Highlight the gaps, inconsistencies and pitfalls. Uncover the gnarly pieces early. You might not necessarily solve these immediately, but surfacing them is a key outcome.

Often the Goals themselves are simply driven out through workshops and consultation. In the ideal case you can derive Goals from sources such as strategic initiatives, requirements definitions, existing systems and business processes. If these aren't available or mature, focus on how the outcomes will be validated.

3. Deep-Dives, with a focus on Cohesion and Coupling
This is really where the heavy-lifting of modeling comes in. A deep-dive is a concentrated effort on an important aspect of the domain. Often the Goal-Based Workshops will help to drive out the topics.

The frequency and function of these will need to vary based upon the topic involved. Our ideal is a set of tight iterations on each topic. However, a common compromise is to run a deep-dive session, have the modeler refine the results offline and then a hold final session to reconfirm the outputs.

4. Revisit the Landscape
Deep-Dives generally uncover a lot of detail, however, before going to far down a rabbit-hole it's good to pause and check "is it really relevant?". The key question here is always "what's the context?".

Ideally you will have introduced Business Processes as a requirements driver earlier. However, if this hasn't occurred, this is a key opportunity to introduce them. The Business Process will let you ask a lot of key questions around model attributes and relationships - Are they used? When and what for? This can lead to surprising results, often for aspects the participants take for granted.

5. Iterate, Evolve & Refine
No model is ever "ultimately complete"; in fact it can be damaging to have this as a goal. It's important to plan for how a domain model can and should change over time. This can include small things - from notes and documentation on why decisions have been made and considerations for the future - through to a full roadmap on how the model needs to progress from here.

That's a very quick wrap for now. If you have any other tips, let us know and we'll see about exploring them over the coming weeks.

Jon