Monday, December 30, 2013

How Edapt Works

The gory details of Edapt migrations


Edapt is an Eclipse technology which enables 'coupled' evolution of an Ecore metalmodel and instances of the model. Coupled evolution means evolving the meta model. (.ecore) and the model instance hand-in-hand.

As we will see, Edapt implements the concept of couple evolution in a very sophisticated way. Edapt provides great flexibility with many re-usable model migration Operations. Additionally custom migration Operations are also supported.

Why this blog

This blog explains how the Edapt Migrator works. (It's not an Edapt tutorial!). The reason I wrote this blog is that I am one of the developers of Edapt and as such, I investigated how Edapt works and made many notes. Not sure where to store this study, I thought others could benefit, because the migration process is rather complex and large and can easily daze you. As it turned out for me, understanding the inner workings also helped me understand when a custom migration Operation is needed and how to implement it.

Prerequisites 


Experience with EMF is recommended. Edapt literally constructs Ecore meta models back and forth, so understanding Ecore is key. There are many tutorials. Here is one of the them.

Experience with Edapt History editor and Operations Viewer. The primary step in model migrations. The Edapt tutorial is a must.

(Note: Edapt can now be installed on the latest Eclipse release named Kepler with this plugin repository ).

Content


  • What is the Migrator?
  • Migrator concepts.
  • Migration example dissected.
  • Migrator in details.

What is the Migrator?


The Edapt documentation explains about the Edapt migrator here. What it teaches us is how to contribute a Migrator and how to execute a migration by i.e. extending editor code to detect if a Migration is needed and actually performing it. It doesn't tell us how it works, and this is what this blog is about.

For us to understand how the Migrator work I will explain the various concepts and show how it acts upon a sample metamodel and model instance.

What happens in a migration process can be summarized as:
  1. Process a History by visiting it's releases and changes.
  2. Create an inner map from the original metamodel to a constructed target metamodel in memory, which we will call the reference metamodel. 
  3. Load a model instance (through a converter) with corresponding meta model of a release if it's not the target release yet.  (Effectively determine if the migration of a model is required based on it's release).
  4. Migrate the model instance and referenced metamodel by applying primitive and operation changes. 
  5. When the target release is reached, finish the process and persist (save) the model back to a target resource. (A file identified by a URI)

The Edapt Migration State Model


One key aspect to understand in this approach is that various metamodels will be maintained during the migration process.
  1. The original .ecore metamodel which is referenced by the changes in the History. 
  2. The 'constructed' metamodel in memory which is in the state of the changes applied to it. This metamodel can be identified as the 'reference' metamodel. It is the reference for applying metamodel changes.
  3. The 'constructed' metamodel referenced by the model instances. (See MMMeta)

Migrator concepts


Edapt defines two abstractions to perform the migration. These are:
  • Metamodel Migration model (MMMeta)
  • Model Migration model (MMModel)
Both the MMMeta and the MMModel are bound together in a Repository for easy access.
Then there are the Reconstructors and Converters.

A Reconstructor processes the History model and reconstructs a certain release going forward or backward for the releases in the history. The reconstruction process acts on the mapping and the reference metamodel in case of primitive changes which do not affect a model instance. It also acts on the MMMeta and MMModel when the actual model instance is impacted.

Note: The reconstruction process is also available through the Edapt UI acting on an Ecore editor. It will in this case only reconstruct the metamodel and not the actual model instance.

Converters are required to convert back and fort a model from a MMMeta/MMModel to an EMF ResourceSet. Finally Edapt deals with persistence through a utility class named Persistency, which we will discuss as well.


   

Migration Example dissected

Here we follow an example which is part of the Edapt tests. The example Model and Meta model can be obtained from here.

History Model


Release 1 (Which is only the metamodel definition).


<releases date="2008-11-23T22:45:42.562+0100">
<changes xsi:type="history:Create" element="component.ecore#/">
(1)        <changes xsi:type="history:Set" element="component.ecore#/" featureName="name"
            dataValue="component"/>
(2)        <changes xsi:type="history:Set" element="component.ecore#/" featureName="nsURI"
            dataValue="http://component/r0"/>
(3)        <changes xsi:type="history:Set" element="component.ecore#/" featureName="nsPrefix"
            dataValue="component"/>
 </changes>

etc.... (The rest of Release 1 further builds up the Meta model


Release 2 (Which actually starts to change Release 1, Model instances conforming to Release 1 will be migratable with Operations from Release 2). 


 <releases date="2008-11-23T22:49:28.078+0100">
    <changes xsi:type="history:MigrationChange" migration="org.eclipse.emf.edapt.tests.migration.custom.ComponentSignatureCustomMigration"
       >
      <changes xsi:type="history:OperationChange">
        <changes xsi:type="history:Create" target="component.ecore#/" referenceName="eClassifiers"
            element="component.ecore#//InPort">
          <changes xsi:type="history:Set" element="component.ecore#//InPort" featureName="name"
              dataValue="InPort"/>
          <changes xsi:type="history:Add" element="component.ecore#//InPort" featureName="eSuperTypes"
              referenceValue="component.ecore#//Port"/>
        </changes>
        <operation name="newClass">
          <parameters name="ePackage">
            <referenceValue element="component.ecore#/"/>
          </parameters>
          <parameters name="name">
            <dataValue>InPort</dataValue>
          </parameters>
          <parameters name="superClasses">
            <referenceValue element="component.ecore#//Port"/>
          </parameters>
        </operation>
      </changes>

 etc...

...or as a screenshot from the History editor:


Following along


Here we illustrate the migration process step by step.  



Release 1 (Here the meta model is constructed).

1. caseCreate()
 

Change => Create (EPackage)
  • create a new EPackage
  • add it to the Metamodel Extend cache.
  • Add mapping between the EPackage in the Create Change and map it to the new factored EPackage. (Later on when loading the MMMeta, the extend is used to get the EPackage).

2 caseSet()

  • Cet the target element from the Set
  • Get the equivalent from the mapping definition.
  • set the attribute (feature) on the target element.
etc... continued construction of the meta model


Release 2 ( Here the model is also migrated, as we use Operations)
 

Change => Custom Migration

2.1 startChange() which delegates to the MigrationReconstructor.

  • caseMigrationChange()
  • load the Custom Migration (In our case "ComponentSignatureCustomMigration")
  • call migrateBefore() only. (migrateAfter is implemented by this custom migration, so nothing really happens here).
2.2 calls switch again but not delegating to other reconstructors.
  • caseMigrationChange() => null (EcoreFwReconstructor returns null).
2.3 .. back in the ForwardReconstructor
  • Iterate over the MigrationChangeChildren()
  • caseOperationChange
  • creates a copyResolve OperationInstace from the ResolverBase (It copies and resolves ECore elements from the mapping).
  • Converts the OperationInstance to an OperationImplementation.
  • calls OperationImplementation instance's checkAndExecute( ) with the model and metamodel
  • We end up on the Operation Implementation which is NewClass for this operation, it calls MetaModelFactory to create the class with all the operation parameters.

2.x endChange() which delegates to the MigrationReconstructor 

etc..

When the target release is reached, the MMModel is converted back to a valid model instance which conforms to the target release of the Ecore metamodel, which completes the migration process. 

Migrator details

We explain deeper the various Migrator concepts.

MMM's

MMMeta

The metamodel migration model (MMMeta) is a migration specific representation of the .ecore metamodel.

MMMeta Instance creation

The MMMeta instance is created with the MetamodelExtent corresponding EPackage for a model nSURI. The Metamodel instance is then available for the migration process, for example to load a model instance with the correct .ecore


MMModel

The model migration model (MMModel) is migration specific representation of a model instance conforming to one of the releases of a metamodel.

The basic idea is to group instances, attributes and references together. So a change can easily iterate over the Instances and change (In one of the changes which affects the MMModel) for example the Type of an Instance in the MMModel. Later on as we will see with the converters, the 'migrated' MMModel will be serialized back into a regular Ecore model instance and can be persisted.  

The following entities exist (Somewhat simplified).
  • Model => The MMModel 
  • Instance => Each EObject in the actual model instance will have an Instance with a Type
  • Type => Each Instance has a Type. A Type has an eClass which corresponds to the original eClass of the EObject
  • AttributeSlot => Each attribute in EObjects has an AttributeSlot with the EStructuralFeature and EJavaObject as values. 
  • ReferenceSlot => Each reference in EObjects has a ReferenceSlot with the EStructuralFeature and Instance pointed to by this ReferenceSlot

Reconstructors


As said the reconstructors take the .history and allow to 'build' a certain release of the Meta model and instance model. The Migrator uses the EcoreForwardReconstructor (As models typically age and need to be migrated forward).

Now the EcoreForwardReconstructor extends the CompositeReconstructorBase which delegates the reconstruction to one or more reconstructors declared with it. This is the typical delegation pattern to allow the reconstruction process to be extended. This is also exactly the way the Migrator works. The Migrator adds the MigrationReconstructor to the EcoreForwardReconstructor so delegation happens when needed.

As we will see in the Reconstruction Process, at some point in the reconstruction we will hit a Change. This definition knows many forms. (Many types of changes). In order to act appropriately on the Change type, the reconstructor typically implements a model object Switch.

In the Edapt case the History code generation produced the HistorySwitch which is extended by the various reconstructors to perform the appropriate action.

The MigrationReconstructorSwitch for example deals with specific Change implementations like an OperationChange and a MigrationChange to add or delete from the MMMeta or MMModel.


Mapping and resolving

Whenever a Migration kicks in it will create a Mapping instance which is initialized through all reconstructors and delegated reconstructors through the init(...) method of an reconstructor.  

The Mapping contains a TwoWayIdentityHashMap for mapping EObjects to each other.

One of the usages is to map Change elements to the created equivalent (The Ecore Metamodel) in memory of these Change elements.

In this case the history model is visited, starting with the initial Release. From this Release, the Ecore metamodel in the form of one or more EPackages is gradually build up to be the first release of the as intended by the history. (With corresponding nsURIs).

Then in subsequent releases and underlying changes, the reference EPackage is adapted gradually. When migrating the actual model instances (Which is only applicable for some changes), the model instance references to the Ecore model artifacts (EClass, EReference, EAttribute) are resolved from this very same mapping. It is therefor absolutely key, that the MMMeta and MMModel are loaded with the same EPackage from the 'extent'. 

Example:

When adding a new feature to a Class, in the mapping the target element from the Change
is looked up. and the new feature is added to the EClass (in the mapping). 

With the Mapping utility there is a  ResolverBase class to resolve elements from the mapping.  The ResolverBase has a special method named:

copyResolve

What it does is for an OperationChange in the History is to perform a copy of the model element but resolving from the mapping at the same time. It descends the hierarchy of features and resolves when an EClass package is of type ECorePackage.

Effectively what this means is that when the OperationChange is a  metamodel definition, the resolver starts resolving from the mapping, making sure the constructed Ecore metamodel is used.

Converters


The are two converters. 
  • ForwardConverter => Converts a ResourceSet to a MMModel
  • BackwardConverter => Converts from a MMModel to a ResourceSet
ForwardConverter

The Model model is populated in the order.

initElements(); (EClassifiers etc..).
initProperties() (EAttribute => AttributeSlot, EReference => Slot / ReferenceSlot).
initResources()

BackwardConverter

The ResourceSet and it's Resources are loaded in the order. 

initObjects(model);
ResourceSet resourceSet = initResources(model);
initProperties(model);

Persistence


Persistence is handled  through a utility class named Persistency. This utility is specialized to deal with the situation whereby loading and saving of Model instances respects the metamodel version for which the model should be loaded/saved.

One aspects of dealing with XMI serialized model is the potential dynamic nature of a Resource load implementation.  EMF support the dynamic creation of an EPackage based on the schemalocation attribute. The schemalocation attribute will potentially point to an instance of the .ecore which is constructed on a certain release of the history, so loading the resource, will auto-create an EPackage for that release. An EObject will have an eClass with parent EPackage for a certain Release of the history.

This is important, as various reflective functions which act on the EPackage should be acting on the exact intended EPackage 'version'.  I ran into this, when trying to copy a loaded model, with ECoreUtil (To load a copy in another Resource). The model had an EPackage which corresponded to the latest release, while the model itself was serialized with a previous release.

Edapt has encountered this issue and deals with this in the following manner; 


When loading a model it provides the EPackage to use from the MMMeta instance (See MMMeta Instance creation).  The EPackage is mapped to the nsURI of the model in the EPackageRegistry of the ResourceSet, so when the resource is loaded, it consults the EPackageRegistry and uses the EPackage instead of dynamically loading the EPackage.
Reconstruction Process
The reconstruction process 'visits' the History model hierarchy, it has hooks for the start and end of Releases and Changes.

When descending the history to the intended release, the reconstructor will delegate call change (CompositeReconstructor) which call the MigrationReconstructorSwitch, which 'switches' the Change. 

 
At a determined point the migration reconstructor loads the 'Model' model and a 'MetaModel' instance. This happens when the end of a Release is reached which is not the targeted release.

If the change is one of the types CompositeChangeMigrationChange or  InitializerChange, then the change also reconstructs the children of the specialized Change instances by the ForwardReconstrutor.

The Reconstrucion process can be represented as:

startHistory History
    startRelease Release
        for Release.changes()
            startChange Change
                startChange (CompositeReconstructor).
                switch Change
                    CompositeChange
                    MigrationChange
                    InitializerChange                 
            endChange change
    endRelease Release =>  (If the Release if the original release, load the model, see MigrationReconstructor )             
endHistory History


Persistence.saveModel();


The switchs process the following Change types:
 

caseAdd()
caseCreate()
caseDelete()
caseMove()
caseRemove()
caseSet()

caseMigrationChange()
caseOperationChange()


Conclusion


The Edapt Migrator and it's concepts have no more secrets! We explored the concepts and how they work. The migration process which couples Meta and Model is quiet impressive. In subsequent posts on Edapt I will elaborate on how to work with a non XMI Resource based Persistency, for example CDO.