Thursday, December 27, 2012

Rolling the Calendar

For NetXStudio , I needed a function which would split a period into sub-periods. The function needs to be generic, so I can pass in a Calendar field which would then split the Period object according to field value.

First some definitions:
  • The DateTimeRange object is a simple Java Object with set and get methods for two Date fields named 'begin' and 'end'. 
  • Day start is defined at 00:00h and the day ends at 23.59h (Yes there is one lost minute but OK for me). 
The Java Calendar  or better the GregorianCalendar is capable to roll forward and backward as a real Calendar would do. The method roll(...) is used for that. The Calendar can be forced into a specific position using the set(...) method.

My method to deal with at least Calendar.MONTH and Calendar.DAY_OF_MONTH ended using the method getActualMinimum(...) and getActualMaximum(...) to set the Calendar to the desired position. The desired positions are:
  • Calendar.MONTH => The first day of the current month. 
  • Calendar.DAY_OF_MONTH => The first hour of the day. 
Now to roll into these positions, I need to know the 'child' Calendar field, to be used in the getActualMin/Max() methods. By 'child' field I mean the following mapping:
  • Calendar.MONTH => Calendar.DAY_OF_MONTH
  • Calendar.DAY_OF_MONTH => Calendar.HOUR_OF_DAY
Calling getActualMinimum(Calendar.DAY_OF_MONTH) and setting this as the new Calendar position, would set the Calendar to the first day of the current month of the Calendar. Calling getActualMinimum(Calendar.HOUR_OF_DAY) and setting this on the Calendar would set it at the first hour of the day.

All fine... The Unit test on a 6 month period which would be asked to split it in first MONTH_OF_YEAR, and subsequently DAY_OF_MONTH generated the intended result nicely.

Full: From: 27-06-2012 @ 00:00 To: 27-12-2012 @ 00:00
 Month: From: 01-12-2012 @ 00:00 To: 27-12-2012 @ 23:59
   Day: From: 27-12-2012 @ 00:00 To: 27-12-2012 @ 23:59
   Day: From: 26-12-2012 @ 00:00 To: 26-12-2012 @ 23:59
   Day: From: 25-12-2012 @ 00:00 To: 25-12-2012 @ 23:59
   Day: From: 24-12-2012 @ 00:00 To: 24-12-2012 @ 23:59
   Day: From: 23-12-2012 @ 00:00 To: 23-12-2012 @ 23:59
   Day: From: 22-12-2012 @ 00:00 To: 22-12-2012 @ 23:59
   Day: From: 21-12-2012 @ 00:00 To: 21-12-2012 @ 23:59

Being confident about my code, I expanded it by adding an option to split the Period in weeks. This is were trouble started. First I had to pick the 'child' field for the day of the week for the 'parent' field  Calendar.WEEK_OF_YEAR.

I quickly choose to use Calendar.DAY_OF_WEEK for this, makes sense right? Well that's what I initially thought. The result for setting the Calendar to the getActualMinimum(Calendar.DAY_OF_WEEK) actually didn't set the Calendar position to what I expected. I took me a while to figure out why. The reason is best explained with the following diagram:




In the diagram, the Calendar is set to today (Which happens to be the publication date of this blog-post, but this purely a coincidence, trust me on this). Now calling getActualMinimum with my freshly developed method for Calendar.DAY_OF_MONTH and Calendar.HOUR_OF_DAY works as expected. the Calendar rolls to the intended position. For Calendar.DAY_OF_WEEK) however, it doesn't. It actually roles forward! What I really expected was the new Calendar position to be the previous Sunday at least. Which was not even correct, as it should be the Monday for my Local.

mmmh... It triggered my curiosity (Ok, first there was a few moments of programmers frustration..). Why does it do that. I read about Calendar and noticed a bit of documentation on Calendar field conflicts. Suddenly I realized I should not use the actualMinimum for rolling the week day. What I should use is getFirstDayOfWeek() and set this on the calendar.

Ok, but this means that depending on the field, my method would need to act differently, which was not my initial goal. Sofar I haven't found another solution with the current GregorianCalendar capabilities. I even had to implement a function getLastDayOfWeek(Calendar cal) to set the end boundary of the week.  I encourage any reader to comment and let me know a better solution. (Note: I know about date/time libraries, but without them, can you make this better?).

Here is the code for the final solution:

public List periods(DateTimeRange dtr, int calField) {

  boolean weekTreatment = false;

  int childField = -1;
  switch (calField) {
  case Calendar.MONTH: {
   childField = Calendar.DAY_OF_MONTH;
  }
   break;
  case Calendar.DAY_OF_MONTH: {
   childField = Calendar.HOUR_OF_DAY;
  }
   break;
  case Calendar.WEEK_OF_YEAR: {
   childField = Calendar.DAY_OF_WEEK;
   weekTreatment = true;
  }
   break;
  }

  List result = Lists.newArrayList();

  if (childField == -1) {
   result.add(dtr);
   return result;
  }

  final Calendar cal = GregorianCalendar.getInstance();
  cal.setTime(dtr.getEnd().toGregorianCalendar().getTime());

  // An end calendar to compare the calendar field, and not take the field
  // maximum but the value from the end calendar.
  final Calendar endCal = GregorianCalendar.getInstance();
  endCal.setTime(dtr.getEnd().toGregorianCalendar().getTime());

  // Go back in time and create a new DateTime Range.
  do {

   // Set the begin to the actual minimum and end to the actual
   // maximum, except at the start, where we keep the actual.
   // At the end, roll one beyond the minimum to set the new actual.
   if (cal.get(calField) != endCal.get(calField)) {
    if (weekTreatment) {
     // :-( there is no method to get the last day of week.
     cal.set(childField, getLastDayOfWeek(cal));

    } else {
     cal.set(childField, cal.getActualMaximum(childField));
    }

   }

   final Date end = cal.getTime();

   // Special Treatment for Week
   if (weekTreatment) {
    final int firstDayOfWeek = cal.getFirstDayOfWeek();
    cal.set(Calendar.DAY_OF_WEEK, firstDayOfWeek);
   } else {
    int minimum = cal.getActualMinimum(childField);
    cal.set(childField, minimum);
   }

   Date begin;
   if (cal.getTime().getTime() < dtr.getBegin().toGregorianCalendar()
     .getTimeInMillis()) {
    begin = this.fromXMLDate(dtr.getBegin());
   } else {
    begin = cal.getTime();
   }

   final DateTimeRange period = period(this.adjustToDayStart(begin),
     this.adjustToDayEnd(end));
   result.add(period);

   // Role back one more, so the new actual can be applied.
   // This will cause the
   cal.add(calField, -1);
  } while (cal.getTime().getTime() > dtr.getBegin().toGregorianCalendar()
    .getTimeInMillis());

  return result;

 }


Notes:

  1. Usage of Google Collect to produce collections.
  2. The DateTimeRange type holds a 'begin' and 'end' member fields. 
  3. The period(Date begin, Date end) is a factory for an instance of of type DateTimeRange. 
  4. The method ignores the beginning of the period, which precedes the intended boundary.
  5. The method getLastDayOfWeek(Calendar cal) looks like this:

public int getLastDayOfWeek(Calendar cal) {
  final int firstDayOfWeek = cal.getFirstDayOfWeek();

  final int lastDayOfWeek;
  if (firstDayOfWeek != 1) {
   lastDayOfWeek = firstDayOfWeek - 1; // One before the first day...
  } else {
   lastDayOfWeek = cal.getActualMaximum(Calendar.DAY_OF_WEEK); // Expect
  }
  return lastDayOfWeek;
 }


Thursday, November 22, 2012

Lazy loading CDO backed JFace Viewer.

This post is about achieving a good user experience in a User Interface while presenting a potentially large set of data which is retrieved from a back-end system. In our case the UI is an Eclipse RCP Application, which uses JFace Viewers and the Back-end system is Connected Data Objects, in short, CDO. CDO is an object-persistence middleware system based on the Eclipse Modeling Framework (EMF).

Objective

The objective is to load data in a UI Widget "Just in time". In this case it means, whenever data becomes visible. The Eclipse JFace library as such a facility for scrolling through a TableViewer or TreeViewer. For this to work the following conditions need to be met:
  1. The Content Provider needs to be an ILazyContentProvider for TableViewers and a ILazyTreeContentProvider for TreeViewers. 
  2. As data is fetched dynamically the content provider is not knowledgeable about the number of items in the viewer. Therefor it is required to call setItemCount(...) on the viewer. 
  3. The TableViewer should be created with the SWT.VIRTUAL style to signal that the viewer should be updated only when data becomes visible.
An example of such a content provider implementation could look like this
public class LazyListContentProvider implements ILazyContentProvider {
 
 private Viewer viewer;
 private List content;

 public void dispose() {

 }

 public void inputChanged(Viewer viewer, Object oldInput, Object newInput) {
  this.viewer = viewer;
  this.content = (List) newInput;
 }

 public void updateElement(int index) {

  if (viewer instanceof TableViewer) {
   ((AbstractTableViewer) viewer).replace(content.get(index), index);
  } else {
   throw new UnsupportedOperationException(
     "Only table viewer is supported");
  }
}


Here updateElement(int index) will be called, whenever more items are needed which stems from the use scrolling the viewer down and revealing more rows.

The code for the table viewer would something like this

tblViewer = new TableViewer(frmTolerances.getBody(),
SWT.BORDER | SWT.MULTI | SWT.FULL_SELECTION |SWT.VIRTUAL| widgetStyle);

tblViewer.setItemCount(((CDOResource)resource)     .eContents().size());

tblViewer.setContentProvider(new LazyListContentProvider());
tblViewer.setLabelProvider(....any label provider...);
tblViewer.setInput(toleranceResource.getContents());

Filters, Sorting and History


Now there is one limitation with using an ILazyContentProvider. It's not capable to deal with the sorting and filtering capabilities of a JFace Viewer. Why not? Well, the viewer doesn't have access to to the complete set of items, and as both Sorting and Filtering require the full set of items, it won't work.

However there is a potential solution to enable these functions, and if you are a user of Eclipse, the chance is you use this already on a daily basis. The same dilema exited for finding types in the IDE and other search operations which require large sets of data to be presented, filtered, remembered and sorted. The implementation of this function is the FilteredItemSelectionDialog. Now if we look at this class we notice that it complies to all requirements for Lazy loading data. It uses a content provider which implements the ILazyContentProvider. It acts on a TableViewer with flag SWT.Virtual and sets the item count. The really nifty thing however is all the additional facilities available as well.

As an example and teaser here is a screenshot of a view (In this case in an Eclipse Form) uses the base class referred further down.




Filtering 

The concept of filtering in the FilteredItemSelectionDialog should be decomposed, to reflect what happens when filtering.

The dialog has a text / search entry widget which allows a filtering pattern to be applied on the data set. The way it works is by adding items to a dedicated filtered items collection which match the filter criteria. For this the Content Provider needs to be fed with items when realized. As the Dialog is abstract, a concrete implementation will implement the methods:
@Override
  protected void fillContentProvider(
    AbstractContentProvider contentProvider,
    ItemsFilter itemsFilter, IProgressMonitor progressMonitor)
    throws CoreException {
  ...
  }

 @Override
  protected ItemsFilter createFilter() {
   return null;
...
  }

The concrete implementation method will feed the content provider with the data, considering the provided filter. Note that the filter is applied already when realizing the Dialog. the method which initiates activities is applyFilter() . It is also called when the pattern matching input text changes.

applyFilter() will invoke a sequence of background activities, which use the Eclipse Job API. The sequence is:

(1) applyFilter()
           |___ (1a) createFilter()
           |
(2) FilterHistoryJob -- (4) RefreshCacheJob -- (5) RefreshJob
           |
(3) FilterJob -- (4) RefreshCacheJob -- (5) RefreshJob
         
(6) RefreshProgressMessageJob

1a ItemsFilter

The ItemsFilter is abstract and should be implemented by Clients. A typical implementation will extract a relevant item attribute like the "name" of the item and match it against the filter.
This is as easy as calling the .matches(String) method on the ItemsFilter class.

Note that the ItemsFilter can be instantiated with a custom SearchPattern(int rule) class. The SearchPattern can hold various matching rules like exact matching or pattern matching and case sensitive matching.

2 FilterHistoryJob

First, calls the createFilter() method, which will call the concrete implementation. Next it will invoke a background Job which populates the items with a potential history. (Still considering the filter). This job is named: FilterHistoryJob. The history is managed by an abstract SelectionHistory class, which clients should extend. The purpose the ability to customize the serialization to XML for the items which need to be remembered. Items can be added to the History as we wish. In the case of the FilteredItemSelectionDialog, selected items are remembered when OK is pressed. This will actually only happen when the SelectionHistory has been set using:


.setSelectionHistory(SelectionHistory);


In our implementation, As we deal with CDO Objects, we use the unique CDOID as a serialization option.

It will already refresh the viewer with the result from History if any and if the filter matches the history item,  but will also as a last step invoke the next Job which is the FilterJob.

3 FilterJob

The FilterJob will do the actual filtering based on the current ItemsFilter as created in the first step. This is delegated to the method filterContent() in the job.

4 RefreshCacheJob

The RefreshCacheJob will refresh the viewer with the result of the filtered items in two steps. First the cache is is refreshed by the procedure described here:

The filtered items is a result collection of several actions:

  1. First the current items are sorted first using a Client specified Comparator. 
  2. Next a potential additional ViewerFilter is applied. The ViewerFilter(s) can be defined by calling addFilter(ViewerFilter).  
  3. Finally a Separator is inserted in the cache between the Historical items and any potential item not in history which as previously made it through the Item filter and the Additional ViewerFilter(s) 

The second and last step is to invoke another job which is the RefreshJob

5 RefreshJob

This is a UIThread Job as it is actually refreshing the TableViewer. As we use a ILazyContentProvider, what happens is to set the item count on the table viewer with method setItemCount(). Also the selection is saved and restored after the TableViewer refresh. Note that a refresh will trigger the ILazyContentProvider update method, which will in turn replace the viewer items with the cached items. 

6 RefreshProgressMessageJob

This job refreshes the loading progress, it's cyclical so it schedules itself every 500 milliseconds while the progress is not cancelled or done. 

Applying the method on a lazy loading 

So now we understand the FilteredItemSelectionDialog inner workings, we can apply this same technique to a UI Component which is not a Dialog, but perhaps a ViewerPart which presents CDO Data.  The steps to do so are:

  1. Extract the relevant code from FilteredItemSelectionDialog
  2. Change the algorithm for populating the initial items list. 

Refactor FilteredItemSelectionDialog 

There is very various things we want to do:

  • Remove Dialog specifics 

Here we want to remove all the Dialog specific stuff, and refactor this is as a "regular" Eclipse UI Component. Various aspects like Saving Dialog Settings, and handling button selection are not required so can be removed. 
  • Support for multiple columns in the TableViewer
Also we want to support multiple columns. For this we need to provide a hook for clients to create the columns, additionally we want the default implementation of the Label Provider. (Which is very sophisticated), to also support multiple columns. This is done by supporting the ITableLabelProvider


With a bit of work, you could get an abstract like this: AbstractLazyTableViewer

This is merely an example, but it is self-contained and should work for clients extending it. 
Note 1: The refactored version assumes it will be based in an Form. 

Algorithm for populating the inital items list

The implementation will, without adaptations, populate the entire content in the viewer, if a pattern like "?" is entered. Although an ILazyContentProvider is used, it's not acting entirely as we might expect. What happens is that the method fillContentProvider expect the following method to be called:

public void add(Object item, ItemsFilter itemsFilter)

Now that's Ok for a data source which is local, but in case of a remote CDO Repository, it forces hard labour instead of laziness. So we need to come up with a different solution. Unfortunately I am running out of time now.. so perhaps thought for another post. 

Tuesday, January 3, 2012

Xtext Connected scoping - CDO and Xtext 2.0

Just last year I ventured into integrating an expression language which can be used to perform operations on parts of an EMF model. The building blocks were decided to be Xtext and CDO, both based on EMF but not interworking out of the box.

There are likely several ways in which Xtext and CDO could integrate. The serialization of an Xtext model into a CDO resource could be thought of. What I needed was different. I wanted to be able to execute expressions on objects stored in a CDO repository. The solution: We would use a regular Xtext resource, but would need to reference other objects stored in CDO. The serialized version of our Xtext resource would need to be stored somewhere as well. For this I decided to simply store it as a String in one of the features of CDO model.


In the diagram here, it shows the idea.



Model A and B are Ecore models which have been adapted to work with CDO. In order for EMF models to benefit from the CDO capabilities, CDO can convert an EMF .genmodel so it can generate CDO optimized java.

The Script.xtext is an Xtext grammar, which implements an expression DSL,  It does arithmetic operations like + - * /.

The Script.xtext grammar includes import statements for model A and B, so the grammar rules reference these external models in the defined grammar expressions. Xtext terminology for these references is a "cross-link".

In Xtext referencing external model objects is a well known pattern which is documented in the great  Xtext Documentation. I won't go into how this is done exactly, but it boils down to telling the workflow where to find the A and B model.

The cross-links pattern, is also explained in the documentation. This is the interesting part. The formal grammar for cross links is:

CrossReference :
  '[' type=ReferencedEClass ('|' terminal=CrossReferenceTerminal)? ']'
;

In our Script.xtext we use cross-links linking to the external models A and B in similar fashion.
Behind the scene Xtext will resolve the links in the so-called linking process. By default Xtext generates a linker named LazyLinker, which works just fine with CDO based resources. the second step in linking is the definition of the linking semantics. The Xtext linker is lazy, it links the object when needed. The proxy URI which is created and set. An Xtext proxy uri looks like this.


"xtext resource.extension"#xtextLink_::0.3.0.0::3::/4


See the class LazyURIEncoder [src] for details on the coding technique. Next the Xtext Scoping API comes in to play and has been the focus of most of my efforts.

In Xtext scoping is either local or global. The local scope is from the own grammar definition, and global is scoping from external models. We are especially interested in the IGlobalScopeProvider. It is the scope provider's responsibility to return an IScope for a given EReference, in our case a cross-link to either model  A or B.

Xtext ships with two GlobalScopeProvider implementations, one is based on explicitly referencing resources by a grammar naming convention. This is the ImportUriGlobalScopeProvider. The other is the DefaultGlobalScopeProvider, which leverages the Java class path to find resources.

What I needed was a GlobalScopeProvider which was none of these two, but would rather use a fixed set of CDO resources as the base to determine the applicable IScope. The CDO resources holding objects from package A and B, were well known to me and resolvable by a URI, so I looked for a way to use  the CDO resource URI as a base for building the IScope. The set of CDO resource URI's in my case is fixed, so I could simply hardcode the URI's in the IGlobalScopeProvider.

With the highly customizable nature of Xtext, it was rather straighforward to replace/enhance the following components.

IResourceServiceProvider

Here I simply enhanced the DefaultResourceServiceProvider to override the method canHande(URI uri)
This method needs to support CDO URI's which look like this:

cdo://"repo name"/folder/resource

In my implementation, I look for a URI scheme starting with "cdo".
The CDO  implementation of the ServiceProvider can be installed in the xxxRuntimeModule Guice class.

public Class<? extends IResourceServiceProvider> bindIResourceServiceProvider() {

   return CDOResourceServiceProvider.class;

}

So this service provider can now handle CDO URI's, next the actual GlobalScopeProvider is needed.


IGlobalScopeProvider



In the CDOGlobalScopeScopeProvider we need to return an IScope based on the requested context.
Our implementation uses the EClass of the referenced type in package A or B, to load the CDO resource.

When initializing the CDOGlobalScopeProvider we build a map of EClass to CDO resource URI(s), which is queried when an IScope is requested. The target EClass is derived from the EReference which is part of the API call for the IGlobalScopeProvider as seen in this method signature.

IScope getScope(Resource context, EReference reference, Predicate<IEObjectDescription> filter);

when the CDO URI is retrieved from the Map, we then invoke (from our customCDO IResourceDescriptions) :

IResourceDescription description = descriptions.getResourceDescription(uri);

IResourceDescriptions

As we don't want to load the CDO resource each time an IScope is requested, we use an index for the IResourceDescription produced by the IResourceDescription.Manager, in our IResourceDescriptions implementation.

Xtext comes with a SimpleCache implementation which can be used for that. Additionally we want the index to be updated when the CDO models change, so I borrowed some of the Dawn CDO listeners, boiling dawn to an implementation of the following method:


public void handleViewInvalidationEvent(CDOViewInvalidationEvent event);


from this, I could tell the CDO URI / IResourceDescription index to clean the URI entry for the CDO event dirty objects or any other objects invalidated by the event. The next time an IScope would be requested for such a URI, the cache would simply rebuild the IResourceDescription

Conclusions so far. 

I am aware this blog doesn't provide the details and actual implementation for people to re-use. The reason is that the code is pretty specific for my CDO models, hence my implementation of the CDOGlobalScopeProvider map which tells me in which CDO URI, an EClass resides is really build for my CDO model and not re-usable in a generic way.


private Map<EClass, List<URI>> eClassToURIMap;

Additionally, I have some custom code to open a CDO session and view, to actually retrieve the CDO resource, this won't be re-usable directly in any other app.

I still wanted to share this experience, and simply state CDO and Xtext work great together! I am well inclined to share more details to anyone interested to learn from this experience.

Next steps

It's pretty cool to commit an object in CDO and seconds later see it appear at code completion in an Xtext editor! however.

  1. For larger resources, building the IResourceDescription is time consuming, and it will take some time to build. It becomes important how CDO Resources are stuffed with objects of certain types. In my experience, at some point CDO resources have to be chopped in CDO folders etc... to make them smaller units which can be worked with. 
  2. related to 1), the index should actually be build in application idle time, and ready for use whenever I want to link or get a proposal for a CDO cross-refed object from Xtext.  A solution could be a background job doing this, similar to the Xtext builders which do this in background. 
  3. I find it's needed to sometimes to limit the scope of exposed CDO objects by a certain context. I haven't figured out exactly if this should be done in the ProposalProvider, by the generated DeclarativeScoping or in the CDO global scope provider. I would appreciate some advise on this. 
  4. Perhaps with some help of the Xtext and CDO guys, we could turn this into an off-the-shelf Xtext fragment, I am personally a bit disappointed by the lack of focus of Xtext for runtime RCP apps, although the way it's build up with Guice, makes it super easy to replace or leave out capabilities.