Using the Ext JS PivotGrid

July 29, 2010 by Ed Spencer · 2 Comments 

One of the new components we just unveiled for the Ext JS 3.3 beta is PivotGrid. PivotGrid is a powerful new component that reduces and aggregates large datasets into a more understandable form.

A classic example of PivotGrid’s usefulness is in analyzing sales data. Companies often keep a database containing all the sales they have made and want to glean some insight into how well they are performing. PivotGrid gives the ability to rapidly summarize this large and unwieldy dataset – for example showing sales count broken down by city and salesperson.

A simple example

We created an example of this scenario in the 3.3 beta release. Here we have a fictional dataset containing 300 rows of sales data (see the raw data). We asked PivotGrid to break the data down by Salesperson and Product, showing us how they performed over time. Each cell contains the sum of sales made by the given salesperson/product combination in the given city and year.

Let’s see how we create this PivotGrid:

var SaleRecord = Ext.data.Record.create([
    {name: 'person',   type: 'string'},
    {name: 'product',  type: 'string'},
    {name: 'city',     type: 'string'},
    {name: 'state',    type: 'string'},
    {name: 'month',    type: 'int'},
    {name: 'quarter',  type: 'int'},
    {name: 'year',     type: 'int'},
    {name: 'quantity', type: 'int'},
    {name: 'value',    type: 'int'}
]);

var myStore = new Ext.data.Store({
    url: 'salesdata.json',
    autoLoad: true,
    reader: new Ext.data.JsonReader({
        root: 'rows',
        idProperty: 'id'
    }, SaleRecord)
});

var pivotGrid = new Ext.grid.PivotGrid({
    title     : 'Sales Performance',
    store     : myStore,
    aggregator: 'sum',
    measure   : 'value',

    leftAxis: [
        {dataIndex: 'person',  width: 80},
        {dataIndex: 'product', width: 90}
    ],

    topAxis: [
        {dataIndex: 'year'},
        {dataIndex: 'city'}
    ]
});

The first half of this ought to be very familiar – we just set up a normal Record and Store. This is all we need to load our sample data so that it’s ready for pivoting. This is all exactly the same code as for our other Store-bound components like Grid and DataView so it’s easy to take an existing Grid and turn it into a PivotGrid.

The second half of the code creates the PivotGrid itself. There are 5 main components to a PivotGrid – the store, the measure, the aggregator, the left axis and the top axis. Taking these in turn:

  • Store – the Store we created above
  • Measure – the field in the data that we want to aggregate (in this case the sale value)
  • Aggregator – the function we use to combine data into the cells. See the docs for full details
  • Left Axis – the fields to break data down by on the left axis
  • Top Axis – the fields to break data down by on the top axis

The measure and the items in the axes must all be fields from the Store. The aggregator function can usually be passed in as a string – there are 5 aggregator functions built in: sum, count, min, max and avg.

Renderers

This is all we need to create a simple PivotGrid; now it’s time to look at a few more advanced options. Let’s start with renderers. Once the data for each cell has been calculated, the value is passed to an optional renderer function, which takes each value in turn and returns another value. One of the PivotGrid examples shows average heights in feet and inches but the calculated data is in decimal. Here’s the renderer we use in that example:

new Ext.grid.PivotGrid({
    store     : myStore,
    aggregator: 'avg',
    measure   : 'height',

    //turns a decimal number of feet into feet and inches
    renderer  : function(value) {
        var feet   = Math.floor(value),
            inches = Math.round((value - feet) * 12);

        return String.format("{0}' {1}\"", feet, inches);
    },
    //the rest of the config
});

Customising cell appearance

Another one of the PivotGrid examples uses a custom cell style. As with the renderer, each cell has the opportunity to alter itself with a custom function – here’s the one we use in the countries example:

new Ext.grid.PivotGrid({
    store     : myStore,
    aggregator: 'avg',
    measure   : 'height',

    viewConfig: {
        getCellCls: function(value) {
            if (value < 20) {
                return 'expense-low';
            } else if (value < 75) {
                return 'expense-medium';
            } else {
                return 'expense-high';
            }
        }
    },
    //the rest of the config
});

Reconfiguring at runtime

A lot of the power of PivotGrid is that it can be used by users of your application to summarize datasets any way they want. This is made possible by PivotGrid’s ability to reconfigure itself at runtime. We present one final example of a PivotGrid that can be reconfigured at runtime. Here’s how we perform the reconfiguration:

//the left axis can also be changed
pivot.topAxis.setDimensions([
    {dataIndex: 'city', direction: 'DESC'},
    {dataIndex: 'year', direction: 'ASC'}
]);

pivot.setMeasure('value');
pivot.setAggregator('avg');

pivot.view.refresh(true);

It’s easy to change the axes, dimension, aggregator and measure at any time and then refresh the data. The calculations are all performed client side so there is no need for another round-trip to the server when reconfiguring. The example linked above gives an example interface for updating a PivotGrid, though anything that can make the API calls above could be used.

I hope you enjoy the new components in this Ext JS 3.3 beta and look forward to comments and suggestions. Although we’re only at beta stage I think the additions are already quite robust so feel free to stress-test them.

Offline Apps with HTML5: A case study in Solitaire

June 21, 2010 by Ed Spencer · 3 Comments 

One of my contributions to the newly-launched Sencha Touch mobile framework is the Touch Solitaire game. This is not the first time I have ventured into the dizzying excitement of Solitaire game development; you may remember the wonderful Ext JS Solitaire from 18 months ago. I’m sure you’ll agree that the new version is a small improvement.

Solitaire is a nice example of a fun application that can be written with Sencha Touch. It makes use of the provided Draggables and Droppables, CSS-based animations, the layout manager and the brand new data package. The great thing about a game like this though is that it can be run entirely offline. Obviously this is simple with a native application, but what about a web app? Our goal is not just having the game able to run offline, but to save your game state locally too.

The answer comes in two parts:

Web Storage and the Sencha data package

HTML5 provides a brand new API called Web Storage for storing data locally. You can read all about it on my Web Storage post on Sencha’s blog but the summary is that you can store string data locally in the browser and retrieve it later, even if the browser or the user’s computer had been restarted in the meantime.

The crucial part of the sentence above is that we can only store string data. In the case of a game of Solitaire we need to store data on the elapsed time and number of moves as well as the location and status of each card. This doesn’t sound like the kind of data we want to manually encode into a string, so thankfully the data package comes to the rescue.

The Sencha Touch data package is a complete rewrite of the package that has been so successful in powering Ext JS 3.x. It shares many of the same philosophies and adds the learning we have gained from developing Ext JS 3.x over the past year. One of the new capabilities it offers us is a Local Storage proxy, which automatically marshalls your model data into local storage and transparently restores it when you need it.

Using the new proxy is simple – all we need to do is set up a new Store, specifying the Proxy and the Model that will be saved to it. Models are the spiritual successor to Ext JS 3.x’s Records. Now whenever we add, remove or update model instances in the store they are automatically saved to localStorage for us. Loading the store again is equally easy:

//set the store up
var gameStore = new Ext.data.Store({
    proxy: new Ext.data.LocalStorageProxy({
        id: 'solitaire-games'
    }),
    model: 'Game'
});

//saves all outstanding modifications, deletions or creations to localStorage
gameStore.sync();

//load our saved games
gameStore.read({
    scope: this,
    callback: function(records) {
        //code to load the first record
    }
});

And just like that we can save and restore games with Web Storage. We can visit our app’s webpage and start a game then come back later and find it automatically restored. But we still can’t play offline, for that we need the application cache.

The HTML5 Application Cache Manifest

The application cache is one of the best features of HTML5. It provides a simple (though sometimes frustrating) way of telling the browser about all of the files your application relies on so that it can download them all ready for offline use. All you have to do is create what’s known as a manifest file which lists all of the files the application needs – the Solitaire manifest looks like this:

CACHE MANIFEST
#rev49

resources/icon.png
resources/loading.png

resources/themes/wood/board.jpg
resources/themes/wood/cards.png

resources/css/ext-touch.css
resources/solitaire-notheme.css
resources/themes/wood/wood.css
resources/themes/metal/metal.css

ext-touch-debug.js
solitaire-all-debug.js

We tell the browser about the manifest file by pointing to it in the tag’s manifest atttibute. When the browser finds this file it downloads each of the listed assets so that they are ready for offline consumption. Note that it does not automatically include them on the page, you still need to do that yourself via the usual link and script tags. Here’s a snippet of the Solitaire index.html file:

<!doctype html>
<html manifest="solitaire.manifest">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title>Solitaire</title>

        <link rel="stylesheet" href="resources/css/ext-touch.css" type="text/css">
        <link rel="stylesheet" href="resources/solitaire-notheme.css" type="text/css">
        <link rel="stylesheet" href="resources/themes/wood/wood.css" type="text/css">

        <script type="text/javascript" src="ext-touch-debug.js"></script>
        <script type="text/javascript" src="solitaire-all-debug.js"></script>

Note the manifest file definition in the html element at the top, and the fact that we still include our page resources the normal way. It sounds easy, but without a little setup first it can be a very frustrating experience. Usually your browser will try to cache as many files as possible, including the manifest file itself – we don’t want this. As soon as your browser has a long-term cache of the manifest file it is extremely difficult to update your application – all of the files are already offline and won’t be updated, and the browser won’t even ask the server for an updated manifest file.

Preventing this behaviour turns out to be fairly easy, and the solution in its simplest form comes in the shape of a .htaccess file with contents like the following:

<Files solitaire.manifest>
    ExpiresActive On
    ExpiresDefault "access"
</Files>

This directs Apache to tell the browser not to cache the manifest file at all, instead requesting the file from the server on every page load. Note that if the device is currently offline it will use the last manifest file it received.

This is half the battle won, but let’s say you change one of your application files and reload – you’ll find nothing happened. This is because when your browser asked the server for the manifest file it actually asked if the file had changed or not. As the manifest itself wasn’t updated, the server responds with a 304 (Not Modified) and your browser keeps the old file.

To make the browser pick up on the change to the application file you need to update the manifest file itself. This is where the mysterious “#rev49″ comes in on the manifest example file above. This is a suggestion from the excellent diveintohtml5 article on the subject – whenever you change any application files just bump up the revision number in the manifest file and your browser will know to download the updated files.

One final detail is that your Apache server probably isn’t set up to server manifest files with the correct mime type, so be sure to add the following line to your Apache config and restart the server:

AddType text/cache-manifest .manifest

Wrapping it up

Offline access is a big deal for mobile apps and Sencha Touch makes them much easier to write. The benefit is not so much that the apps can run without an internet connection (many modern touch devices have a near-permanent connection to the internet already), but that web apps can now be treated as first-class citizens alongside native apps.

The fact that many devices allow your users to save your app to their home screen and load it as though it were native is an important step – you keep all of the advantages of web app deployment while gaining some of the benefits of native apps. As more and more native hardware APIs become available to web apps their importance will only grow.

If you want to check out Solitaire’s offline support for yourself visit the application’s site and save it to your iPad’s home page. Try turning on airplane mode and loading the app and see how it behaves as though it were native. If you don’t have an iPad, you can load the app in up-to-date versions of Chrome or Safari and get a similar experience.

Writing Compressible JavaScript

June 19, 2010 by Ed Spencer · 8 Comments 

Writing a library is a balancing act between the (sometimes competing) interests of API clarity, code clarity, performance and compressibility. In this article I’m going to detail three of the approaches we take to meet this balance and suggest them for your own usage.

1. Collecting var statements

Every time we declare variables we add 4 bytes to the compressed file size. Variables are declared sufficiently often that this can really add up, so instead of this:

var myFirstVar = 'something';
var myOtherVar = 'another thing';
var answer = 42;
var adama = true;

One should use this form:

var myFirstVar = 'something',
    myOtherVar = 'another thing',
    answer = 42,
    adama = true;

When this code is compressed, each variable name above is turned into a single-letter name, meaning that the wasted 4 bytes per useless additional ‘var ‘ in the first example would have contributed significantly to code size with no benefit.

2. Local variable pointers to object properties

The following code (pruned from a previous version of Ext JS) is not as compressible as it could be:

var cs = {};
for (var n in this.modified) {
    if (this.modified.hasOwnProperty(n)) {
        cs[n] = this.data[n];
    }
}
return cs;

We’re better off aliasing ‘this.modified’ to a local variable first. Aside from the performance benefits some JS engines derive from not having to perform object property lookups over and over again, we save precious bytes this way too:

var modified = this.modified,
    changes  = {},
    field; 

for (field in modified) {
    if (modified.hasOwnProperty(field)) {
        changes[field] = this.data[field];
    }
}

return changes;

Again, the minifier will compress those variable names down to a single character each, so for our ‘this.modified’ example we’re going to use 15 bytes to define the variable plus 1 byte each time we use it (totalling 15), vs the 26 bytes for that code previously. This approach scales especially well – were we to refer to this.modified a third time in the function, as our code will now minify to 16 bytes, vs 39 without the variable declaration.

Side note: in the first example here the variable ‘n’ was being used in the for…in loop. We can always safely exchange that for a meaningful variable name (in this case ‘field’) and leave the rest to the minifier.

3. Aliasing ‘this’ to ‘me’

In Ext.data.Record’s markDirty method we have the following code:

this.dirty = true;
if(!this.modified){
    this.modified = {};
}
this.fields.each(function(f) {
    this.modified[f.name] = this.data[f.name];
},this);

There are 7 references to ‘this’ in that method, taking 28 bytes and completely incompressible. We could rewrite it like this (note that the final ‘this’ can be removed in this format):

var me = this; 

me.dirty = true;
if (!me.modified) {
    me.modified = {};
}
me.fields.each(function(f) {
    me.modified[f.name] = me.data[f.name];
});

Again, our minifier will change the ‘me’ var to a single character, saving us 8 bytes in this instance. That might not sound like a lot but after minification it equates to a 7% reduction in code size for this function.

In each of the cases above we’re generally talking about single-digit percentage savings after minification. There is value in that small slice though, especially as more and more applications shift onto bandwidth-constrained mobile platforms.

The first two approaches are no-brainers and must always be done but the third is slightly more controversial. Personally I find it makes the code a little harder to read, largely because my syntax highlighter doesn’t recognise that ‘me’ is now the same as ‘this’. Its value also varies significantly by function – some functions can contain over a dozen references to ‘this’, in which case this approach makes a big difference.

Ext JS 3.2 beta out today

March 9, 2010 by Ed Spencer · 9 Comments 

We pushed out a beta release of Ext JS 3.2 this morning. Although we’ve marked it as beta, it’s a pretty solid release and we expect to release a final version shortly. The DataView transitions are especially fun – watch this space for a fuller example…

Here’s a quick rundown of the features we added:

One of the big projects we’ve undertaken that most people probably won’t find so exciting is ramping up our internal QA efforts. Our unit test coverage has increased dramatically in the past couple of months, and we’ve built infrastructure to run all of our tests on every browser/OS we support in a fully automated fashion. Doing TDD on Ext JS is an awesome feeling.

I’ll talk more in the future about what we’re doing internally to ensure the quality of our code, framework performance and rendering.

Answering Nicholas Zakas’ JavaScript quiz

February 16, 2010 by Ed Spencer · Leave a Comment 

A current meme that’s floating around the JavaScript geek corner of the internet is setting quizzes on some of the more unusual aspects of JavaScript. This time round Nicholas Zakas is providing the entertainment, so I thought I’d provide some answers. Let’s get started:

Question 1

Question 1 looks like this:

var num1 = 5,
    num2 = 10,
    result = num1+++num2;

We’re asked what the values of result, num2 and num1 are. First, let’s deconstruct what that +++ is doing. There is no +++ operator in JavaScript – instead we have a num1++ followed by a + num2.

JavaScript has two ways of incrementing a number by 1 – we can either put the ++ before the variable or after it. The variable is incremented either way – the only difference is what is returned. ++10 returns 11, whereas 10++ returns 10:

var a = 10;

var b = a++; //a is set to 11 now, but b is set to 10
var c = ++a; //a is set to 12 now, c is also set to 12

So ‘result’ is the sum of num1++ (which is 5) and num2, which is 10, so result equals 15. num2 remains at 10 as it was not modified. num1 is now equal to 6 because we incremented it by 1, though the incrementation did not affect the sum passed to result.

Question 2

var x = 5,
    o = {
        x: 10,
        doIt: function doIt(){
            var x = 20;
            setTimeout(function(){
                alert(this.x);
            }, 10);
        }
    };
o.doIt();

We’re asked what is alerted. This is mostly smoke and mirrors – there’s some indirection with all the duplicate names but the important thing here is the setTimeout. The function we pass to setTimeout gets run in the global scope, meaning ‘this’ refers to the window object. Declaring x as a variable in the global scope (var x = 5) is the same as setting window.x = 5, so 5 is alerted.

Question 3

var num1 = "10",
    num2 = "9";

We’re asked:

  • What is the value of num1 < num2?
  • What is the value of +num1 < num2?
  • What is the value of num1 + num2?
  • What is the value of +num1 + num2?

This question is all about type casting.

  • num1 < num2 is true - if both operands are strings JavaScript will compare them alphabetically, and "10" is lower alphabetically than "9"
  • +num1 < num2 is false - placing a "+" operator before a string casts it into a number, so we're actually testing 10 < "9". When testing a mixture of numbers and strings like this, everything is cast into a number, so we're testing 10 < 9, which is false
  • num1 + num2 === “109″ – the plus sign can mean both addition and concatenation, depending on the operand types. Here we have 2 strings so we’re concatenating them together
  • +num1 + num2 === “109″ also – again we’re casting num1 into a number, but the + operator means concatenation if at least one operand is a string

The confusion around this comes largely from the fact that the plus sign is used for both addition and concatenation in JavaScript. This causes the engine to have to test the typeof each operand and cast accordingly. All of the other math operators (e.g. /, *, % etc) cast both operands to numbers.

Question 4

var message = "Hello world!";

We’re asked:

  • What is the value of message.substring(1, 4)?
  • What is the value of message.substr(1,4)?
    • substring and substr do similar things. The first argument to each is the character index to start from, but whereas substring’s second argument is the character index to end at, substr’s second argument is the number of characters to return. Therefore message.substr(1, 4) will return a string of length 4, whereas message.substring(1, 4) will return a string of length 3 (4 – 1):

      • message.substring(1, 4); //”ell”
      • message.substr(1, 4); //”ello”

      Question 5

      var o = {
              x: 8,
      
              valueOf: function(){
                  return this.x + 2;
              },
              toString: function(){
                  return this.x.toString();
              }
          },
          result = o < "9";
      alert(o);
      

      We’re asked the value of ‘result’, and what gets alerted. This requires an understanding of the special valueOf and toString functions available on every object. These functions are used internally by the JavaScript engine to pull out the best representation of an object’s value based on the situation.

      When alerting a value, we want a string representation so toString is called. When comparing the value to another object, valueOf is called instead. So alert(o) alerts “8″, and result is set equal to the result of 10 < "9". The JavaScript engine will decide when to use which option, or we can specify it ourselves:

      var num1 = 8, num2 = 9;
      
      num1 + num2; //17
      num1.toString() + num2.toString(); //"89"
      num1.valueOf() + num2.valueOf(); //17
      

      The 'result' assignment needs a little explanation. First, the engine calls valueOf on the object, which returns 10. Second, because one of the operands to the < operator is a number, the other is also cast into a number, so we are testing 10 < 9, which returns false. We could instead force it to use toString: o.toString() < "9" returns true.

      Quizzes like this are great for getting your teeth into some of the guts of JavaScript, but don't mistake them for a good way to write code. The point is to demonstrate how quirky JS code can be unless you write it in a sensible way.

Next Page »