Open Source Finance2014-11-30T04:28:24+00:00http://t2k.github.ioTed Killilealrrs259@gmail.comasync.parallel image processing2014-11-16T00:00:00+00:00http://t2k.github.io/server-side/2014/11/16/async-parallel-image-processing
<h3 id="processing-file-updloads-on-the-server">Processing file updloads on the server.</h3>
<p>Streams in NodeJS are incredibly powerful. I found out myself for a site I’m working on where users are allowed to upload images from their mobile devices. Images are typically sent from a camera photo but they can also be selected from a local gallery or clould storage. Desktop users can use the ‘file-open’ dialog.</p>
<p>The code below is a request handler written in coffee-script that handles a mult-part http POST (containing a file) from an Express route. Steams are modern, async and event-driven. Instead of writing temp files to the file-system, we stream the file into a <a href="http://mongodb.github.io/node-mongodb-native/markdown-docs/gridfs.html">mongoDB GridStore</a>. Streams are great for uploads, not only because a file doesn’t touch a disk file-system but they are processed in a non-blocking, asyncronous fashion. A stream doesn’t have to process a large file in one buffer either which allows the server to use memory more efficiently. With <a href="https://github.com/mscdex/busboy">busboy</a> the file’s stream <em>pipes</em> it’s way straight into mongoDB via <a href="https://github.com/aheckmann/gridfs-stream">gridfs-steam</a>). </p>
<p>My original goal was to upload a file into a mongoDB GridStore and I was gonna stop right there and call it a day, problem solved. I’d achieved my goal, in fact I’d gone one better and used streams. Good night, Bob’s your Uncle! We can quit now. But noooo, I’d been reading <a href="https://pragprog.com/book/tbajs/async-javascript">Async JavaScript</a> and was very impressed by the <a href="https://github.com/caolan/async#parallel">async.parallel</a> example but I didn’t have a good use case. Plus, I also have a kid in college and I was trying to impress him with my open-source chops. As we began early testing of our app it quickly became apparent the need to support multiple platforms with multiple versions of each image. We needed image versions more sized for Desktop, Tablet, Mobile and really small for Thumb images. You know that commercial where you buy 1 suit at the regular price and you get 3 suits free? Well <a href="https://github.com/lovell/sharp">sharp</a> image processing with <strong>Async.Parallel</strong> is kinda like that. Upload one file and store 4 more for free…</p>
<h3 id="coffee-script">Coffee Script</h3>
<script src="https://gist.github.com/t2k/ae28bda9e194976ced03.js"></script>
<p><em>For extra-bonus points: What does the 201/created JSON response look like?</em></p>
Endless Scrolling in Marionette2014-10-14T00:00:00+00:00http://t2k.github.io/client-side/2014/10/14/filtered-backbone-collections
<h4 id="mvc---modelhttpbackbonejsorgcollection-viewhttpmarionettejscomdocsmarionettecompositeviewhtml-controllerhttpmarionettejscomdocsmarionettecontrollerhtml">MVC - <a href="http://backbonejs.org/#Collection">MODEL</a> <a href="http://marionettejs.com/docs/marionette.compositeview.html">VIEW</a> <a href="http://marionettejs.com/docs/marionette.controller.html">CONTROLLER</a></h4>
<p>My primary goal when developing a client-side web application is to establish a clear separation of responsibilities between <strong>views</strong> and <strong>model</strong> data. In the world of async web apps, a strict definition of the role played by the <strong><em>controller</em></strong> can get a little <em>fuzzy.</em> But I digress… In this example, I’m using the <a href="http://requirejs.org/docs/whyamd.html#amd">AMD</a> specification. and by doing so it enforces modular design concept built into the application architecture. Each module has its own self contained scope and cannot communicate directly with other modules outside of it’s intended scope. This approach may seem like overkil, especially when designing a simple demo, however as client side complexity grows the AMD ‘building-block’ approach pays off.</p>
<h4 id="feeling-the-mvc-backbonemarionette-flow">Feeling the MVC Backbone.Marionette Flow</h4>
<p>There is so much out there on the Internet about <strong>MVC</strong>. If you Google <strong>Model-View-Controller</strong> MVC, MVP, MVVC, MVVM etc. you’ll be overwhelmed. I’ve read most of it and I’m sure you have too. There’s so many perpectives and opinions out there about MVC and I’m not here to solve the debate. Every developer faces unique complexities and must approach their application architecture by taking their own full stack into consideration. We face different complexities whether we live in a JAVA, .NET or Linux/Open Source world. Furthermore, realities change if we’re using an enterprise database such as Oracle, SQL-Server, Postgress, MySQL or if you live in a NoSQL world like MongoDB, Couch, Pouch, Slouch etc. Let’s abstract all the server-side issues away and just assume the back-end in this example is a fully supported REST-API so we can focus on the client-side development.</p>
<p>This post is about client-side development and we’re <strong><em>MVC-flowing</em></strong> with Backbone.Marionette. The abstract <strong>MODEL</strong> is a backbone collection, wrapping an array of data returned from an AJAX call. In Backbone, the MODEL has the responsibilty of making async/AJAX calls against the REST API’s, that’s the Backbone way. The abstact <strong>VIEW</strong> is a class extending from Backbobne.CompositeView wrapping an HTML template which defines a user-interface. The abstract <strong>CONTROLLER</strong> exends from the Marionette.Controller class. The controller creates views and coordinates passing data to those views. The controller also sets up the intra-module, event driven communication between the MODELs and the VIEWs. The controller coordinates the event-api handling and data-marshalling between the REST-API and the VIEW presentation architecture. Have I lost you yet? I’m a little confused too, hopefully a diagram will help.</p>
<h4 id="the-view-a-scrollable-region">The VIEW: A Scrollable Region</h4>
<p>The hardest part for me was understanding the markup and CSS required to make a scrollable region work within a Marionette View. Working with click events, change events, blur events is straight forward. Understanding these user interactions is pretty obvious. In order to create a UI that will emit scroll events will require a bit more explicit HTML and CSS. The interaction between the <em>html</em> in our template and the Marionette View class that repsponds to scroll events took a little extra level of understanding on my part. For this to work, we must basically contrain a <code>div</code> with a <code>max-height</code> setting then have that wrap in inner <code>div</code></p>
<h4 id="html">HTML</h4>
<pre><code class="language-html"><div class="scrollable-container">
<div class="scrollable-inner">
<!-- repeatable item views -->
</div>
</div>
</code></pre>
<h4 id="css">CSS</h4>
<pre><code>// marginal support of endless scrolling
.scrollable-container {
max-height: 500px;
overflow-y: scroll;
}
.scrollable-inner {
max-height:100%;
}
@media all and (min-width: 1023px) {
.scrollable-container {
max-height: 640px;
}
}
</code></pre>
<h3 id="the-compositeview-class">The CompositeView Class:</h3>
<pre><code>
LPList: class _listview extends AppViews.CompositeView
template: _.template(Templates.lpItems)
childView: LPItem
childViewContainer: ".scrollable-inner"
className: "scrollable-container"
ui:
scroll: ".scrollable-inner"
events:
"scroll": "checkScroll"
collectionEvents:
"request": ->
NP.start()
NP.inc()
"sync error": ->
NP.done()
checkScroll: (e) =>
virtualHeight = @ui.scroll.height()
margin = .05 * virtualHeight
scrollTop = @$el.scrollTop() + @$el.height()
@trigger "scroll:more:lpitems" if (scrollTop + margin) >= virtualHeight
</code></pre>
<p>Notice above, the extended CompositeView class wraps its template like this:</p>
<pre><code> className: "scrollable-container"
</code></pre>
<p>This interaction between the view’s rendered template and it’s CSS is critical in enabling the view both emit scroll from it’s markup and listen to events. What we are doing here is wrapping a view in a contrained block element. By setting a <code>max-height:500px</code> and <code>overflow-y:scroll</code> this enables the view to both emit the scroll event and the view’s events hash</p>
<p><code>coffeescript
events:
"scroll": "checkScroll"
</code></p>
<p>can listen for a scroll event and respond with it <em>checkSroll</em> event. The logic: if the scroll bar is close to the bottom, trigger the <code>scroll:more:items</code> event. That’s all we need to support endless scroll.</p>
<p><br /></p>
<h4 id="screenshot-endless-scroll-ui">Screenshot: Endless Scroll UI</h4>
<div class="row">
<div class="col-sm-12">
<img src="/assets/images/scroll-region.png" class="img-responsive" />
</div>
</div>
<hr />
<div role="tabpanel">
<!-- Nav tabs -->
<ul class="nav nav-tabs" role="tablist">
<li role="presentation" class="active"><a href="#model" aria-controls="model" role="tab" data-toggle="tab">MODEL</a></li>
<li role="presentation"><a href="#view" aria-controls="profile" role="tab" data-toggle="tab">VIEW</a></li>
<li role="presentation"><a href="#controller" aria-controls="messages" role="tab" data-toggle="tab">CONTROLLER</a></li>
</ul>
<!-- Tab panes -->
<div class="tab-content">
<div role="tabpanel" class="tab-pane fade in active" id="model">
<script src="https://gist.github.com/t2k/dc5300be5343524656fa.js"></script>
</div>
<div role="tabpanel" class="tab-pane fade" id="view">
<script src="https://gist.github.com/t2k/ccde4a35397ee661b47b.js"></script>
</div>
<div role="tabpanel" class="tab-pane fade" id="controller">
<script src="https://gist.github.com/t2k/762d37c61fb7ab4b62cc.js"></script>
</div>
</div>
</div>
<hr />
<h4 id="templates-module">Templates Module</h4>
<pre><code>define (require) ->
heatmap: require "text!apps/lp/list/templates/heatmap.htm"
lpItem: require "text!apps/lp/list/templates/lpItem.htm"
lpItems: require "text!apps/lp/list/templates/lpItems.htm"
layout: require "text!apps/lp/list/templates/layout.htm"
panel: require "text!apps/lp/list/templates/panel.htm"
title: require "text!apps/lp/list/templates/title.htm"
</code></pre>
<p>lpitem.htm: the ItemView template</p>
<pre><code><div class="row">
<ul class="list-group col-md-3 col-sm-6 col-xs-12">
<li class="list-group-item"><span class="label "><strong></strong></span><span title="process id"> </span></li>
<li class="list-group-item"></li>
<li class="list-group-item"><span title="processed"> </span><span title="value date"> </span></li>
</ul>
<ul class="list-group group col-md-3 col-sm-6 col-xs-12">
<li data-toggle="tooltip" title="SAP-CML#" class="list-group-item"></li>
<li class="list-group-item"></li>
<li class="list-group-item"></li>
</ul>
<ul class="list-group group col-md-3 col-sm-6 col-xs-12">
<li data-toggle="tooltip" title="Murex Simple TradeID" class="list-group-item"></li>
<li data-toggle="tooltip" title="Maturity-Date" class="list-group-item"></li>
<li data-toggle="tooltip" title="Loan-Amount: Change" class="list-group-item">Change<span class="badge"></span></li>
</ul>
<ul class="list-group group col-md-3 col-sm-6 col-xs-12">
<li class="list-group-item"> <span data-toggle="tooltip" title="cash flows affected" class="badge"></span></li>
<li data-toggle="tooltip" title="LS vs LPR" class="list-group-item">&nbsp<span class="badge"> v </span></li>
<li data-toggle="tooltip" title="PV'd Loss/Benefit" class="list-group-item">PV<span class="badge"></span></li>
</ul>
</div>
</code></pre>
<h3 id="mvc-interactions">MVC Interactions</h3>
<p>Understanding the specific interactions among the Model, View, and Controller modules is the key to understanding the larger application architecture. Communication among modules is event driven which also enables us to decouple out modules. Starting with the backbone <strong>Model</strong>, it makes async requests to a REST/API as shown here by setting up a request handler on application’s <code>msgBus</code>: </p>
<pre><code>getLPEntities: (options={}, filter={})->
{skip,take} = options
_coll.setPage skip
_coll.setPageSize take
_coll.setFilter filter
_.extend options, filter
_coll.fetch
reset: true
data: options
_coll
...
msgBus.reqres.setHandler "lp:entities", ->
API.getLPEntities
skip: 0
take: 20
</code></pre>
<p>The <strong>Controller</strong>, instantiated by a modular App, is responsible for requesting data from the <strong>Model</strong> and passing that data to the <strong>View</strong> as shown here from the <strong>controller</strong> initialize function:</p>
<pre><code>class Controller extends AppController
initialize:->
@lpEntities = msgBus.reqres.request "lp:entities"
heatmaps = msgBus.reqres.request "lpheatmap:entities"
@layout = @getLayoutView()
@listenTo @layout, "show", =>
@heatMapRegion heatmaps
@titleRegion()
@panelRegion()
@lpRegion()
@show @layout,
loading:
entities: [@lpEntities, heatmaps]
lpRegion: ->
lpView = new Views.LPList
collection: @lpEntities
@listenTo lpView, "childview:lp:item:clicked", (child, args) ->
msgBus.events.trigger "lp:item:clicked", args.model
@listenTo lpView, "scroll:more:lpitems", ->
msgBus.reqres.request "lp:fetch:more"
@layout.listRegion.show lpView
</code></pre>
<p>The <strong>View</strong> listens at UI level for events triggered by specific user interactions, in this specific case the view is responding to a <code>scroll</code> event and the <em>View</em> triggers an event for the controller to handle. </p>
<pre><code>LPList: class _listview extends AppViews.CompositeView
template: _.template(Templates.lpItems)
childView: LPItem
childViewContainer: ".scrollable-inner"
className: "scrollable-container"
ui:
scroll: ".scrollable-inner"
events:
"scroll": "checkScroll"
collectionEvents:
"request": ->
NP.start()
NP.inc()
"sync error": ->
NP.done()
checkScroll: (e) =>
virtualHeight = @ui.scroll.height()
margin = .05 * virtualHeight # 10%
scrollTop = @$el.scrollTop() + @$el.height()
@trigger "scroll:more:lpitems" if (scrollTop + margin) >= virtualHeight
</code></pre>
<p>to the <em>controller</em> and it gets to decide how to pass event’s to models or in other cases to routeable apps. In this specific case <em>scroll</em> events trigger a <code>scroll:more:lpitems</code> event up the chain. </p>
<p>As seen above, the <strong>controller</strong> monitors this event like this:</p>
<pre><code>@listenTo lpView, "scroll:more:lpitems", ->
msgBus.reqres.request "lp:fetch:more"
</code></pre>
<p>Here, the <em>controller</em> listens to events from it’s <em>views</em> and passes them up the chain. In our case the <strong><em>LP</em></strong> entity model listens to this custom event and springs into action, by fetching another page of items. That’s all there is to it. The extended Marionette.CompositeView automagically updates it’s childview item template because it listens to the @collection <code>reset</code> event causing it to renders itself with <em>more:items</em></p>
<p>…</p>