Microservices architecture presentations

Friend of mine, Paweł, has recently shared few very interesting recordings of presentations about microservices architecture. Follow to his blog posts to learn from the best what microservices architecture is about and how it can make your team be even more agile:

I’ve found the first one particularly interesting as it goes beyond technical side only, touching topics like how microservices architecture can reshape your team/organization. The one by Fred George got me into too – part of it is the story of project Paweł and I we were involved in.

Comment on ThoughtWorks Radar

About twice a year ThoughtWorks publishes so-called Technology Radar. It’s a whitepaper  on latest trends in technology and their practical implementation prepared by ThoughtWorks Technology Board. Must read for every IT engineer. I used to use it as guideline to what to read about next, what should I learn not to stay behind.  To my surprise the latest May 2013 edition described fair list of tools, technologies and practices that I’m using on daily basis. Some of them categorized as “trial” or “asses”. Thus I thought I can share my experiences dealing with them every day.

To put things in context, the project I was working on during past few months is kuapay.com website and public as well internal webapps. So everything spins around browser (HTML5, CSS3, JavaScript) and HTTP (REST APIs etc.). Very front-end things. Kuapay product is mobile payments platform.

Techniques

As someone obsessed about testing I’m happy to see mobile testing on mobile networks as one of techniques recommended for adoption. Having working on system that is deeply rooted in mobile technologies and must work perfectly under any reasonable conditions in any environment I can not imagine how important, not to say critical, mobile applications would not to be tested in the wild. Stubbing simply fails here. If your target environment are smartphones in Chile then you have to book flight to Santiago, buy some prepaid phones and start testing. To your surprise even same phone models will behave differently here and there. I feel like we did big circle and we’re back with on-site manual testing. BTW, don’t forget about breaking things intentionally, as usual.

Still in testing topic suggested to adopt performance testing as first-class citizen is basically about life cycle of your performance tests. Are they brittle little tools your run from time to time just to discover they don’t run anymore or are they important part of continuous integration? I hope the second as I see more and more companies small and big adding suite of performance tests to their CI pipeline, at least in staging and production environments (pre-release and post-release).

Promises are new hip thing talk about with your JavaScript folks. The idea itself is not new. In fact it’s almost 40 years and used to be known as Futures. Thanks to popularity of Node.js and support from  libraries like jQuery more and more developers become fluent with them. I’m going to write a bit more about promises in separate post so just to give quick example compare this callbacks “hell” based code

with promises

I hope that second one looks way more maintainable. Exited by using new cool programming technique don’t forget that the are no silver bullets.

I’m not sure why blue-green deployment has been set on trial. Maybe because of name. I haven’t heard this term before even though I’ve seen this technique for ages. Maybe because it used to be tricky to implement. Or maybe because it’s simply not for everyone. It profits most in environments where zero downtime is a requisite, even during upgrades. If you’re interesting in implementing it seriously consider infrastructure automation tools (Chef, Puppet) and/or cloud services, as well as solid suite of acceptance and performance tests (see already mentioned performance testing as first-class citizen).

I still don’t get why continuous delivery for mobile is such great thing. I simply don’t see it working. Especially if you take into account problems mobile testing on mobile networks solve. Or one week on average for iOS application to be accepted by Apple. I’ve been successful delivering mobile apps to testers this way, but not to customers. If you have successes with CD for mobile write me back please. I would love to hear your story.

Database migrations for No-SQL are in my humble opinion even more important in No-SQL world than in relational databases. I assume ThoughtWorks puts it on trial because No-SQL itself is not that popular yet and usually used to store vast amounts of data. I see it to solve one big problem with No-SQL. Lack of schema. Whenever developers wants to add something to schema he simply does by saving document with more properties. Problem is older documents don’t have such properties. Clients have to deal with it and “upgrade” documents on the fly. Since number of clients we need to support is growing (web, android, ios…) it means “upgrade” code has to be duplicated. With code duplication come bugs, maintenance costs and so on…

One of important features of Kuapay platform is that password never travels across network. And it’s needed in almost any business process. Requirement of our webapps it that they can not send password to server, can not retrieve password from server and need to have access to password during entire session even after refreshing page. My colleague found interesting solution for this problem – HTML5 storage instead of cookies. What started as trick to satisfy business requirements ended up as regular way to store session data. HTML5 session storage is in almost every way better than cookies. Session data is kept securely by client for time span of session, no extra and potentially sensible information is transferred over network with every HTTP request, session data is automatically wiped after closing window and there is more storage. 5MB instead of 4kB. If cleaning session by closing window sounds too inconvenient local storage can help. I’ve got so used to it that it become my first choice for storing session data.

First thing to think when starting new project – mobile first. Below chart says everything

It also implies responsive design (or maintenance hell of even more clients).

I discover more and more new terms for techniques known and practiced since long time. Or call them buzzwords. The latest one is semantic monitoring. Imagine that. Your unit tests in staging environment did pass, acceptance tests as well, days after release all the monitoring lights are green yet it’s 10pm and you’re still fixing this bug no one noticed. What’s missing? Checking on production workflows crucial for business. Not just once after release, in one environment, but as part of your monitoring tool. I did not practice it recently but I did found it very useful in past to run carefully tailored suite of acceptance tests simulating typical behavior paths of types of users and alerting when threshold of failures crosses unacceptable number. Small advice, don’t send alerts when anything fails. Re-run failed tests few times, or even dozens of times, to see if you had bad luck or things are really going bad.

My personal list of techniques to try:

  • capturing client-side JavaScript errors (and mobile apps errors)
  • minimizing application configuration
  • machine image as build artifact

Last but not least, exhaustive browser based testing. I used to be big fan of them but now I have to agree with ThoughWorks. Too brittle and to expensive compared to profit. Invest your time somewhere else.

Tools

There is more tools recommended by ThoughtWorks I would like to try than I have experience with. However… Recently I begun playing with D3 library for data visualization. I still have too little experience to recommend it for adoption or not, but I would definitely suggest to play with it. Makes your life easier and more beautiful. Try it by yourself. I’ll keep using it and learning.

Unit testing JavaScript used to be real headache for me. Immaturity of tools was leaving me with running unit tests in real browser via Selenium or similar tool accompanied with custom hack to get CI consumable results (brittle, slow, false positives) or running them against VMs that were not used by target audience (even more false positives). Functional testing was beyond my reach. Until PhantomJS. It’s a scriptable (read “easy to integrate”) headless WebKit  browser (read “realistic target”). Combined with Jasmine and JSCover made it possible for me to write and run with trust hundreds of JavaScript tests. My Eclipse runs whole test suite whenever file is saved and I get results in matter of second(s). I’m already working on separate post about my setup.

Personal list of tools to evaluate:

  • Gradle
  • immutable servers
  • Gatling
  • Locust
  • Light Table
  • Snowplow Analytics

Platforms

Reading platforms section of the report ensured me that recently I’m spending too much time working with front-end, missing all the beautiful things happening in backend world. I need to catch-up with:

  • PostgreSQL for No-SQL (big fan of PostgreSQL)
  • Hadoop
  • OpenStack
  • Calatrava
  • PhoneGap/Apache Cordoba
  • Zepto

Last one is JavaScript library that is aiming to replace jQuery. I just begun using. Lets see how it goes.

For now I had enough pleasure with set of WS-* standards. Last time was about WS-Security, my younger peers failing several times to integrate with partners’ API and me coding in C++ adding missing features to gSOAP and patching others where external SOAP-API was not following the standard. Even though it did not take me that much time to get things done seeing junior developers not being able to even start coding, not talking about debugging problems, makes me thing that WS-* stuff is too complicated, thus too expensive.

I’ve put in grave Node.js. Maybe too soon. ThoughtWorks still recommends to trial it. My strongest argument not to used it? Well, it’s JavaScript, with all the good and bad parts. To start with, no integers… With some many better options I don’t see why someone would use it on the server side. Command line “helper” tools, fine. Server side? I pass. I’ll keep watching how Node.js grows in the meantime I prefer to invest into learning Erlang.

Language and frameworks

And we’re back to front-end development, with just a little bit of lightweight back-end. No surprises here.

HTML5 is growing fast and so JavaScript and CSS3. I’ve already mentioned Jasmine for automated testing and I backup ThoughtWorks with adopt recommendation. There is no competitor for Jasmine. It’s easy to integrate with other tools, fast, focused development on important things (behavior!) and fits well both unit testing as well acceptance. In my setup runs every time I hit Ctrl+S. JavaScript developers must have.

Other must have for JS is requirejs. I know that there are many people who genuinely hate it, some popular libraries even moved back from AMD but hey. It’s reality. There is no large-scale software made well without proper code organization. And that’s what requirejs gives you. Isolated modules and ability to import things you need. No global scope.

Report just mentions HTML5 for offline applications. It’s still novelty, mainly used by Mozilla/Firefox OS. But it works. It might change disrupt world of mobile applications. When I was writing my first website 17 years ago my friends were joking this platform will never work, never mature. Todays world is run by web technologies, everything spins around HTTP. HTML is moving to offline applications and with it millions of developers. Why? It’s proven to work (web), faster to learn, faster to develop. In many cases converting your existing webapp into offline app is just a matter of creating manifest file and zipping all the code and resources together. Why to spend money on web developers, iOS developers, android developers, Windows developers and Mac developers and spending some more on maintenance of all these clients made in very different technologies if you can do the same thing with one team and one technology stack? HTML5 will be for small to mid apps what Java is for enterprise level software.

Once it gets there. HTML is very mature in classic web but offline requires very different approach. For example, hand written CSS is over. If you’re not using some CSS framework like SASS you’re just wasting your time and introducing bugs. But that’s just a tool to organize CSS like you organize your code – modules, inheritance, parametrization. The goal is CSS UI toolkits like Twitter Bootstrap. Even though I do not recommend TB in current form I suggest taking a look at direction it’s setting. What’s wrong with TB? Impossible to customize for real-world projects. Works as is, fails when you want to do things differently. Too much coupling.

World of JavaScript MV* frameworks looks similar. It shows us direction but does not give solid solutions. ThoughtWorks even puts Backbone.js on hold and I agree. Quoting previous report

Backbone.js is great example of an abstraction pushed too far. While we initially liked the ease of wire-up, in practice it suffers from them same issues as all such data-bound frameworks form WebForms to client/server tools. We find that it blurs the framework and model too much, forcing either bad architectural decisions or elaborate framework hackery in order to preserve sanity.

I find Knockout, EmberJS and AngularJS walking same path. I didn’t try the last one yet, but other too in same ways are even worst than Backbone.js.

This leads me to big missing piece of the report. Lack of entire application framework for JavaScript apps. Something similar to Android SDK. We have some UI widgets libraries, some broken MV* frameworks but we’re missing bigger picture. The application. How to wire things up, bottom to top? The path from user input down to HTTP request and back to user requires lots of machinery that is missing. To mention just few: events that are commonly understood, localization and internationalization, filter, formatters, specifications/validations, authentication, authorization, data mapping… At Kuapay we’ve spent about two months trying to marry existing tools and filling in gaps, sometimes reinventing wheel, to create a framework allowing us to develop fast and with trust.

I feel like we’ve made circle in how we develop applications. 20 years ago development was predominated by desktop applications and fat client side frameworks communicating with server services via SOAP or CORBA. With web booming we’ve moved most of the client side code to the server. Yes, popular tools like Zend Framework or Rails are much about client side logic made on server-side and deliver to client (browser) as ready to use HTML and CSS. Now it’s changing again, webapps and HTML5 offline apps are taking back much of client side workload letting server-side to be thin again. No surprise micro-frameworks like Sinatra are gaining popularity.

Personal list of tools to evaluate:

  • Scala, Clojure
  • micro-frameworks (more like pick one)
  • Dropwizard
  • Play Framework

Conclusions

It took me few years to blog again and ThoughtWorks Radar was the trigger. Reason is when I was reading it I’ve discovered that many tools and technologies report suggests to adopt or try are bread and butter for me, and the ones it puts on hold I’ve abandoned too. In some areas made me feel being ahead. Mainly on front-end side. Thus I wanted to share my experiences to add a little bit to the report.

It also surfaced where I got behind. Back-end. Well, it’s time to catch-up.