Welcome to NTCoding
Weclome to version 3 of NTCoding now built using FubuMVC and RavenDB.
I'm Nick, a developer working with .NET technologies 9 - 5 but always taking an interest in a more diverse range of technologies - particularly Erlang and Scala at the moment. My outlook is the more I learn, the more I can deliver for myself and employers.
My currently employer being 7digital. I'm very grateful for all the things I've learned whilst being part of an incredible team of business and technical people.
Recent Blog Entries
Cross-platform communication with dbus
5/24/2013
Up until recently I'd never hit the use case where I needed an application written in one programming language to send messages to an application written in another programming language on the same machine.
Initially, I had two options in mind - sockets or HTTP. Both would have been a lot of work I was hoping to avoid. Fortunately Linux has dbus which I was vaguely aware of. After some investigation, dbus turned out to be my dream choice.
I'll show you how easy it is to send messages between applications in different languages, by using dbus's RPC capabilities to implement publish/subscribe.
What we want to achieve
Take a look at the diagram below. You can see that we want to create an application called publisher that will allow other applications to register as subscribers and therefore have messages sent to them.
Implementation-wise, the publisher will be written in scala, whilst subscribers will be in ruby and scala (could be any language). Additionally, messages will be sent to the publisher for publishing from the command line.
.jpg)
.jpg)
Structure of dbus services
Before building the applications, it's really important to understand the structure of a dbus service. Luckily, it's really simple as the diagram below illustrates.
Each service has a unique name – this is how other services can find it on the bus. Each service also has a number of paths – analogous to buckets or namespaces which contain methods that other services can call.
So when a service wants to call a method on another service, it:
- Looks up the service by name
- Selects a path
- Selects an interface inside selected path
- Selects a method on selected interface
- Invokes the selected method with correct argument
Creating a publisher
I'm not going to cover setting up projects, or configuring dependencies, I'm just going to show the actual code with explanations. There will be links at the bottom of the page to help get started and you can download this example from github.
First thing the publisher does is register itself on the bus with the service name dbus_demo.publisher (line 10). Then it exports – meaning other services can call the methods on it - the "Publish" interface in the path "/" (line 11).
Publisher lives inside the package "dbus_demo" (line 1), so it's full interface name on the bus will be dbus_demo.Publisher.
A really important feature to note occurs on line 57. Here, the publisher is creating a local Handler object which represents an identical object on the subscriber. To be identical the path, interface name, and method name must all be the same in both applications.
Below is the handler. Remeber, this is a trait (interface) because the implementation lives on the service being called - essentially it's used to create a proxy of it.
Below is the handler. Remeber, this is a trait (interface) because the implementation lives on the service being called - essentially it's used to create a proxy of it.
Creating a scala subscriber
This client will be a completely separate application, with unique code, running inside a different JVM, running inside a different OS process.
Similar to the publisher, the subscriber first registers itself with the bus. Then it needs to register itself with the publisher by calling the publisher's register method. To do this, it again uses a local interface (Publisher) to create a proxy of the remote implementation using getRemoteObject().
Here is the Handler for this project - it has the same package name and method signatures as the interface in the Publisher application. Therefore, it will have the interface name on dbus - bus_demo.client.Handler - which the publisher looks for (publisher code line 57).
Creating a ruby subscriber
the ruby client provides the same functionality as the scala client. It makes a remote call to the Publisher to register as a subscriber and it exports a method called "Handle" on an interface called "dbus_demo.client" in the path "/". Remember, the publisher looks for exactly this when publishing a message to clients (publisher code line 57).
It's nice to work with the ruby library because you can see it makes good use of dynamics instead of requiring mirror classes.
Testing it all out
To show this all working, I'm going to:
- Start the publisher (top panel)
- Start the scala subscriber (2nd top panel)
- Start the ruby subscriber (3rd panel)
- Send a message from the command-line to the publisher (bottom panel)
- Watch as each terminal window updates – confirming the message reached the publisher and was delivered to all the subscribers. (all panels)
You can get all of the source code used for this example from my github. Why not extend it and add clients in C, Python, or your favourite language(s)?
Links
How I build software: Enough design to clear my mind
5/16/2013Following on from the last post where I set out the vision and motivations for my project, now I'll talk about why I needed to design the system and how I started to do so.
As was a theme in the last post, the purpose of design is mostly to clear up my mind – to turn feelings about how the system will work, into visual designs that can be reasoned about, resulting in mental clarity and less reliance on my memory - so I have confidence to start building.
I blogged previously about diagrams if you want more information.
Mental clear out
Even though I had a vision and list of features, I wasn't comfortable to sit down and start coding - too many unknowns. From experience, I know that coding in this state would leave me with scatter brain and lots of erratic hacking. Instead, I wanted a theoretically-plausible design to guide me.
I sketched out a few diagrams and continued to revise them over the course of a week or two. Over the same time period, and often guided by question marks in the design I did research about related topics (such as remote akka).
Let me show you just one example of how the diagrams helped me get a fundamental decision right, and prevented me from spending weeks of my spare time writing unnecessary code.
Instant payback from the diagrams
One big question I was contemplating was how I was going to transmit messages across the network. My gut reaction (ergo emotion) gave me a strong feeling that json over http was the best option by a mile.
After I created the first version of the diagram I could visually see that I was going to be communicating between two akka applications using json over http. At that point I felt like (especially with the misspelled “you're”):
Like Erlang, akka has it's own protocols and delivery mechanisms for sending messages between remote instances that are part of the same cluster - this is a much smarter solution to my problem. The updated version of the diagram (below) shows this
.
 |
| click image to view full size |
I just saved myself a LOT of wasted effort (weeks of spare-time coding). Life-lesson: don't trust your instincts or emotions.
You can see other significant decisions on this diagrams that give me lots of time to research and re-consider them before I start to code. Although code can always can be changed, it's cheaper to fix problems the earlier I find them.
Just to give a full picture of what I considered on this project to be “enough design to clear my mind”, here are the remaining 3 diagrams.
As you can see, there aren't pages and pages of highly-constrained diagrams – just enough to make me feel confident that I can start to build the system with a general outline in mind.
All of these diagrams were created with guidance from the vision, motivations, and high-level traits to ensure they meet the project's goals. For me it would be very difficult to do all of this just from working memory as I sit at a computer coding.
I also find that ideas come to me when I'm in the gym, in the kitchen, or in any other situation where I'm thinking about something totally different. That's good, because a bit of design like this doesn't force me to think about everything when I'm sat at my desk.
These diagrams make it easier for other people to understand the system. Be that people wanting to use it, or people wanting to help me build it (get in touch if that's you). I blogged before about using diagrams to on-board team members.
Often I think to myself how much easier it would be for me to use an open source project if there were a few diagrams about how it worked. So I know there is value here.
I can also use these diagrams to collaborate in other ways, too. For example, If I need help building the system I can use these diagrams as the basis for conversation with other developers.
These few diagrams have given me a lot of mental clarity, because I'm comfortable that what I'm about to build, theoretically, should work as well as meeting all of the requirements.
Remember, though, that the design is not fixed. Even better, any time I do make a high-level change I have to update the diagrams. This forces me to see how changes fit into the bigger picture. Down at the code level and without any diagrams, I'm at the mercy of my cognitive abilities to perform all this work.
Having a basic design means I can break up the project and finally start to plan how I'm going to start to carry it out; sounds like my next post.
You can see other significant decisions on this diagrams that give me lots of time to research and re-consider them before I start to code. Although code can always can be changed, it's cheaper to fix problems the earlier I find them.
A few more diagrams, much more mental clarity
 |
| click image to view full size |
 |
| click image to view full size |
 |
| click image to view full size |
As you can see, there aren't pages and pages of highly-constrained diagrams – just enough to make me feel confident that I can start to build the system with a general outline in mind.
All of these diagrams were created with guidance from the vision, motivations, and high-level traits to ensure they meet the project's goals. For me it would be very difficult to do all of this just from working memory as I sit at a computer coding.
I also find that ideas come to me when I'm in the gym, in the kitchen, or in any other situation where I'm thinking about something totally different. That's good, because a bit of design like this doesn't force me to think about everything when I'm sat at my desk.
Helps bring people on board
These diagrams make it easier for other people to understand the system. Be that people wanting to use it, or people wanting to help me build it (get in touch if that's you). I blogged before about using diagrams to on-board team members.
Often I think to myself how much easier it would be for me to use an open source project if there were a few diagrams about how it worked. So I know there is value here.
I can also use these diagrams to collaborate in other ways, too. For example, If I need help building the system I can use these diagrams as the basis for conversation with other developers.
I'm now comfortable to make a plan and then start coding - exciting
These few diagrams have given me a lot of mental clarity, because I'm comfortable that what I'm about to build, theoretically, should work as well as meeting all of the requirements.
Remember, though, that the design is not fixed. Even better, any time I do make a high-level change I have to update the diagrams. This forces me to see how changes fit into the bigger picture. Down at the code level and without any diagrams, I'm at the mercy of my cognitive abilities to perform all this work.
Having a basic design means I can break up the project and finally start to plan how I'm going to start to carry it out; sounds like my next post.
How I build software: Start with a vision
5/14/2013I'm just starting a new project which is a great opportunity for me to reflect on how I go about building software. Blogging about it means I have to be even more analytical, and also means I can get feedback if I'm lucky.
You'll see how my primary focus is working around the inherent limitations of my cognitive abilities - I take many opportunities to replace emotion with logical reasoning in decisions making. Also, I try to un-burden my (short, working and long-term) memory as much as possible.
This post shows how I start out on a project by being clear about what I want to achieve.
You might enjoy a quick read of this wiki page about emotion and memory first.
If you look at the github repository for this project you’ll see that my vision is:
To create a service bus that facilitates the creation of highly-decoupled (temporal, platform & implementation) business services that can scale horizontally and are highly-resilient to faults.
When I’m designing and building the system I can keep these goals in mind. Any time I need to make a decision, I can use this vision (and motivations below) to guide it.
Without a clear vision I will just have a feeling. Subsequently, this means my current emotional state has a chance to influence the vision every time I think about it. This can result in erratic focus, indecision, or lack of coherence over the life of the project.
Visions are allowed to change. Importantly though, having a vision makes sure that I am logically disagreeing with a written-down statement, not changing goals based on my current emotional state (aka being whimsical).
My studies of psychology again lead me to question my emotions and favour a clear, written list of motivations for the project... I don’t want to feel like this project is a good idea - I want written down logical reasons so I can see a purpose to the project and can enjoy working on it (good use of emotion).
As with the vision, a written list of motivations helps decisions to be more logical. For example, the question: “Should I continue with the project?”, can be logically answered by: “do I still agree with the motivations I’ve stated?”.
Here’s what I have for this project at the moment:
I benefit from keeping a small list of high-level properties the system should have which support the vision. Again, these are all about keeping the project focused on solving a certain set of goals. I'm taking work away from my memory and offloading it to a text file.
Importantly I have to think about the traits in more detail too as I write them down. Sometimes they just don't make sense. Other times they conflict. It's not until they are written down and logically evaluated that I notice this.
Here’s what I’ve got so far:
Any time I'm unsure about something I just pop it in the questions document. For instance "should network communication be platform agnostic or remote akka?". I can come back to it later whilst staying focused on the current task.
What's really important about these simple text files is that they remove the opportunity for my memory to forget the information. Also, just the act of writing down my thoughts forces me to think about them more deeply.
Sometimes, my mind will pick a really stupid answer. But when I write down a problem the obvious solution often stands out. Try it yourself during day-to-day development tasks.
At this stage of a project I have a good idea of what I want to build and why I want to build it. These reasons are all written down for me to clearly understand over the course of a project – not held in memory as feelings, then re-interpreted through my current state of emotion each time I think about them.
You might think all this sounds like unnecessary work, but for me the pay-off is grand. I work best and feel happiest when my mind is clear to focus on one thing and is not stressed about having to remember important details.
I'm not the smartest developer so I have to get the best out of what I've been given. Hopefully you enjoyed seeing how I try to achieve this. In the next post I'll show I started to design this project and the thought-processes that drive me.
You'll see how my primary focus is working around the inherent limitations of my cognitive abilities - I take many opportunities to replace emotion with logical reasoning in decisions making. Also, I try to un-burden my (short, working and long-term) memory as much as possible.
This post shows how I start out on a project by being clear about what I want to achieve.
You might enjoy a quick read of this wiki page about emotion and memory first.
What do I hope to achieve?
If you look at the github repository for this project you’ll see that my vision is:
To create a service bus that facilitates the creation of highly-decoupled (temporal, platform & implementation) business services that can scale horizontally and are highly-resilient to faults.
When I’m designing and building the system I can keep these goals in mind. Any time I need to make a decision, I can use this vision (and motivations below) to guide it.
Without a clear vision I will just have a feeling. Subsequently, this means my current emotional state has a chance to influence the vision every time I think about it. This can result in erratic focus, indecision, or lack of coherence over the life of the project.
Visions are allowed to change. Importantly though, having a vision makes sure that I am logically disagreeing with a written-down statement, not changing goals based on my current emotional state (aka being whimsical).
What is the motivation?
My studies of psychology again lead me to question my emotions and favour a clear, written list of motivations for the project... I don’t want to feel like this project is a good idea - I want written down logical reasons so I can see a purpose to the project and can enjoy working on it (good use of emotion).
As with the vision, a written list of motivations helps decisions to be more logical. For example, the question: “Should I continue with the project?”, can be logically answered by: “do I still agree with the motivations I’ve stated?”.
Here’s what I have for this project at the moment:
- Want to learn more about building distributed systems and middleware
- Want to improve my scala and akka skills
- Need a service bus similar to NServiceBus that works on linux and for any language, for my future projects
What traits should the system have?
I benefit from keeping a small list of high-level properties the system should have which support the vision. Again, these are all about keeping the project focused on solving a certain set of goals. I'm taking work away from my memory and offloading it to a text file.
Importantly I have to think about the traits in more detail too as I write them down. Sometimes they just don't make sense. Other times they conflict. It's not until they are written down and logically evaluated that I notice this.
Here’s what I’ve got so far:
- Highly fault-tolerant - as soon as business intent is captured, it should never be lost. e.g. at least once message delivery
- Enables horizontal scalability (as much as it can)
- Easily monitorable (performance of system, errors)
- Primarily to run on linux
- Can be used in any programming language
Todo list and questions list
Any time I realise I need to do something on the project I'll just put it on the todo list. I avoid holding it in memory or losing focus on what I'm doing at all costs.
Any time I'm unsure about something I just pop it in the questions document. For instance "should network communication be platform agnostic or remote akka?". I can come back to it later whilst staying focused on the current task.
What's really important about these simple text files is that they remove the opportunity for my memory to forget the information. Also, just the act of writing down my thoughts forces me to think about them more deeply.
Sometimes, my mind will pick a really stupid answer. But when I write down a problem the obvious solution often stands out. Try it yourself during day-to-day development tasks.
I try to reduce mental workload as much as possible
At this stage of a project I have a good idea of what I want to build and why I want to build it. These reasons are all written down for me to clearly understand over the course of a project – not held in memory as feelings, then re-interpreted through my current state of emotion each time I think about them.
You might think all this sounds like unnecessary work, but for me the pay-off is grand. I work best and feel happiest when my mind is clear to focus on one thing and is not stressed about having to remember important details.
I'm not the smartest developer so I have to get the best out of what I've been given. Hopefully you enjoyed seeing how I try to achieve this. In the next post I'll show I started to design this project and the thought-processes that drive me.