Importance of Messaging for Enterprise applications

Communication happens between multiple applications using four major approaches.
1. File Transfer.
One application writes a file and another application reads it later. Applications need to agree on file location and name to exchange information.

2. Shared Database
Multiple application share information using a common database. This way application doesn't have to exchange information as it is a single source of data.

3. Remote Procedure
One application exposes some functionality remote so another application can access it remotely as a remote procedure call. This is real-time synchronous communication

4. Messaging.
One application publishes or sends a message to common message channel (Queue) and another application cand read or consume message from the message channel later. This application must agree on channel and Message format to do this asynchronous communication.

Challenges of communication between applications.

As a developer, while developing a software we are taking many assumptions.
For example, If we are developing an application which has to run an individual machine then we assume that while running that application it will always get enough memory and CPU cycle.  An application has never had to wait for any other resource etc.

While developing a software system which has to run on the different machine for different part of the application, we assume that communication between two system is reliable.  The network is efficient with good latency. TCP communication over network runs in a reliable manner.

Applications rarely live in isolation. any application has to communicate with other application directly or indirectly.  while working on this kind of distributed application, developers make a false assumption.  This is called "Fallacies of distributed application"

Fallacies of distributed application

https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

The network is reliable.
There can be many problems with a network like a cable cut, switch power cut, network h/w failure, firewall issue etc.  Also, there can be some security attack. Applications like e-commerce highly affected by this issues.
As a developer, we need to consider that network is unreliable and we need to address that too.

Latency is zero.
Latency is a how much time it takes for data to move from one place to another.
We develop an application on a local machine and considering LAN latency or local machine latency and considering latency is zero.  But when an application goes into production environment then the scenario is different and application results into slower in performance or transaction failure.
Ajax and Async/await are patterns are little use in this scenario.

Bandwidth is infinite.
Bandwidth is capacity of a network to transfer data (how much data can be transferred).
Bandwidth is continuously improving the network nowadays. Sometimes when bandwidth is lower then network packet loss ratio is high.
Considering bandwidth is not infinite, we should limit the size of information we send over the wire.
Always test application in a production environment for this kind of reason.

The network is secure.
The Network is far from secure.
1. General Internet attack trends are showing a 64% annual rate of growth
2. The average company experienced 32 attacks per week over the past 6 months
3. Attacks during weekdays increased in the past 6 months" [RipTech].

As a developer, we always think of security and awareness of all security and the implications. Sometimes provides message-level security for more confidential information considering network is unsecured.

Topology doesn't change.

When we deploy an application on production environment its topology may change. on Network we have to change servers,  load balancers,  Active directory settings etc. Sometimes we change the whole vendor with different network architecture (AWS/Azure).
The application should work in any of the topology change.

The application should not depend on specific endpoints or routes. It should be always prepared for renegotiating endpoints or provide discovery services.
Another strategy is to abstract the physical structure of the network. Instead of depending on specific IP address, the application should depend on DNS name.

There is one administrator.
If we have an application subsystem which outside of your administrative control then it can be an issue for your application.
We have integrated third-party pricing supplier API. Sometimes they change their security protocol and strategy without even informing us in advance.  That results in failure of a part of our application. Sometimes they change data model also without informing in advance. Sometimes they let us know in advance about their change.
This type of changes is not in control of one administrator.  We should develop our application in a way that it can very less depend on their model structure. Also, we can provide two version of implementation. One can work with an older version and another for the new version so the application can decide automatically which one needs to use until it finals all the changes.

Transport cost is zero.
When information transfer over the network it has to be serialized into bits. This conversion takes server resources and also add latency.
Also, All networking instrument like the router, subnet, load balancer has its own cost. Some cloud providers also charge based on data transferred across applications.
Developers normally ignore this costs.  They should consider this aspect also and develop the cost-effective application.

The network is homogeneous.
The network is not same everywhere. Even at home in the Similar network also may use different systems like Linux and windows. This can be an issue of data formate, protocols etc.
The application should use a standard protocol like XML/JSON.  The application should be used both according to client request it should respond in a particular format


Messaging System can save some of the above scenarios.

What is a messaging system?

We can take an example of telephone voice mail system.  One person calls to another person and leaves a voicemail. That voicemail queued to a message queue(mailbox). once person wanted to listen to voicemail, it dequeues a message from the queue.

Messaging definition: Messaging is a technology that enables high-speed, asynchronous, a program to program communication with reliable delivery.

Applications communicate by sending packets of data called Messages to each other.
Channels(queues) are logical pathways that connect that program and convey a message.
Sender or producer is an application that sends messages by writing the message to a channel. Reciever or Consumer is an application receives a message by reading from the channel.

Messaging capability provided by separate software called Messaging system or Message Oriented Middleware (MOM).  NServiceBus, MassTransit.

Messaging system manages messages while send/publish a message to consume the message. It also uses a database to persists message and state. An administrator has to configure a messaging system with messaging channel that defines the path of communication between sender application and receiver application. The primary purpose of the messaging system to move messages from sender application to receiver application in a reliable fashion. While transmitting a message from sender to receiver, at any point, failure occurs then Messaging system retry to transmit until it succeeded.

Message transmits in following steps:

1. Create: sender creates a message using message model class and populates with data.
2. Send: The sender add a message to the channel.
3. Deliver:  Message system moves the message from senders machine to receivers machine.
4. Receive: The receiver reads the message from the channel.
5. Process: Receiver extract data from the message.

Advantages of Messaging system

1. Remote Communication reliable
Messaging enables separate applications to communicate and transfer data.

2. Plateform / Language integration.
Messages can be consumed by an application written in a different language and deployed on a different platform as it message formate is standard like XML or JSON.

3. Asynchronous communication
Messaging provides fire and forgets communication. The application doesn't need to wait until it finishes execution.

3. variable timing:
In synchronous communication, both applications have to wait for each other to complete a transaction.
In messaging scenario, both applications can work on at their own pace.  no one has to wait for each other.

4. Throttling
In an RPC application it is a single point of the receiver of requests and When there is too many requests comes to a single receiver then it may crash the application. In Messaging scenario it only accepts message from queue whenever it is ready.

5. Reliable communication.
    Covering all fallacies,  Messaging system uses to store and forward approach to transmitting messages. Sending system stores messages and receiver system also stores a message. Messaging system resends messages until it received by receiver successfully.

6. Disconnect operation:
Some applications are designed to work in a disconnected environment.  Messages are queued. Whenever it connects to a network it can start consuming or sending messages.

7. Mediation
Messaging system can act as a mediator between two application.

8. Thread management.
By Asynchronous communication, Threads doesn't need to be blocked until other applications finish its work.  Request/Response scenario also can be implemented by the messaging system.

Challenges of Messaging system

Complex programming model
Implementation of messaging system is little complex then RPC or another synchronous system. Debugging is complex.  Requires little learning curve for new developers.

Sequence issues
Messaging system guarantee message delivery but not in proper sequence.  You need to take special care of it. Implement saga and correlation can resolve this issue.

Synchronous scenario
In many cases, we need to provide instant feedback to the client about performed operation result.
We need to implement request/response or Push notification to cover up this.


Ref: Book: Enterprise-Integration-Patterns-Designing-Deploying
https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
www.rgoarchitects.com/Files/fallacies.pdf

Polling from angularJs ($interval)

Recently come across the situation that we need to get status for one of disconnected service and show on browser.  We end up doing polling from angular Js $interval service. We could have use SignalR features of asp.net for same thing  but wanted to do it more simple way.

We can do it polling easily using $interval service.
here is detail document about $interval  https://docs.angularjs.org/api/ng/service/$interval

Problem using $interval is that if you are doing $http service calling or server interaction and if server delayed response more than $interval time then before your one request completes, it starts another request. your next interval will be in race with previous response.

Solution:

1. To avoid  above problem, polling should be simple status getting from server like a single bit or very lightweight json so it should be complete within your defined interval time. You should also define time of interval appropriately to avoid this issue.

2. If somehow it is still happening due to any reason, you should check a global flag that checks previous request finished or not before sending any other requests. It will miss that time interval but it won't send request prematurely.

Meaning if you set $interval time as 1000 milliseconds and it sent first  get request to server and that request didn't sent response yet before 1000 milliseconds and your second interval starts before completes previous one.

Someone has resolve this issue using just $timeout service. here is the blog.  This guy has used recursion to resolve this issue. meaning you can call same polling function from resulted promise. This is good but i believe that recursion saves function call stack.  All browser has its one limit of stack.  browser stack limit also depend on other factors  like memory, open number of tabs etc.

Someone has done very nice research on it.

https://www.nczonline.net/blog/2009/05/19/javascript-stack-overflow-error/
https://web.archive.org/web/20110128022845/http://www.javascriptrules.com/2009/06/30/limitation-on-call-stacks/

You can also perform some test using following url
http://adamrich.name/recursion.html

Another problem with using $interval service is, We should stop interval after some threshold value because it may happens that server has some issue or error and it may never turn in to your desired result from server to stop your interval.

Here is working example where you can use $interval solve both of above problems.

'use strict';

angular.module('myApp.view2', ['ngRoute'])

.controller('View2Ctrl', ['$scope', '$timeout', '$interval', '$http', function ($scope,  $timeout, $interval, $http) {

        $scope.title = "Test Title";

        $scope.data = [];

        var hasvaluereturnd = true; // Flag to check

        var thresholdvalue = 20; // interval threshold value

        function poll(interval, callback) {

            return $interval(function () {

                if (hasvaluereturnd) {  //check flag before start new call

                    callback(hasvaluereturnd);

                }

                thresholdvalue = thresholdvalue - 1;  //Decrease threshold value

                if (thresholdvalue == 0) {

                    $scope.stopPoll(); // Stop $interval if it reaches to threshold

                }

            }, interval)

        }

        var pollpromise = poll(1000, function () {

            hasvaluereturnd = false;

            //$timeout(function () {  // You can test scenario where server 

                                      takes more time then interval

            $http.get('http://httpbin.org/get?timeoutKey=timeoutValue').then(

                function (data) {

                    hasvaluereturnd = true;  // set Flag to true to start new call

                    $scope.data = data;

                },

                function (e) {

                    hasvaluereturnd = true; // set Flag to true to start new call

                    //You can set false also as per your requirement in case of error

                }

            );

            //}, 2000);

        });

        // stop interval.

        $scope.stopPoll = function () {

            $interval.cancel(pollpromise);

            thresholdvalue = 0;     //reset all flags.

            hasvaluereturnd = true;

        }

    }]);

Reference:

Above code just wrote on angular js seed boilerplate  code for angularjs 1.x. Github location is here
Also, we have used http://httpbin.org/  for any third party test call. Which is very nice  to test all http library. you can send any type of request and get response.

Session Helper

I am writing this for my own repository of  using session variables and though may be useful to others friends also.  Over use of session variables are not good. Try to avoid using it. It takes server memory. If you want to use it then this code example shows how to use it without worrying to remember Session keys and simply access like a property with appropriate type.

I am explaining everything using code and code comments. 

/// Session Helper class
   public static class SessionHelper
   {
 
       /// The userinfo Key
       const string USERINFO = "USERINFO";
 
       /// The userclaiminfo Key
       const string USERCLAIMINFO = "USERCLAIMINFO";
 
       
       /// Property to Get or sets the user claim information from or to Session.
       public static UserClaimsInfo UserClaimInfo
       {
           get
           {
               return GetInfoFromSession<UserClaimsInfo>(USERCLAIMINFO);
           }
           set{
               SetInforToSession<UserClaimsInfo>(value, USERCLAIMINFO);
           }
       }
 
       
       /// Property to Get or set the user information from or to session.
       public static Userinfo UserInfo
       {
           get
           {
               return GetInfoFromSession<Userinfo>(USERINFO);
           }
           set{
               SetInforToSession<Userinfo>(value, USERINFO);
           }
       }
 
       
       /// Priavate generic method to Get information from session using Key.
       private static T GetInfoFromSession<T>(string key)
       {
       if (HttpContext.Current.Session[key] != null && HttpContext.Current.Session[key] is T)
           {
               return (T) HttpContext.Current.Session[key];
           }
           return default(T);
       }
 
       
       /// Priavate method to set information to session using Key.
       private static void SetInforToSession<T>(T info, string key)
       {
           HttpContext.Current.Session[key] = info;
       }
   }

In above code block You can add individual property for each session variable.  You can to define constant variable for defining key.  Session variable property can use generic method  GetInfoFromSession<T> and SetInfoToSession<T> to get and set values to and from session. 

Thanks.

Code snippet search

Code snippet search is going to be very easy now and in future with our favorite IDE like visual studio.

It is using Microsoft Bing. 

Bing searches code from coding community like 

•   MSDN
•   StackOverFlow
•   Dotnetperls
•   CSharp411

To search a code snippet, You can go to following Url.

http://codesnippet.research.microsoft.com/

It is looking like this.

All instruction is written there like how to get code snippet.

Start with special comment "///".

Then you just need to type your code snippet query in English like you search in Google. and press Tab will show you intellisense like window. which also show code in next window beside the main window.

You can select and  get the code snippet. 

You can select and  get the code snippet.   

You can also do directly from Visual Studio directly.   

You can download VS extension from here. 

Check this blog for more 

detail http://blogs.msdn.com/b/visualstudio/archive/2014/02/17/introducing-bing-code-search-for-c.aspx

Who want to remember all the code all the time. 

There are some other websites also there for code snippet repository

Like 

• http://snipplr.com/
• http://css-tricks.com/snippets for CSS.

Happy Coding....

TFS with VS 2013 tutorial

Microsoft has provided online Visual studio support for Team Foundation Server and Git.
You can create your visual studio account and access your own TFS. You can use your own code repository online anytime anywhere. It is free for up to 5 Users.

Go to following url : http://www.visualstudio.com/

Click on "Get Started for free" highlighted in green background.

Sign in using your any windows live account user id/ password. If you don't have windows live id then you can create one by clicking "Sign up now".

After login, You will get this page. You will find your name as logged in User. If you want to use Visual studio 2013 trial version then you can download from here. 

Click on your user name and you will get following page.

You can find URL starts with your username like {your username}.visualstudio.com. It is your online TFS access link. click on that link. You can add up to 5 Users for free.

As I was the early adopter of this feature, Microsoft has extended my "Visual studio online Ultimate" version for free. After 62 days it will be a standard account with limited feature. You can get to know about all features or get it from here.

Features:

Categories and capabilities	Visual Studio Online Advanced	Visual Studio Online Professional	Visual Studio Online Basic	Visual Studio Premium with MSDN
Maximum number of users with this plan on an account	Unlimited	10 users	Unlimited	Unlimited
Unlimited team projects and private, hosted code repos using TFVC or Git
Project planning and bug tracking tools, including Kanban boards and team velocity forecasts
Integration with popular development tools, including Visual Studio, Eclipse, and Xcode
Work in one IDE to create solutions for the web, desktop, cloud, server, and phone
Support for Office 365 business apps
Track complex projects with hierarchical portfolio backlogs
Visualize project data with work item chart authoring
Discuss projects with teammates and monitor project events in Team Rooms
Send feedback requests to users and stakeholders
Manage and run tests and test plans
Host team projects on-premises as well as in the cloud

Learn

Microsoft has provided detail documentation about how online TFS works. 

LIke following. I have provided Link for each. I will still give overview of How to setup Code with Visual studio.

• Setup
• Code
• Work
• Build
• Test
• Deploy
• Insights
• Cross-Platform

   

Create Project 

Go on your visual studio url {your username}.visualstudio.com.  This is dashboard of all your project and project activities.  Find "New" link under "Recent project & teams" to create New project. Check highlighted "New" link in following screen shot. 

Click on "New", It will open Modal window to create Project. You can insert Project Name, Description. You also can select Process Template and Version control.  Microsoft has provided following Process Template. 

• Microsoft Visual Studio Scrum 2013
• MSF for Agile Software Development 2013
• MSF for CMMI Process Improvement 2013

These are the process followed by Software development team to develop software. 

Microsoft has come up following two options for version control.

• Team Foundation Version Control
• Git ( Github)

After insert of all required details, you can create project by click on "Create project" button. It will crate project using selected Process template.

Visual Studio with Online TFS

Go back to Visual studio.  I am using Visual studio express 2013. I am going to show how can you connect your visual studio with your online TFS which you have configured above.

Open Visual studio and click Menu link "TEAM".  You will find  sub menu link "Connect to Team Foundation Server..".

click on it.

It opens "Team Explorer" tab at right side of VS window.  Find connector icon on top. If you mouse hover on it it shows "Connect to Team Projects". Click on it. It opens small "Connect" tab. 

Click on "Connect" and it will popup screen for "Connect to Team Foundation Server".

Click on "Servers.." and it opens following another popup where you can add your own visual studio url which we have created before explained in above section.  {yourusername}.visualstudio.com.  Once you enter your url it will disable all other options like Path, protocol. online visual studio uses HTTPS.  No need to select anything just your correct url is enough.

Once you added TFS server. you can select same from servers drop down. It will show  DefaultCollection in left tab and all projects on left side. You can find Project you have created online and explained in above section. check below screen shot.

On Team Explorer you will find your project.  Click on Source Control Explorer.

You can create your project folder and map it with your local drive. 

Follow my previous blog to create project in TFS for more details here.  

Online TFS has all features of TFS including Project Management like Scrum dashboard, Tracking Product Backlog items, Sprint Backlog items, Scrum velocity charts, Bug Tracking, Automated Unit Testing, Load testing, Automatic Build Deployment,  also has support with Windows Azure.

In my next blog I will try to cover how to manage Scrum Dashboard, Create Product backlog items, how to track Bug etc.
Thank you.
~Atul