Git branching strategy

This post explores different Git branching options to optimally manage code base. Primarily, there are two options which most of the teams use - Git flow and GitHub flow. Let's cover them in detail.

Git Flow

Git Flow works on the idea of releases; you have many branches each specializing in specific stages of application. The branches are named as master, develop, feature, release and hotfix.

 You create a develop branch from the master branch which will be used by all developers for feature development and bug fixes for a specific release. If a developer needs to add a new feature then he will create a separate branch from develop branch so that both branches can evolve independent of each other. When feature is fully implemented, that branch gets merged back to develop. Idea is to have a stable and working develop branch; so feature gets merged back only when it's complete. Once all the features related to a release are implemented; develop branch is branched to release branch where formal testing of the release will commence. Release branch should be used only for bug fixes and not development. And finally, when it's ready for deployment it will be tagged inside master branch to support multiple feature releases. 

http://nvie.com/posts/a-successful-git-branching-model/

This flow is not ideal for projects where you do releases quite often. This helps where you have releases happening like every other month or even every month. 

GitHub Flow

https://guides.github.com/introduction/flow/

GitHub has popularized it's own flow which is quite effective for open source projects and other projects which doesn't have longer release cycles (in this, you could release every day). It's based on the idea of features. 

https://guides.github.com/introduction/flow/

It's light weight version of git flow which removes all unwarranted overhead of maintaining so many branches (It's just have feature branches apart from master). If you want to work for a feature or bug you start a separate branch from master. Once feature/bug is implemented it get's merged back to master branch. Feature branches are given quite descriptive names like oauth-support or bug#123. Developers can raise pull request to collaborate with co-workers or to do code review on the feature branch. Once code is reviewed and sign off is received the featured branch is merged back to master branch. You can have as many feature branches as you wish and once feature branch is merged it can also be deleted as master branch will have all commit history (unless you don't want). 

http://scottchacon.com/2011/08/31/github-flow.html

GitHub flow assumes that every time you merge changes to master; you are ready for production deployment.

There is 3rd branching strategy as well; GitLab Flow.


References:
https://about.gitlab.com/2014/09/29/gitlab-flow/

--happy branching !!!

Show Directory Tree structure in Unix

Graphics based operating systems provide the option for you to see the complete directory structure graphically. But, in console-based Linux/Unix operating systems this is not possible (by default). At most, you can go inside each directory and do ls or if you are a scripting geek you can create one to do the same for you.

Recently, I came across directory listing program, tree which makes this job possible in Unix/Linux from the terminal. 

Installation

CentoOs/RHEL: $sudo yum install tree -y

Ubuntu: $sudo apt-get install tree

Using Tree

Use tree and the directory name. That's it!

$tree cookbooks/

cookbooks/

`-- learn_chef_httpd

    |-- Berksfile

    |-- chefignore

    |-- metadata.rb

    |-- README.md

    |-- recipes

    |   `-- default.rb

    |-- spec

    |   |-- spec_helper.rb

    |   `-- unit

    |       `-- recipes

    |           `-- default_spec.rb

    `-- test

        `-- recipes

            `-- default_test.rb

7 directories, 8 files

How Couchbase identifies Node for Data Access

This post talks about how Couchbase identifies where exactly the document is stored to facilitate quick read/update.

Couchbase uses a term Bucket which is equivalent to the term Database in the relational world to logically partition all documents across the cluster (or data nodes). Being a distributed DB, it tries to evenly distribute or partition (or shard) data into virtual buckets known as vBuckets. Each vbucket owns a subset of the keys or document id (and of course corresponding data as well). Documents get mapped to vBucket by applying the hash function on the key. Once the vBucket is identified there is a separate lookup table to know which node hosts the vBucket. The thing which maps different virtual buckets to nodes is known as vBucket map. (Note: Cluster Map contains the mapping of which service belong to which node at any given point of time)

Steps Involved (as shown in the diagram):

1. Hash(key) to get vBucket identifier (vb-k) which hosts/owns Key.
2. Looking up vBucket map tells vb-k is owned by node or server n-t
3. The request is sent directly to the primary server node, n-t to fetch the document. 

Both hashing function as well number of vBucket is configurable. By default, Couchbase automatically divides each bucket into 1024 active vBuckets and 1024 replica buckets (per replica). When there is only one node, all vBuckets reside on that node. 

What if a new node gets added to the cluster?

When the number of nodes scales up or down the information stored in vBuckets are re-distributed among the available nodes and then the corresponding vBucket map is also updated. This entire process is known as rebalancing. Rebalancing doesn't happen automatically; as an administrator/developer you need to trigger it either from UI or through CLI.

What if the primary node is down?

All read/update request by default go to the primary node. So, if a primary node fails for some reason, Couchbase takes off that node from the cluster (if configured to do so) and promotes the replica to become the primary node. So you can fail-over to replica node manually or automatically. You can address the issue with the node, fix it and add it back to the cluster by performing rebalancing. Below table shows a sample mapping.

+------------+---------+---------+
| vbucket id | active  | replica |

|     0      | node A  | node B  |
+------------+---------+---------+
|     1      | node B  | node C  |

+------------+---------+---------+
|     2      | node C  | node D  |

|     3      | node D  | node A  |

References:

http://docs.couchbase.com/admin/admin/Concepts/concept-vBucket.html

http://docs.couchbase.com/admin/admin/Tasks/tasks-rebalance.html

http://www.couchbase.com/sites/default/files/uploads/all/whitepapers/Technical-Whitepaper-Couchbase-Server-vBuckets.pdf

http://dustin.sallings.org/2010/06/29/memcached-vbuckets.html

https://medium.com/@sent0hil/consistent-hashing-a-guide-go-implementation-fe3421ac3e8f

---

happy learning !!!

Why Multi-Dimensional Scaling in Couchbase

Couchbase has been supporting horizontal scaling in a monolithic fashion since its inception. You keep adding more nodes to the cluster to scale and improve performance (all nodes being exactly the same). This single dimension scaling works to a great extent as all services - Query, Index, and Data scale at the same rate. But, they all are unique and have specific resource requirement.

Let's profile these services in detail and their specific hardware requirements to drive home the point - why MDS is required! 

This feature got added in Couchbase 4.0.

Query Service primarily executes Couchbase native queries, N1QL(similar to SQL, pronounced as nickel - leverages the flexibility of JSON and power of SQL). The query engine parses the query, generates execution plan and then executes the query in collaboration with index service and data service. The faster queries are executed, the better the performance.

Faster query processing requires more CPU or fast processor (and less memory & HDD). More cores will help in processing queries in parallel. 

Reference on  - Query Data with n1ql

Index Service performs indexing with Global Secondary Indexes (GSI - similar to B+tree used commonly in relational DBs). Index is a data structure which provides quick and efficient means to access data.  Index service creates and maintains secondary indexes and also performs an index scan for N1QL queries. GSI/indexes are global across the cluster and are defined using CREATE INDEX statement in N1QL. 

Index service is disk intensive so Optimized storage / SSD  will help in boosting performance. It needs a basic processor and less RAM/memory.  As an administrator, you can configure GSI with either the standard GSI storage, which uses ForestDB underneath (since version 4.0), for indexes that cannot fit in memory or can pick the memory optimized GSI for faster in-memory indexing and queries. 

Data Service is central for Couchbase as data is the reason for any DB.  It stores all data and handles all fetch and update requests on data.  Data service is also responsible for creating and managing MapReduce views.   Active documents that exist only on the disk take much longer to access, which creates a bottleneck for both reading and writing data. Couchbase tries to keep as much data as possible in memory.
Data refers to (document) keys, metadata, and the working set or the actual document.   Couchbase relies on extensive caching to achieve high throughput and low read/write latency. In a perfect world, all data will be sitting in memory.

Data Service: Managed Cache (based on Memcached) + Storage Engine + View Query Engine

Memory and the speed of storage device affect performance (IO operations are queued by the server so faster storage helps to drain the queue faster). 

Why MDS

So, each type of service has it's own resource constraints. Couchbase introduced multi-dimensional scaling in version 4.0 so that these services can be independently optimized and assigned the kind of hardware which will help them excel. One size fits all is not going to work (especially when you are looking for higher throughput i.e. sub-milliseconds response times).  For example, storing data and executing queries on the same node will cause CPU contention. Similarly, storing data and indexes on the same node will cause disk IO contention.

http://blog.couchbase.com/introducing-multi-dimensional-scaling

Through MDS, we can separate, isolate and scale these three services independent of each other which will improve resource utilization as well as performance.

http://developer.couchbase.com/documentation/server/4.5/concepts/distributed-data-management.html

References
http://www.couchbase.com/multi-dimensional-scalability-overview
http://developer.couchbase.com/documentation/server/4.5/concepts/distributed-data-management.html

---
happy learning !!!

IP Address in Private Range

Ah, again; I forgot the range of private IP address. So, no more cursing of my memory. Now...instead of googling I will search on my blog :D

Below are permissible private IP ranges:

• 10.0.0.0        - 10.255.255.255
• 172.16.0.0    - 172.31.255.255
• 192.168.0.0  - 192.168.255.255

These IP address are used for computers inside a network which needs to access inside resources. Routers inside the private network can route traffic between these private addresses. However, if they want to access resource in outside world (like internet) these network entities have to have a public address in order for response to reach to them. This is where NATing is used. 

Representing IP address in CIDR format

The number following the forward slash (/) is the prefix length, the number of shared initial bits, counting from the most significant bit of the address. 

Thus, a /20 block is a CIDR block with an unspecified 20-bit prefix. 

In below example, 10.0 is the network address and last two segments are device addresses. 

Reference: http://whatismyipaddress.com/nat