A Beginner’s Guide to Garbage Collection in .NET

Ronnie Mukherjee 0 Comments

Dustbin

Garbage collection is often seen as a kind of black magic in the world of .NET, particularly by junior programmers. There is no real need to understand how it works in order to build most applications, therefore it remains a mystery until a time comes when you think you need to know how it works, or else you decide that it might be interesting to learn about. I remember early in my career being part of a team faced with a performance problem which was due to a memory leak. We decided to respond by littering the code with GC.Collect calls to see if that helped. Recalling this makes me wince. It is a classic example of “hit and hope”, when really we should have taken more time to try and understand and diagnose the problem. At the very least, having a basic understanding of garbage collection may prevent you from attempting to solve a performance problem in this way, without understanding what that call to GC.Collect is actually doing.

What is garbage collection in .NET?

Garbage collection is the automatic process of freeing up (or deallocating) memory which is being taken up by objects your application no longer needs. Most developers know this much at least.

Every time you instantiate an object in .NET some memory is allocated to store that object. But at some point that object may no longer be needed by your application. Only by deallocating its memory does it become available again for your application to reuse. If memory is not deallocated, then sooner or later your application will run out of memory and stop working.

So garbage collection is a must?

No, garbage collection is just one approach to the problem of how to deallocate memory. In some languages, such as C and C++, it is the responsibility of the programmer to keep track of which objects are no longer needed, and to explicitly deallocate memory as required.

How does garbage collection work?

Inside the CLR lives a ‘garbage collector’. Unsurprisingly it is the responsibility of the garbage collector to manage garbage collection. It does this by periodically performing ‘garbage collections’. Every time a garbage collection is performed, the garbage collector basically looks at the memory (i.e. the managed heap for your application) and frees up memory which is no longer needed, that is, memory which is occupied by ‘dead objects’.

How does it know when an object is ‘dead’?

An object is dead if it is unreachable by your code. The obvious example is a local variable inside a method. There is no way for your code to access that variable once the method has returned, so it becomes ‘dead’.

How often does the garbage collector perform a garbage collection?

There are three ways a garbage collection can be triggered.

Firstly, if your system has low physical memory, this can trigger a garbage collection.

Secondly, a threshold is defined which indicates an acceptable level of memory on the heap which can be used by allocated objects. If this threshold is surpassed, then a garbage collection is triggered.

Finally, a garbage collection can explicitly be triggered by calling the GC.Collect method. Only under very rare circumstances is this ever required.

So ignoring the case of calling GC.Collect, a garbage collection is basically triggered when the garbage collector figures that it might be useful to free up some memory?

Yes.

So whenever a garbage collection is triggered, it frees up all memory on the heap that is occupied by dead objects?

No, because scanning the entire managed heap for dead objects might take a long time, and thus affect performance. Every time a garbage collection is triggered, execution on all other threads is paused until it completes.  So the garbage collector tries to be efficient in the way it looks for and deallocates dead objects. It is selective.

How is it selective?

Basically, every object on the heap is categorized into one of three ‘generations’. These are called ‘Generation 0’, ‘Generation 1’ and ‘Generation 2’. The generation of an object indicates how ‘old’ it is, that is, how long it has been since it was created. Broadly speaking, Generation 0 is for younger objects, Generation 1 is for middle-aged objects, and Generation 2 is for older objects.

When a garbage collection occurs, it does so on a specific generation, and deallocates any dead objects on that generation and on all younger generations. So a collection on Generation 1 will deallocate dead objects in Generations 0 and 1. The only time that all dead objects on the heap are deallocated is if a collection is performed on Generation 2.

Garbage collection is generally performed on Generation 0, that is, on short-lived objects. This is based on the reasonable assumption that the objects that are most likely to be dead are the ones which have most recently been allocated.

If an object ‘survives’ a garbage collection, it is ‘promoted’ to the next generation. So Generation 0 objects which are still alive at the time of a garbage collection are promoted to Generation 1. The assumption here is that if it is still alive after this collection, there is a good chance that it will still be alive after the next one, so we will move it out of our ‘top priority’ Generation 0 collection.

Presumably new objects are allocated into Generation 0 then?

When a new object is allocated, it goes into Generation 0, with one exception. Large objects (larger than 85,000 bytes) go straight into Generation 2. This decision is based on an assumption that large objects are more likely to be long-lived.

…and that’s pretty much it as far as the basics of garbage collection go.

As we can see, the garbage collector makes a few assumptions about your application to help it decide how to behave. Only when these assumptions turn out to be inappropriate for your application do you need to consider the possibility of manipulating it. For example, you can configure the ‘intrusiveness’ of the garbage collector (how often it is triggered), or explicitly trigger a collection on a specific generation.

The fact that many developers never feel the need to understand how garbage collection works is perhaps an indication that it does its job quite well. The garbage collector does the work so you don’t have to.

My 3 Favourite Productivity Tools

Ronnie Mukherjee 0 Comments

Over the years I have been something of a productivity tool junky. Like many programmers I love trying out new productivity tools and utilities, but 90% of them don’t stand the test of time. I will generally try something out, fail to be impressed, and cast it aside to make way for something more interesting. However, there are three tools in particular which have stood the test of time, and which I have been using on an almost daily basis for years. These are ‘secondary’ tools – they are not essential to my performing my work, but they really make my work easier and help me to get things done more quickly. Each of them is lightweight, minimalist in terms of functionality and free of charge. I have found that paid-for tools are often overloaded with complexities which aren’t needed, perhaps in an effort to justify the price tag. There are so many great free utilities available now that I very rarely find the need to pay for any productivity tool.

Without further ado, here are my three favourite software productivity tools, in no particular order.

1. SlickRun

I hate having to use the mouse. It’s a lot easier to achieve a state of flow by using the keyboard exclusively, and it can really breaks your stride when you have to move your hand over to the mouse and move the cursor or click. Maybe it utilises a different part of the brain. In any case, any tool which helps me to avoid using the mouse is likely to win my approval, and SlickRun does just that.

SlickRun allows you to define commands for common tasks, in particular for opening applications. For example, I have defined a command ‘sql’, which opens up Oracle Sql Developer (as I am working with Oracle databases at present). A small command prompt floats on your desktop, by default just above the date and time in your Windows taskbar.

software_tools1

 

It it is small enough to be barely noticeable, but large enough to let you see what you are typing. To focus your cursor in this command prompt you just hit Win+Q.

To manage your shortcuts you simply type ‘setup’.

software_tools2

In addition to this primary function, allowing you to quickly and easily open applications and run commands, SlickRun includes a couple of other very useful features which I use regularly.

When your cursor is not focused on the command prompt, SlickRun displays the percentage of RAM on your machine which is currently available.

software_tools3

I know that when this figure falls below 10% my machine will start to run slowly, and I need to close some applications down or end some processes. If my system is grinding to a halt, I will often see that I have only 3% or 4% of RAM available.

The final feature of SlickRun which I find immensely useful is something called SlickJot. Hitting Win+J opens up a ‘JOT’ where you can store notes and other useful bits of random text. Whatever you type here is automatically saved, and you can quickly close the ‘JOT’ by hitting Escape. It is roughly equivalent to having a Notepad text file which you can open and close with keyboard shortcuts, and which saves automatically.

software_tools4

2. Ditto

An even simpler utility is Ditto, which does one thing but does it well.

Ditto is an extension to the Windows clipboard, which simply allows you to not only paste the last thing you copied, but also to paste anything you have copied prior to that, looking back as long as you want. This is a particularly useful utility for programmers, which you may not realise until you try it. Again it requires no mouse interaction. You cut and copy stuff as normal, but when you then want to paste something which you have previously copied, you hit CTRL+’ which opens a list of things which you have placed on the clipboard, with the most recent first. This list is quickly filtered as you type, and unless you clear it, will store items that were placed on your clipboard days or weeks ago.

software_tools5

3. Workflowy

A far more popular tool than SlickRun and Ditto, but equally simple and effective, is Workflowy. It is a web-based application which allows you to write and store notes in a hierarchical structure. We all make lists containing items and sub-items, and workflowy is the most usable system for doing so that I have come across. The beauty of workflowy is the speed with which you can use it. Its keyboard shortcuts allow you to instantly expand, collapse, traverse, delete or move items. If you have a mind which can move quite quickly like mine, you will love it. Mindmaps are undoubtedly an effective tool for brainstorming and organizing your ideas, but there is no mindmapping tool which allows you to capture, process and organize your thoughts anywhere near as quickly as workflowy. I still sometimes use mindmaps with a pen and paper, but once you’ve drawn a branch on paper, you obviously can’t instantly move, edit or delete it.

software_tools6

Like my other favourite tools, workflowy has a very minimalist feel about it, favouring simplicity over bells and whistles. You can’t write text in different colors or fonts, which I think is great. I use it on a daily basis, whether I am outlining a blog, writing a to-do list, or brainstorming. Workflowy is free for up to 500 list items per month, but you can gain additional items by referring friends. I currently have 1500 items per month available per month, free of charge, and this is more than enough for my needs.

I sporadically use the Getting Things Done methodology for organising my tasks and ideas, and I find workflowy is a great tool for helping me to do this.

Final Words

Using the right software tools and utilities can have a huge impact on your productivity. As a general rule, anything which allows you to use keyboard shortcuts to perform common tasks is probably going to help you a lot. The three tools I have described here do just that. They are all free to use, lightweight and are either web-based or can be downloaded and installed in seconds. If you haven’t tried them already, I urge you to do so. I would also love to hear of any lightweight and free tools and utilities which you find invaluable in your work.

B-tree Indexes

Ronnie Mukherjee 0 Comments

Round bookshelf in public library

You probably know that database indexes are a means for improving database performance. But surprisingly few people understand the different types of indexes there are, how they work, and how to choose an appropriate index for a specific performance problem – or indeed whether an index will help at all. This article may help you get started. It will focus on B-tree indexes, the most commonly used index type in both Oracle and SQL Server databases. Some terminology, such as ‘branch blocks’ and ‘leaf blocks’ is specific to Oracle databases, as these are my primary interest at present.

Why Database Indexing?

In short, indexes help our database to find rows quickly.

Without an index, our database must perform a full table scan every time it needs to find a row or a collection of rows. A full table scan describes a process where the database just looks through a table one row at a time, looking for the desired row or rows. In large tables this can be slow. An index provides shortcuts which the database can use to help find what it is looking for more quickly.

One analogy is a library. Imagine how long it would take you to find a particular book if the books in your local library were arranged in no particular order. You would have to examine one book at a time until you found what you were looking for. Thankfully, libraries are organized in such a way as to provide clues on where to look. Books are typically arranged by genre and by author. In most libraries there are computer terminals dotted around which you can use to access a catalog. You type in a title or author, and the catalog tells you where exactly to look for your book. In this way the catalog is much like a database index.

The key of an index is the column or group of columns to which the index is applied. An index can only help speed up a query if its key includes one or more of the columns used in the query’s predicate (i.e. in its WHERE clause). In a library, if all you know is a book’s title, then having all books arranged solely in order of the author’s surname isn’t going to help you.

B-tree Indexes

B-tree indexes are a particular type of database index with a specific way of helping the database to locate records. B-tree stands for ‘balanced tree’ (not ‘binary tree’ as I once thought). A B-tree index orders rows according to their key values (remember the key is the column or columns you are interested in), and divides this ordered list into ranges. These ranges are organized in a tree structure, with the root node defining a broad set of ranges, and each level below defining a progressively narrower range.

Consider again our library example, and imagine that the books in the library are arranged purely with regards to the author’s surname. Now let’s say you are looking for a book by an author named Osman.

On entering the library you see that the ground floor contains books by authors named A-G, the first floor is H-N, the second floor O-U and the third floor V-Z. So you go directly to the second floor to search for books by Osman. On the second floor you see a sign which describes how the books are arranged on this floor. There is one aisle for O-R, and another aisle for S-U, so you progress to the O-R aisle. In this aisle there is a shelf for each letter, so you find the shelf for O, and here the books are ordered alphabetically so you can quickly find your book by Osman.

Now let us consider the equivalent database search. Imagine all the books in the library were contained within a Books database table, with a B-tree index on the author_surname column. To find your book by Osman, you might perform the following query:

SELECT * FROM Books WHERE author_surname = ‘Osman’

First the database would examine the root node of the B-tree index, which might define four ranges corresponding to our four floors in the library (A-G, H-N, O-U, V-Z). It would then progress to the node in the next level of the tree representing the O-U range, corresponding to progressing to the second floor of our library. Within this node would be a further set of ranges (O-R, S-U), corresponding to the aisles on the second floor, and so on.

Each node in the B-tree is called a block. Blocks on every level of the tree except the last are called branch blocks, and those on the last level are called leaf blocks. Entries in branch blocks consist of a range and a pointer to a block on the next level of the tree. Each entry in a leaf block represents a row and consists of a key value (e.g. ‘Osman’) and a rowid which the database uses to obtain all other data contained within the relevant row.

Traversing Across Leaf Nodes

In addition to traversing up and down the B-tree, the database also has the ability to traverse horizontally between leaf nodes. This can be useful, for example, when processing a query containing an ORDER BY clause on the indexed column. Consider the following query:

SELECT * FROM Books ORDER BY author_surname

With our B-tree index, the database can simply start at the first leaf node in the tree, and traverse horizontally across all leaf nodes in sequence to obtain the requested ordered collection. Without this index, sorting the rows by surname would obviously be a much more complicated and slower operation.

Traversing from leaf node to leaf node is also useful when executing queries which return a range of rows based on a predicate. For example, consider the following query:

SELECT * FROM Books WHERE author_surname > ‘O’

Using our B-tree index, the database will traverse down the tree from the root until it reaches the leaf node containing rows where author_surname begins with ‘O’, and then simply traverse to the right across all remaining leaf nodes to obtain books for ‘P’, ‘Q’ and so on.

Final Words

There are a number of ways you can configure a B-tree index to meet your specific needs. In Oracle you may want to look into index-organized tables, reverse key indexes, descending indexes and B-tree cluster indexes. In addition, you may want to consider Bitmap indexes, which are an alternative to B-tree indexes. These topics may be covered in future blog posts, but hopefully this article has given you a quick introduction to B-tree indexes.

A Tale of Optimization (part 2)

Ronnie Mukherjee 0 Comments

Click here for Part 1

Chapter 3: The Select n + 1 Problem

The Select n + 1 problem describes a situation where you are hitting the database many times unnecessarily, specifically once for each object in a collection of objects. As I mentioned in part 1, I had been struggling to diagnose a performance problem with an operation which involved retrieving several thousand objects from the database. My NHibernate log file showed me that this single NHibernate request was resulting in thousands of database queries being executed, one for each object in my collection. So why was this happening?

Below is a simplification of my class structure, starting with my root class, Equipment.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
public class Equipment
{
   public virtual IList<Asset> Assets { get; set }
}
 
public class Asset
{
   public virtual Location Location { get; set; }
}
 
public class Location
{
   public virtual IList<PositionPoint> PositionPoints { get; set; }
}
 
public class PositionPoint
{
   public virtual double X { get; set; }
   public virtual double Y { get; set; }
}
public class Equipment
{
   public virtual IList<Asset> Assets { get; set }
}

public class Asset
{
   public virtual Location Location { get; set; }
}

public class Location
{
   public virtual IList<PositionPoint> PositionPoints { get; set; }
}

public class PositionPoint
{
   public virtual double X { get; set; }
   public virtual double Y { get; set; }
}

Each instance of Equipment has a collection of Assets, each of which has a Location, which in turn has a collection of PositionPoints. The problem this structure presents to NHibernate is that the root class has a collection of objects in a one-to-many relationship, each of which has another collection of objects in another one-to-many relationship. My mapping classes had been set up to explicity turn off lazy loading for Assets, Locations and PositionPoints, therefore NHibernate was obliged to find a way to fetch all this data, and it chose to do this by first retrieving the data for Equipments, Assets and Locations, and then executing a single query for each Location to retrieve all of its PositionPoints.

I couldn’t remember why exactly I had turned lazy loading off for these relationships (perhaps I should have commented my mapping file with an explanation). Therefore I modified the mapping file to turn lazy loading back on. As expected this solved the Select n + 1 problem, as NHibernate was no longer obliged to fully populate Locations and PositionPoints. However, this change caused an exception to be thrown in the business layer, a LazyInitializationException. This was caused by logic in the business layer which was attempting to read the PositionPoints property of a Location after the session which originally obtained the root objects had been closed. Indeed this exception may well have been the reason I had previously decided to turn lazy loading off for these objects. So the idea of using lazy loading was not a viable solution, at least not without some other changes being made. A little research around the lazy initialization problem led me to the idea of injecting my NHibernate session into the data access layer from the business layer, allowing me to use the same session for the lazy loading, but I really didn’t want my business layer to know anything about database sessions.

I reverted my code to switch lazy loading back off and continued to investigate my original problem. I tried instructing NHibernate to eagerly load my objects using a HQL query to eagerly fetch associated data, but this resulted in a cartesian product issue, where the returned collection contained duplicate objects.

Then I discovered a page on ayende.com on NHibernate Futures.

Chapter 4: Futures

NHibernate Futures are a variation on the MultiCriteria feature, which allow us to combine queries to eagerly load one-to-many associations in exactly the way I needed. I would have to define a Future query to retrieve all of my Equipments, then another to retrieve my Assets, and another to retrieve my PositionPoints. These queries would then be combined my NHibernate to retrieve and combine all the required data in a single roundtrip. Finally it seemed like I had found a solution to my problem. I modified my code to use Future queries and tested it.

But it didn’t work!

Stepping through the code and checking my log file revealed that each Future query was causing a trip to the database. Future queries are not supposed to result in an immediate trip to the database, execution should be delayed until the last possible moment.

Again I had hit a brick wall. So again I started googling for answers.

After some time I very fortunately stumbled upon an explanation – NHibernate Future queries do not work with an Oracle database. This was disappointing.

Chapter 5: Getting Futures to Work with Oracle

So now I had reached a point where I had discovered an NHibernate feature which would seemingly allow me to eagerly populate my collection of objects in an efficient way. But it wasn’t supported with Oracle. I did however discover a method of getting Futures to work with Oracle on stackoverflow.

I would need to extend NHibernate’s OracleDataClientDriver and BasicResultSetsCommand classes. I followed the instructions, and updated my NHibernate config file to use the new driver. I reran my code using Futures, and it worked! All of my data was returned in a single trip to the database! But it wasn’t quick. In fact it was very slow. The whole point of this was to try to optimize my code. The Select n + 1 problem seemed to be an obvious reason for its slowness. I had solved that problem. But my code was still slow. Why? The reason was that the solution I had found on stackoverflow to get NHibernate Futures to work with Oracle used cursors. And cursors are slow. The built-in Futures feature results in SQL which does not use cursors. I had found a workaround but it wasn’t a good solution for my problem. Yet again I felt like I was back to square one.

Chapter 6: Rethinking My Approach

Having gone down various rabbit holes and tried a number of potential solutions, it was time now to take a step back from the problem.

What had I learnt?

I needed to obtain a large collection of Equipments, and their associated Assets. The operation was too slow because of a Select n + 1 problem. I needed to read PositionPoint data in the business layer.  I couldn’t lazy load this data because of a LazyInitializationException. I couldn’t use NHibernate Futures because the result was still too slow (with the Oracle workaround at least).

But what exactly did I need to use my PositionPoints for? I reviewed my business layer code and then it hit me. Of the several thousand Equipments and Assets I was retrieving, I only actually needed to access the PositionPoints of a small number of them! Less than ten in fact. Therefore if I turned lazy loading back on, which would result in a fast select query to obtain my root objects, I could identify in my business layer which objects I actually needed to access PositionPoint data for, and hit the database again (using a new session), just for those particular objects.

A few minutes of coding later and at last, I had my solution. The operation which had previously been taking around one and a half minutes was now taking around 30 seconds – an acceptable level of performance.

Conclusions

Looking back on this journey, I must admit I feel a little stupid. I had been looking at the problem in the wrong way. I had assumed that my approach was correct, and that I needed to optimize my use of memory or NHibernate, when in fact it was my algorithm which was inefficient. This is the main lesson I will try to take from this experience. When faced with a database performance issue, first review your algorithm and consider whether you are retrieving data you don’t actually need, particularly when using an ORM framework. There are also a few other things I will take away from this. Perfview is a great tool which I am sure will use again. NHibernate logging is an equally valuable tool for understanding what is going on under the hood. And it remains a mystery how anyone ever coded before the existence of the Internet!

A Tale of Optimization (part 1)

Ronnie Mukherjee 0 Comments

trying to get an award

I intended for this article to be contained within a single post, but it turned out to be too long for that. Click here for part 2.

Introduction

Over the past couple of days I have been on quite a journey in trying to optimize a method in my current project. I’ve known this operation was slow for the past few months, but it has never really been a priority to address the issue.

On my daily commute I often listen to podcasts, my favourites being .NET Rocks, Radiolab and The Guardian Football Weekly. Earlier this week I listened to a .NET Rocks episode titled Making .NET Perform with Ben Watson. Watson is the author of Writing High Performance .NET Code and during the show he and the guys discussed the topic of performance optimization from a number of different angles. The show inspired me to finally look into this annoying performance issue which had been on my radar for months. As I was listening I contemplated the problem, and in particular took an interest in a discussion on a favourite tool of Watson’s: PerfView. I had never heard of PerfView, but it sounded like the perfect application to help me understand my performance issue, and if nothing else offered the chance to try out a new tool. Apparently it was free and lightweight – two characteristics I always love in a development tool.

Chapter 1: Adventures in PerfView

Later that day, sitting at my desk, I downloaded PerfView and read through its tutorial. What a great tool! I had previously used Red Gate’s ANTS Performance Profiler, admittedly to a limited extent, but PerfView seemed easier to use, just as capable and a great deal more lightweight. Essentially PerfView helps you to explore two aspects of your application – CPU usage and memory allocation. My problem operation involved retrieving several thousand rows of data from an Oracle database, to automatically populate a collection of complex C# objects, using NHibernate. I had a hunch that it was the complexity of each object, with several layers of inheritance and multiple associations, that was the problem. I was perhaps slightly biased having just heard Watson emphasise the importance of memory allocation in .NET applications and how slowness was often a result of memory issues. Indeed, the PerfView tutorial states:

If your app does use 50 Meg or 100 Meg of memory, then it probably is having an important performance impact and you need to take more time to optimize its memory usage.

So I loaded my application in Visual Studio, paused execution with a breakpoint and used PerfView to take a snapshot of the heap, when I knew my large collection of objects would be in memory. I discovered that although IIS express was indeed using over 100MB of memory, only a fraction of this (around 10%) was being used by the heap. So maybe memory allocation wasn’t the problem at all?

Next I decided to use PerfView to analyse the CPU usage during my long operation. In total the operation was taking around one and a half minutes. I ran an analysis and was not surprised to find that the bulk of this time was taken up in the data layer, retrieving data from the database and implicitly populating my collection of several thousand objects. This was just as I had feared. Would I have to redesign my database and class structure to remove some layers of complexity? This would be a huge task. However, on closer inspection, I realised that although over 80% of the CPU usage during this operation was taken up inside my data retrieval method, the total CPU usage time was in fact only 15 seconds or so. Surely this could mean only one thing – it must have been the database query which was taking so long, which suprised me as several thousand rows is of course not much to ask of a database.

Chapter 2 – NHibernate Logging

This project is the first time I have used NHibernate. While I think I can see its benefits, or rather the benefits of using an ORM tool in general, I am not totally convinced. I come from a traditional background of writing and calling my own stored procedures, and miss that complete control of how and when the database is called. There have been a few times when I have wrestled with NHibernate to achieve the desired results, but perhaps this is just a part of getting to grips with it and learning how to use it to my advantage. In any case, having concluded that the problem involved my database query, I wanted to know exactly what query was being executed. After some googling I found that I could use NHibernate logging to obtain this query, by adding a couple of lines to my web.config file.

Using breakpoints to isolate the data access method, I was able to examine my log file to obtain the database query in question. It was indeed a very complex query, with many joins corresponding to the multiple layers of inheritance defined in my class and database structure. However, I noticed that stepping from my data retrieval line of code to the next line was in fact pretty quick, less than 5 seconds in fact. Copying and pasting the cumbersome SQL query into Oracle SQL Developer and executing it from there confirmed that the query itself was indeed very fast, despite its complexity. So my assumption was proved wrong again. It was not memory allocation that was the problem, it was not my data retrieval query, yet it was not CPU usage that was taking up so much time. So what was it? I hit F5 to continue execution from my breakpoint, let the operation complete, and then reexamined my NHibernate log file. To my surprise I discovered that the database had been hit several thousand times, running a relatively simple query each time to populate a collection property on one of my classes. It seemed that, without my knowledge, I had fallen victim to the Select n + 1 problem.

Click here for part 2.

Advice to My Younger Self

Ronnie Mukherjee 0 Comments

grandpa dj

In professional terms, at the age of 33 I am still relatively young. I have more of my career ahead of me than behind me. Nevertheless, when I look back, I can see that my perspective has changed considerably. With this in mind, I thought I would consider the question: if I could give some professional advice to myself as a fresh-faced graduate entering a career in programming, what would that be? This post is my answer to that question.

At the start of a project, things always look rosy

One of the most satisfying parts of life in software is beginning new projects. You have a blank sheet of paper upon which you can create the greatest system ever developed. The possibilities are endless. You will get everything done on time, under budget, and you will be a hero. It is difficult to avoid this kind of wishful thinking at the start of a project, in fact such optimism is a good thing in some respects. All great achievements start with a lofty vision. However, without being too pessimistic or miserable, I believe it is important to temper that early enthusiasm with a dose of realism. There will be difficulties, disagreements and unexpected obstacles. People will over-promise and under-deliver. Some things will take much longer to complete than expected. This is just how projects unfold. If we acknowledge this reality from the outset, we are more prepared for difficulties, even if only on a subconscious level. This is something I have learnt from experience. Things never run completely smoothly. One common mistake that people make is to believe that by following a particular methodology or project management method, difficulties can be largely eliminated. This is simply not the case. There will be difficulties, and a degree of stoicism is required to handle and overcome these difficulties.

Time and experience are the best teachers

Early on in my career, I was desperate to make progress. I could see that the people around me were better than me and I wanted to bridge that gap as quickly as possible. I studied and tried hard but just couldn’t move forward at the speed I wanted to. Now I see that the reason those around me were ahead of me was simply that they were more experienced. There is a reason that job descriptions tend to require a certain number of years of experience. Reading a book or completing a tutorial is incomparable to real experience. In addition, personal and professional development take time – not only the hours spent at work, but the years spent developing an overall picture of work, people and life. As you encounter and overcome obstacles, your brain forms new connections which make sense of those problems and prepare you for their reoccurence in the future. There is no other way to form these connections than to gain experience and wait. Success in my opinion is essentially about learning to solve problems, whether the problem is a bug in your code or a difficult colleague. You can read about bug-fixing or relationships, and this can help to some extent, but to really develop you need to face and overcome these problems.

Satisfaction at work is down to the people around you

As a junior programmer, I greatly underestimated how important people are to your level of satisfaction at work. From your peers to your boss to your customers, the people around you are the biggest influence to your day-to-day levels of job satisfaction. You can be faced with a failing project or a seemingly insurmountable snag list, but if the people around you are intelligent, positive and understanding, you will be able to cope and learn. Equally, you can have access to all the latest tools and technologies, and use them to design and deliver a brilliant system, but if you are working with difficult people, you won’t enjoy the experience. It is easy to think of life in programming consisting simply of a programmer and his machine, joining forces to conquer the world (or at least his project). Indeed, programmers are perceived stereotypically as geeks, because they are seen as lacking in social skills. This stereotype comes from the fact that introverts are often attracted to computers as a possible alternative to having to deal with actual people. I see myself as something of an introvert and this is possibly what drew me to computers initially, but there is simply no escaping the fact that you can’t get  very far alone. The good news is that human relationships offer rewards far greater than anything offered by a machine, and the real value of a successful project comes from sharing satisfaction with your colleagues and your customers.

Don’t just code

As I have progressed in my career, I have learnt that actually writing code is a small part of being a good programmer. A much more important skill is the ability to think well. That is, the type of thinking required to take a vague task or problem and turn it into a plan of action. Sometimes we need to step back and perform a ‘brain dump’ of everything on our minds. We need to learn to capture, organize and prioritise ideas, and also to let go of some of our desires. For example, upon experiencing a desire for our code to be supported by unit tests, a seemingly reasonable next step would be to start writing unit tests. But we need to learn to view that desire relative to the bigger picture, to make sacrifices and realise that we can’t achieve everything. As much as we would like our code to be faster, is performance optimization the best way we could be spending our time right now? The only way to effectively consider such questions, I have found, is to stop coding for a while and start thinking. Make a list on paper, in notepad, or on a whiteboard. Draw a mind map, be messy, write down your thoughts. Think about how much time you have, make some estimates, make some sacrifices and decide upon the best course of action. Then go ahead and start coding in the right direction.

I hope you have found something interesting or useful in this post, particularly if you are new to programming. I have no doubt that in ten years time, my view will be quite different. As we progress through our careers and our lives, our experiences will inevitably reshape our views. It would be nice to know what my future self would advise me right now, but I guess I’ll just have to wait and see.

 

Pareto Programming

Ronnie Mukherjee 0 Comments

Pareto principle - the key is in the twenty percent

One of my favourite principles is the Pareto Principle, also known as the 80-20 rule. The principle states that frequently, around 80% of the effects come from 20% of the causes. I love this principle because it is so simple, and it can be observed and applied to so many different areas in our professional and personal lives. It is powerful enough to fundamentally change the way we approach our work, and encourages us to accept that the world we live in is unbalanced. Things aren’t evenly distributed.

The principle is named after an Italian economist named Vilfredo Pareto, who observed that 80% of the land in Italy in 1906 was owned by 20% of the population. More recently, a report produced by the UN in 1992 stated that the richest 20% of the population of the world controlled around 80% of the world’s income. The principle can also seen in nature, indeed Pareto is said to have observed that 80% of the peas in his garden were produced by 20% of the pods.

But how can the Pareto Principle help us to produce better software?

Here are some ways we can apply the Pareto Principle at work to ensure we are spending time and money where it will have the greatest impact, and to accept that imbalance exists and there is no point pretending otherwise.

80% of value is delivered by 20% of requirements

This idea is particularly useful at the start of a project. Generally speaking the initial functional specification in any software project is produced from an optimistic point of view. What developers and customers often fail to realise is that around 80% of value, whether we measure that in terms of customer satisfaction or profits, will be delivered by 20% of the requirements contained in that specification. This doesn’t mean that the other 80% of requirements are pointless or a waste of time, it just means that they are less important. For this reason it is crucial to prioritize requirements. This may sound obvious, but it is surprising how little time is often dedicated to prioritization. The false assumption made at the start of a project is that we will have time to do everything, therefore it doesn’t really matter what we do first. The reality is of course that we will be faced with unexpected obstacles, therefore it pays to get the most valuable stuff done first. If we then don’t have time to do everything, at least we will not be ommitting any of the vital 20%.

80% of errors and bottlenecks will be contained in 20% of the code

As a codebase ages it is common for developers to fantasise about ripping it up and starting over. It seems to be riddled with bad code and inconsistencies. It is slow and difficult to read. The reality is that we just like to start with a blank page as this is more exciting for most people than having to drag an old codebase forwards. In truth, by rewriting 20% of the code we can remove 80% of the solution’s ‘badness’. This does not mean that we should indeed go ahead and rewrite 20% of the code, it just means that we shouldn’t assume it would be best to start again. Again we need to prioritize. If we are determined to improve things, we should work on identifying where the bad 20% is, and decide whether improving that will be enough.

80% of your productivity will be delivered in 20% of your time

As desirable as it might be, you simply can not function at 100% productivity, 100% of the time. Creativity tends to come in bursts and learning to recognise when you are feeling creative and when you are not is a valuable skill. Expecting yourself to constantly produce brilliant work is a recipe for stress. It might appear to work for a period of time, but soon your body will force you to stop and take your foot off the gas. Luckily, in any job, there are lots of menial tasks which need doing and are perfect for those times when we are not at our most creative. This might be writing some code which is not challenging, adding comments or reviewing your work or someone else’s. One technique I have used in the past is to categorise items on my to-do list as ‘creative’ or ‘menial’, and to choose tasks based on how creative I am feeling. Humans function in cycles whether you like it or not, and by learning to work within these cycles you can maximise your overall levels of productivity in a sustainable way.

80% of the value of a team is in 20% of its members

This may be controversial, but in every project I have worked on, success has depended primarily on the abilities of a small portion of the team. There are generally a couple of ‘stars’, usually including the team leader, who are there for other team members to lean on and whose experience and skill carries the project over the line. If that 20% of the team were to leave, the project would probably fail, whereas much of the rest of the team are more easily replaceable. These are the team members who have the greatest overall understanding of the project, allowing them to intuitively make important decisions and identify potential pitfalls. This does not mean that the other 80% of the team are useless. A good team needs reliable workers who can take designs created by others and make them a reality. There have certainly been times in my career when I have been part of that 80% of the team.

I really believe that understanding the Pareto Principle and its ramifications is crucial in software development. Whether we are considering requirements, bugs, team members, tasks or customers, all things are not equal. By acknowledging and accepting this we can adjust our behaviour according to the relative importance of a particular thing, and embrace imbalance rather than deny it.

Keep Your Blood Flowing

Ronnie Mukherjee 1 Comment

docs_thumbs_up

One of the drawbacks of life as a programmer, along with most other professions, is that we are required to spend most of our day sat at a desk. There have been a lot of studies indicating that this is not good for our health, and common sense tells us the same. Sitting down for hours on end leads to tension, which is not only detrimental to our health, but also to the quality of our work and our vitality. Computers have a way of drawing you in, to the point where you become so engrossed in your task that you forget about everything else, including your body. It is only at the end of the day, when you finally leave the office that you realise how stiff your shoulders are and how stressed you feel.

I have struggled with this problem to varying degrees throughout my career, and have experimented with various remedies to address the issue, from ergonomic keyboards and footrests to exercise and meditation. Most of these measures have helped me to some extent, but one simple technique which I have been using for the past few months seems to have really had an effect, so I thought I would share it with you. It is this:

Every half an hour, I get up and out of my seat.

My intuition has always told me that it is important to take regular breaks, but I have only recently realised that getting up and doing something makes those breaks more effective in relaxing and refreshing my muscles and mind.

I have tried using the Pomodoro technique in the past, and much of it didn’t suit me. I never felt any benefit from logging interruptions, and I found its method of task management too simple, however the one rule that did stick was to work in 25 minute bursts, and I have been doing this for several years. I have noticed that by working in this way I am able to maintain a better level of focus during the time when I am actually working. Only recently however have I added the rule of getting up out of my seat after every ‘pomodoro’.

So what do I do when I get up? I might go to the toilet, go and get myself a glass of water, go and look out of the window, go wash a cup, or if no one is watching just stand up and do a few stretches. The Pomodoro technique recommends taking a break of 3 to 5 minutes between each pomodoro, which is what I used to do, but a better rule I find is to just make myself get up out of my seat, even if only for a minute or so. Once I am up I often realise that I need a slightly longer break. If I stay in my seat, I no longer count that as taking a break.

I use the online pomodoro timer at tomatoi.st, which is nice and simple and also shows me how many pomodoros I have completed each day, but there are many pomodoro websites and apps to choose from. A stopwatch would work just as well.

I’m not completely sure why this technique works for me, but it does. In my mind it is a case of keeping the blood flowing, which relieves tension and refreshes the mind. If I sit for hours I notice that I lose mental sharpness and begin to struggle to make progress on problems. If we think a little about how our bodies have evolved, we are clearly not built to remain in the same position for long periods of time. Taking a walk at lunch time is another technique which I frequently use. Again, it just gets the blood flowing.

Applying this technique is probably only going to work for you if you feel there is actually a problem there that needs solving. I recently started to experience neck and shoulder pain, not for the first time, and I am certain this was related to my work. I tried an ergonomic keyboard for a few days, but if anything this only seemed to make matters worse. At the same time, I bought a footrest, which does seem to have helped me adopt a more comfortable sitting position, but it is getting up out of my seat frequently which I am sure has made the biggest difference.

There are other measures I am taking to minimize the effects of sitting for long periods.

One conclusion I have come to is that a stronger core would help me to sit correctly, therefore I am looking at how I can modify my exercise routine accordingly. I’ve tried Pilates which initially struck me as a slightly feminine form of exercise, but it does appear to be very effective at isolating the core and strengthening the stabilizing muscles between your shoulder blades. I attended a Pilates class last week and it was certainly not easy. I also get a full body massage every month.

The key message here then is to be aware of the need to keep your blood flowing if you sit at a desk all day. You may not need to get up every 30 minutes, but consider getting up more often than you currently do, particularly if you feel like you are getting nowhere with your work or you are starting to feel tense. If this is a real problem then I would recommend trying an approach similar to mine. Use a timer and make yourself get up periodically. You may be surprised at just how effective this is.

Five Lessons From JavaScript: The Good Parts

Ronnie Mukherjee 0 Comments

js_good_parts

I have just finished reading JavaScript: The Good Parts by Douglas Crockford. It is a quite illuminating book, from which I learnt a number of interesting and useful lessons about JavaScript. I have chosen five to share with you below. If you are interested in JavaScript, you might find something useful here and I would strongly encourage you to buy and read the book yourself.

undefined and null

If I had been asked what the simple types in JavaScript were before reading this book, I would have replied with number, string and boolean. In fact there are two more simple types in JavaScript: null, and undefined. Having worked with JavaScript for many years, I have of course encountered null and undefined many times, but had never really considered their types, or that they are indeed simple types themselves.

So what is the difference between undefined and null?

When you try to retrieve the value of a property or variable which has either not been assigned a value, or which has not been declared, you will receive the undefined value.

The null value must be explicitly assigned to a property or variable before it is encountered.

However, matters are complicated by the fact that the expression (null == undefined) returns true, which means that in the following code, our alert is raised.

1
2
3
4
5
var notAssigned; // will return undefined
 
if(notAssigned == null){
  alert(‘it is null!);
}
var notAssigned; // will return undefined

if(notAssigned == null){
  alert(‘it is null!’);
}

…which brings us to the two sets of equality operators.

== and ===

For programmers like myself who come from an object-oriented language background, it is easy to fall into the trap of making a lot of assumptions about the JavaScript syntax without really taking the time to check those assumptions.

An easy mistake to make is to assume that the “double equals” equality operator == has the same basic meaning as in JavaScript as it does in C#. However, there is an important difference.

In C#, the double equals will only return true if the two operands either point to the same reference (for reference types) or contain identical values (for value types). In JavaScript, the double equals can return true even if the two operands are of different types. The reason for this is that when the double equals is presented with operands of different types, it will attempt to convert one of the operands to the type of the other, and compare the result. Below are examples taken directly from the book, which demonstrate some of the strange consequences of this behaviour.

1
2
3
4
5
6
7
8
9
'' == '0' // false
0 == '' // true
0 == '0' // true
false == 'false' // false
false == '0' // true
false == undefined // false
false == null // false
null == undefined // true
' \t\r\n ' == 0 // true
'' == '0' // false
0 == '' // true
0 == '0' // true
false == 'false' // false
false == '0' // true
false == undefined // false
false == null // false
null == undefined // true
' \t\r\n ' == 0 // true

The other equality operator offered by JavaScript is the triple equals, ===. This displays behaviour which is more like what we would expect. It does not attempt to do any conversions when presented with operands of different types, it just returns false. So in each of the examples listed above, false would be returned.

The double equals operator is identified by Crockford as one of the bad parts of JavaScript, and he advises against using it under any circumstances.

Objects are Containers of Properties

Having confirmed that all values in JavaScript are either simple types or objects, an “aha” moment for me was reading that all objects in JavaScript are simply containers of properties. It’s a satisfying feeling being able to abstract a seemingly complex or unclear concept into a simple model (isn’t this in fact the whole purpose of science?). From my experience with JSON and object literals, I was quite familiar with the concept of properties and values in JavaScript. However it had never dawned on me that objects are simply containers of properties, with each property consisting of a name and a value. That value can of course be another object, which contains its own properties, and so on. Objects in C# are more complicated . A C# object can have a variety of different types of members, including methods, events, delegates and properties. Furthermore, each of these members is associated with a visibility level, such as ‘public’ or ‘private’. Behaviour according to differing levels of visibility can be emulated in JavaScript, as I discussed in a previous post, however this feature is not built in to the language. I find the fact that objects are such simple constructs an almost beautiful feature of JavaScript.

Another important feature of objects is the prototype linkage feature, but that is probably a topic for a separate blog post.

Functions are Objects

Functions in JavaScript are themselves objects, which means, as we have seen, that they are simply containers of properties. How is a function simply a container of properties? In a nutshell it has hidden properties for the function’s context and its enclosed statements. The important difference between a function and any other object is that a function can be invoked.

The fact that functions are objects means that they can be assigned to a variable, stored in an array, passed as an argument to a different function, or used in any other way that a ‘regular’ object might be used.

An interesting aspect of functions in JavaScript is the value of this inside of a function, which I discussed in a previous post.

Function Scope in JavaScript

This is something I feel I really should have known about JavaScript, but I must confess I didn’t.

Variables in JavaScript have function scope, unlike C# which has block scope. What this means is that when you declare a variable in JavaScript, it is accessible by any code within the same function, even if that code exists outside of your current block, as defined by your curly braces. This is probably best explained by an example:

1
2
3
4
5
6
7
8
var myFunction = function()
{
  if(10 > 5)
  {
    var message = ‘hello there!;
  }
  alert(message); // alerts 'hello there!'
}
var myFunction = function()
{
  if(10 > 5)
  {
    var message = ‘hello there!’;
  }
  alert(message); // alerts 'hello there!'
}

If JavaScript had block scope, like C# and Java do, then invoking myFunction would cause undefined to be alerted. In fact our message ‘hello there!’ is alerted.

For reasons of clarity Crockford advises us to always declare our variables at the top of the containing function. So our code would then look like this:

1
2
3
4
5
6
7
8
9
10
11
var myFunction = function()
{
  var message;
 
  if(10 > 5)
  {
    message = ‘hello there!;
  }
 
  alert(message);
}
var myFunction = function()
{
  var message;

  if(10 > 5)
  {
    message = ‘hello there!’;
  }

  alert(message);
}

Having function scope rather than block scope is identified by Crockford as one of the ‘awful’ parts of JavaScript.

Final Words

In summary, this is a book that is well worth reading for anyone interested in the finer points of JavaScript. My next goal in developing knowledge and understanding in my chosen niche is to start looking at angular in more detail. I’m not yet sure how I will go about this, but the free tutorial over at codeschool.com looks like it could be a great place to start.

Is Contracting For You?

Ronnie Mukherjee 2 Comments

contract

I  have been contracting for around two and a half years now. This isn’t long at all in the grand scheme of things, nevertheless I thought I would share some thoughts on contracting compared with permanent employment, from my slightly limited point of view. If you are considering taking the plunge into the world of contracting, you may find this useful.

Being an Outsider

Becoming a contractor requires a slight psychological adjustment. As a permanent member of staff you become used to playing by the same rules as your colleagues. Everyone adheres to the same policies and principles, from working hours and holiday allowance to pension schemes, performance appraisals and career development plans. This leads to a subtle sense of belonging to a team, which needs to be recognised and compensated for by contractors. I am of course a part of my client’s team and I work hard to achieve success in my projects, but there is no denying that in some respects I will always be an outsider compared to permanent staff. It took me some time to get used to this.

Lack of Security

I have been fortunate enough to have never been short of work since becoming a contractor. However, by the very nature of contracting it is necessary to always keep one eye on your next job. When times are hard in industry, contractors are often the first to go, and it is important to never take one’s position for granted. It is true that anyone can be made redundant, however contractors obviously change their place of work more frequently, and this means more job hunting, more interviews, and potentially more stress. The idea of being out of work is frightening for anyone. We all have responsibilities to meet and bills to pay.  For contractors this fear is likely to rear its head more often than for permanent employees.

More Paperwork

If you set yourself up as a limited company, as most contractors do, there will be some additional paperwork for you to do to manage your business finances and payment of taxes. If you hire an accountancy this will help, however you will still have at least a little admin work to do each month, and accountants obviously charge you for their time.

Expectations

As a contractor you will be expected to have a certain degree of technical expertise. You will be expected to hit the ground running. Your employers will not hire you based on potential or willingness to learn, you will be expected to have already fulfilled much potential and learnt a great deal. This can be daunting, but equally it can motivate you to actually spend some time and effort on developing and maintaining your skills, and thus speed up your development. Since becoming a contractor I have spent more of my personal time learning than I ever did when I was a regular employee.

Getting a Mortgage

One aspect of contracting I had not considered was the added complications when applying for a mortgage. I have recently moved house and my first mortgage application was rejected because I did not have two full years of accounts to show the bank. Thankfully another bank was more flexible. If you are thinking of applying for a mortgage in the next two to three years then now may not be the best time for you to become a contractor. At the very least do some research. Any other application which requires proof of a steady income may be subject to similar complications.

More Money

This is perhaps the biggest draw to the world of contracting. It is no secret that contractors are paid more for their time than permanent members of staff. It is easy to overestimate the difference however. We still need to pay taxes and we do not receive any paid holidays, pension contributions, company car or other benefits. Furthermore, there will probably be times in between contracts when we are looking for work and obviously not being paid at all. That said, my earnings through contracting are higher than they would be if I were a permanent employee. If you can repeatedly find work, then you will likely receive more money as a contractor, but you must be confident in your ability to find that work.

Varied Experience

As valuable as book-reading and personal projects are, in my opinion there is no substitute for commercial experience when it comes to your professional development. As a contractor you will experience a great deal of variety with regards to projects, people, working environments and working practices. This will allow you to better understand what actually matters and what doesn’t when it comes to achieving commercial success. You will probably work with a variety of different tools, on projects of different lengths, and will have to communicate with different types of colleagues and customers. You are less likely to get bored and as scary as it can be not knowing where your next job will come from, the anticipation of a new challenge can be exciting.

Freedom From Office Politics

Successful organisations are of course made up of people who want to get ahead. Naturally everyone is aiming for that big promotion and pay rise. This is just human nature and there is certainly nothing fundamentally wrong with it. One consequence of this however can be a high level of unhealthy competitiveness in the workplace. Sometimes employees can be so eager to impress the boss that they will engage in questionable behaviour. This might be taking credit for someone else’s work, failing to accept responsibility for mistakes made or even worse, blaming someone else. As a contractor there is no guarantee of freedom from such political games, but as you are outside of the race for promotion, you are less likely to become embroiled in them.

Autonomy

Permanent employees are often guided towards learning about particular technologies or programming languages, according to the needs of the business. Contractors on the other hand have a greater degree of control over their professional development path. We can assess the market and find a compromise between which skills are in demand, and which areas we are interested in learning about. This freedom to be your own career development manager can be quite liberating.

Final Thoughts

If you are thinking of becoming a contractor then I hope my thoughts have helped you. It is not for everyone. There are certainly advantages to permanent employment, and the right decision will depend upon your individual circumstances, including your financial responsibilities, your skill set, and your geographical location. I took all these things into account when I made my decision, and I certainly have no regrets, but I would encourage anyone to look very carefully before you leap.