The Financial Information Exchange Protocol is defined on the homepage of the protocol standards web site as “The Financial Information eXchange (FIX) Protocol is a messaging standard developed specifically for the real-time electronic exchange of securities transactions.”

However, I think this definition is very limited. The protocol does not just define a messaging standard but also how the two systems communicate including session management, encryption and login details.

In my own words “FIX is a protocol which defines the messaging standard and how those messages should be passed back and forwards, encryption and communication failure recovery between two or more financial systems specifically for the real-time exchange of a number of different financial instruments.”

Why build a FIX engine in .NET

Several attempts have been made to build open source .NET implementations of a FIX engine. However, these engines have limitations which include:

• Written in old versions of .NET so they don’t take into account generics or the new thread-safe libraries
• Are just a wrapper around a different technology such as Java or C++
• Are not open source and therefore just supplied as binaries
• Inefficient or un-configurable logging and auditing systems

How will the engine be used?

The FIX engine will be used in a part manual and part automated trading system to manipulate orders on one or many remote broker systems.

Which implementation of FIX will be used?

As different broker systems use different versions of FIX, the .NET FIX engine should be able to communicate with any version from 4.0 to the latest version.

Important features of the engine

The engine should be fast, stable and provide an easy to use API. All messages should be logged for replay, order history and audit purposes. It should include all of the functionality to enable session creation, replay of missed messages, encryption and checksum validation. It should be easy to debug, maintain and have 100% test coverage.

How to build for an entire protocol

Covering an entire protocol is a large and daunting task. So as with any large and difficult task it is good to break it down into smaller pieces. The first instance will be to create a machine readable specification of the protocol so that we can auto-generate the large number of message types needed to cover the different versions of the protocol. We may be able to get this or some of this from the FIXimate part of the protocol website or use the word documentation which forms the protocol definition for FIX.

As well as coding the messages we will also need to provide an engine to create an open session for communication including logon and encryption and the ability to send and replay messages should they not be received by the target host. We will write this engine from scratch using the recommended approach defined on the FIX web site through their training sessions.

This should give us a good start to providing an engine which we can refine in a test harness / sandbox on a broker account.

Testing the solution

We will use TDD for testing unit functions through unit tests and provide full integration testing against a broker’s sandbox system.

In part 1 we defined a standard producer – consumer queue, however, there was nothing that particularly made this stand out from a queue written in a previous version of .Net. In this post, I am going to upgrade the queue to allow for the following:

• Take advantage of the thread safe collections in .Net 4
• Allow for the producer to know when a task has been completed
• Cancel un-started tasks
• Handle exceptions thrown by the tasks

The code for this example has mostly come from the book C#4 in a nutshell with some modifications from me.

First we will modify our queue class. I have renamed the class to PCQueue4 to differentiate them.

The first thing to notice about this Class is that instead of queuing actions directly on a queue, we are encapsulating these actions in a workitem wrapper.  I have made this wrapper a Struct instead of a class to avoid have this set of objects on the heap to avoid excessive garbage collections.

///

        /// Class to encapsulate an item on the queue
        /// 

        struct WorkItem
        {
            public readonly TaskCompletionSource TaskSource;
            public readonly Action Action;
            public readonly CancellationToken? CancelToken;

            public WorkItem(TaskCompletionSource taskSource, Action action, CancellationToken? cancelToken)
            {
                TaskSource = taskSource;
                Action = action;
                CancelToken = cancelToken;
            }
        }

The wrapper workitem has three properties. They are the action enqueued by the producer, a CancellationToken and a reference to the actual Task itself through the TaskSource<T>  class.

We will use these other properties to achieve our goal of cancellation, exception handling and for progress reports.

The next change is to change the collection that we are using from the Thread Unsafe Queue, to the new .Net 4 Threadsafe collection BlockingCollection<Action>.

 

BlockingCollection<WorkItem>
 _itemQ = new BlockingCollection<WorkItem>();   

The next change is to the construction of our worker threads. In our previous example we created the thread directly using the Thread class, however, this time we will use the new Task.Factory.StartNew() method to create our thread as a Task object on the thread pool. Having these threads in the pool will also improve performance.

For our Dispose() method, we will take advantage of the method CompleteAdding() which is called on the BlockingCollection. This tells the collection that no more items will be added (trying to add will throw an ObjectExposedException) and it also stops any thread waiting on the collection if it is empty.

 

public void Dispose(){_itemQ.CompleteAdding();}  

To enqueue a Task we have an overloaded method EnqueueTask() which allows the producer to add an Action to the queue. The overloaded method is to allow for cancellation through a cancellation token.

Finally the last change is the Consume() method. This is where the really cool changes are made and the reason why this collection is so compelling to use.

The first thing you notice about this method is that we are not calling Thread.Wait() and we are not entering into an infinite loop with while(true)

The foreach loop which calls _itemQ.GetConsumingEnumerable() is the key to how this queue stays alive. This method is a blocking method. Build into the collection is a Wait() when the collection is empty. It is practically made for consumer / producer queues.

We then check the cancellation token to see if this task has been cancelled. If it has been we set the transition the underlying task into the Cancelled state.

This will throw an operationCancelledException which will be rolled up in the AggregateException which we will handle later.

If the Task has not been cancelled we can process the Task (Action) and mark it as completed using the SetResult method on TaskCompletionSource object which again transitions the Task into a new state. (RanToCompletion)

The full source is here:

PcQueue4:

using System;

using System.Collections.Concurrent;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading;

using System.Threading.Tasks;

namespace StreamingProducer

{

public class PcQueue4 : IDisposable

{

/// <summary>

/// Class to encapsulate an item on the queue

/// </summary>

struct WorkItem

{

public readonly TaskCompletionSource<object> TaskSource;

public readonly Action Action;

public readonly CancellationToken? CancelToken;

public WorkItem(TaskCompletionSource<object> taskSource, Action action, CancellationToken? cancelToken)

{

TaskSource = taskSource;

Action = action;

CancelToken = cancelToken;

}

}

BlockingCollection<WorkItem> _itemQ = new BlockingCollection<WorkItem>();

public PcQueue4(int workerCount)

{

for(int i = 0;i<workerCount;i++)

{

Task.Factory.StartNew(Consume);

}

}

public void Dispose(){_itemQ.CompleteAdding();}

public Task EnqueueTask(Action action)

{

return EnqueueTask(action, null);

}

public Task EnqueueTask(Action action,CancellationToken? cancelToken)

{

var tcs = new TaskCompletionSource<object>();

_itemQ.Add(new WorkItem(tcs,action,cancelToken));

return tcs.Task;

}

void Consume()

{

foreach (var workItem in _itemQ.GetConsumingEnumerable())

{

if(workItem.CancelToken.HasValue && workItem.CancelToken.Value.IsCancellationRequested)

{

workItem.TaskSource.SetCanceled();

}

else

{

try

{

workItem.Action();

workItem.TaskSource.SetResult(null);//indicate completion

}

catch (Exception ex)

{

workItem.TaskSource.SetException(ex);

}

}

}

}

}

}

And the way to call this from the Producer:

Task task = EventQueue.EnqueueTask(() => _service.UpdateListItem(update));

			if(task.Exception != null)
			{
				AggregateException ex = task.Exception.Flatten();
				foreach (var innerException in ex.InnerExceptions) { }//do logging
				_tmr.Stop();
			}

I’ll put this into context and then all will become clear. I came across this interesting problem recently when playing around with a set of streaming real time data and attaching an observable collection to it so that I could view the changes in a WPF application.

The application was holding on to objects which should have been disposed after they went out of scope. These objects did not have their memory de-allocated by the Garbage Collector but nothing seemed to be referencing them. (i.e. they had no GC Root object or referee that I could see)

It turned out that the objects had subscribed to an event on an object which was still alive. (in scope) and this was not being cleaned up when the objects went out of scope. I wanted a way to overcome this problem and also to test for this kind of problem in the future as I am likely to forget to remove subscriptions again.

I did a bit of research and came up with the simple solution which is to implement IDisposable on each object and remove the event subscription on each call to Dispose().

The consumer of these objects can do something like this

Array.ForEach(obj, o=> o.Dispose());

And the Dispose() method would have something like:

_memoryLeak.Click -= ItemClicked;

This sorts me out this time, but what about testing for this condition across my unit tests?

I did some more Googling and thanks to Joseph Albahari I discovered the call GC.GetTotalMemory(true);

This static call to the GC class with the parameter true, performs a full Garbage collection and then determines all of the memory allocated to the program.

However, it is virtually impossible to test your code as is, using this method. The memory usage will change because of the test itself. I found that I could get the result I wanted but the tests have to be very granular indeed and not introduce any additional overhead or variables in otherwise it won’t work.

Debugging memory management is more about tuning and finding the big problems. The smaller the problem the more time spent on it and each fine tune has a cost-benefit analysis involved.

Here is my example:

The classes to test which have a memory leak:

using System;  namespace MemoryLeamSimilator {  public class MemoryLeak  {  public event EventHandler Click;  }   public class MemoryItemsToBeDisposed :IDisposable  {  private MemoryLeak _memoryLeak;   public MemoryItemsToBeDisposed(MemoryLeak memoryLeak)  {  _memoryLeak = memoryLeak;  _memoryLeak.Click +=ItemClicked;  }   void ItemClicked(object sender, EventArgs e)  { //do something }   public void Dispose()  {  _memoryLeak.Click -= ItemClicked;  } } }  

The unit test for memory leak detection:

using System; using NUnit.Framework;  namespace MemoryLeamSimilator {  [TestFixture] public class MemoryLeakTests {  private static MemoryLeak _memoryLeak = new MemoryLeak();  private long _memoryUsedBefore = 0;  private long _memoryUsedAfter = 0;   [Test, Description("Test the memory is de-allocated")]  public void CreateItemsMemoryTest()  {  _memoryUsedBefore = GC.GetTotalMemory(true);  var ml = new MemoryItemsToBeDisposed(_memoryLeak);  ml.Dispose();  ml = null; //need this  _memoryUsedAfter = GC.GetTotalMemory(true);   Assert.AreEqual(_memoryUsedBefore, _memoryUsedAfter);  } } }  

You can see this test fail by commenting out the .Dispose() line.

Of course, this problem will still exist if the developer who uses the library forgets to call dispose. You can address this with the Dispose pattern. (Just substitute the release of the COM object with the code to release the event handlers in this example)

Note:

The test will also fail is you do not set the reference of the item to null.

Profiling:

To profile the application you can use Jet Brains’ Dot Trace, MS CLR Profiler or Red Ant’s Memory Profiler.

As this solution is very tricky to test and will only work if there is almost no set up required in your unit test to call the method under test, I need another way of testing this. The next approach is to use Weak References.

Weak References are a way of referencing a heap object in code but have the GC ignore this link when analysing an object graph. By setting up Weak Reference objects and then letting the underlying object go out of scope, doing a GC.Collect() and testing the WeakReference.Target for null, we can see if the object still exists on the Heap.

The unit test for this is here:

 [Test, Description("Test the memory is de-allocated")] public void CreateItemsWeakReferenceTest() {  var ml = new MemoryItemsToBeDisposed(_memoryLeak);  var weakReference = new WeakReference(ml);  ml.Dispose();  ml = null;   GC.Collect(); Assert.IsNull(weakReference.Target); } 

Again, if you comment out the Dispose() line you will see that the GC did not clean up the object and the test will fail.

Conclusion

I think for most cases for specific memory de-allocation tests I will use a WeakReference object, but it is good to have the bytes solution there in the testing armament.

Note:

One other solution for this problem as provided in the C#4 in a nutshell book, is to use WeakReferences instead of event handlers, which will remove this problem altogether.

]]> http://codingpowers.wordpress.com/2011/12/31/unit-testing-for-managed-memory-leakage/feed/ 0 mrsimonpowers http://codingpowers.wordpress.com/2011/12/29/simple-producer-consumer-queue-in-net-4/ http://codingpowers.wordpress.com/2011/12/29/simple-producer-consumer-queue-in-net-4/#comments Thu, 29 Dec 2011 21:15:32 +0000 mrsimonpowers http://codingpowers.wordpress.com/?p=462 ]]> Another back to basics post for reference material . I will follow this post with how to improve this basic pattern with .NET 4 objects.

In building a real time news aggregator which requires a producer – consumer queue to provide up to the second news feeds for a trading engine, I have updated my general pattern for .NET4. So this seemed like a good excuse for a post.

The pattern is used for allowing a multi-threaded execution of a set of tasks which are being produced, usually continuously and the tasks are discrete functional items that do need to interact with each other.

The idea is that the producer of the tasks simply loads the tasks as it produces them to a Queue. The Queue has a number of threaded workers that pull items of the queue and execute them as fast as they can.

The queue can be implemented using the .NET Queue class which is a first in, first out collection of items which we can load Action or Delegate types to.

The implementation of the Queue for this pattern is as follows:

public class PCQueue : IDisposable {  private Thread[] _workers;  Queue<Action> _itemQ = new Queue<Action>();   public PCQueue(int workerCount)  {   _workers = new Thread[workerCount];  for(int i = 0;i<workerCount;i++)  {   (_workers[i] = new Thread(Consume)).Start();  }  }   /// <summary>  /// Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.  /// </summary>    public void Dispose()  {  foreach (var thread in _workers)  {  EnqueueItem(null);  }  }   private void Consume()  {  while(true)  {  Action item;  lock (_itemQ)  {  while (_itemQ.Count == 0)  { Monitor.Wait(_itemQ);  }   item = _itemQ.Dequeue();  }   if(item == null) { return; } item();  } }   public void EnqueueItem(Action Item)  {  lock (_itemQ)  {  _itemQ.Enqueue(Item);  Monitor.Pulse(_itemQ);  }   }  }  

The PCQueue object is instantiated with the number of worker threads that we want the queue to use. Each worker thread is immediately started with the Consume() method of the queue.

The Consume() method enters an infinite loop which continually tries to ‘consume’ or execute the tasks on the Queue. When the Queue has no more items on it, the thread blocks and waits until it receives a signal (pulse) from the EnqueueItem method. The method also has a way of exiting the loop. When a null item is placed on the Queue, the loop with exit and the method with terminate which will also terminate the worker thread.

The EnqueueItem method, simply add a new item to the Queue and uses Pulse to signal to the next waiting thread to try to execute (dequeue) the next item.

The producer code can go something like this:

 EventQueue.EnqueueItem(() => _service.UpdateListItem(update)); 

This line could be called in a timer or when stock data or in my case when a new news article is found.

Watch for the next post on .NET 4 improvements.

“This above all: to thine own self be true,
And it must follow, as the night the day,
Thou cans’t not be false to any man”.

Which is of course Shakespeare.

Life is good and you know you are on the right track when people say things like that to you.

My first draft at architecture is to just get the different functional parts of the program identified. Then later we can see how we might host these on the appropriate hardware and also how we are going to build in the cost model to enable us to charge for the application.

The functional parts of the application are shown here:

I will now discuss each functional block of the application to detail what its function is and how it communicates to the rest of the system. Later we will review the performance when building large sets or complicated relationships and we are retrieving lots of documents.

The internet

The internet will be the source of information for the aggregator. In future we may want other streams of data such as actual market data, but we will cross that bridge later. For now we will make sure we design the application to allow easy plugging in of different data sources.

We have a few options of how we get data from the internet. They are summarised here:

• Pre-defined URLs which return one page. These might be RSS feeds.
• Pre-defined URL starting places for web crawling, such as a company website or a blog.
• A search engine. Sending search terms to a search engine as using these results
• A subscription service, such as paid news or squawk data
• Push data that notifies us of changes to specific records in a datastore

There are probably others, but we will look at these sources of data to start with.

Web Client

The web client is the .NET library we will use to send http requests to the internet. We may use other libraries as well, such as our custom RSS reader or web services (WCF) for other sources of data. This container on the diagram represents all these libraries as a source of data entry point to the application.

Producer – Consumer Queue

This is the work horse of the application. It queues requests to the web client library and executes the filter and relationship engine on return of the data. It is a standard threading pattern. I am looking forward to investigating some of the new .NET 4 objects which encapsulate this pattern.

F# Filter and algorithm

The filter system will be written functionally and use the trader defined relationships to determine relevance of data and how to link this data together.

Relationship Manager

The relationship manager is the container and rule set for how the traders world view is put together. This will require GUI elements.

GUI

The GUI will be a WPF application that will display a rich drag and drop interface of the trader’s world view. I am thinking of using reactive extensions to encapsulate the IObserver / IObservable pattern to allow the UI to respond instantly to updates.

Conclusion

I am sure there will be many changes to this, but this is the first iteration of the design. We now need to think how to make this into a payable service.

The key differentiator of this project than other news aggregators (of which there are hundreds) is that this allows a configurable set of relationships and correlations to be defined to represent the trader’s world view. The news then streams in and is matched against these relationships to enable a challenge or proof in real time of that view.

The ultimate purpose of this tool is to guide traders in placing trades and making investments by using this up to date and very focused picture of global events.

I thought this would make a great series of blog posts to discuss the architecture, code patterns and methodology used in building this very cool tool. Also when the tool goes to general market, this will be a log of the quality that has gone in to the tool to raise confidence in it as a day to day tool in the traders armoury.

Being a Microsoft guy I will of course use C# to build the application. I’ll use WPF for the GUI and try to build the application using the best practices that I know.

The projects I work on generally have the following life cycle:

• A requirement or need is identified by one of my clients
• Discussion takes place to identify what the need really is, and how this can be best serviced by a technological solution if appropriate
• A proof of concept is built which uses best practices as though it was production code.
• This POC is then used to build out the application in an iterative way
• Each iteration generally lasts from 2 – 4 weeks and has the following parts:
  • Discussion with the client determines the priority of each of the functional items
  • The highest priority items are scoped and planned into the next iteration
  • User Acceptance Tests are written by the Client or Tester if we have one
  • Developers use TDD (tests first) to develop the code
  • Clients are involved with team meetings answering questions as they arise
    (This reduces developer decisions on business items and removes gold plating)
  • Developers show case the work they have done to client
  • Feedback then gets either built in, or scheduled for re-prioritization next iteration
  • UAT  happens
  • Application changes are rolled to production for the client to start using it

They also have the following key elements:

• All team members having access to ask questions to the client and vice versa
• Developers all use the same ubiquitous language as the client
• A rich domain model built on top of the common language
• Every team member sharing responsibility of make working software on time
• Commitment of everyone to deliver working software that is fit for purpose
• Dedication to best practices including TDD
• Sustainable development
• Clear communication of what is being delivered and when
• Trust built by the development team from the client who can clearly see the above steps taking place

This creates an exciting enthused environment for achieving success. This is the environment we will be building our news aggregator. Next post will cover the general code design.

I have been following Kyle for about six months and he is certainly a character. I have seen him on various financial chat shows and a crazy interview on BBC where the presenter who clearly hadn’t got a clue about world markets just spent the whole time trying to make him out to be a big bad hedge fund villain. The best bit being his ‘good bye’ at the end.

His latest shareholder’s letter which  pretty much sums up his position covers a lot of my own beliefs about Euro and Japanese Debt. I am not convinced we will see a war which will affect us in the west, but given Europe’s history of fighting amongst them nothing would surprise me. The Syria problem is also slightly worrying. Some people on ZH are calling this the US / European plan B.

Kyle is also mentioned in the book Boomerang by Michael Lewis, which is an entertaining read about how different countries got into the mess they are in now. Would have been good if this book had covered more countries, like the UK, France and Italy and Japan.

Kyle Bass, is the founder of Hayman Capital in Dallas. What I like about him is the way he researches his market view. He gets a view and then researches it and then he goes out and talks to all the people involved to make sure he has the right facts, including Congress, The Fed , Japanese officials etc.

If I had a Hedge Fund I would have a similar approach.

I have recently been playing around with this concept and have come across a cool trick to save typing and make the XAML a bit tidier.

The problem with using IValueConverter for example is that you have to add the class to the resources in the XAML and then reference it with a string key. This is a bit clunky in my opinion.

If I forget to put the resources in I get the error I am trying to use my binding like an Extension property. It then came to me why don’t I make it an extension property.

So I now have a base class that all my converters derive from as follows:

public abstract class BaseConverter : MarkupExtension {
 public override object ProvideValue(IServiceProvider serviceProvider)
 {
 return this;
 }
}  

I can now reference my converter from XAML like this:

  
<Label Content="{Binding Task1,Converter={Converters:TaskConverter}}"/>
 

A much cleaner solution.

It is no good for changing the properties of another class, such as an element on a domain object, but is quite good for incrementing a counter for example.

The methods on the Interlocked class:

Interlocked.Add()

Interlocked.CompareExchange()

Interlocked.Decrement()

Interlocked.Exchange()

Interlocked.Increment()

Interlocked.Read()

It is quite limited in its usage. The example for usage is given by MS as an incremental counter.

These methods are supposed to be better performance than the using lock(), but the Exchange methods use a ref variable and so this is a limitation.

Worth knowing about for higher performance threaded algorithmic calculations though.

Amendment: 7/12/2011: I realised what these are for today. They are avoiding the double write / read of 64 bit integers on 32 bit systems when performing increment, decrement, addition or multiplication on Int64 types. You can use lock to do the same thing, but these methods do not use the blocking  Full Memory Barriers / Fences that Lock uses, hence their increased performance.