Heap memory usage

To figure out how the heap memory is being used, using windbg.exe:

.load clr10\sos.dll

!dumpheap stat

High throughput server apps and exception handling

(or how I took down the server yesterday)

Yesterday, there were multiple failures leading to a single catastrophic failure.  We transitioned our SQL server to SQL Server 2008, so our database server changed.  I typed in the new address in the config file for my app, started the app, and checked the log for errors.  I was clear so I went home.

The next Monday, the site was down because it couldn’t receive any quotes from my server.  The application couldn’t reach the SQL server because the name wouldn’t resolve!  DNS configuration was mistake #1.

Why didn’t the error get logged?  The database calls were only made during market hours, so the code never got hit.  Assuming that checking the log would be enough was mistake #2.

I had actually anticipated an error like this, but my code wasn’t sufficient.  I caught the exception thrown during the database call, and wrote code to do this only once a day… except during the error condition.  So I’d atttempt to make the database call every single time a quote was requested (which is about 500-1000 per second).  Making so many calls is mistake #3.

The database call would take time to complete.  Even if it was fast, it wasn’t fast enough to handle the volume.  Overwhelmed, the server would freeze up and would stop responding to requests.  Not anticipating a response delay in a critical section (both in the threading sense and in importance) of code was mistake #4.  I fixed this by using a TryEnter block of code so that only one thread could do the database request at a time, and supplying an estimated value if a thread couldn’t enter that section of code, or if an exception happened.  To avoid hitting the TryEnter line every time, I used a form of double check locking.

Evolution of a subscription based real-time streaming server

I’m writing a server application which streams data to clients based on filters that they specify.  For example, a user can specify that they want to see out-of-market trades on stocks which have a volume above 100k, and under $40.  Or just stocks that have just hit their daily high.  I wrote a few different versions of this and ran into different problems each time.

For background, the technology I’m using here is WCF, which has one way calls, but are not truly fire and forget.  If the connection fails, then an exception is thrown when the message is sent, and much cpu time is taken.

Attempt #1: All requests go into the same thread pool

Each client has a specific callback function which is used to send a quote.  It was necessary to decouple the incoming data feed from the quote sending mechanism, and to make sure I was making good use of multiple processors (I’ve found earlier, that WCF can be quite processor intensive).  Essentially what I did, was a work item was queued in the threadpool, and had the atomic function of sending that one quote.  The problem that I ran into was that with multiple clients, a single bad client can severely impact the system.  Eventually, as items get queued and executed, if one connection causes a slowdown for the quote that is being sent, all the available threads will be waiting on callbacks to that one connection.  Even though I was catching the exception to flag the connection, many quotes can be sent to that same connection before the exception is thrown.

Attempt #2:  Single thread creates async calls to callback functions

When I asked my boss, this is how he said most server apps were designed.  I rewrote the code to make the notifications asynchronous, and a single thread was used to kick off the asynchronous operations.  Since the exception would never reach me if the connection failed, I also used a synchronous polling mechanism in a separate thread to check if the connection was still okay, and killed the connection if it wasn’t.  I discovered that the internal exception handling of the failed send was taking a lot of cpu time, and was unacceptable.  Maybe I should have figured out that this solution would not have worked, but I didn’t understand how the async operation was implemented behind the scenes.  My boss suggested that it could be at the OS level before I tried it, but now I believe it was a standard IAsyncResult type of operation.

Attempt #3:  A dedicated queue and a threadpool work item for each client

I rewrote the code again, this time to solve the problem of one bad client clogging the system.  Each client would have a queue of quotes which needed to be sent, and a threadpool work item which would take care of sending the message.  If one threadpool work item stopped working, others could still work.  I knew that having an individual real thread for each client would be bad, since each thread takes about 1 megabyte, and too many threads causes thrashing at the cpu level.  When I thought about this, I realized that it wouldn’t work.  Once you go beyond the 25 worker threads, the queued items would be on pause, and would have to wait their turn before they are run.  Not exactly real time streaming data for everyone.  I tested this by running 400 threadpool work items which would sleep and increment a static value each time.  I should have gotten +400 each time, but I didn’t.  This design was also inherently flawed in it’s misuse of the threadpool, which is designed to do discrete tasks.

Attempt #4:  A dedicated queue for each client, and a single thread that kicks off threadpool work items.

I believe that this is my final and best solution.  One thread is used to determine whether quotes are eligible for being sent, and if they are, they are queued in a short queue (items are dequeued if it gets too long), one for each client.  Another thread loops through each queue of eligible quotes, sees if there any available, and queues a work item in the thread pool if there is one, then it sleeps (how long depends on the level of throttling).  Another strength of this solution is that I can flag a queue for being mid-send, so other threads wouldn’t dequeue from it.  This seems similar to the first solution, except that not every eligible quote gets queued… only the ones that should be sent.  The first solution queues a work item regardless, and does the throttle checking afterward.

Digging for undocumented system calls

After much reverse engineering and stepping through MS source code looking for how the .net framework renders the simple black X close button for maximized MDI windows, I’ve found something which is well documented. DrawCaptionButton. Sigh. Well, here it is in case I forget.

Image ControlImage;

protected override void OnPaint(PaintEventArgs pe)
{
pe.Graphics.Clear(BackColor);
pe.Graphics.DrawImage(ControlImage, 0 , 0);
}

Image GetImage()
{
Bitmap image = new Bitmap(0x10, 0x10);
using (Graphics graphics = Graphics.FromImage(image))
{
ControlPaint.DrawCaptionButton(graphics, new Rectangle(Point.Empty, image.Size), CaptionButton.Close, ButtonState.Flat);
graphics.DrawRectangle(SystemPens.Control, 0, 0, image.Width - 1, image.Height - 1);
}
image.MakeTransparent(SystemColors.Control);
return image;
}

Example of visual style applied to a button

public partial class Expando : Button
{
public bool Pressed { get; set; }
public Expando()
{
InitializeComponent();
}

protected override void OnPaint(PaintEventArgs pe)
{
VisualStyleRenderer renderer;
if(Pressed)
renderer= new VisualStyleRenderer(VisualStyleElement.ExplorerBar.NormalGroupExpand.Normal);
else
renderer = new VisualStyleRenderer(VisualStyleElement.ExplorerBar.NormalGroupCollapse.Normal);

pe.Graphics.Clear(BackColor);
renderer.DrawBackground(pe.Graphics, this.DisplayRectangle);
}

protected override void OnClick(EventArgs e)
{
Pressed = !Pressed;
base.OnClick(e);
Refresh();
}
}

System.Windows.Forms.VisualStyles.VisualStyleElement

I was writing a windows app and wanted to add in buttons that look like the built-in windows buttons. For example, the MDI close window button, and the gripper ui texture. Fortunately through the power of reflection, I’ve found that the .net framework defines these in System.Windows.Forms.VisualStyles.VisualStyleElement. Unfortunately, these are poorly documented, and I couldn’t even find samples of what the styles look like.

So here they are, in one monolithic jpg file.

Dumb mistake of the day: why is the form rendering 7 times?

This one was odd.  I noticed that OnPaint was called 7 times per normal refresh, and only after an unrelated exception occurred somewhere else (WCF connection faulted).  This happened because I was switching ForeColor of the control, and the control refreshed each time it was changed.  Don’t know why it only happened after disconnect, though.

How to add scrollbars to a custom control

Turns out you don’t really need to do this (unless you want to scroll by something other than pixels like lines of text).  Panel implements ScrollableControl, which takes care of the scrolling for you as long as you adjust the height of your control!

Reader/writer locks and favoritism

Something that I never really considered when using an implementation of reader/writer locks is whether it favors readers, writers, or tries to take a balanced approach. I waited to use this feature in .net until recently since the basic reader/writer was implemented poorly

So 3.5 has ReaderWriterLockSlim, which is really handy, except in my case, it doesn’t work the way I want it to.  I’m writing a quote server which requires high read availability.  After I implemented ReaderWriterLockSlim, I couldn’t get a read lock in some cases when there was only one other read lock.  Turns out that I had one other request for a writer lock pending, which blocked the 2nd read lock. 

I read this in the documentation, but I guess it just didn’t click at that time.

minus 10 points for me (and minus 3 hours of my life)

while(currentThread == backgroundThread)

Since Thread.Abort is so bad, the normal alternative is to do:

volatile bool runBackgroundThread;

then

while(runBackgroundThread) {…}

In my code, there is a start and stop function which would set runBackgroundThread to different values, then create a new thread in the start function.

I was worried about the scenario where the user stops and starts really quickly, which would mean that I would have to wait until the 1st thread dies before starting the 2nd.  Then I have to consider what happens if the background thread takes too long on it’s last iteration, and then use some sort of blocking.  Otherwise it’s possible to have 2 background threads running… one being a zombie thread.

So what I did was:

public static void Heartbeat()
{
Thread currentThread = heartbeatThread;
while (currentThread == heartbeatThread)
{ … }
}

Where heartbeatThread is defined in the class.  I think this should work fine except for the odd case when the new thread is assigned to the same pointer as the old thread.  Is that possible?

I think I have to think about this more.

edit: I think a better way is to actually pass the thread pointer in to the function which will run in the thread instead. This avoids the possiblity of the thread being assigned to between initialization and the thread running and acquiring the pointer to its own thread.

example:

backgroundThread = new Thread(Heartbeat);
backgroundThread .IsBackground = true;
backgroundThread .Start(backgroundThread );
…
public static void Heartbeat(Object currentThread)
{
while ((Thread)currentThread == heartbeatThread)
{ … }
}

edit 2: use Thread.GetHashCode() instead