Elcom
ELCOM: More Microsoft 2008 Stack testing results
We’ve done a little more testing of our product on the Microsoft 2008 stack and I’m pleased (and a little worried) with the results.
The pleasing part: we are now showing results on the 2008 stack as being 16X faster than the 2003/2005 stack.
The worrying part: results of 16X faster are very high – what if we’ve tested something incorrectly…
So, the goal of this post is to explain what we did and how we measured the results. Then, I’m after your feedback on what else we should be checking.
Background
First, the background.
In January we moved our Elcom web site over to the 2008 stack. That is, we recompiled our entire Community Manager application to target the .NET 3.5 Framework, moved to SQL Server 2008 CTP5 as the back-end and hosted it on Windows 2008 Server RC0. We immediately noticed that the site was much quicker. Browsing the site was faster and managing the content (through all the web based administration tools was very snappy). We then did some basic testing by launching 1,000 concurrent requests and measuring the delivery times. You can read the full results on Alan’s blog, but in summary, the 2008 stack was 5 times faster. Impressive stuff.
We always wanted to do some more detailed testing, but it wasn’t until Windows Server 2008 finally RTMed two weeks ago that we started testing again. This time round the techies wanted to beef up the testing a little.
Latest testing
Whereas our initial testing had been a simple comparison between Windows 2003 Server, SQL 2005 and .NET 2.0 versus Windows 2008, SQL 2008 and .NET 3.5, this time Alan set up testing for all 8 scenarios so that we could isolate which components of the 2008 stack were contributing the most. eg is Windows, SQL or .NET that is giving the improvement.
He also changed the testing method. This time, instead of just firing 1,000 requests he kicked of a loop of 1,000 sets of 20 concurrent requests. This is a more realistic scenario for what our sites experience. He fired this via a wget based Linux bash script.
The machines he used are as follows:
2008 stack machine: Intel Core2 Dual core 2.12 GHz, 2GB RAM, 160GB Seagate SATA drive.
Testing (Linux) machine: Intel P4 3.0 GHz, 1GB RAM, 80GB Seagate SATA drive [ie This machine is acting as a client]
As you might notice, the 2008 stack machine was acting as both the IIS and SQL server. Although not a production architecture, this was deliberate to ensure that network affects were minimised.
Results
Here’s the results so far:
| Windows | .NET | SQL | Duration (h:mm:ss) | X faster |
| 2003 | 2.0 | 2005 | 3:37:59 | – baseline – |
| 2003 | 2.0 | 2008 CTP5 | not yet | |
| 2003 | 3.5 | 2005 | 3:59:14 | 0.9 |
| 2003 | 3.5 | 2008 CTP5 | not yet | |
| 2008 | 2.0 | 2005 | 0:13:55 | 15.6 |
| 2008 | 2.0 | 2008 CTP5 | 0:14:17 | 15.2 |
| 2008 | 3.5 | 2005 | 0:13:43 | 15.9 |
| 2008 | 3.5 | 2008 CTP5 | 0:13:33 | 16.1 |
A few observations:
- We haven’t finished all our Windows 2003 testing yet. They take almost 4 hours to run, and the guys are fitting this testing all in amongst their normal daily work 🙂
- The main improvement is (obviously) due to Windows Server 2008, and IIS7
- Running the 3.5 Framework on 2003 with SQL 2005 is actually slower than with 2.0
- Some of the other results are also puzzling… for example:
- On Windows 2008, using SQL 2008 is slower than SQL 2005 when we target 2.0 but faster when we target 3.5
- Happily, the best result is achieved when using the full 2008 stack
If I put the results another way, we can see that adding 2008 components (Server, then .NET, then SQL) has incremental improvements eg
| Windows | .NET | SQL | Duration (h:mm:ss) | % increase |
| 2003 | 2.0 | 2005 | 3:37:59 | – baseline – |
| 2008 | 2.0 | 2005 | 0:13:55 | 1566% |
| 2008 | 3.5 | 2005 | 0:13:43 | 1589% |
| 2008 | 3.5 | 2008 CTP5 | 0:13:33 | 1609% |
But there’s no escaping that the major improvements are all thanks to Windows 2008 Server. In general Windows 2008 is much faster, and our techies have already switched to using it as their desktop OS.
FAQ
Some questions. I’m trying to look at our testing as critically as possible. After all, 1600% improvement is something not to be taken lightly, and there will likely be a few people reviewing these results. So, I want to be clear on where we stand.
The first thing to state is that we are not a certified testing lab. So, this testing has been conducted on simple server hardware, and with tests that are geared to how we operate. We didn’t have any of our big rack servers available for testing, so the results are pretty ‘real world’.
Q: Why does it take almost 4 hours to do 20,000 requests on Windows 2003? Something must be wrong…
A: Yes, this concerned me as well. But here’s what I didn’t mention earlier. Each request we submit is returning our home page, plus graphics, plus all links from that page and any graphics and documents from those pages as well (ie wget with the u directive is basically spidering from the page). It turns out that each request is returning 6.4MB of data. Thus, 20,000 request in total is returning about 120GB. This is to emulate load testing the server. With Windows 2003 this takes over 3 and a half hours.
Q: Did you test on a smaller page?
A: Yes, we also performed testing on a landing page that returned 70KB. The results were much quicker, but in the same proportions.
Reducing page size down further to very small (eg an empty page, but still delivered by our Community Manager product) brought the comparisons closer, with Windows 2008 being only 4X quicker than Windows 2003. The 1,000 X 20 concurrent requests took 51 seconds on Windows 2003, but only 12 seconds on Windows 2008.
Q: Are the actual durations meaningful?
A: No, the durations on their own are not that meaningful, since changing hardware will obviously affect the results. What is meaningful is comparing the durations between the different scenarios.
Q: Did you try on different hardware?
A: Yes, we’ve managed to get our hands on a beefier machine (Quad Core 3.5 GHz, 8GB RAM) for a few days. Initial testing has shown quicker results on Windows 2003, but in the same order of magnitude (ie hours). Full results from that machine will be available in the next few weeks.
Q: How many times did you run these tests?
A: Most tests have been run a few times as the techies were setting them up and performing initial loops. All the test results are from testing scenarios run by Alan. Brad was then asked to run a few of the tests again independently. He verified the baseline 2003 Server result and the first of the 2008 Server results (ie with 2.0 and SQL 2005)
The results above are based on the last and most thorough pass (by Alan). Thus, it is the actual result, not an average of repeated tests. Ideally multiple results would be better, and as time permits, we’ll re-run all the tests, and provide averages.
Q: Did you perform any optimisation of the 2008 setup?
A: No, all test scenarios were conducted on freshly installed, out-of-the-box setups.
Q: Did you try optimising the Windows 2003 setup?
A: Yes, we did. The techies tried different combinations of turning on compression, increasing memory, CPU monitoring, tweaking application pools, and other memory changes. In all cases there was very minimal change in Windows 2003 performance. The results tabled above are based on the untweaked out-of-the-box installation.
Q: What version of Windows Server is the Elcom web site running on?
A: The actual live Elcom web site is still on Windows 2008 RC0. We aim to cut over to the RTM version once our SPLA license is finalised… 2008 SPLA licensing should be available this month (fingers crossed).
Q: Are you using any .NET 3.5 specific features in these scenarios?
A: No, we’ve made sure that the code base is exactly the same, the only change has been the framework version targeted during compile (and of course the IIS settings).
Next steps for Elcom
So, where to from here?
We’ll be moving a few of our client sites over to Windows 2008 hosting in the coming weeks (assuming licensing is all sorted). A few of our staging servers are already in transition. And from April onwards we plan to move all our hosting onto Windows 2008.
[We’ll still be supporting clients who host on Windows 2003 of course.]
Next, we are upgrading clients to .NET 3.5.
Our next Community Manager.NET release (in mid March) will be available in both 2.0 and 3.5 versions. The 3.5 version (which we are referring to as Community Manager 2008) will be essentially the same code base, with only a few new 3.5 features used for off-line services (the new System.AddIn stuff).
Our code base will be branched into the two versions for approximately 3 months until we’ve completed our testing and moved all clients over to .NET 3.5. After that we will return to having the one code base (ie just targeting 3.5).
SQL Server 2008 is still too early to make plans for. Whilst we will continue testing with it (especially with CTP6 just released), we won’t be moving any production systems (other than our own web site) over to it until much later in the year.
Over to you
There’s our situation. I hope it is helpful (or at least of some interest).
I’m keen to answer any questions, hear any criticisms and take on board any suggestions.
Please contact me via any method on my Contact page.
ELCOM: Career Plans for Developers – Part II
Further to my post about career plans for developers at Elcom, I wanted to mention the approach of encouraging staff to build their personal profiles.
It can be summed up in this attitude:
Your job (as a developer and employee) is to be so good that you are being chased by head-hunters all the time.
My job (as Technical Director) is to make the work environment so good that you don’t want to leave.
I feel this is a sensible approach, for a number of reasons:
Firstly, I don’t want staff who aren’t improving. Rather I want them continually pushing themselves to do bigger and better things. Momentum must be maintained. If we aren’t improving, then realistically we are going backwards…
Secondly, the better skilled they are, the better the result for Elcom.
Thirdly, the better well known they are, the more well known Elcom becomes.
Fourth, the better well known Elcom is, the better the caliber of candidate we can attract to join and grow Elcom.
Keeping a great team in place is a tricky thing, and I’m not claiming to be an expert at it. But in case you think that encouraging developers to be head-hunted is a crazy notion, here’s a few tidbits I’ve observed over the years.
The job market is tight
The first is that the IT market is very tight. If a developer wants to find a new job, they won’t have much trouble. And *some* recruiters are becoming more ruthless and unethical by the day…
So, if you assume that your staff will stay on board simply because they can’t get another job – you are deluding yourself – either that or they are so bad that they really can’t get another job and you should have fired them ages ago anyway!
Feeling valued
The second is that most developers want to feel as though they are making a contribution that is both valuable and valued. (Note: This, sadly, is the reason many look for another job – they feel a sense of being valued when some other company offers them a job). Thus, by encouraging developers to better themselves, and perhaps become well known in the community, we are encouraging them to feel valued.
We’re people after all
Thirdly and most importantly, developers are people. Sure, they are an asset, resource, body, project plan item, etc too. But the important thing is: we as people grow via our relationships. At work we need to remind ourselves every day that our relationships with our colleagues are what affect our morale, productivity, generosity and ultimately longevity in a company. Common sense I know. But easy to forget in the over-worked, email deluged, constantly interrupted hours between breakfast and bed time.
Have I missed something obvious? Please leave me a comment.
ELCOM: Career plans for Developers – Part I
Part of my brief at Elcom is to look after all the developers (ie people management). This is something I enjoy, especially when seeing developers grow and improve themselves.
During November and December I sat down with each of my staff and started preparing their career plans. It is their responsibility to work out their goals and aspirations, it is my job to facilitate that happening (within reason of course).
The process involves preparation by the developer (approx 1 hour), then a chat with me (1-2 hours) and then some updating of notes, consolidation of ideas, etc.
During the chat we cover everything, including likes and dislikes, books, training, aims and goals, money, gripes, and then finish with some KPIs.
I thought others might be interested in how we conducted the process. Attached is the Career Plan brief, which outlines the method. I’ve included my original one, which has a few problems.
Based on feedback from the team during the process, I will be updating it to fix a few things. For example, the doc has various time frames that it asks about which can be confusing, however my point is this: when setting goals we often over-estimate what we can achieve in 1 year, but under-estimate what we can achieve in 3 years. So we need to have short term and longer term goals continually in mind.
The file is a Word document. Feel free to take it, improve it, use it if useful to you and your role. I’m open to any comments and suggestions you have…
(Here’s the books we have ordered so far, here’s the ones still to be ordered)
Read Full Post | Make a Comment ( None so far )ELCOM: More on Windows 2008, SQL 2008, VS 2008
Earlier this week I mentioned Elcom had moved everything over to 2008 (that is, our product – Community Manager.NET – is now targeting .NET 3.5 in Visual Studio 2008, and our site is running on Windows Server 2008 with SQL Server 2008 on the back end).
Initial thoughts were that the site was running a bit quicker. Turns out we were wrong. The site is actually running a lot quicker.
One of our techies, Alan Lee, set up a testing environment so that we could start measuring the performance improvements. You can check out his method here, and his results here. This is only the start of his testing mind you, and there are a few caveats to note (in regard to CPU usage).
But the results are stunning.
Bottom line: The Elcom site is now running over 5 times faster.
+
=
ELCOM: Windows Server 2008, SQL Server 2008, Visual Studio 2008
Elcom is all 2008.
The Elcom web site is now running on a Windows 2008 Server, with SQL Server 2008 as the back-end and Community Manager.NET completely recompiled under Visual Studio 2008 (targeting the 3.5 Framework) delivering the content.
Community Manager.NET is Elcom’s content management system (CMS), which we have now re-compiled, tested and released on the .NET 3.5 Framework.
Now, you might think this is a pretty simple thing to do, but there’s actually a reasonable amount of work involved to ensure it is all performing smoothly. I won’t go into the details here, but suffice to say, when you have a mature product (Community Manager is in its 5th year of development) there’s a fair bit of code that needs to be checked. Third party tools, and rarely used custom modules only add to the mix.
Additionally, running it on two products that are still not yet released (Windows 2008 and SQL 2008) brings its own share of fun 🙂 The main hassles have been in getting up to speed with a few of the Windows 2008 Server settings, some of which Brad and Alan have noted.
And, to be honest, there are still a few little issues with the site… (can you find them?).
It’s a little early to be talking about the advantages and performance improvements that the updates provide, but we’ll report back in the coming months with issues and improvements.
ELCOM: Developers going dark
Microsoft developers (I’m told) have the option to ‘go dark’ at times. At least that’s what Joe Schwetz indicated at a recent Sydney Deep meeting. The aim is to allow them to catch up on projects that are running behind. The process involves killing all distractions (email, blogs, phone calls, IM, etc) and focusing on priorities.
I’m sure many software companies use this technique in various guises. Here’s Angus, one of our Devs, in a dark moment:
I wonder if Michael Kordahi will notice the sticker on Angus’ PC, just under the book shelf…
I write this on Friday evening as I head home for two weeks holiday break. The last two weeks have been incredibly hectic. Anyone else found this year to be busiest in a long time?
Read Full Post | Make a Comment ( 2 so far )PERSONAL: Elcom – The first month
I’ve managed to make it through the first month of my new job :-). And I love it. I’ve held off posting about the new role until I felt I had a good understanding of the company, people and clients.
First off, the job. I’ve joined Elcom Technology as their new Technical Director. This means I’m looking after their Dev and Tech teams, and responsible for the technology strategy of the company. It may sound a little daunting, but in reality it is a perfect fit for where I am at the moment.
Second, the company. Elcom are in the Enterprise Content Management space, and have their own product – Community Manager.Net – built on .Net and SQL Server. The product is mature, having been around over 5+ years, and is feature rich. It has over 50 different modules, and from what I’ve seen is pretty impressive. It’s not all fantastic of course, and part of my job is to manage how we smooth out the rough edges as well as enhancing it for the future.
Third, the processes. A good product is all well and good, but we all know a company’s success lies in the execution. And for this reason I’m pleased to see the company has a very strong foundation. They have a sales team, a marketing team, a projects team, a creative team, a customer support team, and of course the development and tech teams. They have everything in place for proper execution. To be honest, this was probably my biggest concern/requirement for my new job, and I was very relieved to find they have this all managed well.
Lastly, and perhaps most importantly, the clients. I’ve now met with a number of Elcom customers and am pleased they are all highly professional and understand the process of software development. They all by and large understand the costs of developing proper web sites and systems. Our clients are more interested in fast tracking their systems into production than they are about haggling on price. This in turns engenders a strong ethic of service from the Elcom staff.
Finally, you’ve heard me say this many times… I’m the luckiest guy I know.
Craig Bailey on Microsoft 