Finally implemented the MetaWeblog API.

Posted about this over at Dot Net Solutions. If you haven't already subscribed there, here is the RSS link!

I am hoping to get the Dot Net guys blogging at long last.

Wikipedia Explorer

Just posted about this over on the Dot Net Solutions blog.

Book Review

Started reading 101 Reasons to switch to a Mac (Mark McElroy).

And I stopped reading it at page 7 (First reason given).

"I'd done this before on my Windows-based PC. Because Photoshop or Microsoft Publisher would crash every other hour, the project took two days to complete. But my Mac didn't crashnot during editing, not during saving, not during printing. As a result, the job took just two hours."

This kind of bullshit makes me really angry. It's totally unsubstantiated tripe that Mac users love to throw around. Insecure as they are; they make a full time job out of comparing their platform with Microsoft's. Even at their conventions they have big posters up slanderously trying to imply that Microsoft "copied" their features in Vista.

Funnily enough - Microsoft never makes any poor mention of Macs. You never hear Microsoft people poking a hole at Apple. In fact; Microsoft writes great software for the Mac too.

Tags: mac

Powershell, String Similarity and well, etc.

Sup 'peeps!

I realised that I haven't blogged in a little while and it was time to give you fine people the low down on some of the totally cool shiznat I have been playing with recently. You got it; when I'm not doing paid-for project work, reading books on Workflow Foundation or Design Patterns I'm busy applying cool technology so I've actually got something to talk about on our office Friday technical shindig in the office.

The current hot topics of exploration in our office are WPF, WPF/E, Microsoft Expression Blend, WF, C# 3.0/Orcas tech-stack and Powershell (I must make people sick of Powershell with all my incessant rambling about it).

I'm always cautious not to bore you, my precious few (if any readers); with stuff that other people have already blogged about. So that rules out WPF/E!

Today I would like to discuss a little PowerShell scenario that worked pretty well for me.

Time Keeping (The Story)

Everyone at Dot Net Solutions logs their time. Unfortunately the GDI+/WinForm app we have for this is super-laborious and I like to log time in my Outlook calendar (and all of my meetings are there anyway). It's also super-crucial that the data is in the Time Keeping database because thats where we do a bunch of analysis like profitability per staff member and even invoicing management.

The Solution

My first problem was that the GDI+ app had no API (translate: automation model). Times like this you have divine confirmation how important the Model View Controller (MVC) design pattern is. The bottom line is that in the situation where I would like to make a web page or powershell script for TimeSheet stuff - I would have to hit the database directly which would bypass any critical validation or business rules.

I started to work on an API using C#/.NET. The great thing about PowerShell is that you can start calling into your .NET code at any time. Think of it as the VS immediate window on steroids.

#load the assemblies in

gci *.dll | % { [Reflection.Assembly]::LoadFrom($_.FullName) }

#call my .NET function

[TimeSheetAPI.TimeSheetTools]::GetProjects()

Loading Assemblies (note the LoadFrom instead of LoadFile, the latter will search elsewhere than the current directory):

Calling .NET Static Functions:

Getting members:

Query:

Logging time from Powershell!

/// <summary>

/// Insert an entry into the TimeSpent table

/// </summary>

/// <param name="salary">annual salary in i.e. 40000</param>

/// <param name="description"></param>

/// <param name="employeename"></param>

/// <param name="chargable">is it chargable</param>

/// <param name="startdate">from</param>

/// <param name="end">to</param>

/// <param name="hours">how many hours</param>

/// <param name="projectname">name of project (case insensitive) will create if no match found</param>

/// <param name="clientname">name of client (case insensitive) will create if no match found</param>

/// <param name="clientname">commit to db?</param>

/// <returns></returns>

public static TimeSpent[] InsertTimeInformation(

int salary, string description,

string employeename, bool chargable,

DateTime startdate, DateTime enddate,

decimal hours, string projectname,

string clientname, bool commit)

With this function cooked up I hit the ground running. At this point I was still exporting data from outlook and massaging the data before importing. This diagram shows the use of import-csv data.csv | % { }.

Lets automate the whole thing

We can get the data out of Outlook really easily:

$o = New-Object -com Outlook.Application;

$ns = $o.GetNamespace("MAPI");

$ns.GetDefaultFolder(9).Items | where {

$_.Body -inotmatch "ADDED TO TIMESHEET" -and $_.Subject -imatch "^TS:" } | % {

#from here, we can get all of our calendar items!

write-host $_.Subject

[...]

}

Houston, we got a problem

The main reason I would have a staging area for the data before committing it to the database was because if I didn't type the project name in exactly right (or couldn't remember how it was worded in the database) it would create a duplicate. What I really needed was a fuzzy look-up so I could type in roughly what it was and get the match with some degree of certainty.

I decided to do this in a really simple way. I downloaded C# implementations of Levenshtein distance and Soundex. I then wrote a similarity class that would break up search terms into words and run a Levenshtein distance on every combination of word and full phrase (of the original terms and the soundex representations). I have implemented a point scoring system where matches of Levenshtein distance 0 are 45, and matches of distance 1 are 10 etc. Turns out it works phenomenally well!

Similarity is always something that has fascinated me (text, images & video). We could easily step this up with better feature detection i.e.

• Word Shape
• Lexical Analysis
• Phonemes
• N-Grams
• (and many more)

But luckily in small domain environments it's easy to tweak your algorithm with heuristics/special cases (not really needed for this case though). Another point is adding in more complicated vertical-domain feature detection would require better aggregation of the n-dimensional space created:

• Clustering
• Artificial Neural Networks
• Support Vector Machines
• Bayesian Networks
• (etc)

public static int GetSimilarityScore(string input1, string input2, bool debug)

{

if( String.IsNullOrEmpty(input1) || String.IsNullOrEmpty(input2) ) return 0;

///we run LD processing based on soundex keys of words/entire phrase

int[] SoundexScoring = ProcessIndexesLD(GetWords(input1, true), GetWords(input2, true), debug);

///we run LD processing based on words/entire phrase alone

int[] WordScoring = ProcessIndexesLD(GetWords(input1, false), GetWords(input2, false), debug);

List<int> scoreing = new List<int>();

scoreing.AddRange(SoundexScoring);

scoreing.AddRange(WordScoring);

///tally up relevant scores

int level1 = scoreing.Count<int>( i=>i==0 );

int level2 = scoreing.Count<int>( i=>i==1 );

int level3 = scoreing.Count<int>( i=>i==2 );

int level4 = scoreing.Count<int>( i=>i==3 );

int level5 = scoreing.Count<int>( i=>i==4 );

int boost=0;

///if we have an exact match, we want to show an extremely high score

///i.e. the weightings become irrelevant and we dont want to risk other things

///having higher score due to many words @ level1/2 etc

if( input1 == input2 ) boost= 1000;

else if( input1.ToLower() == input2.ToLower() ) boost= 500;

return ( level1*45+level2*10+ level3*5+ level4*3+ level5*2+boost );

}

private static int[] ProcessIndexesLD(string[] keys, string[] inputkeys, bool debug)

{

List<string> dedupe = new List<string>();

List<int> scoreing = new List<int>();

for (int x = 0; x < keys.Length; x++)

{

string key = keys[x];

for (int y = 0; y < inputkeys.Length; y++)

{

string inputkey = inputkeys[y];

string combination1 = String.Format("{0}_{1}", x, y);

string combination2 = String.Format("{0}_{1}", y, x);

if (!dedupe.Contains(combination1))

{

scoreing.Add(LevenshteinDistance.LD(key, inputkey));

}

dedupe.Add(combination1);

dedupe.Add(combination2);

}

}

return scoreing.ToArray();

}

Let's have some fun from PowerShell:

function fp {

$what=$args[0]

tgp | select Name, @{Name="Score";Expression={[TimeSheetAPI.SimilarityTools]::GetSimilarityScore( $_.Name, $what, 0 )}} | sort Score -des| where{$_.code -ne 0} | select -fir 30 | ft -auto;

}

I could show you some other examples, but damn. It's almost bullet-proof!

With this in place we can now automatically pull stuff from items in the Outlook calendar. If we don't have a decent match, we can simply assume that a new project should be created.

Final Powershell Script:

function SyncTimeSheet{

$o = New-Object -com Outlook.Application;

$ns = $o.GetNamespace("MAPI");

$projregex = [Regex] "\[(?<name>.+?)\]"

$clientregex = [Regex] "\{(?<name>.+?)\}"

$ns.GetDefaultFolder(9).Items | where { $_.Body -inotmatch "ADDED TO TIMESHEET" -and $_.Subject -imatch "^TS:" } | % {

if( $_.AllDayEvent ) {

$hours=8;

}

else {

$hours = ( [Convert]::ToDateTime($_.End) - [Convert]::ToDateTime($_.Start) ).Hours

}

$chargable = 1;

if( [Regex]::IsMatch( $_.Subject, "\[c0\]" ) ) {

$chargable = 0;

}

#perform fuzzy match on project

if( $projregex.IsMatch( $_.Subject ) ) {

$projectname=$projregex.Match( $_.Subject ).Groups["name"].Value

tfpraw( $projectname ) | where{ $_.Score -gt 100 } | select -fir 1 | % { $projectname=$_.Name }

write-host $_.Subject +"_"+ $_.Start +"-->MATCH FOUND: "+ $projectname

}

$client = "Internal";

#fuzzy match the client

if( $clientregex.IsMatch( $_.Subject ) ) {

$client=$clientregex.Match( $_.Subject ).Groups["name"].value;

tfcraw( $client ) | where{ $_.Score -gt 100 } | select -fir 1 | % { $client=$_.Name }

}

#the timesheet app want to see time 00:00:00, god knows :)

$start = [Convert]::ToDateTime( [Convert]::ToDateTime($_.Start).ToShortDateString() )

$end = [Convert]::ToDateTime( [Convert]::ToDateTime($_.End).ToShortDateString() )

[TimeSheetAPI.TimeSheetTools]::InsertTimeInformation(

100,

$_.Subject +" "+ $_.Description,

$_.Organizer,

$chargable,

$start,

$end,

$hours,

$projectname,

$client,

1) | ft -auto

#///ensure it doesnt get hit again

$_.Body = $_.Body + " [ADDED TO TIMESHEET]"

}

}

What would be nice

When you write "proper" command-lets you can do far more clever and expressive things i.e. implementing whatif and confirm extensions (as well as pipeline input although that is possible from straight functions).

Imagine some of the expressive possibilities:

Get-TimeSpent -Today | Log-Time -whatif

Get-Client | where{ $_.Name imatch "Microsoft" } | Delete-Client -confirm

We have a sophisticated (home-grown) database proxy code-generation system in the company. You point it at the database and it gives you C#.NET. Now I'm thinking about it giving me powershell commandlets too.

Powershell has much in common with C#3.0/Language Query because of it's functional rather than imperative approach to programming.

Think of it as a half way house between SQL and C#2.0.

Get-Customers | where{ $_.Orders gt 300 } | % { Delete-Customer $_ }

Tags: powershell similarity levenshtein soundex csharp powergadgets outlook automation search

Presentation Technologies

Phil Winstanley posts about presentation technologies.

"I think HTML is the easiest for developers to use (think back to the 90's every man and his dog was writing HTML) and that the web is the most comfortable for 99% of users out there, they know and understand the web and how to use it, Windows Applications are much less comfortable to most."

I very much disagree with this. I think it's roughly the other way around i.e. WinForms is by far the easiest and quickest/least extraneous presentation technology. I think DHTML/AJAX is by far the hardest (for LOB apps, we are not counting DX, MDX or XNA) and most extraneous with WPF sitting exactly half way in the middle. We even have a rule of thumb at Dot Net Solutions i.e. "relative to WinForms; DHTML takes 4 times as long and WPF takes 2 times as long to develop".

Phil linked to a post by Mike Taulty about Rich vs. Reach

I also think this is incorrect. The key point with WPF is that it is both rich and reach i.e. the pitch is this: "In the old days you had to select a presentation technology based on rich vs. reach; now you have WPF - which works in any scenario only drawbacks being it takes two times longer than WinForms to develop and client prerequisites"

Mike also omits DirectX, MDX and XNA which are by far the richest (and the hardest to develop in).

From Mike Taulty:

"It comes up in debates about the various client technologies that we've got kicking around at the moment. I think you can argue as to whether Windows Forms and "WPF/E" should trade places and, also, whether there should really be 3 separate HTML-based entries and, also, Office would be another possible inclusion here as it's a client platform that sits on the screens of millions of users every day so it's very much a real platform."

There does seem to be some confusion on the reach aspect of WPF i.e. we have WPF/e and Xbap. The only differences being feature set and object/embed tag deployment.

WPF should not be on any rich vs. reach diagram because it's both.

Here is a slide from my DDD3 WPF presentation at Microsoft:

Tags: wpf xna aspnet ajax winforms mtaulty directx

Scoping in JavaScript

Omar posted a few weeks back about when this is not this.

i.e.

In the anonymous function; "this" will refer to global scope.

His solution was this:

I sent him a mail about this and may as well spread the love!

In the DHTML world of past, this was always the solution. JS is a very advanced language and supports higher order functions/closures.

Important to note that in JS the "." (dot/period) symbol sets the scope therefore any function run from a setTimeout etc will run in global scope (by definition).

In Omar's example, one function still ran in global scope (highlighted):

I liked his solution though; typically I may have done something like this:

The problem with closures and DHTML is you inevitably get circular references on DOM objects and your memory usage goes sky high i.e. they are best avoided by using some reference cache and not bothering about scope. Erik Arvidsson is the world authority on this stuff (@webfx.eae.net) (plug).

From JavaScript: The Definitive Guide, 5th Edition (David Flanagan):

8.8.4.2. Closures and memory leaks in Internet Explorer

Microsoft's Internet Explorer web browser uses a weak form of garbage collection for ActiveX objects and client-side DOM elements. These client-side objects are reference-counted and freed when their reference count reaches zero. This scheme fails when there are circular references, such as when a core JavaScript object refers to a document element and that document element has a property (such as an event handler) that refers back to the core JavaScript object.

This kind of circular reference frequently occurs when closures are used with client-side programming in IE. When you use a closure, remember that the call object of the enclosing function, including all function arguments and local variables, will last as long as the closure does. If any of those function arguments or local variables refer to a client-side object, you may be creating a memory leak.

A full discussion of this problem is beyond the scope of this book.

See http://msdn.microsoft.com/library/en-us/IETechCol/dnwebgen/ie_leak_patterns.asp for details.

Stopping Skype from opening up your firewall

I had a little rant about Skype last week.

For those of you unlucky enough to be in a situation where you have to run Skype please run it without administrator privileges using the SysInternals PsExec tool.

"C:\Program Files\Sysinternals\PsTools\psexec.exe" -l -d "C:\Program Files\Skype\Phone\Skype.exe"

This will give you the following benefits:

• Stops Skype from opening up your firewall EVERY time you start it
• Acts as a security sandbox for when Skype.exe gets compromised

Skype opens your firewall up in the worst possible way i.e. for Skype.exe rather than a port (see below).

Thanks to Kazi (one of our resident code gurus) for this insight!

He also adds these comments (MSN):

Kazi says:

and i'm sure it opens the ports on router's too via UPnP, but I didn't check it

it's clear skype is the biggest security risk today, because you can't filter viruses centrally, because it is a p2p network

Once it was called Sparkle.

Then, Microsoft Interactive Designer.

Now it looks like they are calling EID "Blend". And making it look pretty hip too.

Long Zheng got the skinny.

"We all know Microsoft Expression is a set of graphics-creation applications for uses of image composition, interface design and website design, with the names of Graphics Designer, Interactive Designer and Web Designer respectively. So what is Blend? It was clear from the functionality and tools (e.g. timeline editor), this was is Interactive Designer. Therefore, I speculate Blend is the official name crafted by the branding team to make the product hip. Although I think Interactive Designer makes a lot more sense, but when has common sense ever been marketable, right?"

No wonder they haven't been spreading the love on the CTPs recently (which is a shame because running Sept CTP of EID made my .NET 3.0 RTM machine go bananas when I tried it).

While we are talking about Long Zheng (I love his blog) check out his piece on the Vista multi-dimensional scrolling and Royale Noir, the secret (black) XP theme that we werent supposed to see.

Chris Pirillo, Robert Scoble and Open Source.

Chris Pirillo Talks CMS found from Robert's blog.

Saw this video of Chris Pirillo talking about a new WCMS/Publishing system he has decided to create (yes, yet another web content management system).

Needless to say I disagreed with just about everything he said but I was shocked that it would be possible for a passionate technologist such as me to be worlds apart from another guy in a similar space.

He kept ranting on and on about open source. He doesn't like WordPress because it's not "open source". Naturally he tries to imply that had it been open source he would have ripped it apart himself and made it better. He is now going to create his own publishing platform because nobody else gets it and he is going to do it better.

I don't buy it.

Don't get me wrong. He did make the point that most publishing platforms don't support the concept of publishing to many sources of information (and he would like to incorporate that). That's cool. I even decided to make a CMS at Dot Net Solutions and that was based on the concept that publishing content should be as simple as a "publish now" button in your Microsoft Word toolbar.

But why can't Chris do it in Word Press (or whatever). I strongly believe in Plug-ins, Extensibility and APIs/Web Services (so does Microsoft funnily enough). This gives you the best of both worlds i.e. good quality, supported software that can still do whatever you want it to do.

Software projects are immensely complex. If I was using a product I would far rather it was proprietary anyway because it would install some confidence in me about the quality of the internals, competence of the developers, procedures/methodologies in place during its development, the level of support I am going to get etc. i.e. "Joe Bloggs" hasn't been hacking around with the source code in his bedroom.

I strongly believe that good quality software is incredibly expensive to produce and for this reason alone the open source model is disastrously flawed. The average community technology preview that Microsoft releases is of higher quality than the production software of most ISVs or (in my opinion only) a lot of open source software.

I wouldn't be in such a rush to put this up had Chris not shown such a disregard for proprietary software (and making throw away comments like "PHP is the path of least resistance").

Flame me all you like but this is a reality check from my point of view (for what it's worth).

PowerGadgets Rocks!

If you love Powershell as much as I do; stop what you are doing (after reading this entire blog entry of course) and check out the trial of PowerGadgets.

"PowerGadgets is a revolutionary new data visualization product that utilizes Windows PowerShell, Microsofts new scripting shell, to allow the creation of Gadgets in Windows XP, Windows Vista, Windows Server 2003 and Windows Server "Longhorn". PowerGadgets requires no complex development environments, servers or browsers to run real-time Gadget components such as charts, gauges and maps on your desktop or in the Windows Sidebar."

I discovered it after watching Jeremy Snover (Powershell Architect) demonstrate it during an IIS7 admin Channel9 video.

Summing up grossly; PowerGadgets gives you dash-boarding capability from the power you have in PS using several visualization "gadgets":

• Chart (out-chart)
• Gauge (out-gauge)
• Map (out-map)

get-wmiobject Win32_PerfRawData_PerfOS_Memory | out-gauge -value AvailableMBytes -floating

Gives me this floating gadget on my desktop:

I can even get it to refresh every second, with this extension:

get-wmiobject Win32_PerfRawData_PerfOS_Memory | out-gauge -value AvailableMBytes -floating -Refresh 0:0:1

As a side note; there are millions of different styles for these gadgets, even a designer tool to let you create templates which you can reference on the cmd-lets with -template x.pgt

How about this one (display of top 20 processes, ordered descending by private bytes):

gps | sort PM -des | select Name, PM -fir 30 | out-chart

The seriously clever thing is that it can visualize anything you could possibly chuck at it i.e. imported CSV file, objects from Active Directory, the file system, anything!

Anyway I thought I would create a simple digital gauge displaying the ping time to www.dotnetsolutions.ltd.uk updating every second. Unfortunately because I was calling into a console application (.exe) I would get an annoying black box flick up every second.

#getping1.ps1

$pattern = New-Object System.Text.RegularExpressions.Regex "\d{1,4}ms"

$pattern.Match( ( ping ( args[0] +" -n 1" ) ) ).Value

#called with ./getping1.ps1 www.dotnetsolutions.ltd.uk | out-gauge

So the solution? Well I'm glad you asked!

#getping2.ps1

$pattern = New-Object System.Text.RegularExpressions.Regex "\d{1,4}ms"

$psi = New-Object System.Diagnostics.ProcessStartInfo "ping.exe", ( $args[0] + " -n 1" )

$psi.CreateNoWindow=1

$psi.RedirectStandardOutput = 1

$psi.UseShellExecute=0

$process = New-Object System.Diagnostics.Process

$process.StartInfo=$psi

$process.Start() | Out-Null

$process.WaitForExit()

$match = $pattern.Match( $process.StandardOutput.ReadToEnd() )

if( $match.Success ) {

$match.Value

}

else {

$args[0] +" is not responding to ICMP echo requests (pings)..."

}

No annoying black box on that one.

.\getping.ps1 www.dotnetsolutions.ltd.uk | out-gauge -Template GaugeTemplate.pgt -refresh 0:0:1

You can annotate the gadgets with titles so you know which is which. In Vista the gadgets can even live in the side bar!

Check Out Photosynth. Now!

From: http://labs.live.com/

Today were releasing our first Technology Preview of Photosynth. Photosynth combines hundreds or thousands of regular digital photos of a scene to present a detailed 3D model, giving viewers the sensation of smoothly gliding around from every angle. The scene can be constructed regardless of whether the photos are from a single or multiple sources. Its like a hybrid of a slide show and a gaming experience that lets the viewer zoom in to see greater detail or zoom out for a more expansive view. By viewing the photos in a 3D context you are able to get a better sense for the place where they were captured. Photosynth is a collaboration between Microsoft and the University of Washington based on the groundbreaking research of Noah Snavely (UW), Steve Seitz (UW), and Richard Szeliski (Microsoft Research). To find out more about how this collaboration came about visit our recently updated Video section.

Image feature detection and classification has always been a fascination of mine. These guys wrote feature detection algorithms that could take in hundreds of images and extract vertices in 3d space. Combined with a 3d viewer and some cool JPEG2000 powered image deferred loading technology they acquired from a company called SeaDragon about 10 months ago they have created a killer application.

Let's take a look at image classification as it is on Flickr now. It's basically tagging and simple meta data like file name, camera model etc. If I take 100 pictures around a town in Rome, I would have to tag all of them as "Rome". However, with this technology, if some one else has already tagged a Rome picture, my images might get tagged automatically.

Discovery gets about a million times richer when you have clever classification. I hope the next 5 years will see photo communities get more classification capability i.e.

• Face recognition
• 3d vertex/scene recognition like Photosynth

Microsoft continues to innovate. Only last week they beat Google Earth by adding far more sophisticated 3d buildings to Virtual Earth (and the technology to get all the major cities in the world by the end of 2007 due to a recent company acquisition). The obvious next step is a killer 3d application combining an online community, Virtual Earth and Photosynth. It's an exciting time to be in the industry.

Killer, immersive applications like this also remind us that the writing's on the wall for HTML/Web applications as we know them now.

Live Messenger Problems, Fixed.

Several weeks ago I was over at iNNN in Reykjavik, Iceland. While there I made friends with all the developers (seriously cool bunch of guys they are) but was gutted that my MSN/Live Messenger would not receive or send invitations to them. Other than that I have also noticed other strange behaviour with my current contacts.

I tried saving my contacts to an XML file using Contacts->Save.. and creating a brand new account. I didn't get too far down this road because Live Messenger wouldn't let me even sign into the new account. It would just say "Signing in" forever. One of the guys at iNNN too said he recently had the same issue and was forced to set up a new account. At this point I got really depressed and assumed the MSN network itself was in a bad state and even started using Skype (I didn't use Skype for long though!).

Hope came when I signed into another machine that I hadn't used before the invitations came through and all seemed OK. However on my home and office machine, I still couldn't see the new contacts. I tried using Meebo and that worked too. Next I tried re-installing the Live Messenger client, and that had no effect. It was clearly some corrupt database in my profile area or something like that.

I fired up trusty Sysinternals FileMon/RegMon and signed in to see where it was referencing.

On the file system side - I found two profile areas used by Live Messenger.

C:\Documents and Settings\Tim Scarfe\Local Settings\Application Data\Microsoft\Windows Live Contacts\[email address]

C:\Documents and Settings\Tim Scarfe\Contacts\[email address]

My first hunch was to delete the contents of these folders and try signing in again. When I did so it just said "Signing in" forever (as with the new account I created).

After some trial and error I discovered that renaming the registry entry:

HKLM\CU\Software\Microsoft\MSNMessenger\Policies\contacts.msn.com

Would allow me to sign in!

The strange thing is that I can't find any reference to this key in my Regmon trace but I am quite certain it was responsible for fixing the sign-in problem.

I assume this registry key was also responsible for me not being able to sign in on the new account I created too but I haven't verified that yet.

With that problem fixed, feel free to come and have an IM chat J

QuakeCon 06: John Carmack Keynote Address

This evening I had the great pleasure of watching John Carmack go into detail about various technical issues pertaining to:

• Increasing processor cores
• Futures
• Scalability
• Parallelism
• Harder to code than you may think
• We are reaching diminishing returns
• Games on mobile phones
• Xbox360/PS3 hardware issues

This video really resonated with me and it touches on many relevant issues that are going to become increasingly prevalent in the near future.

At Dot Net Solutions we tend to write business middleware and use high level languages and object frameworks. Much of our stuff runs single threaded and when we do multi-threaded apps they're "coarsely" threaded. The kind of parallelism John is talking about (computationally intensive applications) is seriously non-trivial and would soon hit diminishing returns when you ramp up the cores anyway. While some applications spread out nicely over multiple processors; games are usually an example of processing where split items of execution require constant synchronisation.

John points out that we can't expect much more brute force performance from processors; and we may only expect to get another single order of magnitude before hitting limits. The near future will see 4 and 8 core processors and contrary to what some non-techie people may believe; it will not mean 4 or 8 times the performance. Not even close.

This came at a good time for me because only last week Anandtech posted a great parallelism article about how Valve software are trying to solve this problem in their next game. We had quite the little e-mail thread in the office about it at the time.

An interesting paper about programming Threads in C# (Andrew D. Birrell) that Mike Taulty linked to some time back. Multi-threaded programming may seem easy, but it's not.

Skype causing considerable network slowdown

I got back from work this evening and discovered that my home Internet connection was running very slowly. Considering it's a 10Mb connection I decided it was not due to millions of UK teenagers visiting their myspace.com accounts at the same time.

I noticed that there was a hell of a lot of activity through my NIC by glancing at the status panel.

Digging a little deeper; I fired up Microsoft Network Monitor 3.0. I was horrified to see an incredible amount of traffic to hundreds of machines all over the internet. At a glance it looked like p2p/bit torrent traffic. Obviously I don't run any p2p software due to network slowdowns and security concerns. Netstat.exe confirmed that there were about 800 socket connections all over the place and this immediately confirmed the slow down.

So what process what hosting all this activity, and why?

I fired up good old Sysinternals Process Explorer to identify the culprit. I hardly run any non-Microsoft signed images so it didn't take long to track down the offending process.

Skype.

I have no idea why Skype would hold all these connections open to people out there on the Internet but I was horrified. It's a massive security concern. There is every chance it's just me being stupid but until someone gives me a decent explanation I won't be using Skype again. It's shame because Live Messenger isn't working well for me either right now; for some reason I can't add new people (and some people can't see me, or vice versa). It appears to be an installation problem because on new machines (that I haven't used for MSN) - it works.

Damn!

Update 15^th Nov 18:00

I have just done a little due diligence/research on this one.

Turns out that Skype is written by the people out of Kazza and works on a peer to peer model. I was only using Skype for IM, not phone conversations or file transfers though. A little bit of Googling pulled up this article about why you may want to block Skype.

"The bandwidth used during file transfers or during internet telephony can be tremendous, depending on the amount of usage--and Skype can turn your network into a "Supernode" without your consent, using it as a relay station for calls that do not originate or terminate on your site. The impact on productivity is severe. The reason Skype calls work so well is that it uses "intelligent routing," which sounds like a good idea until you realize what that means: Skype routes calls over the most effective path possible, leveraging available bandwidth from users on the Skype network."

Emphasis mine.

All I can say is that I hate malware and Skype.exe is the dictionary definition.

Update #2 15^th Nov 23:00

"Security questions have some companies steering clear of the Internet phone service [...] On Nov. 10, Info-Tech Research Group in London, Ont., issued a report under the headline "Ban corporate Skype usage immediately" that cited a litany of potential security risks. "

Sean in the office pointed me to another article about this.

Skype Supernode problems

From the Skype Forums:

"[...] be a supernode. This requires that you have a fast internet connection (256kbit upstream or more), your firewall allows incoming TCP&UDP to the port you see (and could change) in Skype options, AND that you run Skype like that for days or weeks continuously without restarting. So if your firewall blocks incoming TCP or UDP part of the day, then your computer does not qualify. In any case, supernodes are elected on demand, and only those with fastest internet connections are elected -- so even if your computer does qualify, becoming a supernode is not guaranteed. A friend of mine tried a while ago, and didn't succeed This actually shows there is no shortage of supernodes in Skype. There is no option to voluntarily become a supernode, as it is best to leave the decision up to automatic network management algorithms.

"[...] be a relay node. Calls and Instant Messages are sometimes relayed over random Skype nodes (in encrypted form, of course, so relay nodes can not eavesdrop). Again you need a fast internet connection, and your firewall must not block incoming TCP&UDP. As opposed to supernodes, there is no special "relay node" status, it's just that your services will be used by those in need when necessary"

" When you installed Skype you agreed to let Skype use some spare processing power and network bandwidth (see EULA), although that does not mean that you cannot take actions on your own to discourage your own system from being promoted to Supernode status. It's just that you are not likely to get support from Skype on how to do this because doing so would be against the fundamental philosophy of how Skype was designed to work (from Skype's perspective this is kinda like shooting oneself in the foot)."

"Probably the easiest way to prevent a PC from ever becoming a Supernode is to place it behind a NAT router."

From http://www.skype.com/company/legal/eula/ :

Article 4 Utilization of Your computer

4.1 Utilization of Your computer. You hereby acknowledge that the Skype Software may utilize the processor and bandwidth of the computer (or other applicable device) You are utilizing, for the limited purpose of facilitating the communication between Skype Software users.

Dan in the office made these comments in an email:

"Thinking further this really does bring the whole of Skype's business model crashing down around it. You ask any user are they happy to be a relay, they're bound to say no. The only way Skype gets away with it is because so many people just dont realise (like Tim). It may well have been discussed in techie circles, but Im doubt most users would realise[...]."

I don't need to do any more research here. In my opinion Skype is at best malware (on a par with win32 iTunes!) and worst a security/bandwidth nightmare waiting to happen. I'm livid as I feel mislead by their marketing. I have already uninstalled it (along with the hole it poked in my Windows firewall) but actually feel like going out of my way to lobby for corporate policy on Skype usage where we conduct business.

Automation

I love rich user experiences as much as the next guy but the problem is their inherent manual, sequential process. If any single item in the process takes a long period of time then it blocks you (the user) from being able to do other things. Like sleeping!

Enter Powershell and a raft of good quality command like tools for everything ranging from parity checking/repairing, RAR compression, downloading files, copying files etc.

Here was my last nightly powershell script J

#move gymnastics vids over to f to save space on g

#this should result in 18gig to play with on g

robocopy 'G:\Gymnastics Videos' 'F:\Gymnastics Videos' /MOV /LOG:G:\gymtransferlog.txt /S

#verify all the downloads

sl "F:\transfers\newsgroups\p\"

gci | where { $_.Name.EndsWith(".par2") } | foreach-object{ &'C:\bin\par2cmdline-0.4-x86-win32\par2.exe' r $_.FullName }

#extract some things we have space for

& 'C:\Program Files\WinRAR\UnRAR.exe' x "Armin_van_Buuren_-_A_State_of_Trance_273_(DI.FM)_2006-11-02.part01.rar" "G:\MP3\!MP3_ForApproval"

& 'C:\Program Files\WinRAR\UnRAR.exe' x "Armin_van_Buuren_-_A_State_of_Trance_274_(DI.FM)_2006-11-09-.part01.rar" "G:\MP3\!MP3_ForApproval"

#we got a bunch of oct trance singles in seperate rar files

gci | where{ $_.LastWriteTime -gt [DateTime]::Parse("13/11/2006 23:06:00") -and $_.LastWriteTime -lt [DateTime]::Parse("13/11/2006 23:33:00") -and $_.Name.EndsWith(".rar") } | foreach-object{ & 'C:\Program Files\WinRAR\UnRAR.exe' x $_.FullName "G:\MP3\!MP3_ForApproval\Trance Singles Oktober 2006 (Vinyl,CDM,CDS,CDR,WEB)" }

& 'C:\Program Files\WinRAR\UnRAR.exe' x "house okt deel 1.part001.rar" "G:\MP3\!MP3_ForApproval\house okt deel 1"

& 'C:\Program Files\WinRAR\UnRAR.exe' x "House okt deel 2.part001.rar" "G:\MP3\!MP3_ForApproval\house okt deel 2"

#not sure if this little lot got par checked (all 8GB of it...)

&'C:\bin\par2cmdline-0.4-x86-win32\par2.exe' r "F:\transfers\newsgroups\p\Future Of Trance\MorXMXPX3X0001.par2"