Task.ContinueWith() [Parallel Extensions June 2008 CTP]

ActivityExecutionStatus.Executing + WorkflowQueue.QueueItemAvailable [Windows Workflow]

(Almost) the same thing.

That is all.

My side project TextFlow is coming along nicely... I've still got a lot of work to do there, but it's surprisingly functional for something I've only been tinkering around with for a few weeks.

TextFlow is (among other things) a language compiler that generates Windows Workflow programs from a text-based language syntax. Those familiar with WF will know that ordinarily you create WF programs from code, or using XAML. I like the idea of using a language syntax as yet another means to define workflows... some potential workflow authors will be comfortable neither with code or XAML; the graphical WF designer can help here, but even that requires either Visual Studio, SharePoint Designer (which has a crippled graphical workflow designer, and is SharePoint-specific to boot), or you need to host the WF designer in your own app.

WF is a somewhat under-utilized and misunderstood technology, IMO. There are several reasons for this... confusion with Biztalk, lack of prescriptive guidance on implementing systems using WF, the fact that once you get beyond the graphical designer the sexiness factor goes way down (but that's precisely when a technology like WF starts to shine). It's plumbing, and plumbing isn't very sexy.

Another difficulty with WF adoption is audience. WF is plumbing, and is ideally used to provide higher-order abstractions through which business users can express intent. But those business users can't express their intent without the abstractions in place; waiting for programmers to recognize this deficiency on their own and do something about it is mostly a non-starter. You get what we have today... most higher-order workflow logic is expressed as low-level control flow in languages like C# or Java. Enter TextFlow.

TextFlow is (hopefully) one example of these higher-order abstractions. It's intended to be more approachable and business-user-friendly than writing .NET code (certainly) or interacting with the workflow designer (hopefully). It's also intended to be more powerful and general-purpose than the SharePoint Designer.

As I've said, the key characteristic of TextFlow is that it compiles down to a WF program, instead of a "raw" CLR assembly. What this means is that, when a TextFlow program is compiled, the output is an in-memory .NET object graph that represents the logical sequence of actions described by the original syntax. This in-memory graph can be executed, serialized into XAML, or even displayed in the WF designer.

So one obvious question is "why would you compile to WF, instead of the CLR itself?" Great question!

Advantages of compiling to WF

1. WF allows you to work at a higher level of abstraction when writing your compiler. For example, control flow is built into WF via a set of pre-canned atomic elements (activities); if your language syntax has looping or branching constructs (as most will), its trivial to use WhileActivity or IfElseActivity (among others) to realize these. To do the same in MSIL is entirely possible, but arguably more difficult and tedious (though admittedly well documented). In effect, the WF "opcodes" (activities) more directly translate to procedural language constructs than do IL opcodes.
2. The CLR is a huge leap forward for language designers and compiler writers because of the set of built-in services; garbage collection, a unified type system, a ubiquitous security model, etc. Platforms can leverage these services instead of writing them over and over again. The WF infrastructure leverages the basic CLR services, but also provides an array of services above and beyond the CLR that are powerful and easily leveraged from your own WF-targeting language:
   • inherent support for transactions and compensation
   • inherent support for long-running workflows, where periods of inactivity (waiting for human input, etc.) result in automatic serialization of the workflow to a persistent store
   • the ability for workflow execution to load-balance across multiple machines, so that a single workflow instance might hop from machine to machine during its lifetime
   • the ability to pause or cancel running workflows
   • the ability to modify the inherent logic of a running workflow
3. The extensibility model of WF is a big win for compiler writers, too. The atomic unit of execution in WF is an Activity; there are many pre-defined Activities shipped with WF, but it's trivial to create your own, too. In essence, if you're writing a compiler that targets WF, you can extend the set of opcodes that your compiler supports! Try that with MSIL! Another great thing about this is that the built-in activities have no special significance to the WF runtime; they don't leverage any "hidden" APIs that are unavailable to yours, etc.

Disadvantages of compiling to WF

1. WF doesn't offer much help if you want to expose a user-extensible type system from your language. As I mentioned above, you can certainly create your own custom Activity types and execute them in WF; but definition of your Activities is done via traditional subclassing of the .NET Activity class. WF has no magical CreateNewActivityType() method to help you here. So, WF is most appropriately targeted by procedural languages where the emphasis is on control flow and sequences of actions, as opposed to more pure OO-style languages (for the record, I think targeting WF with an OO language is possible... you've just got more typing ahead of you :-) ).
2. I hesitate to mention this, because most obsessing over software performance is misguided and misinformed... but it's true that WF will impose some performance tax on your code, relative to execution of equivalent logic directly against the CLR. I have no numbers to quote you (and wouldn't do it even if I did), but it stands to reason that since WF is in essence a virtual machine running on the CLR, there will be some performance penalty, even if small. That said... if you're launching the space shuttle or doing DirectX 3D graphics, you probably shouldn't be running on WF anyway. For true workflow-style programs that interact with external systems and (gasp!) actual users, whatever performance penalty may exist is totally dwarfed by the benefits provided by the platform. If your workflow program sends an email to a user and waits two weeks for a response before then proceeding with execution, who cares if the email was sent in 50ms or 5 seconds???
3. One final thing to note... to truly take advantage of WF by compiling to it from your language, you'll obviously need a firm grasp of the semantics and underpinnings of the WF runtime and ancillary services. And since WF runs atop the CLR, you really need a similar grasp of the basic CLR services, too. So in all fairness, the barrier to entry isn't small; but neither is it insurmountable. In fact, I would venture to say that if you're capable of designing a language and writing a compiler, then grokking WF isn't going to seem like a big deal.

There's more to say, but that's a start. I highly recommend Essential Windows Workflow Foundation by Shukla and Schmidt as the definitive guide to WF and its underpinnings.

And if you're intrigued by any of this, I invite you to take a closer look at TextFlow... let me know what you think!

Sad to see that Gary Gygax has died.

Man, I gave many Sunday afternoons to D&D during my formative years... Casey, Chris, Zoob, Pete, Kessel, and me had some fun times. Then Pete got a girlfriend, and that was the beginning of the end.  :-D

I played a few different characters over the course of several campaigns... but my favorite has always been my first, Bandaar the dwarf fighter. Life was pretty simple for Bandaar... ale good, battle axe good, orcs bad. He wasn't much of a talker, and he certainly wasn't the shiniest gem in the pouch (8 for intelligence, around there for wisdom, IIRC)... but we had good times, me and him.

Here's to you, Gary. Thanks for the memories.

My little side project is finally ready for others to poke around with:

TextFlow - www.codeplex.com/TextFlow

The main idea behind TextFlow is to allowing authoring of WF programs (workflows) via a language syntax, as opposed to (or, eventually, in addition to) using the drag-and-drop designer. The designer is great for many users, but there are times when a terse language syntax would be useful too.

Here's a (mostly) simple Hello World in TextFlow syntax:

SET x = 10

INPARALLEL

START IF x > 100 THEN [HelloWorld] ENDIF END

START IF x < 100 THEN [GoodbyeWorld] ENDIF END

ENDPARALLEL

This is a slightly gratuitous example that executes a WF ParallelActivity with two IfElseActivity children; the square bracket syntax allows execution of configured external activities (in this case, ones that simply display messages of some sort). You get the idea. The actual generated workflow looks like this:

The project includes an authoring tool that let's you toggle between syntax view and workflow view; workflow view is read-only at this point, but I hope to enable bi-directional updating eventually.

The code is a definite work in progress... it's not much more than prototype quality at this point. So don't expect to launch the space shuttle with it anytime soon, but if you have an interest in any of the following, you might find something worth your trouble:

- using ANTLR-generated lexers and parsers in C# (if you've only ever used lex and yacc, you need to see ANTLRWorks)

- hosting the WF designer in a WinForms app (not sure my code should be reference material for this :-) )

- a real-world use for .NET 3.5 expression trees (very cool stuff)

I'll be blogging more on this in the coming weeks, as I continue to work on the code. For further details, visit www.codeplex.com/TextFlow.

Feedback is welcome and appreciated!

Several days ago I mentioned that I'm working on an idea that combines ANTLR and Windows Workflow to demonstrate "business process authoring for analysts".

In hindsight, that's a pretty bombastic thing to say. So instead, let's just say that the tool I'm working on could be used by semi-technical folk to author executable business processes. There are many other fine tools in this space; mine incorporates a few (IMHO) clever ideas, but I'm not breaking any monumental new ground.

I'm getting closer to having something for public consumption... I'm trying to walk that line between making it solid enough for folks to actually use of it, without crossing every T and dotting every I. I'm very eager for feedback and input, but I want a minimal level of utility before it goes out the door.

So if I've piqued your interest at all, stay tuned.

Oh, and by the way... .NET 3.5 expression trees are FANTASTIC.  Except that they don't serialize without writing your own surrogate. Bah.

In the interest of not scaring off employers of both the "current" and "potential future" variety, it's perhaps useful for me to follow up and clarify my comments originally made here.  :-)

In general, I meant what I said... code reuse is usually more trouble than it's worth. Usually, you're better off not trying too hard to make it work, because you'll probably fail, and waste a lot of time. Even if you're smart, and certainly if you're not.

I come by this point of view honestly... I've personally failed more often than not in trying to build "The One Reusable API to Rule Them All". I don't think it's a big secret that most others fail at this, too. I like to think I've gotten better as I've aged... but it's still a tough nut to crack.

IMO, the characteristic most often desired in "software that shall be reusable" is FLEXIBILITY. Truly flexible code that is widely applicable across multiple disparate problem domains is a thing of beauty... and extremely rare. The fact that any such code exists isn't the telling point; the fact that it represents an infinitesimally small percentage of all code ever written is.

Most "flexible" code I've personally witnessed was anything but. I've concluded that flexibility (and the goal of wide-scale reuse) is wildly overestimated as a desirable (and achievable) code characteristic for most classes of software.

So I say all this to say, reusable software is hard. It has instant sex appeal for non-technical manager types (which makes it even more dangerous), which is probably one reason it perpetuates. And certainly, there exist many fine counter-examples in the wild (none of this really applies to SDK vendors, obviously).

But I think the world would be a better place if we software types focused on building for today, and not For All Time To Come. Try it, and use your extra free time to read up on refactoring and test-driven development... there's your flexibility, cowboy.  :-)

I understand that "sorry it's been awhile since I posted on my blog" posts are considered bad form by the blogging elite... oh well.

I started a new job last October, working here. Very interesting company, very interesting work (rules engine technology)... but they were quite eager to throw me immediately into the deep end of the pool, so I haven't had time for much of anything since I started. Just now starting to find some energy for endeavors beyond, like this blog.

I've got an interesting new project idea percolating... I'll be posting more on that very soon. I also hope to have a CodePlex project up soon with some prototype code.

As a teaser, think ANTLR + Windows Workflow = "business process authoring for analysts".

ASP.NET as WF host is an important and useful scenario for WF adoption, but there are some interesting details that are worth understanding before you tackle this yourself.

The first thing to understand is that WF itself imposes no specific threading model; rather, it must be configured to conform to any specific threading requirements of its host process.  This means that WF can be hosted in a single-threaded console or WinForms app, and also can scale up to be hosted inside ASP.NET, with all of its multithreaded goodness (thankfully hidden under the covers from the average web developer).

But there's the rub... ASP.NET uses the .NET ThreadPool to dispatch request processing.  The default WF scheduler also uses the ThreadPool to execute individual workflows; in combination this would be bad, as it results in 2 ThreadPool threads consumed per web request.  What we need is a way to re-use the ASP.NET thread to also execute the workflow.

Enter ManualWorkflowSchedulerService.  As opposed to the DefaultWorkflowSchedulerService, the manual service grants explicit control over which thread is used to execute a given workflow.  Its usage is a bit goofy... there's an extra call to ManualWorkflowSchedulerService.StartWorkflow() that comes after calling Start() on the workflow object itself.  But it is the thread on which StartWorkflow() is invoked that executes the workflow... so, a necessary evil.

Okay, all of this is fine, and frankly already welldocumented.  The purpose of this post is to highlight a few subtleties.

When you create an instance of ManualWorkflowSchedulerService, you must specify whether you're using active timers, or not.  The idea here is that, for workflows that might go idle (as a result of DelayActivity, or any other activity that implements IEventActivity), the manual scheduler itself cannot know when to resume such workflows (as the default scheduler can do).  So you have two choices... choosing "active timers" means a separate thread will spin up, periodically check for expired activity wait timers, and resume workflows with expired timers.  Choosing "no active timers" means you must write extra code to check for expired timers, and manually resume them when necessary.

The primary issue with active timers is that, since two threads are now involved, it's not very difficult to create a race condition where your primary ASP.NET processing thread completes before the active timer thread wakes up your idled workflow and allows it to complete its work.  The resulting problems are implementation-specific, but not good regardless.

The way to solve this problem is naturally to use some sort of thread synchronization mechanism to ensure that the main ASP.NET thread waits on the active timer thread to restart your idled workflow.  See the sample code referenced at the end of this post for an illustration of the issue, and one potential solution using thread events.

An important observation here is that ManualWorkflowSchedulerService, used in this way, implies synchronous execution of workflows.  This is counter to the relative "fire-and-forget"-ness of the default scheduler.

On the other hand, if you don't use active timers with the manual scheduler, it is *your* responsibility to detect expired workflow timers, and to then resume the idled workflows (by calling WorkflowInstance.Resume() ).  This generally means spinning up your own secondary thread (might as well just use an active timer) or periodic polling from your main thread (ick).  Again, see the sample code for an illustration of the issue and one possible resolution.

A final note... given the synchronous nature of ASP.NET request processing (setting aside async ASP.NET handlers for the moment) I can't generally recommend the use of async activities in workflows used inside ASP.NET.  By all means, use WF to define processing logic for HTTP requests inside ASP.NET.  Just tread lightly with the use of DelayActivity, HandleExternalEventActivity, etc. inside such workflows.

Here's the sample code I refer to above, that illustrates what I'm describing here (requires VS.NET 2008 beta 2).

Enjoy!

http://www.infoq.com/news/2007/07/worthless-code

This parallels my general take on code reuse...

Josh's 1st Axiom of Code Reuse

In order to be reusable, code must first be good.  Since most code is not good, most code is not reusable.  This means you.

Josh's 2nd Axiom of Code Reuse

Most of the time, idea reuse is infinitely more beneficial and productive than code reuse.

Josh's 3rd Axiom of Code Reuse

Most of the time, don't bother... it's not worth the trouble.

...and it's incredible.

http://www.amazon.com/Essential-Workflow-Foundation-Microsoft-Development/dp/0321399838/ref=pd_bbs_sr_2/104-2057410-5880735?ie=UTF8&s=books&qid=1183209452&sr=8-2

A few Amazon reviewers poo-poo it for being too esoteric and not dealing with "how to get work done in WF", or something.  This is definitely not a WF cookbook.  But the explanations behind bookmarking and continuations as the underpinnings of the WF architecture are fabulous.  Quite simply, if you don't understand this stuff, you don't understand WF.

The more I dig into this technology, the more fascinated I am.  The service model, the support for arbitrary execution semantics (but lack of preference for any baked-in ones), the multiple abstraction levels at which you can participate, the simplicity and yet power of the bookmark metaphor... it's really excellent stuff.

If you have even a casual interest in "next generation programming models" (as the tagline goes), read this book.

Well now, this was a fine how-do-you-do on a Wednesday morning...

http://footheory.com/blogs/bennie/archive/2007/05/12/simple-application-extensibility-with-wf-rules.aspx

Holy cow!  Turns out the rules engine that ships with WF is usable in your own application, even if you're not using workflows at all!  This is incredibly cool.  A little background for the uninitiated...

Windows Workflow is a technology that ships with .NET 3.0.  It enables you to express program logic and control flow visually, in what looks a bit like a Visio diagram.  It directly enables things like continuations and passivation in your software.  It is, in general, a Very Big Deal... though it usually plays second fiddle to WPF and WCF when folks start discussing .NET 3.0.

One of the features of WF is a business rules engine that can be used to author and execute rules against workflows in your application.  The rules work against internal workflow state, and produce side effects during execution to modify that state in some meaningful way.  For example, your workflow might expose properties such as AccountName or OrderAmount, so a simple rule might be (in pseudo-code):

if this.OrderAmount > 100

then this.Discount = 10%

else this.Discount = 0%

The idea is that you express such rules out-of-band from your code, which provides really nice maintenance and visibility benefits.  Since rules tend to be the core value of a software system, they're best not buried inside layer after layer of C# code where only a programmer can maintain them.

You get an out-of-the-box editor with Intellisense for rule authoring.  In addition, there are standard hooks in WF (through the use of PolicyActivity, typically) for invoking the rules at runtime.  This all works great assuming you're authoring and using workflows in your application.

Okay... so here's the neat part.  You can use this rules engine even if you aren't using workflows.  There's a great sample on the WF community site that demonstrates how to re-host the rules designer in your own application, as well as how to use alternate storage mechanisms for rules (by default, rules are saved to a file in your workflow project, and the file is compiled as a resource).  But, that code is still relying on workflows and custom policy activities to execute rules.  So the code from this sample is used in the other sample found at the link I mentioned at the top of this post to demonstrate rule execution without workflow involved at all.  I also re-purposed the base sample code to whip up my own demo app.

Here's a screenshot of rule authoring against my own "User" type (which has properties like Name and Age):

Notice the full Intellisense support for my custom type... this makes rule editing a breeze.

Once you edit and save your rules (storage is just a single table in the SQL database of your choice), you can then reference the rules (sets of rules, actually) by name at runtime, for execution.  Here's a code snippet:

              //
create my user object...
            

User joe = new User();

joe.Name = "Joe";

joe.Age = 35;

joe.Valid = false;

              //
next several lines execute the ruleset called "User2"...
            

RuleSetService svc = new RuleSetService();

RuleSet ruleset = svc.GetRuleSet( new RuleSetInfo( "User2" )
);

RuleExecution exec

    = new RuleExecution( new RuleValidation(
joe.GetType(), null ), joe );

ruleset.Execute( exec );

              //
okay, ruleset has executed, let's see what the outcome is...
            

Console.WriteLine( joe.Valid.ToString() );

And that's it... about 4 lines of code to invoke a ruleset at runtime.  And again... no workflow in sight.  Not that I have anything against workflows, it's just nice to know you can use the rules engine by itself if/when needed.

I've uploaded my own sample application here.  Grab it, unzip, restore the database backup, check your config file connection strings, and party on.  There are 5 VS.NET 2005 projects included... 4 of them are modified versions of the code found in the previously mentioned WF community site sample, the other (called "Tester") is my own sample that exercises some rules I created through the designer.  So just to be clear... most of this is not my own code.  But that also speaks to the beauty of this approach... most of the heavy lifting is done for us already, so it's easy to get up and running.

Enjoy!