November 2018
Volume 33 Number 11
.NET - Create Your Own Script Language with Symbolic Delegates
By Thomas Hansen | November 2018
I love my C# compiler. My girlfriend claims I love it more than I love her, although, for obvious reasons, I’d never admit to that in public. Strong typing, generics, LINQ, garbage collection, CLI, the list of compelling traits goes on. However, every now and then I need a couple of days off from the comfortable securities with which my compiler normally provides me. For those days, I choose to code in dynamic programming languages—and the more dynamic the language, the more dangerous and fun.
OK. Enough introduction. Now let’s do what real programmers do when left alone—let’s create code:
hello world
What? You thought I said “code”? Yup, I certainly did, and that text actually is code. To understand, let’s see what you can do with this code. Take a look at Figure 1, which shows a minimalistic example of what I refer to as symbolic delegates. Create an empty console app in Visual Studio and type in the code from Figure 1.
Figure 1 A Minimalistic Example of Symbolic Delegates
using System;
using System.Collections.Generic;
class MainClass
{
public static void Main(string[] args)
{
// The "programming language"
var keywords = new Dictionary<string, Action> {
// The "hello" keyword
["hello"] = () => {
Console.WriteLine("hello was invoked");
},
// The "world" keyword
["world"] = () => {
Console.Write("world was invoked");
}
};
// The "code"
string code = "hello world";
// "Tokenising" the code
var tokens = code.Split(' ');
// Evaluating the tokens
foreach (var token in tokens) {
keywords[token]();
}
}
}
In just 25 lines of code I’ve created my own micro programming language. To understand its advantages, realize that the “code” variable can be sent over the network; it can be fetched from a database; it can be stored in a file; and I can dynamically change the string from “hello world” to “world hello,” or to “hello hello world world,” for that matter, and completely change the result to which it evaluates. This ability means I can dynamically combine delegates to end up with a list of function objects, sequentially evaluated, according to the code’s content. All of a sudden I’ve turned a statically compiled programming language into a dynamic scripting language, simply by splitting a sentence into its words, and using the individual words as lookup keys into a dictionary. The dictionary in Figure 1, therefore, becomes a “programming language.” In effect, it’s a symbolic delegate.
This example is obviously not that interesting and is only meant to illustrate the core idea. I’m going to create something slightly more intriguing by chasing these ideas a little bit further down the rabbit hole, as shown in Figure 2.
Figure 2 Dynamically Chaining Delegates
using System;
using System.Linq;
using System.Collections.Generic;
public class Chain<T> : List<Func<T, T>>
{
public T Evaluate(T input)
{
foreach (var ix in this)
{
input = ix(input);
}
return input;
}
}
class MainClass
{
public static void Main(string[] args)
{
var keywords = new Dictionary<string, Func<string, string>>
{
["capitalize"] = (input) =>
{
return input.Replace("e", "EE");
},
["replace"] = (input) =>
{
return input.Replace("o", "0");
}
};
string code = "capitalize replace";
var tokens = code.Split(' ');
var chain = new Chain<string>();
chain.AddRange(tokens.Select(ix => keywords[ix]));
var result = chain.Evaluate("join the revolution, capitalize and replace");
Console.WriteLine(result);
}
}
Now I have something that looks almost useful—the ability to dynamically chain together delegates, which results in a lambda object of loosely coupled functions. Thus the code is declaring a chain of functions that transforms an object sequentially, according to the symbols I choose to put into my code. For the record, you shouldn’t normally inherit from List, but to keep the example short, I decided to do it to illustrate the main idea.
Expanding upon this idea is simple. The exercise is to find the smallest common denominator for a generic delegate that can describe any programming construct you know, which just so happens to be the following delegate:
delegate object Function(List<object> arguments);
This delegate can represent nearly every programming structure ever invented. Everything in computing can take a list of input arguments and return something back to the caller. This delegate is the very definition of input/output fundamental to all computing ideas, and becomes an atomic programming structure that you can use to solve all your computing problems.
Meet Lizzie
As I wrote this article, I created a programming language embodying the preceding idea. I wrote the entire language—which I call Lizzie—after my girlfriend, Lisbeth—in a couple of all out, full-moon weekends. The language is entirely contained in a single assembly, roughly 2,000 lines of code. When compiled, it’s only 45KB on my disc, and its “compiler” is only 300 lines of C# code. Lizzie is also easily extendable and lets anyone add their own “keywords” to it, allowing you to easily create your own domain-specific language (DSL). One use case for such a language is a rule-based engine, where you need to tie together code more dynamically than C# allows. With Lizzie you can add dynamic script code to your statically compiled C# application that’s Turing-complete snippets of functionality. Lizzie is to C# what spice is to your dinner. You don’t want to eat only spice, but if you add some spice to your steak, your experience obviously becomes more pleasant. To try Lizzie out, create an empty console application in C#, add Lizzie as a NuGet package, and use the code in Figure 3.
Figure 3 Creating a Domain-Specific Language
using System;
using lizzie;
class Demo1
{
[Bind(Name = "write")]
object write(Binder<Demo1> binder, Arguments arguments)
{
Console.WriteLine(arguments.Get(0));
return null;
}
}
class MainClass
{
public static void Main(string[] args)
{
var code = "write('Hello World')";
var lambda = LambdaCompiler.Compile(new Demo1(), code);
lambda();
}
}
In just 22 lines of code I’ve arguably created my own DSL and added my own domain-specific keyword to the language.
The main feature of Lizzie is that you can bind your Lizzie code to a context type. The LambdaCompiler.Compile method in Figure 3 is, in fact, a generic method, but its type argument is automatically inferred by its first argument. Internally, Lizzie will create a binder that it binds to your type, making all methods with the Bind attribute available to you from your Lizzie code. When your Lizzie code is evaluated, it has an extra keyword called “write.” You can bind any method to your Lizzie code as long as it has the correct signature. And you can bind your Lizzie code to any type.
Lizzie has several default keywords, which it makes available to you for your own code, but you don’t have to use these if you don’t want to. Figure 4 shows a more complete example that uses some of these keywords.
Figure 4 Using Some Default Keywords
using System;
using lizzie;
class Demo1
{
[Bind(Name = "write")]
object write(Binder<Demo1> binder, Arguments arguments)
{
Console.WriteLine(arguments.Get(0));
return null;
}
}
class MainClass
{
public static void Main(string[] args)
{
var code = @"
// Creating a function
var(@my-function, function({
+('The answer is ', +(input1, input2))
}, @input1, @input2))
// Evaluating the function
var(@result, my-function(21,2))
// Writing out the result on the console
write(result)
";
var lambda = LambdaCompiler.Compile(new Demo1(), code);
lambda();
}
}
The Lizzie code in Figure 4 first creates a function called “my-function,” then it invokes that function with two integer arguments. Finally, it writes out the result of the function invocation to the console. With 21 lines of C# code and eight lines of Lizzie code, I’ve evaluated a piece of dynamic code that creates a function in a dynamic scripting language, from within my C# code, while adding a new keyword to the scripting language I’m using. It took just 33 lines of code in total, including comments. These 33 lines of code allow you to claim you’ve created your own programming language. Anders Hejlsberg, move over rover, and let little Jimmy take over …
Is Lizzie a “Real” Programming Language?
To answer that question, you need to think about what you consider a real programming language to be. Lizzie is Turing-complete and, at least in theory, allows you to solve every computing problem you might imagine. So, according to the formal definition of what constitutes a “real” programming language, it’s certainly as real as any other programming language. On the other hand, it’s neither interpreted nor compiled, because every function invocation is simply a dictionary lookup. Moreover, it features only a handful of constructs, and everything is centered around the “function(arguments)” syntax. In fact, even if statements follow the function syntax previously defined in the generic delegate:
// Creates a function taking one argument
var(@foo, function({
// Checking the value of the argument
if(eq(input, 'Thomas'), {
write('Welcome home boss!')
}, {
write('Welcome stranger!')
}) // End of if
}, @input)) // End of function
// Invoking the function
foo('John Doe')
The syntax of if is as follows:
if(condition, {lambda-true}, [optional] {lambda-else})
The first argument to the “if” keyword is a condition. The second argument is a lambda block that’s evaluated if the condition yields non-null (true). The third argument is an optional lambda block that’s evaluated if the condition yields null (false). So the “if” keyword is in fact a function to which you can supply a lambda argument to, using the “{ … code … }” syntax to declare your lambda. This might feel slightly weird in the beginning, because everything happens in between the opening and closing parentheses of your keywords, unlike other programming languages that use a more traditional syntax. However, in order to create a programming language compiler in 300 lines of code, some bold decisions simply had to be made. And Lizzie is all about simplicity.
Lizzie’s functions are surprisingly similar to the structure of an s-expression from Lisp, though without the weird Polish notation. Because an s-expression can describe anything, and Lizzie’s functions are simply s-expressions with the symbol (first argument) outside of its parentheses, you can describe anything with Lizzie. This arguably turns Lizzie into a dynamic implementation of Lisp for the Common Language Runtime (CLR), with a more intuitive syntax for C#/JavaScript developers. It allows you to add dynamic code on top of your statically compiled C#, without having to read thousands of pages of documentation to learn a new programming language. In fact, the entire documentation for Lizzie is only 12 pages of text, which means a software developer can literally learn Lizzie in about 20 minutes.
Lizzie—JSON for Code
One of my favorite features of Lizzie is its lack of features. Let me illustrate this with a partial list of what Lizzie can’t do. Lizzie can’t:
- read or write from your file system
- execute SQL queries
- ask you for your password
- change the state of your computer at all
In fact, a piece of Lizzie code out of the box can’t be malicious, not even in theory! This lack of features gives Lizzie some unique abilities that Roslyn and C# scripting don’t provide.
In its original state, Lizzie is completely safe, allowing you to securely transmit code over a network, from one computer to another computer, the same way you’d use JSON to transmit data. Then at your endpoint that accepts Lizzie code, you’d have to explicitly implement support for whatever functions you need your Lizzie code to have access to, in order to make it work for your use case. This might include a C# method that reads data from a SQL database or the ability to update data in a SQL database or to read or write files. However, all of these function invocations can be delayed until you’ve made sure that the code trying to do whatever it’s trying to do is actually allowed to do it. Hence, you can easily implement authentication and authorization before you allow it to, for example “insert sql,” “read file” or anything else.
This property of Lizzie allows you to create a generic HTTP REST endpoint where the client layer sends Lizzie code to your server and where it’s then evaluated. You can then have your server create a JSON response that it sends back to your client. And more interestingly, you can implement this securely. You can implement a single HTTP REST endpoint that accepts only POST requests containing Lizzie code, and literally replace your entire back end with a 100 percent dynamic and generic Lizzie evaluator. This allows you to move your entire business logic and data access layer into your front-end code, such that your front-end code dynamically creates Lizzie code that it transmits to the server for evaluation. And you can do this securely, assuming you authenticate and authorize your clients before you allow them to evaluate your Lizzie code.
Basically, your entire application is, all of a sudden, easily built in JavaScript or TypeScript or ObjectiveC or whatever, and you can build clients in whatever programming language you like that dynamically creates Lizzie code and send this code to your server.
Lessons from Einstein
When Albert Einstein wrote down his famous equation to explain our universe, it had only three simple components: energy, mass and the speed of light squared. That equation could be easily understood by any 14 year old with a decent grasp of math. Understanding computer programming, on the other hand, requires thousands of pages and millions if not trillions of words, acronyms, tokens, and symbols, plus an entire WikiPedia section on paradigms, numerous design patterns, and a multitude of languages, each with completely different structures and ideas, all of which require “crucial” frameworks and libraries you need to add to your “solution” before you can start working on your domain problem. And you’re expected to know all these technologies by heart before you can start referring to yourself as an intermediate software developer. Am I the only one who sees the problem here?
Lizzie is not a magic bullet, and neither are symbolic delegates, but they’re definitely a step in the direction of “20 GOTO 10.” And sometimes, in order to move forward, you need to start by taking one step back. Sometimes you need to neutrally observe yourself from the outside. If we, as a professional community, do that, we just might realize that the cure for our current madness is simplicity, and not 50 more design patterns, 15 new query languages, 100 new language features, and a million new libraries and frameworks, each with a trillion moving parts.
Less is more, always, so give me more less, and less more! If you want less, join me at github.com/polterguy/lizzie. That’s where you’ll find Lizzie, with zero type safety, no keywords, no operators, no OOP, and definitely not as much as a single library or framework in sight.
Wrapping Up
Most of the computing industry tends to disagree with most of my ideas. If you asked the average software architect what he thinks about these ideas, he’d probably throw entire libraries in your face as sources to prove how wrong I am. For instance, the prevalent assumption in software development is that strong typing is good and weak typing is bad. For me, however, simplicity is the only game in town, even when it requires throwing strong typing out the window. Keep in mind, though, that ideas such as Lizzie are intended to “spice” up your existing statically typed C# code, not replace it. Even if you never use the coding ideas presented in this article directly, understanding the key concepts may help you write standard code in a traditional programming language more effectively, and help you work toward the goal of simplicity.
A Programming History Lesson
Back when I was a junior developer, I used to create 3-tier applications. The idea was to separate the data layers from the business logic layers and the UI layers. The problem is that as front-end frameworks grow in complexity, you’re forced to create a 6-tier application architecture. First you need to create a 3-tier server-side architecture, then a 3-tier client-side architecture. And, as if that weren’t enough, you then have to port your code to Java, ObjectiveC or whatever to support all possible clients out there. Sorry to be blunt here, but this type of architecture is what I call “insanity-driven design” because it grows the complexity of apps to the point where they’re often almost impossible to maintain. A single change in the front-end UI often propagates through 15 layers of architecture and four different programming languages, and forces you to make changes in all those layers just to add a simple column to a data grid in the front end. Lizzie solves this by allowing your front end to send code to your back end, which your back end evaluates and returns to your client as JSON. Sure, you lose type safety, but when type safety comes at the cost of having to tie together the different layers of your applications, such that changes in one place propagate to all other layers in your project, the cost of type safety is simply not worth paying.
I started coding when I was 8 years old, on an Oric 1 in 1982. I clearly remember the first computer program I wrote. It went like this:
10 PRINT "THOMAS IS COOL"
20 GOTO 10
If an 8-year-old kid today wants to reproduce that experience, following all best practices, using a client-side framework such as Angular and a back-end framework such as .NET, this kid would probably need to know thousands of pages of technical computer science literature by heart. In contrast, I started out with a book that was roughly 300 pages, and a handful of computer magazines. Before I made it to page 100, I had created my own computer game, at the age of 8. Sorry if this makes me sound old, but this is not evolution and improvement, this is “devolution” and madness. And, yes, before you start frenetically writing down your objections, this problem has been scientifically researched, and neutrally observed and confirmed, by professors and Ph.D.s, all much smarter than me.
The fact is that computer programming as a (human) profession is at the brink of extinction, because the complexity that’s continually being added may soon reach the point where it surpasses the human brain’s capacity to create computer programs. Because programming is becoming so demanding from a cognitive perspective, it may be that no human being will be able to do it 10 years down the road. At the same time, humans are becoming ever more dependent on computers and software every single day. Call me old fashioned here, but I kind of like the idea that a human being somewhere out there is able to understand the things that are crucial to my happiness, existence and quality of life.
Thomas Hansen has been creating software since he was 8 years old, when he started writing code using the Oric-1 computer in 1982. He refers to himself as a Zen computer programmer who seeks to reduce the complexity of modern programming and refuses to proclaim any belief in technological dogmas. Hansen works for Bright Code in Cyprus where he creates FinTech software.
Thanks to the following Microsoft technical expert for reviewing this article: James McCaffrey