protobuf-net cannot serialize everything you throw at it. It’s picky about what it does because it wants to be fast. If it were to accommodate every type of object it would have to sacrifice speed. The author did however create a hook so that things it can’t serialize can be turned into something it can serialize. This is done with what it calls Surrogates. As you can tell from the name, we tell protobuf-net that for a given type (that it can’t serialize) we want to use a surrogate type (that it can serialize).

Take this class for example:

MyNameValueInfo can’t be serialized because it doesn’t know how to serialize the Value property (typeof object). It will throw an exception: “No Serializer defined for type: System.Object”

To get around this we need to provide a surrogate for MyNameValueInfo that protobuf-net can serialize. First register the surrogate type (only needs to be done once):

RuntimeTypeModel.Default.Add(typeof(MyNameValueInfo), false).SetSurrogate(typeof(MyNameValueInfoSurrogate));

Then implement MyNameValueInfoSurrogate so that it can be transformed from/to MyNameValueInfo and is serializable by protobuf-net:

WARNING
Doing binary serialization like this will include Type information in the serialized byte array. This is only useful if the receiving system is also in .NET. For a more universal approach you could use JSON serialization.

Bootstrap is a handy tool and I use it a lot. I decided to use it with a WordPress plugin I am developing but when I included bootstrap’s css file in my plugin page it blew up the wordpress admin panel’s design. Thus started my journey of hacks to get it working. Here is how it’s done…

This is what my plugin folder looks like:

  • Stylesheets (and a less css file as you will soon find out) live in the “css” folder.
  • Bootstrap fonts in the “fonts” folder.
  • Javascript in the “scripts” folder.

In your plugin file/installer/whatever you probably have a line which was loading the bootstrap css in the html head element…

wp_enqueue_style('admin_css_bootstrap', plugins_url('/myplugin/css/bootstrap.min.css'), false, '1.0.0', 'all');

Get rid of that. You can load the bootstrap javascript file this way but for the stylesheet we begin the hacks…

Create a script called bootstrap-hack.js and load it with your plugin.

wp_enqueue_script('admin_js_bootstrap_hack', plugins_url('/myplugin/scripts/bootstrap-hack.js'), false, '1.0.0', false);

The content of that file is this:

As you can see first we are dynamically adding a .less (LESS CSS) file. Next we are loading the LESS Javascript to transform the .less file. Then we are loading any stylesheets that may override bootstrap styles. This is the main part of the hack but there is a little more to it as I will explain.

The content of bootstrap-wrapper.less is this:

What this does is load the bootstrap css file as LESS and then outputs it with all of the styles wrapped with the “.bootstrap-wrapper” class. This means you have to add a div that wraps your content so that the bootstrap styles will be available to it. It will look something like this:

Now back to bootstrap-hack.js… It’s loading less.js file so download and include it in your scripts folder.

Make sure you load any stylesheets that override bootstrap styles in the same way bootstrap’s CSS is loaded afterwards. You don’t have load the stylesheet using less – we just did that because we needed to wrap bootstrap’s styles with another class so that they won’t conflict with wordpress’ styles. Don’t forget that your overrides must be prefixed with .bootstrap-wrapper now as well.

I was working on RulePlex this week and came across a couple of things I wanted to share. First is a change in the way that rules are “compiled”. You can’t really compile JavaScript per se but this is the process of how things came about working the way they do…

In the first iteration, JavaScript rules were executed individually. If there were 100 rules in a Policy I would execute 100 little JavaScript snippets for an incoming request. The snippets were “compiled” when the call to the API was made. I soon realized that this might be okay if a Policy had a few rules but for large Policies it was slow – even if I executed all of the snippets in parallel.

For the next iteration I took all of the rules and compiled them into one big script. In order to do this I had to wrap the rules with some advanced JavaScript techniques. Because this big script contains the results of every rule I had to append something unique to each result’s variable name – the rule’s Id. This makes the script look horrifying but I am okay with it for now (it’s not hurting performance). Executing one big script increased performance tremendously. Here is an example of what a Policy with 1 rule that simply returns true looks like compiled:

At the same time I had extended this technique to the C# rule engine. I took it a step further though and actually compiled C# rules into a DLL. I took the binary code for the DLL and stored it along with the Policy. I did the compilation whenever a rule in the Policy changed – not when the API was called like I had been doing with the JavaScript engine. When the API was called, I got the DLL’s binary data and loaded it into memory to be executed against the incoming data.

I mimicked the binary compilation and loading of the C# rules into the JavaScript engine as well. The thing is, I never really liked doing it in the JavaScript engine because I had to convert the compiled script (text) to binary, so I could store it in the same database field, and then from binary back to text when it was time to be executed. In C# it made sense but not in JavaScript. Now that the C# engine is gone I had a chance to go back and change this.

Presently, when rules are changed (or added/deleted/etc), RulePlex will compile it’s big script during the save of the rule. I saves it to the Policy as text. When the API is called the script is retrieved and executed.

I haven’t thought about tweaking this process any more but I may in the future. Instead I have been thinking about how this affects the workflow from a business perspective. The more I think about it the more I like the changes that I’ve made. If I ever change how I “compile” the big script it won’t affect policies that are currently working (a certain way). What if I’ve got a bug in the script that you’ve accounted for in you’re rules, knowingly or unknowingly. If I change how the script is compiled and it’s being compiled during the API request then it could be different day-by-day without any actions by you. This is bad because I may have fixed or introduced a bug that changes the results. Now the application you’ve integrated with RulePlex is broken!

The ideal workflow is that there are two copies of the same Policy, maybe even three, or N. One copy would be designated as a Production copy, while the others are for Dev/Staging/whatever. When the engine changes, you want to test those changes in a non-Production environment first. When you’ve verified that the changes do not affect your application then that non-Production copy can be promoted to Production. This also applies to the workflow of building out a Policy too, not just back-end changes to the engine. The concept of environments will be included in the next version of RulePlex.

If you wanted to launch a REST based API today, what technology would you use? For most the answer would be Web API. Have you ever thought about what Web API consists of though? I mean, do you know how much code a request has to go through until it reaches your controller method?

While designing an API recently I forwent Web API and tried to get as low-level as possible in hopes that my service would be faster. I read an interesting tutorial on how this could be done in Azure using OWIN self hosting capabilities, but, for no good reason I am not a fan of OWIN. I get the sense that there is still a lot of other people’s code between the incoming request and my code. In my quest to get as low-level as possible I stumbled upon the HttpListener class which is essentially a wrapper for HTTP.sys. Surely this is as low-level as I can get without getting too carried away.

So, which method out of these three will serve HTTP requests the fastest: Web API, OWIN Self Host, or HttpListener? My hypothesis is that HttpListener will be because it the most low-level. The tests for each method will consist of returning the current date and time. There will be no input (no post data or query string) and the result will be returned serialized as JSON. JSON.NET will be used for serialization in each of the projects for consistency. You can get faster performance by using Jil but we’ll leave it alone for this run. I want the out-of-the-box Web API project to be the baseline because that’s what most people are using. The Web API project will be hosted in IIS while the others will be hosted by a Console app. A fourth project will be created which makes 1000 HTTP requests to each host and records the results. The requests will be made from the same machine the servers are running on to eliminate network latency.

Here is my solution with source code for the servers and tester if you’d like to try it for yourself – WebServerComparison.zip

Here are my results: (all times are in seconds)

Web API OWIN HttpListener
Run 1 0.8059442 0.6924348 0.2742231
Run 2 0.6600578 0.3289284 0.1906594
Run 3 0.640202 0.3297216 0.1872897
Run 4 0.6189885 0.3406656 0.1953822
Run 5 0.6118996 0.3280714 0.1898794
Avg 0.66741842 0.40396436 0.20748676

It looks like the first run primed our servers since it took considerably longer to complete compared to the ensuing runs. I won’t throw this run out because in the real world you will have to prime your servers for various reasons. It doesn’t affect our outcome either. My hypothesis was correct in that HttpListener was the fastest option. Keep in mind that the difference between HttpListener and Web API/IIS is less than half a microsecond per request but it is a difference nonetheless. I did not show the raw responses but the Web API responses were larger in size because IIS tacks on a couple of headers. This would have made a greater difference if we weren’t making request from the same machine.

As with anything there are some trade-offs. With IIS you get a lot of management features that you would never get by running your own web server. It also has a lot more security and is more robust. It will log requests and handle errors for you. Writing your own web server will give you faster responses but you’ll have to spend time solving problems that IIS has already solved. The trade-off is yours to decide upon. In the case of RulePlex or other extreme performance needy services I think it’s better to go with the faster option.

The OWN self hosting option is neither the fastest nor does it give you any management features. It does mean you can setup your server in more of a Web API way and gives you some added security, but, I don’t think this middle-of-the-road option is worth much. You either want the performance or you want the management. Right?

Other notes

If you have an API that is used by your web app’s front-end via ajax requests and both are on the same domain you should pay attention to the cookies being sent in the request. If possible host the API on a different domain to avoid the cookies from being sent with the request.

Compression may also play a factor in larger requests. My next post will explore compression options.

In my previous post I wrote that one of the decisions I made about RulePlex was to only support one rule language. This will make the engine intrinsically faster and I’ll show you why.

When you create a Policy (a set of rules) one of the options you are given is to chose a rule language. When it’s time to run your rules you provide the data and the Name or Id of the policy. From that Name/Id the Policy is found which will tell us the rule language. Once we know the language we can begin processing the rules.

At first you might think of writing code for this that fits into a switch pattern. I might look something like this:

Some things to note here – I am using enums and I left out the implementation of how the rules are processed. That’s because I just want to talk about the switch pattern. It’s not the cleanest way of writing this. My example only has 3 cases but the worst switch I’ve ever seen had over 80 cases, used strings for the case values, and had an average of 12 lines per case – over 1000 lines in total! This is an amplification of my point which makes it very clear, that, switches can become unreadable and hard to maintain. Whenever you have a switch statement you should ask yourself if using a Strategy pattern makes more sense.

In RulePlex I used the Strategy Pattern, something to the effect of this:

The downside to this pattern is that a dictionary lookup is slower than a switch statement. The upside is that it cleans up code very nicely. I could have added support for 10 new rule languages and the lookup code would have stayed the same. It’s up to you to decide between these trade-offs. My original goal was to support as many languages as possible in RulePlex so using the Strategy Pattern would have saved me headache down the road.

That all changes now that RulePlex is only using JavaScript rules. I don’t need the strategy pattern and I don’t even need a switch statement. Instead I can new-up a JavaScriptRuleEngine and call Execute() on it. Faster and cleaner!

On a side note (back to my comment on using enums): You should never use “magic strings”. Your world will be a much better place without them.

Almost a year ago I began on a mission to create the first cloud-based rules engine. I called it RulePlex. After a few months of work I had succeeded. A lot of rules engines can be run (on virtual machines) in the cloud but mine was designed with a cloud-first approach in mind. On top of being cloud-based I had a couple of other goals I wanted for the engine:

The Wright brothers first flight.

The Wright brothers first flight.

  • Allow rules to be written in any language
  • Allow integration with any app
  • Make it incredibly secure
  • Keep the cost low

I accomplished all of these. The engine was awesome. When I was at a good place to ease up on the development, I did, and started to connect with as many people as I could trying to drum up business. I contacted all of my developer buddies, business contacts, previous employers… but no one was interested. I tried to generate business by presenting my software at business start-up events like 1MillionCups. I tried Google Adwords. I threw a hail mary to Scott Hanselman hoping he would do a hanselminutes with me. I even gave the service away for free just to get signups so people would look at it… but all of my attempts at getting the business going failed. I don’t think I did enough even though it may sound like I had.

I’m not giving up.

I’m changing things up!

Instead of being cloud-based I am falling in line with all of the other rule engines. RulePlex will be an on-premise install. It can still be run in the cloud but it won’t be touted as the first cloud-based rules engine any more. It’s nice that I was able to accomplish that feat but ultimately I think proximity to where it will be used will benefit companies more. The latency between the cloud an a corporate network is too much when apps need instant results.

Another thing I am changing is the ability to write rules in any language. I had support for JavaScript, C#, VB.NET, Ruby, and Python – but from now on rules will only be written in JavaScript. By going with only one language it will save development time tremendously. I chose JavaScript because it is the most widely used language today. It’s also not going anywhere anytime soon. It has become an incredible server side language thanks to V8 and the popularity of Node.js. It’s flexible/forgiving thanks to dynamic typing and business users can learn the basics easily.

The last thing I will be doing is blogging a lot more about RulePlex. I’ve been working on it for almost a year and I haven’t written anything (directly) about it. I want people to follow along with it’s progress and see how it is evolving and why things work the way they do. I want you to see what is influencing me into making design choices. Hopefully this will transform into a community of RulePlex users but I won’t get too ahead of myself. Let’s just start with clicking publish on this post.

I’ve been working on a project that saves and reads blobs from Azure. I created a unit test for this functionality. Instead of making calls to Azure proper, I am running the Azure Storage Emulator so that I don’t incur any cost. “It works on my machine!” …but when I check my code into TFS and do a build on the server, the unit test fails because the Storage Emulator is not running. If I remote desktop into the server and start the Storage Emulator then the unit test passes. Starting the emulator manually is a problem because if the server reboots then I have to remember to start that up again and I won’t know about it until the test fails. Other developers might not know to do this either.

To combat this problem I tried starting the emulator with a scheduled task that runs when the server starts. This did not work. I’m not sure why but it just didn’t. Task Scheduler says the task is running but I don’t see it in Task Manager and my unit test fails. I can only assume that the Task Manager status is wrong and something that it doesn’t know about isn’t working. It would be nice if Microsoft created the emulator so that it runs as a Windows Service instead.

I Googled for a solution and came up empty except for one article that mentioned starting the emulator from within code. In order to detect if the emulator is running or not, first you need to access a storage container. If you get a StorageException then the emulator isn’t running and you know to start it. This seems like a hacky solution but I tried it and it works. Here is what I ended up with:

The first line of that method checks a config setting that will let me turn the hack on/off. You won’t need it when you move to production because Azure is always on.

One thing to note is that your build controller must be using a user account and not a built-in account. This is because the storage emulator stores it’s settings in the user’s directory. I created a local administrator account on my build server to run the build controller as.

I am a big fan of Google Fonts. There are some really nice fonts on there. While easy to use on the web, there is no easy way to install fonts on your computer to use in Word or whatever app you’re using locally.

I take that back – there is an “easy” way. It’s called SkyFonts. It’s a third party app (not Google) that will install all of the Google Fonts for you… but there are some drawbacks. Their motivation is money. They want you to use their service to discover OTHER fonts and PAY for them. Their app also runs on your computer 24/7. Isn’t that silly? Just install the fonts and move along. So if you’re like me you’ll say NO THANKS to SkyFonts.

So without SkyFonts there is no other option to easily install all of the Google Fonts locally… Until now! I’ve written a small app to download all the fonts and install them. Half of the credit goes to this guy on github called w0ng. My app automates the download of his zip file and does the installing.

.NET Framework 4.5 is required to run the app (comes with most Windows PCs)

DOWNLOAD: google_font_installer_1.0.zip

I setup a brand new TFS 2013 server and was in the process of creating my first team project when I received this error:

TF54000: Cannot update data because the server clock may have been set incorrectly. Contact your Team Foundation Server Administrator.

I had changed the time zone on the server after installing TFS so I figured this error was because of that. In TFS there is a table called tbl_Changeset. If you check it there is a CreationDate column which is the date at which you checked in something to TFS. When installing TFS it creates the first changeset automatically. The date for that first record was now 3 hours after the current time. I couldn’t create a team project because changeset 2 would have occurred before changeset 1. The changeset numbers and CreationDate both need to be chronologically in the same order for things to work.

To fix this I ran an update statement to set the first record back a day (it could have been 3 hours but 1 day works just as well). This is the query I used:

UPDATE tbl_Changeset SET CreationDate=CreationDate-1 WHERE ChangeSetId=1

I went back to Visual Studio and was then able to create my team project without issues.

I use text editors a lot. I use them for taking notes, writing code, writing long emails before I send them out, and much more. For a long time I’ve used Notepad++ because it can handle all kinds of text-based files. Quite often the files I create in Notepad++ I also encrypt. I use a plugin called SecurePad for this.

Lately I have been using MarkdownPad to replace a lot of the things I do in Notepad++. Markdown (the syntax) is great because it gives you guidelines on how your text should be formatted. It’s used all over the place: reddit, github, and stackoverflow to name a few. Because of the guidelines markdown provides it can be easily transformed into HTML and rendered in an eye-appealing fashion. MarkdownPad splits it’s window into 2 panes: the left pane is a text editor and the right pane is the HTML formatted view. MarkdownPad is very basic as is markdown in general – because it was intended to be simplistic and only used for basic formatting of text. MarkdownPad has no way of encrypting files.

So I got to thinking… Why is MarkdownPad a Windows application? It’s so basic that it could easily be browser based. It would also be nice if it encrypted files. In today’s landscape you can never be too careful when it comes to protecting your personal data, thoughts, and daily habits. I also thought, if you could make a browser based markdown editor where would you open and save files from? …the cloud!

After doing a quick search on Google I couldn’t find anything that remotely resembled the online tool I wanted, so I made it myself.

It’s called Secure Markdownhttps://securemarkdown.azurewebsites.net

It’s a work in progress but the basics are there:

  • Text editor (left pane)
  • Markdown syntax is shown in HTML format as you type (right pane)
  • Files are read and written to Dropbox
  • Files are encrypted (in the browser) before being saved to Dropbox

In the future I would like to add:

  • Buttons for markdown-formatting text, like MarkdownPad.
  • Option for saving unencrypted.
  • Overwriting files (when you save a file with the same name as an existing file in dropbox now it appends a number to the file name)
  • Seperate save / save as options.
  • Option for viewing HTML source and downloading the HTML version.
  • Option for saving HTML view as a PDF
  • Move to own domain name if warranted

If you have any suggestions please leave a comment.