Sunday, July 27, 2014

How I rolled my own simple (DSL) Domain specific Language and you can too!



FYI I'm talking about this....

Preamble

So a project required me to take the following statement
"C#" or "Vb.net" and Not "Java"
and create a filter based on it that equates to the following logic.
if((file.Contains("C#")||file.Contains("Vb.net"))&&(!file.Contains("Java")))
{
  SearchResult.Add(file);
}
This is what we in the biz refer to as a Mini-Language. Essentially an especially small and (hopefully) easy-to-understand programming language that we use for a very specific purpose, a type of Domain Specific Language(DSL for short). To achieve this we turn to a process called Lexical analysis.
  1. Breaking down the expression into manageable parts. Referred to as the "Scanner" or "Tokenizer"; This is the first half of the lexical analysis process.
  2. Transforming those parts into a form can be evaluated by the system that needs to be interacted with a.k.a. a "Processed Value". (be it db access, programming logic, a modeling schema whatever) Referred to as the "Evaluator" or "Parser" This is the second half of the lexical analysis process.


Why This? Why Now?

There are numerous ways that this can be done (and I was back and forth on a couple) and this is not the first time I've had the pleasure of working through them however, its not terribly often so I'm writing this as a means of solidifying it into my memory or at least having something readily available so I don't "Forget to remember" how to do this sort of thing.


The Plan


Stuff You should know

The programming logic doesn't contain anything that is overtly language specific(Though I do use linq at spots for filtering out results. This can just amount to looping through a series of value the filtered action based on a specific set of criteria). The most specialized tool in my arsenal for this project is Regular Expressions (an exceptionally powerful string search and evaluation DSL that IMO you should really have some familiarity with as a programmer as they have totally saved my ass a ton of extra time and work in many cases). If your existing language does not support them or something equally powerful you are gonna have a bad time.


What will it look like?

Hacking away at this in Linqpad the following represents the way this would be envisioned So that is our path to paradise.

The Tokenizer AKA Scanner's job is breaking down string expressions into manageable parts or "Tokens".

The Tokenizer AKA Scanner's  job is breaking down string expressions into manageable parts or "Tokens".





The Tokenizer

So begins the tokenization process. It amounts to taking our users expression
"C#" or "Vb.net" and Not "Java"
and splits it into pieces such that we end up with a collection of strings that looks like the following.
  1. C#
  2. or
  3. Vb.net
  4. and
  5. Not
  6. Java
If you still haven't learned Regular Expressions (seriously go freakin learn them), this is a summary or what it does (more or less).
Looks for an optional not followed by whitespace as a capture group to store the "not" then an expression starting with a double, containing any number of characters and and ending with a double quote " (this is our representation of a string) followed optionally by at least one character of whitespace and a logical operation that identifies the relationship of the condition to it sibling on the right if there is one. This will evaluate the entire length of the string and capture each fitting expression accordingly. See line 12. and with that we have our collection of tokens.

Make the required preparations. For we shall parse as dawn!

So before we can parse our tokens into something of interest we have to decide what data we need to retrieve and what will we need to do with it. Looking once again at our requirements and text

"C#" or "Vb.net" and Not "Java"
if((file.Contains("C#")||file.Contains("Vb.net"))&&(!file.Contains("Java")))
{
  SearchResult.Add(file);
}
So we established before that we were going to need a basic string to hold the content that we were filtering but we also have theNOT (!), OR (||) and AND (&&) to deal with. In computer programming as well as natural language (I.E. the English language in this case) and just about any other Syntax these are referred to as Logical Connectives or more commonly Logical Operators. Logical operators are what allow us communicate multiples of ideas without a lot of additional dialog (i.e. Compound Sentences). Without them the sentence

"Jack and Jill went up the hill."

Becomes

"Jack went up the hill."
"Jill went up the hill."

(UGH.... I think I'd go a bit crazy if people talked like that all the time.)


Logical Operators do a ton more than that but, for our requirements this should be enough to give you an idea of their significance at the core of most any and all forms of language, programming or otherwise.

So with that we have a class with which to parse.

To recap:This is what we have so far is our "condition". What gives our parser its reason for being.

class Condition
- string expression
- bool HasNot
- LogicalOperator LogicalOperator

To elaborate on the "LogicalOperator" type is just an enumeration of all the possible Logical Operators we are concerned with and looks like this.

enum LogicalOperator
{
None = 0,
And = 1,
Or = 2,
}

Now we can probably stop here and just make a collection of "Conditions" and maybe get away with it for now. However that would not accurately reflect the relationship between the items in the collection and will, more than likely, make implementing our parser (and thus maintenance it) much more difficult in the long run should we want to implement more language features (such as additional operators) to make our baby DSL into a strong and able bodied language one day. Because of that we are going to be building a Tree data structure using Nodes



Nodes .... NODES EVERYWHERE!!!!



For our purposes a node is a data structure that can have a concept of its relationship to its neighbor(s). A group of these nodes with relationships to each other become a tree. Now, as with most of the point of discussion, there are tons of other areas of study and topic of discussion in this area of computer science and I encourage you to continue forward in checking out all the neat stuff people do with nodes and trees but, like so many of the things covered in this post, I aim to keep it simple.

So with that I have here a simple node example

in C#

At its heart that is pretty much it. The more interesting pieces or on how one goes about navigating these things. Not a lot too that either...

in C#


Note* that this does not account for Null nodes. It will break on a null node and throw a null ref exception.

If you indented to navigate to the last node in a tree it would just be a matter of recursion.
//goes to the last node in the tree


Same thing if I wanted to get all the values in a tree


Converting a collection of strings to nodes


Here is a link to the full example typed up in linqpad.

Taking it Home


Armed with additional node and tree xp lets go back and preform some "augmentations" to our condition class.

We now have...

Note* the order was used as a means of validating the order of the node tree from the token collection and is completely optional.

The traversal methods are just about the same.



Finally we just need to Transform our tree into whatever form we require.
in this case we will be using it to build out our C# expression's if statement .... just because.


And that ladies and laddies is but one way of building out your own itty-bitty baby DSL.


Additional things I'd like to point out.





  • Instead of a condition class the entire class could be split into node tokens with even simpler parts. This implementation works for me but you are more than free to experiment with more tree node goodness.
  • There are tools out there such as

    that preform many of the parsing and tokenizing related tasks mentioned in this post with ease and then some now that you have a handle on a hand rolled implementation.

I first learned about Mini-Languages, DSLs,among a ton of other awesome stuff reading The Art Of Unix Programming. A freely available book by Eric Steven Raymond(one of the godfathers of the open-source movement).


Friday, May 16, 2014

A Giant Woman


So happy Friday All ! I guess this post will mark my first official dev update on my upcoming release "FearTheLight". So "A Giant Woman" (bonus points if you can guess the ref I was thinking off) is the first tech demo of the game showing of the awesome main character. "Aurea" in all her "bad-ass-ness". I say giant because she still looks this good even though she is 20x the original spec'd out model, In other words she is huge! The support assets are all placeholder and some of her animations are missing (namely climbing and ledge hanging) but with that I'm still pretty pumped about the development. 



With it we have some pretty basic character movement and collision detection. I am so freakin excited to have even gotten to this point with her looking so good. That being said there are a couple choppy points where the animation resets in particular when she walks up "irregular terrain" as my college put it. My main focus at this point is "FEEL" I want you as a player to feel good controlling her and while we are not at 100% yet for that I think we are at a good enough a starting point to show something.

But don't take my word for it try it out.

My "Hello World"  for "Fear The Light"

A Giant Woman

*Note I will eventually be moving game related updates to www.hackerferret.com once I work out all the bugs on that site and getting it to "FEEL" good as well. Don't get me wrong, blogger has treated me kindly over the years but I'm starting to feel a bit limited as to what I can do with the site.

Wednesday, May 14, 2014

Playing with Unity3d and the 4.3 2d tools



Test project I put together using Unity 4.3 2d tools. Built this to teach myself the basics. Yes there are random particle effects everywhere ,you can stand on the enemies heads, yeah its super short and you can't die.
Controls
  • ctrl - strong attack
  • alt - quick attack
  • space - jump
  • a - Move Left
  • d - Move Right


Play it in a new window



Things of note.
  • The hit detection mechanism is RayCasting. Check out the OnAttacking method for details on that. (RayCasting is super easy with unity)
The Player Control Script









Friday, April 18, 2014

Util Regex functions for Javascript


Just a bunch on quick 1 liner validation functions I wrote a bit back that keep making themselves useful. Thought I'd share.
Because everybody needs some validation sometimes and because more specifically I needed these today and I had this snippet in the queue.
And also because everybody needs somebody sometimes is stuck in my head.



Feel free to play with these in jsfiddle.

Wednesday, March 19, 2014

Advice for those just starting out in Software Development





So this is a question that a friend asked me in an email. I though it was worth sharing.

Hey, how are you? I hope everything is going well for you. Im writing for advice or suggestions. I started taking classes for programming. I was wondering if there is anything languages I should really focused on or any reading material you suggest?

Okay here goes....




Rule of Thumb

It depends on what you want to do. However the key thing is to remember when starting out is Depth over Width. With that I mean learn one (or more than likely two) programming language well and its concepts will transfer to other languages. When starting out it is better to just learn how to do one very well that it is to know a little bit of a lot of different programming languages. After you have a pretty solid understanding of the core fundamentals of a language, its inclinations for certain architecture decision, design patterns, data structures, and general methods of code reuse etc..... So in short you can "think" in that language pretty well.




Talkin' Bout Some Relations... Database relations that is.





To add to this, early in your career you will more than likely need to know something about relational database management system. Those bad boys are used to store the data that you would be playing with and unless you are building tools for other developer (even sometimes then) you are going to have to have a fair bit of knowledge about how to preform Creation, Removal, Updating and Deletion queries using a special type of programming language. More than likely this will be a dialect of Sql short for Structured Query Language. As to which SQL there is some debate but I'm of the opinion that MySql or Microsoft Sql would server you well in the beginning as this is what I have seen most commonly in the wild (and are freely available).(this can vary by location so if you have your heart set on a particular location I'd do some research by looking at the job sites and querying for sql or database and see what shows up )In any case you should know something about relational databases.



Major Areas of Specialization



Web of Lies

In some cases just learning one language deeply is not practical. This is especially true for web development. Noting that IMO the additional web languages you work with aren't all that hard in comparison to learning other programming languages. Speaking in overly broad terms and possibly making a number of generalizations I'd say that if you are aiming to be a web developer (even just starting out) I'd expect you to know the following.




HTML

  • You should be able to stub out a basic html page with a header, body, form, a couple of input elements
  • Stub out a table
  • Uses for a div
  • Uses for a span
  • Uses for an anchor tag

CSS

You should be able to use a class, id and element selector. Create a even layout for a page just with a style sheet and little to no inline css or element attributes on your html.

Javascript

  • You should know what the DOM is and be able to access DOM elements by id and name,
  • Some jQuery because its so common
  • JSON, What it is and how is it used
  • Ajax what it is and how to use it.

Javascript - bonus

  • Object Extensions
  • the difference between "==" and "===" when talking about comparison,
  • what closures are and what the are good for
  • Prototypical Inheritance
  • General OOP concepts and Design Patterns and how to apply them in javascript. I.E. how do I make a method or property private.
  • What Underscore.js is
  • Common pitfalls in javascript
  • Some basic knowledge about SPA applications. What they are, when to use them, what are some libraries in the wild.
  • General code organization, possibly using tools like require.js

There are a ton of places to go learn these skills if you don't already know them. Additionally you will want to have a good grasp of a server side language like php, java, c#, ruby, python or even node.js (special case)

Other Areas of Application Development


Often Desktop applications are written in a language like C#, vb6, vb.NET, Java, c++ or some other language with a SQL database back end. Not as many as there were in the late 90's due to the growing popularity of the web but are still around.

The Tiobe index is one kind of indicator for the popularity of programming languages
(does not necessarily reflect the industry as a whole but you can't go too bad from learning one or two of the languages on the list)


Mobile development
Mobile development is mainly done with Java ( for Android) or Objective-C (for IPhone) but can also be done with web knowledge to some extent and work on both (http://phonegap.com/)













Easy to Hard

I listed some languages in order of easiest to most difficult from my experience,This is highly speculative and just my opinion.

  1. HTML CSS (go hand and hand)
  2. SQL
  3. PHP Python Ruby  (about the same level of difficulty)
  4. C#  Java (about the same) 
  5. Vb.NET  
  6. C
  7. C++
#3-#5 are beyond debatable.
#2  is easy to learn the basics but has more than a fair bit of depth


Research
In any case whatever you decide I'd recommend a few spots

For FAQs
For learning online

For lots of free books






  • http://programming-motherfucker.com/become.html
  • https://github.com/vhf/free-programming-books/blob/master/free-programming-books.md











  • Asking Questions Online
    For actually asking questions your best bet might be a chat room like an IRC I don't suggest starting out asking stuff at a place like stackoverflow just because they aren't really that friendly to newbies* and its kind of hard to find places that are. Maybe user groups on facebook or google plus might be better but those are kind of hit and miss as well.....

    *note* Stackoverflow is an awesome resource, just not the greatest place to start out asking questions. That being said if you
    are going to ask about something on SO I'd recommend you look at this on Stackoverflow and this article from Jon Skeet, the record holder for most rep on so





    Remember Remember.....

    Okay so thats a lot to process... but it should be a good start. Biggest thing to remember again is to pick one language and area and stick with it until it becomes easy.... Which takes a considerable amount of time.Also don't just read about the field or language actually make stuff, not big stuff but small simple things..... really small simple things and finish those small simple things and share them so others can critique.

    Best of luck to you in programming....
    Its hard but rewarding both financially and mentally and at times spiritually.

    Tuesday, November 5, 2013

    Respawning a Player in a Relatively Safe Spot in C#

    My rudimentary (kinda brute force-ish) way of re-spawning the player in a spot away from enemies and thus my fix for instant-death-spawning.This is something from  my bacon jam 6 entry for fixing a major bug after the contest. It does not account for things like falling enemies but, it should do the trick in a pinch.

    Considerations:

    So performance wise I'm pretty sure Linq isn't great for xna with respect to garbage collection and number of instructions but, baring that Im going to ignore it for now seeing as premature opt being the route of all evil  or whatever..

    Enjoy


    Link to code


    Sunday, October 27, 2013

    Bacon Jam 6 Entry: Slay The Rainbow


    Slay The Rainbow

    Ello all, Just posting all the links for my Bacon Game Jam 6 entry. I will follow up with a post mortem a bit later. (I.O.U.)

    FAQ





    About my Entry 



    Description

    Slay The Rainbow:Ninja vs Rainbow?
    You fight a legion of rainbow monsters armed with nothing but a sword and your ninja agility.
    20 seconds to clear the level.
    Times running out..... What are you still reading for?????




    Technology used

    C#, xna, (hak'd together a mod of the starter kit)


    Controls

    Xbox Gamepad

    Move -joystick or directional pad 
    Attack - X 
    Jump - A

    Keyboard 
    move- arrow keys, 
    Left - left arrow key ,a 
    Right - right arrow, d, 
    Jump- space bar, w 
    Attack - f

    How to play my game. 


    ***Installation Note Read me first** 
    oalinst.zip  this needs to be installed before before running the exe. To install just download and double click.

    If you are running some form of windows go ahead and download and extract The executables from the link. If you have a problem please let me know. Thanks.