Saturday, 31 October 2020

Flock Day 8

 It's been a very long time since I last posted, although I have been spending on average 2-3 hours a day cutting code since the last post, I did not feel I had made relevant progress until now.

In regards to language creation not much has progressed, I have been sidelined in working out the best way to create a syntax tree. I have switched to an EBNF style recursive descent parser, This was a touch reinventing the wheel, others exist, for c++ the boost libraries provide one. However having done this before in Java, I felt there were some things missing from these libraries.

I invented a new pattern (in a personal sense, I can't be arsed to see if I am the first), I like to call the visitor strategy pattern, an amalgamation of the visitor and strategy pattern. The main problem with the visitor pattern, is that you tend to have to bundle all the functionality for all the behaviours into one unholy and massive class. This makes code organisation a nightmare, and means you end up implementing way more than you need. It also tends to force a certain way of navigating the object hierarchies. Often they also manage state, and so can get quite confusing to know where you are.

In my version the visitors are very light weight containers of strategies, they effectively look at the object, and find the strategy to use against it. The strategy accepts the object, rather than the object accepting the visitor, the strategy will then do its thing, if we need to traverse the object hierarchy, it merely goes back to the visitor and asks it to visit the new object. This means each piece of logic is self contained, and can be extended or reused across different types. This has already proved useful, as my rule printer, navigates all nodes in the rule tree, whilst my rules evaluator only navigates a branch until it fails. On top of that the rule printer is remarkably small.

Strategies are extendable, and via wrapping I have also implemented rule histories, again something I believe unique to my vision of how these parsers work. The theory goes, that if a stateless rule is executed at a particular point in a token stream, (in this case token=character). then if that rule is executed at the same point, it should return the same result. By effectively caching the result we gain massive time savings, at the expense of a small amount of memory, (a freedom not given to those who originally had to write parsers). Recursive descent parser's can be notoriously slow on complex grammars, as they will often ask the question multiple times, hence the time savings. They are also prone to infinite loops, and stack overflows, this technique prevents the last from ever happening, asking the same question of a rule, whilst it is still evaluating at that point counts as a cyclic failure when asked additional times, until the original request has been satisfied. At some point I plan to fix that problem which is inherent in both the visitor pattern and recursive descent parsers which is that of their tendency to stack overflow. The removal of logic from the visitor should allow me to create execution queues instead of recursive calls, as a provided implementation.


Above is an example of the parser at work, I have, unashamedly, used ANSII colours to help work out what is going on. At the top, in ISO EBNF syntax are the rules, printed out by my rule printer. At the bottom is an evaluation of textual input use (me=that,and, (this=that)); this then results in the printing of the syntax tree, each node showing the full textual output of what made up those rules.


Friday, 11 September 2020

Flock Day 7

Its been 3 days since I have done any development. This is to be expected I am no longer on holiday, and my priority list goes in the following top to bottom order, family, work, house, sanity, personal projects.

In reality I have been procrastinating, I have loads of great ideas, but instead of writing it down, I have been making mayonnaise, badly. I roasted some garlic in olive oil last weekend, and a friend has invited me over for some tomahawk steak, so I thought the least I could do is steal some of my partners tomato relish, and make some garlic mayonnaise. I rushed it in the vain wish that I could do some programming, ended up with some vile curdled liquid. My dad who had taught me how to make the stuff aged 6, would be disappointed. If at first you don't succeed, play minecraft instead. Then make a second batch and hope the food processor doesn't wake the family, (see priority one).

However It got me thinking about the relationship of writing code and making mayonnaise; in that there isn't one, unless you like PHP, and even then it doesn't matter how patient you are you, will always end up with a runny curdled mess.

Ok, so maybe I should talk about Flock. To be fair one of the key concepts for flock is around polymorphism, I want separation of structs and traits, you can define a class that has both, but under the hood it is going to get split. That's because I want structs to be about as anemic as is possible. In fact if I had a name struct, that extended String, that would use the same struct as any other tiny type that extended string. The same with traits. 

So why bother? Well from an engineers point of view types are important, they allow me to subdivide what things do and how they look, but the computer/compiler it isn't important. Engineers also tend to have logic that is expressed against the types they use. That might be calling a method existing in a base trait/class and expecting any class that extends that method to have its version called instead. It might even be logic based on what type it is. Don't hold your nose, we have all done it, it is often used in most underlying libraries. I want to support 3 types of class extension, 

  • Inheritance: ClassA : ClassB
  • Prototyping: objectA : objectB
  • Duck-Typing: MethodA<ClassA> : MethodA<>

Notice how I am using the : to express extends. That might actually make it into the language. Thanks c++. Ok the last one probably doesn't make sense, even if you know what duck typing is, but I couldn't just leave it without some form of example.

Every OO object will have 3 distinct components, it will have its data (struct), its functions (traits), and its type. I am hoping to leverage some serious capabilities. So imagine I have a generic/template ClassA<ComponentB T> I could also say that ClassA<ComponentB&ComponentC T> to express a union of types alternatively I could also say ClassA<ComponentB|ComponentC T> to express the acceptance of one or the other, the methods that use this type, would only have access to the common method names or variable names. Hopefully that might give you a clue into why I want my compiler to only have one copy of a particular function or data point to worry about. Why waste time checking types if I don't have too.

I suspect a large part of generating the Abstract Syntax tree, will be around type inference, I have a fear I may have to navigate the whole tree several times just to nail down the exact types available. I have a greater fear that I will use some form of machine intelligence to predict the most optimal tree navigation. That is way too much navel gazing even for me.

Wednesday, 9 September 2020

Flock Day 6

 At this rate I may have the most tinkered with Tokenizer known to man. Coding is a touch like wading through mud if you don't know the intricacies and limitations of the language. C++ has a lot of both, it really doesn't like it if you are not explicit with your types, and the memory management has taken a while to get used too.

However I found a great little site the other day, I have yet to use it, but it contains about every pattern I have been trying to achieve, with some examples. C++ Patterns Over next few days, I will refactor again, but this time, maybe I will PIMPL it up or get down with some SFINAE. Certainly I think my constructors need some work.

You may of noticed that I have improved the blog after your feedback, it's cleaner, and got some relevant background picture. If anyone wants to design me a simple logo... Maybe that's too much to ask.

Sunday, 6 September 2020

Flock Day 5

 So I managed to get a basic tokenizer running, it goes a bit beyond the tutorial, in that it is the startings of an LL(*) parser as opposed to the LL(1) proposed in the tutorial. For those not wanting to go into the deep specifics of parsers, the difference is how far we look ahead when reading text, an LL(1) parser will only evaluate on the next letter, whilst an LL(*) parser can evaluate as far ahead as it wants. I have actually written these before, and there is a Rules based evaluation pattern I built that I think will make parallel parsing very easy, in the new language. I am not confident enough in the way c++ manages its memory to bother until that point.

So it looks like I have decided to go with c/java style comments /* and // I have started tokenizing them already. I had a plethora of different options, # or -- or <!-- just for starters. I may go revisit that decision, but familiarity is important, and they are not the worst comments in the world.

I have also been thinking about memory management, generally the way memory is managed in modern languages is roughly equatable, and in my mind fundamentally flawed, it uses ownership semantics, which are hierarchical in nature. I have a thread, it starts a function, the function owns object A, object A owns Object B and Object B owns Object C, and so on, sometimes C can even own A, and we have some horrible problems trying to resolve what needs to be freed or allocated or passed. In reality there are only 2 owners of memory that are important, there is the global space, and the thread space. The global space is where we store our things accessible by all, they have atomicity constraints as multiple threads may have to access them at the same time. The thread space, is memory that belongs to a flow of execution, I use the word flow purposefully here, it can do what it likes with it, until it decides to move it to a different space (thread of global). Part of the problem is how do you clear data up when it is no longer needed. it is probably not good enough to wait till a thread ends its execution. However I am building a language that is moving out of the thread concepts, instead I have workers and flows, in effect I am creating a third space, the worker space. When a worker finishes, it clears its memory, rather than at the boundary of a function, we do it at the boundary of an execution. 


This is just working thoughts, what I don't want to do is just do a fuck load of memory copies.

Saturday, 5 September 2020

Flock, Day 4

So gave up on vcpkg, not that it wasn't working, but I found NuGet, this did everything I wanted pretty much instantly and seemed to just work with no effort on my part. Next step, create a repo for the cpp compiler, at some point this will be written in flock, but clearly, I can't write a compiler in my new language if I can't compile it. The old chicken egg conundrum.

So here is my repo https://github.com/flocklang/flock-compiler-cpp

Depending on when you read this, you may find very little in there, if there is (hopefully) alot, and you really are bored, I am sure the git history will help aid remove those brain cells. Just remember I am not a c++ programmer by trade, I am sure even with the single header, and default cpp file, I have done something hilarious. Always open to opinions.

The days on the top of the heading are a lie, I might spend multiple evenings working on flock before blogging, or maybe just an hour, it really depends on the actual progress. I am glad I spent such a long time setting things up, it did two things. Firstly it made me realise how much work was going to be needed, and questioned my commitment, I passed. Secondly I was able to identify problems, I might not of come across until much later, and thought it was something I had or hadn't done. Hence why the first file is a CompilerFix.h file.

Wednesday, 2 September 2020

Flock, Day 3

A few years ago, I decided to leave the apple ecosystem behind, me at home at least. I found the hardware for the spec was overpriced, and my dual booted window on my macbook pro, started up substantially faster. I spent less and less time in macosx and more and more time in windows 7 and then 10. In my mind windows 10 just runs better. Under the hood its messy, although to be fair to a lesser extent so is macosx. But you can get linux subsystems etc running on it, so now I have 3 shells to content with. Anyway to cut a long story short, I have a laptop with windows on it, I use it to game, surf the net and cut code. I am used to cutting code in modern languages, ones with good package managers, that doesn't require OS or hardware specific flags, or complicated tool chains. I don't want to dual boot linux, or have a second laptop. So I am a touch lost at sea. If you are reading this and fancy giving pointers comment away.

So fighting with visual studio to work with LLVM has not been pleasant. I imagine this is more the laxidaisical support of LLVM than microsoft, as they do seem to attempt to support their tools. 

In the process of mucking around I came across vcpkg this helpful little utility, is a bit like maven or npm or cargo, but for c++, and surprisingly multiplatform support, yay. I made the mistake of downloading the latest tag version, if you are foolishly following in my footsteps, go with the master, at time of writing the tag version failed to compile the llvm dependency. So you download this little app, run a batch script and then with a few simple instructions (all documented on that link), you can build projects with llvm as a dependency. Its great even if it takes almost 2 hours to download, compile and then install it. 

vcpkg install llvm

To test that things are working correctly, I have been taking the llvm tutorial, chapter by chapter and running them in new standalone projects. 4 days ago I was struggling to get chapter 2 to build, chapter 3 was even more of a nightmare. I have got all the way to chapter 8 building happily. Chapter 9 builds but doesn't play nicely, and gets upset about memory assignments. However Chapter 9 is Just In Time compilation, something which I will want, but not for  looong time, hopefully my c++ chops would of improved. Whenever I have programmed in c++ in the past, it has  been for mobile development in xcode, or writing java bindings. I feel for engineers that have to build multiplatform applications in c++, it is painful, I am also not a fan of visual studio, its clunky.


Monday, 31 August 2020

Flock, Day 2

So I decided to rebuild LLVM following the instructions, it's taking a while to rebuild, but I think its better this way as LLVM version 10.0.1 has come out and the windows version needs a source build. I appear to of made a couple of errors in my first built.

  • Using the cmake gui, made life a lot easier, (although I had to learn the tool)
  • Ensuring the install and build directories don't have whitespace in the path also seemed to make a difference. If at first you don't succeed read the instructions, as my dad says. Although in my case, read the instructions more thoroughly.

So I think I want to cover some thoughts on syntax, in particular how we deal with removing Null/Nil.


I don't want nulls to be in the language. I could just insist all variables gets instantiated, and throw an exception if they aren't. However this is a pain as a developer, as often forcing instantiation, makes for some awkward code with a significant amount of error handling. 

The typical approach in languages and APIs is to specify an Option(al) variable, one that is either set or not. So your variable effectively wraps the instantiated one. If the instantiated version doesn't exist it is represented by 'None'. if it does it is represented by the wrapped variable, like 'Some' or 'Any'. Rust does this particularly well, by representing the 2 states as an enum. 

There is still some exposition style bagage here, in java you would call isPresent() or in Rust use some form of deconstruction, often you stick this logic into a switch, match or if statement. Which brings me to the use of syntax.

A common way to infer this logic is via the Elvis operator ?: this basically says, if not present default to the next bit. Alteratively ?. Says if present do the next bit, otherwise stop. This is definitely the way to go, and this short hand is top of my list. I would like to add some other syntactic sugar here for conformity. Additionally since we are a reactive language, being able to syntactically support errors is also quite useful, now using ! could also mean there is a problem, and we would use it a bit like onerror, or the catch, so maybe :! Defaults as on error, and .! Continue if no error. However we also use ! in logic statements, so this could get confusing. So I need to spend some time thinking about the cleanest way of differentiating.

You might at this point go 'doesn't this fly in the face of the rule to be symbol minimalist?' My answer 'Um.... yes, but I also want it to be easy.' Part of making a language is having to decide what feature is more important, and because its my language (for the moment) I get to make that choice.


It looks like it has just finished building, but family have returned and I have a dishwasher door to replace, the time just creeps away!


Flock Day 8

 It's been a very long time since I last posted, although I have been spending on average 2-3 hours a day cutting code since the last po...