13 Replies - 539 Views - Last Post: 17 January 2019 - 08:08 PM

#1 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

My Video Tutorial Series..

Posted 14 December 2018 - 11:11 AM

Make enumerations first class citizens in C++

I've linked a Youtube video that might be of some interest here. It's about how to make enumerations first class citizens in C++. Enumerations are pretty useful in C++, but still quite weak compared to what they can be. This video demonstrates how I take them up a few orders of magnitude in usefulness.

https://www.youtube....h?v=AF186FraxS0

I am the author of a large, software based home automation system, called CQC. My code base is about a million lines of C++ code now, about half of which is a soup to nuts general purpose part, and the other half is the automation system built on top of that.

One of the very useful things the general purpose system provides is a quite full featured ORB (object request broker.) I have a video on that that I'll post later, but ORBs typically use an IDL language (interface description language) to describe the form of the calls to be made to remote machines and tells the ORB's engine how to pass parameters, return parameters, and so forth. But, it can also do other things. In my case it can generate types and constants and is often used for that purpose.

For enumerated types, it can provide a lot of functionality that makes life far easier for a C++ programmer, particularly when working on the large scale that I do. And it doesn't take a lot of code to create such a tool and integrate it into your own development process. There's actually more functionality that this video covers, but I didn't want it to get too long so I stuck to the core stuff.

* Since I made this video, I've updated the ORB to be able to mark enums so that they use the 'enum class' scheme. This ORB stuff was first written about a decade before enum class was available. I've been incrementally switching over to the new, typesafe scheme.

Is This A Good Question/Topic? 0
  • +

Replies To: My Video Tutorial Series..

#2 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 15 December 2018 - 03:08 PM

Implementing a full featured ORB (object request broker)

In a previous post, I posted a video where I demonstrated the use of our IDL compiler to really crank up the functionality of enumerations in C++, which are fairly weak on their own. That post is here:

https://www.dreaminc...-citizens-in-c/

But the real purpose of the IDL compiler is to support our Object Request Broker (or ORB), which is a mechanism for creating client/server applications. ORBs act as a sort of glue between clients and servers, and make it as though you are doing local calls when really they are remote calls. It handles sending parameters and returning output parameters, return exceptions and all that. An ORB is not just a remote procedure call scheme, it's far more than that. It creates a 'party line' between each client and server and multiplexing communications between them, so everything is done asynchronously with queuing and streaming of messages in both directions. A server farm on the far side grabs incoming client msgs and dispatches them to the appropriate registered handler. Typically (as with us) there is a means for servers to advertise available services on the network, so that they can moved as needed, i.e. the clients don't directly find the server services they need, the look them up.

https://www.youtube....h?v=vjUg8klBLLM

C++ generally has an issue with ORBs, at least the ones I've been exposed to. Since C++ has no over-arching, tightly integrated object framework, it's difficult for an ORB to really operate gracefully as it might in some other languages. Of course you can do things like generate wrappers around SOAP or something like that, but it's not really close to the same.

In a tightly integrated, full featured framework like ours, every object that implements the binary streaming interface can be passed as parameters to remote calls via the ORB. It also understands enums and fundamental types as well. That makes such a difference. There is no requirement to do anything special to a class to support the ORB. And of course it's not being translated to some higher overhead text format or anything like that either. The flattened objects are transmitted as is, and resurrected on the other side.

The linked video demonstrates a simple client/server program. It shows how straightforward writing typesafe client/server programs can be in this sort of system.

This stuff is part of my large (million lines) code base, half of which is general purpose and half of which is the implementation of our CQC home automation platform built on top of the general purpose stuff. CQC is highly network distributed, with lots of remote administrative interfaces and such. Without the ORB, it would be somewhat of a joke to try to create something of that size and complexity and networked nature.
Was This Post Helpful? 0
  • +
  • -

#3 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 17 December 2018 - 01:50 PM

Creating a 'virtual kernel' platform abstraction layer in C++

Here is a video, one in a series that I'm posting, that explores some of the technologies developed as part of my large and long-running C++ code base. It is about a million lines of code, half of which is a comprehensive general purpose system from the ground up, and the rest is an automation system built on top of that general purpose layer.

https://www.youtube....h?v=seXk3RbAjNU

This video looks at how I've created a 'virtual kernel' which abstracts my code away from the platform details. I limit access to any platform or language headers to within this virtual kernel, so that everything else is written purely in terms of my own interfaces. I have my own standard libraries, system functionality wrappers and all that, all designed for portability as well, which is another big point of having such at hing.

Of course nothing is ever perfect in the software world, so there are various bootstrapping issues and such that have to be dealt with when you do something like this. And there are other ways you might approach it, but this is mine. OTOH, by keeping system headers down in this low level layer, that means that 90'ish percent of the code can be compiled in 'pure ANSI C++ compliance' mode, which is nice.

The video goes fairly quickly so that it won't be too long. Just want it through and get the general feel of it. If you want to dig in more, go back and pause it where desired and read more.

Here are the other videos previously posted, which you might find interesting and/or educational. Feel free to ask questions or comment of course.

https://www.dreaminc...request-broker/

https://www.dreaminc...-citizens-in-c/

This post has been edited by modi123_1: 24 December 2018 - 09:59 AM

Was This Post Helpful? 0
  • +
  • -

#4 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 15434
  • View blog
  • Posts: 61,848
  • Joined: 12-June 08

Re: My Video Tutorial Series..

Posted 17 December 2018 - 02:10 PM

I am pondering about bundling these all together...
Was This Post Helpful? 0
  • +
  • -

#5 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 17 December 2018 - 02:40 PM

It might be a bit of a slog to get through. Having them separate lets folks pick the ones they are interested in. And I have others yet to come, so it would get long.
Was This Post Helpful? 0
  • +
  • -

#6 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 15434
  • View blog
  • Posts: 61,848
  • Joined: 12-June 08

Re: My Video Tutorial Series..

Posted 17 December 2018 - 02:43 PM

Sure.. but they are all centered on promoting your youtube channel. Let me think on it for a bit.
Was This Post Helpful? 0
  • +
  • -

#7 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 17 December 2018 - 03:00 PM

My personal Youtube channel has fewer views than most folks would get if they posted a video of themselves asleep in a dark room and took it down the next day. I couldn't buy a cracker even if I monetized it. And posting here would be horrible promotion if that's what it was about, since you can see how few views these posts have gotten. It's a topic of fairly limited appeal, and this section of your form is obviously fairly lightly traveled.

I've just developed all of this stuff and never showed it to anyone at all. I just wanted to demonstrate some of it so that folks can see how these things have been done. At some point we might also open source the general purpose code, so it also helps to maybe gauge what interest there might be in that.
Was This Post Helpful? 0
  • +
  • -

#8 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 15434
  • View blog
  • Posts: 61,848
  • Joined: 12-June 08

Re: My Video Tutorial Series..

Posted 17 December 2018 - 03:05 PM

You lost me.. who is 'we'?
Was This Post Helpful? 0
  • +
  • -

#9 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 17 December 2018 - 03:15 PM

That's just habit, speaking in the 'royal business plural'. I do have a company. There's another guy who handles the books on the side, but it's just me doing all of the coding, testing, documentation, videos, etc... When you have a company you get into the habit of speaking in the company plural. It just sounds more official and whatnot.

I just decided to start showing some of the stuff I've created. My company doesn't sell any code libraries or any code or anything like that, so I just put these on my own channel, which I've had for decades but never used for anything. My company's product is BUILT on top of this code, but otherwise this technical coding architecture stuff is unrelated to what my company sells.
Was This Post Helpful? 0
  • +
  • -

#10 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 19 December 2018 - 10:52 PM

Creating an X-Window'ish touch screen client for my automation sys

This one of a set of posts where I'm covering some of the ways I've created some software technologies used in my automation platform. This one demonstrates how the best laid of plans can go awry, but also how having control over your own software destiny can make these things survivable, and even ultimately triumphant (in a geeky, software kind of way.)

The video looks into the code, but because I've provided some background story below, which would have been too much to get into in the video.

https://www.youtube....h?v=ZpTbU16tfOY

So, to start with, one of the key things in automation systems is graphical touch screens. Of course voice control has gotten a lot better of late, but they don't in any way get rid of the need to have powerful touch screen systems. I may do a separate post on some other aspects of the touch screen system, but in this case it's more about the creation of a sort of X-Windows type thin client system and how that let us support other touch screen client platforms (our product is Windows based.)

Our system has been around a good while, long before people started having cell phones stapled to them at birth. So in the early days we didn't need to worry about them. But of course there is always great evil at work in the world, and suddenly everyone wants to have our touch screens on Android and Apple phones. So we had to consider how to do that.

Our own touch screen system is purely graphical, basically implementing our own windowing type system in graphics, so that we are unconstrained in terms of use of transparency in the layers. We could have just said, OK, we'll just do this whole other thing that is much simpler without all the powerful graphics, which just provides basic access using native platform UI widgets and a completely different design tool.

But I really didn't want that. It would mean two whole different design efforts and all that work we did to create a powerful touch screen system wouldn't be apparent to non-Windows users. So, I hatched a plan to be able to display our existing touch screens on those platforms.

Since we have enormous control over all of our code, because we use almost no third party stuff, that means that we have our own wrapper around the graphical device output functionality. Since almost everything involved in the touch screen system is graphical output, the only real exception being touch screen input coming the other way, it seemed like we should be able to redirect that graphical output to a very thin client.

So, I factored out an abstract interface for the graphical device class, so that it could target something other than an actual graphics output device. The normal one is now of course is just a concrete implementation of that interface, for window/printer/etc... based output. But we can now have another one that just queues up graphics commands. Those commands can be spit out to a thin client, which can perform the equivalent operations on that platform.

We could then, via some more generalization and abstraction of the touch screen engine, wire up one that creates a window based view, and uses the real graphics output, and one that creates a remote view and spools out commands to a transmission queue. And similarly the real one uses our standard touch screen input classes, and uses another derivative of that which accepts transmitted touch screen input commands and feeds them into the engine as though they came from a local touch screen.

So that was all well and good, then I hit a huge road block...

The problem was user rights. Our automation system is highly networked, and multi-user, so everyone has to log in and what they can do is limited by their user account type. And the checking of those user rights is done all over the place. If any of you were probably implementing such a thing you would probably immediately think what I thought when I first implemented the foundations of this system, you store the user rights in a place that it can be gotten to by all the code in the current process.

And that worked well for years, and of course got used in ever more places. But, now we are suddenly no longer going to have one user session per process. We are going to have to virtualize the entire system wrt to user rights. A server is going to be creating multiple touch screen engines, each of which is serving some remote client. We could have spawned off one process per client connection, but that would have introduced a whole other raft of complications. And it would have been very inefficient in terms of memory usage because there are a lot of resources involved, which are downloaded from our 'master server'. These things are cached locally, making things far faster for the clients and vastly reducing load on the server. If we used multiple processes we'd have had to have multiple caches, and various other complications.

There isn't really any way to associate user rights with a specific thread or anything like that. Our product is highly multi-threaded, and various things are done in the background or via thread farms, so it would have really been messy to try to do that.

So, ultimately, I bit the bullet and completely reworked the system so that user rights, and other user context info, is passed along to those things that need it. It's not nearly as clean in one way, but in other ways it's cleaner because means that multi-session virtualization becomes very doable. Or even 'user impersonation' type features later if that is useful. And, in the end, it wasn't as bad as I thought it would be.

So now our Web Server can host multiple virtual touch screen engines. All that remained was to write a Javascript (well Typescript really) client to implement the thin client. That client uses an HTML canvas to do the graphics commands, and it takes user touch input and sends it back to the client. It uses a WebSockets connection back to our Web Server for a permanent connection. Of course Javascript's totally annoying async programming model makes something complicated like this far more painful than it would otherwise be, and the frustrations of how the Canvas does certain things like clipping drove me crazy. But it's doable.

Because there's a lot of constants and structures involved in defining this remote interface, I defined it in an XML file and wrote a little program that uses our XML parser to parse that file and to spit out stuff for both the C++ side and the Javascript side, to insure that they are both in sync and to avoid a lot of brain frying manual checking of stuff.

Anyway, the moral of the story is that we now have the same touch screen interfaces (they look exactly the same) on both Windows and within the browser. There are a couple limitations. One is that we don't get smooth scrolling in the browser. As you might imagine, that could be hugely difficult to make work in this kind of situation. So on the browser it's flicks and paged based moves, whereas on Windows it is dragging and flinging. And currently no embedded web browser or web camera pages.

But the performance is great. Ultimately the amount of traffic is not large and since the commands are small they can be chunked for transfer pretty easily. The protocol is binary and pretty tight.
Was This Post Helpful? 0
  • +
  • -

#11 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 23 December 2018 - 10:58 PM

Creating an object oriented language with VM and IDE

As part of a series of posts I've been making, where I examine some of the technologies I've created as part of my large (million line) C++ code base I've created over the last many years, this one digs into some of the tools and techniques required to create my CML language.

https://www.youtube....h?v=nKveNsTA1zY

The video goes fairly quickly to keep it from being weeks long. This is a heavier subject than any of my previous ones. But you can go back and pause it here and there to read more of the code and such.

Half of our code is a general purpose layer, and the other half is an automation system built on top of that. Though the bulk of customization users do in our automation system is point and click, we wanted to provide a programming method as well for those who feel comfortable with that. And we also needed a way for third parties to be able to write device drivers so that they can integrate devices into CQC for control and monitoring.

Since we don't expose any of our C++ to users, we needed another means, so I created CML (CIDLib Macro Language.) It is object oriented and compiles down to high level opcodes that our CML VM runs. I also created a graphical IDE for the language. Both are implemented as engines that can be embedded into any of our applications that might need CML support.

It's pretty full featured, but it does have some limitations relative to a more general purpose language. Since it's not for writing applications, it is single threaded, it doesn't support aliasing, and it is single pass (so no forward references.) Some of that is for safety and some for speed. Having a single pass compiler is important because these are often downloaded, compiled and invoked in response to a user pressing a button or speaking a command and so forth, so it needs to be really fast. And keeping it very safe (hard to shoot yourself in the foot) is important because the folks using it typically are not professional developers.
Was This Post Helpful? 0
  • +
  • -

#12 modi123_1   User is online

  • Suitor #2
  • member icon



Reputation: 15434
  • View blog
  • Posts: 61,848
  • Joined: 12-June 08

Re: My Video Tutorial Series..

Posted 24 December 2018 - 09:57 AM

Yeah, let's keep the posts in one thread. I'll take the time to merge them.
Was This Post Helpful? 0
  • +
  • -

#13 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 02 January 2019 - 10:56 PM

Here is another one. This one looks at how I created my test framework. Obviously, for this large a code base, I need to be able to run unit and regression type tests in a coherent way. So I needed a test framework, and created this one to serve that purpose. It's a nice little tool and the amount of code involved in surprisingly small.

https://www.youtube....h?v=0Yzcwk95bm8

In my case, since I'm creating my own test framework with my own code, there are bootstrapping issues. I have to limit the test framework itself to quite simple stuff, so that it uses as little of the code as possible. There are a few lower level test programs that do really basic tests of the core stuff that the test framework itself uses.
Was This Post Helpful? 0
  • +
  • -

#14 droddey   User is offline

  • D.I.C Head

Reputation: 7
  • View blog
  • Posts: 51
  • Joined: 25-September 16

Re: My Video Tutorial Series..

Posted 17 January 2019 - 08:08 PM

One thing that's not been done so far is to go through a program that is small enough to bite off and fully understand what all is going on without being an expert in our system, but large enough to be reasonably representative and demonstrate how this stuff all works in a practical program.

Here is video that does that. It goes through all of the code of this program, so you can stop and read it all if you want. Obviously feel free to ask any questions.

https://www.youtube....h?v=qcgEefvTASU

Since we don't use the STL, some folks might find this interesting just to see what life outside of the STL might look like. This is one possibility. We have our own soup to nuts development environment, which one of the videos above (the virtual kernel one) covers.

This little program is a server that accepts socket connections from one or more 'smart sensors' each of which reports five data points. They aren't real sensors, I just did a little program to represent such a sensor for demonstration purposes.

This post has been edited by droddey: 17 January 2019 - 08:08 PM

Was This Post Helpful? 0
  • +
  • -

Page 1 of 1