Dylan Vorster

As the saying goes, time flies when you are having fun!

Here over at Labs, we have been hard at work since the last blog post, cutting up bits and pieces of our internal repos and moving them over to Labs repos. There are now a total of 7 OSS repos that contain an increasingly large amount of code that was previously considered proprietary and strictly internal.

Some of the new repos:​

Micro​

https://github.com/journeyapps-labs/micro

While currently a WIP, this repo now contains (at the time of writing) 4 of our internal JourneyMicro framework packages that (complete with the internal version) provides all the tools required to build cutting-edge modern micro-services with enormous development velocity. These packages were moved over to enable moving over common-sdk.

Common > SDK​

https://github.com/journeyapps-labs/common/tree/main/packages/sdk

A new package in the common project that enables high-quality SDK clients that run in both Node.js & web environments, are fully typed and support pagination via async-generators. This project was moved over to enable us to OSS and make available the V4BackendClient.

Clients​

https://github.com/journeyapps-labs/clients

This project contains only a single client at the moment (V4BackendClient) but will soon contain more of our SDKs that speak to the shared infrastructure at JourneyApps. This will eventually also include the AccountsClient which provides nice typed, paginated access to organizations, projects and developers!

When coupled with the DeployService APIs, creating cloud integrations with JourneyApps becomes much, much simpler.

Why are we doing this?​

In the last post, we announced an early version of Sector which we decided we wanted to build in the open, and since then Sector has evolved dramatically. As part of that evolution, the OSS version of Sector now shows indexes and (very shortly) will also integrate the OpLog. We figured that now is a good time to move over a bit more of the technology so we can make use of our typed SDK clients to enable this.

What's next for labs?​

Right now we are working on getting the cloud version of Sector up and running, but we are working on improving all the current Labs projects as we go. While shipping all versions of Sector, if we have an opportunity to improve a feature / add a missing feature in a way where we can move more code over to Labs, then we almost always now take that route.

Some examples of this:

• Big improvements to Reactor around themes, tables, and large data sets.
• Moving over Micro
• Moving over SDK Core (common-sdk)
• Moving over clients (V4BackendClient)
• Bumping package dependencies across all the Labs repos to stay in sync with active development happening over at Sector.

You can also expect to see more SDKs moved over to the clients repo since the core SDK tech is now available in common.

Micro is going to be fleshed out with more packages such as our router and mongo packages. When we do so, we will also provide example projects that showcase how Micro can be used to set up and deploy micro-services.

We also have some CLI tools that will be heading over soon, and will probably land over at a journeyapps-labs/dev project in the near future. Keep an eye out for those!

That's all for now, but make sure to keep an eye out on the various Labs repos as we push updates there almost every day.

https://github.com/journeyapps-labs/sector

For almost a decade now, JourneyApps has been systematically improving and overhauling all of its core developer tools that contribute to all aspects of the app development lifecycle. We first introduced OXIDE (built on-top of the Reactor IDE framework) which represented a major overhaul of the app editing experience. We updated the Admin Portal and improved the administration parts of the platform making it easier to switch between projects, set-up integrations, manage developers and work with roles. Recently we also announced version 5 of the Runtime which represents a massive step forward to app performance and reliability due to the almost completely overhauled engine which now adopts more declarative and deterministic engineering concepts.

Well today we are pleased to announce that we are making Sector open-source! Sector represents the future of the data browsing experience and like OXIDE, builds on-top of the Reactor framework super-charging the user experience. With tabs, workspaces, cmd palette, SPA, improved performance and multiple deployments, developers can now work with deployment data in ways never seen before on the platform! Sector represents a large step forward for the product as it is one of the first pieces of end-user software that can be self-hosted on your own infrastructure.

Over the next few weeks we will be shipping docker images and providing tools such as Kubernetes configuration and docker compose files to help get developers up and running quickly.

Sector is modular technology​

Sector's core capability has been envisioned from the beginning to be a Reactor module that will be embedded inside of OXIDE. While Sector will be deployed internally as part of the JourneyApps Platform cloud toolset (and eventually replace the current backend UI as we get closer to feature parity), we will very shortly also be embedding the module inside of OXIDE with deep vertical integration making easier than ever to work with data without needing to switch software.

This is just the beginning​

Today marks the start of Sector's journey and there is still much work to do to get Sector shipped as part of the core product, but we are pleased that we are able to already share our development in the open so early on. As we ship updates to Sector and make it more generally-available across the product, we will continue to share updates and document our development journey here on the labs blog.

Sector is still very much a work-in-progress, and therefore we value early product feedback. If you start using Sector and have thoughts & suggestions, please let us know over on the JourneyApps discord server and log some issues directly on the sector repo!

Today we are announcing that the Carbon library is now open source!

https://github.com/journeyapps-labs/carbon

Carbon is a visual programming language and execution engine that was developed in-house as a solution to an enormously complex customer requirement that almost made its way into production.

The goal was simple:

• Allow apps deployed in the field, to be reprogrammed by technical users using visual means.
• Allow developers to create the default programs for such technical users as part of the application code.

While the goals were simple, creating such a feature and integrating it into the code-first platform was no easy task. As with almost everything we create these days however, we decided to invest development time into a solid platform-agnostic technology layer as the first step. Back then, we had hoped to license this technology in the future as a stand-alone product, but today we are pleased that it is now open source!

Carbon is also a user-interface library agnostic programming language and engine, meaning the way it is rendered is completely pluggable. Since I am also the original author of react-diagrams (https://github.com/projectstorm/react-diagrams), we decided to do our rendering layer using that library (represented as the carbon-ui-storm package within the repo). React-diagrams was always intended to be used in a technology such as this, and it was also originally modeled over LabVIEW, UE blueprints and the Blender compositor which are stellar examples of declarative visual programming technologies.

So how can I use Carbon today?​

At the moment, the best way to use Carbon, is to check out the carbon-demo folder which provides a demonstration molecule (our terminology for a wired up collection of atoms, which we also call nodes). This demo also wires in to the default molecule the optional atom builder package which allows custom atoms to be created using an XML parser (which we had originally developed as a way for developers on the platform to define custom atoms within OXIDE). The debug demo shows how you can create a program from a molecule, compile it, debug it using the debugging canvas and see its execution in realtime with the browser.

Ultimately, you would want to serialize the molecules created in the browser, send and store them into a database (as JSON blobs), deserialize them on a server and then compile and run the programs in threads. In the future we will most likely provide an out-of-the-box execution Docker image for this exact purpose allowing Carbon programs to be used as a cloud execution programming system.

How extensible is carbon?​

Carbon is very extensible, as is demonstrated by the elements and elements-ui packages within the repo. These packages represent the 'standard library' but also serve as a demonstration for how to create your own atom packs. At JourneyApps we extended this in-house with a schema atom pack that used the platform schema.xml to create database model CRUD atoms. In the future we may open source these libraries too!

Will this feature ever be released to the JourneyApps Platform?​

We are busy with some big plans for the platform, and are currently hard at work on rolling out V5. While not on the immediate roadmap, Carbon and its vertical integration within the platform were already near close to full completion at the time. We are considering this as a possible future next step if there seems to be interest in the feature. If you are a platform customer and wish to see this technology released, please reach out to us!

Some original screenshots of the technology inside the platform stack:

Many years back when we started working on the successor to the JourneyApps editor (now called OXIDE), we set a goal for ourselves of building a great IDE framework that could power the next generation of online development environments.

At the time, we were not yet open-sourcing many of our lower level libraries, but we still made sure to separate the underlying framework from the final IDE. We did this not only because we wanted to use Reactor for other in-house software such as a tool called Override (for internal platform administration which we may release in the future for enterprise customers), but because we have always said "build everything as if it were a product."

Well today marks that day, and we have finally decided that it's time to release Reactor to the world! Reactor can do some pretty cool things and allows engineers to build highly-capable software at incredible speed. We called the framework 'Reactor' because the more entities you describe to it (along with the relationship between entities), the more Reactor will automatically do for you and the more powerful it becomes (entities react with each other). We won't dive into too much that Reactor can currently do as we will save that for a future post, but if you have used the JourneyApps platform or PowerSync, then Reactor has been at the heart of the developer experience.

The packages and demo folder are fully operational and represent the same code that powers the cloud versions of OXIDE and the PowerSync Dashboard. While the demo is a bit crude at the moment, we will continue to flesh it out and provide more concrete solid examples that can showcase the best that Reactor has to offer. Here are some screenshots of all the software built on-top of Reactor:

OXIDE:

PowerSync Dashboard:

Override: