#Key Achievements

#Project Overview

ACE is an experimental side project focused on a simple question: what if an AI-native workspace lived directly on top of the desktop, with local tools, persistent runtime context, and a UI that feels like part of the operating environment instead of a separate app?

The project combines an Electron desktop shell, a React renderer, a background DeepAgents runtime, and a kernel-like control plane for memory, process orchestration, filesystem access, configuration, and package-driven extensibility. The goal is not just chat, but a unified agentic surface where windows, tools, memory, and execution all cooperate inside one environment.

From a product perspective, ACE is intended to reduce context switching. Instead of jumping between IDE panels, terminals, browser tabs, and disconnected assistants, the user gets an overlay-driven workspace where AI interactions, live tool output, and runtime observability are integrated into the same desktop experience.

#Challenges & Tradeoff

The primary challenge in ACE is architectural how can we make a windowing system in one browser window, and also creating a package registry system so that future extendability is easy.. below is couple of architectural decision & RnD that i made..

Decision Windowing System

For the windowing system, i choose to render every window as one part of browser window is that to prevent bulking memory usage since ACE was intended to be used as an overlay tool, imagine if one window equal to one browser window, imagine each one browser window is a standalone browser, and each browser can take up around 100-300mb, imagine if we have one window for dockbar, window for the prompt, and window for the sytemsettings, this would easily goes to 800mb usage, that's why i choose this way since it only takes 200mb in total even though we have multiple windows..

Why Choose Electron ?

At first i got into Tauri since many people said it was lighter compared to Electron, but after couple of iteration and couple of benchmark using the same code, there's a major issue in Tauri, which on the rendering side.. i tested it using the same feature and same code it just a overlay click with windowing drag, and electron outperformed the tauri, especially in the case of window that occupied large area on the screen, in Tauri it easily dropped to 20-30 fps, while Electron can easily render it on 60++ fps...

Why Choose DeepAgents ?

Its also the same way as Electron, at first i was develop my own Custom ReAct loop, and turn out managing parsing, context, memory and etc are kinda horrendous to manage as a Sole Developer, and then i tried couple of iteration of DeepAgents on python at first, which turnout is really good for agentic workflow, and hugely easy to manage compared to build my own ReAct, and then after that i ditch the python and convert it into typescript environment since at that moment i want to shared the schemas between the DeepAgent, managing multiple schemas on each environment is kinda confusing, and i also didn't want the DeepAgent kinda becoming its own server but i want it built into the main electron runtime..

Why Use React ?

I also make the decision to use React is that due to extensive library and mature ecosystem on it, and also it kinda easier to handle multiple reactive components.

Why Built Kerneling, Event Bus And Ram Storage On Electron Overlay ?

The reason i kinda build the whole ecosystem of kernling, event bus, ram storage is to make every components, every window, can read on the same single source of truth, for example the window system were need a process_uid to spawn, and the information related to the windowing system are putted into the storage memory_uid which also need process_uid, why i make this way is that...

Imagine u exit the window, or closing process_uid for example, by the memory_uid and the window_uid are already tagged with process_uid, from the Kernel can run a cleaning up mechanism to clean all memory_uid and window that tagged with the process_uid..

Resulted :

All of those decision that i made even in uncompressed condition or not production build yet, still using "npm run start" for the runtime resulted in ACE can run in 60++fps and smooothness drag and resize windowing, and also allowing Agentic even can interact with resizing, moving the window.. the memory usage was also in multiple windows is 400mb which kinda small with all the feature and functionality provided, in extreme case in 12++ window and it occupies large area, the fps when dragging drop to 30fps+- but this is due to Oclusion Culling hasn't implemented yet..

After all this can be lighter after the production build already implemented and the occlusion culling mechanism too..

#Some Demontration Below

#Simple Tool Agent And Window Example :

Below is the demonstration of the overlay ui and windowing system, and the agent doing simple things such as list folders..

#Agentic Tasks Example : Organizing File And Folder

Below is the demonstration when the agent doing complex things, which the task is organizing file and folders, which are common usecase...

#Technical Deep Dive

#Overlay Desktop Architecture

ACE runs as an Electron-hosted desktop environment with a React renderer and a custom windowing layer.

• Window System: The UI is structured around draggable and focus-aware window shells, widgets, dockbar interactions, and specialized system windows for chat, settings, runtime monitoring, and internal tooling.
• Rendering Strategy: The renderer keeps high-frequency interaction state local while durable runtime state is mirrored into shared memory surfaces for orchestration and observability.
• UX Goal: The overlay is designed to stay available without fully taking over the desktop, allowing the workspace to feel ambient, inspectable, and fast.

#Background Agent Runtime with DeepAgents

The active AI layer runs in a dedicated background runtime rather than inside the renderer itself.

• Runtime Ownership: Planning, prompt orchestration, stream handling, and tool execution are handled in the background process, while the renderer focuses on interaction and presentation.
• Streaming Model: Agent lifecycle events, tool start and finish events, and assistant text are streamed back into the desktop UI to support live feedback rather than waiting for a final response blob.
• Local Context: The invoke path carries local runtime metadata such as desktop state and user context so the agent can reason about the environment it is operating inside.

#Kernel-Like Control Plane

ACE includes a control-plane style runtime that manages system memory, process lifecycle, configuration, registry loading, keybind routing, window state, and shared engine boundaries.

• Memory and Process Model: Runtime domains can register memory surfaces and process ownership through shared engine contracts, making long-lived state easier to inspect and coordinate.
• Event-Driven Coordination: Internal features communicate through engine boundaries and event-style orchestration rather than hard-wiring every UI surface together.
• Observability: System monitors and dev surfaces make it easier to inspect thread state, event flow, render counts, and runtime status while the platform is still evolving.

#Package and Tool Extensibility

One of the most important design goals in ACE is that the environment should not be closed around one hardcoded assistant experience.

• Registry Model: Packages can contribute windows, widgets, tools, and related metadata through a registry-driven architecture.
• Tool Surfaces: Filesystem actions, search flows, shell execution, and other structured capabilities can be rendered back into the UI through dedicated tool result components.
• Future Direction: This package model creates a path toward a richer internal ecosystem of desktop-native workflows, not just a single chat screen.

#Current State

ACE is already usable as a local-first experimental workspace, but it is still clearly transitional. The desktop shell, background agent runtime, control plane, and package system are real, while some contracts, schemas, and permission boundaries are still being refined.

That early-stage quality is intentional. The current phase is about proving the interaction model, validating the developer experience of an overlay-based agentic desktop, and identifying which abstractions deserve to become long-term primitives.

Want to discuss the local-first runtime, overlay UX, or background agent architecture in more detail? Let's talk.