JoyDemo Cuts Bugs 95% by Coding Directly on Production

Overview

A software development team behind the product demonstration platform JoyDemo has reported a dramatic reduction in production bugs, attributing the improvement to a novel workflow where AI coding occurs directly on the production infrastructure. By eliminating the traditional handoff between separate development and production environments, the company claims to have cut its bug count by approximately 95 percent.

This approach challenges long-standing industry conventions that strictly segregate development and production servers. Instead, the team utilizes isolated Git worktrees on the same host that runs the live application. This allows their AI assistants, including the Codex CLI, to operate within the exact runtime environment and dependency stack used by customers.

The move comes after inspiration drawn from independent developers who advocated for coding directly on production servers. However, JoyDemo adapted the concept for team scalability, introducing rigorous safeguards such as automated testing, preview environments, and strict isolation between individual developer sessions and the live app code.

The End of Environment Drift

For years, the standard software development lifecycle has involved building code in one environment and deploying it to another. This process, often referred to as environment drift, occurs when differences in dependencies, configuration settings, or runtime versions cause code that passes local tests to fail in production. JoyDemo identified this handoff as a primary source of reliability issues, particularly problematic for a product focused on software demonstrations where uptime and flawless execution are critical.

To address this, the team restructured their workflow so that AI agents work in the same runtime context as the live product. This ensures that any code generated or modified by AI is immediately compatible with the production environment. There is no need to translate or adapt changes for a different setup, thereby removing a significant category of potential errors.

"We used to build in one environment and deploy to another. A change could pass in development, then fail in production because the dependencies, settings, or runtime were different. That handoff was a steady source of bugs."

Despite the shift to production-hosted development, the team maintains a clear separation between the code currently serving customers and the code being modified. Changes are not pushed directly to the main branch. Instead, they are contained within isolated workspaces, tested extensively, and reviewed before being promoted to the live application. This ensures that the reliability of the product remains uncompromised while benefiting from the consistency of a unified environment.

Scaling for Teams: Isolated Worktrees

A common concern with developing directly on production servers is the risk of multiple developers interfering with each other’s work or destabilizing the live service. JoyDemo solved this by implementing a model where each team member operates in a separate Git worktree on the production host. A worktree is essentially an independent checkout of the repository that shares the same object database but maintains a separate working directory.

This architecture allows each developer to have their own branch and AI coding session without colliding with the files currently in use by the live application or by other team members. Each worktree can run its own preview process, leveraging the full power and service access of the production host. This gives developers a realistic sandbox to validate changes while ensuring the live app’s codebase remains untouched and stable.

The workflow supports parallel development. Multiple team members can work on different features or fixes simultaneously on the same machine. The critical boundary is no longer between a developer’s laptop and a production server, but between an individual’s workspace and the shared live code. Once a change is ready, the developer runs the full test suite and reviews the preview. Only after this review is the change promoted to the live app.

If a bug does escape into production, the remediation process is significantly accelerated. Because the development environment is identical to the production environment, a developer can reproduce the issue on the host, implement a fix in their personal worktree, and verify it through tests and previews. This eliminates the lengthy cycle of trying to recreate production conditions on a local machine or a staging server, allowing the team to release fixes within minutes rather than hours or days.

Safeguards and Strategic Inspiration

The decision to adopt this high-risk, high-reward model was influenced by public discussions from prominent figures in the tech industry, including Pieter Levels and Amjad Masad. These discussions highlighted the friction caused by environment differences and questioned whether the risks of production development could be mitigated with the right safeguards. JoyDemo’s leadership concluded that the benefits of consistency outweighed the risks, provided that robust safety mechanisms were in place.

The safeguards implemented include extensive automated testing and mandatory preview reviews. No change reaches the live app without passing these checks. Furthermore, the AI is not given unchecked autonomy. While AI assists heavily with implementation, the human team retains ultimate decision-making power. Developers decide what to build, review every change, and own the final output that reaches customers.

"I don't mean that AI never gets things wrong. I attribute the improvement to removing environment differences and checking each change before it ships."

This approach does not eliminate all bugs, but it drastically reduces those caused by environmental mismatch. For a product like JoyDemo, where reliability is paramount for customer demonstrations and training, this consistency is invaluable. The ability to investigate and fix issues rapidly ensures that teams can continue to showcase software effectively without downtime.

As the software industry continues to explore AI-assisted development, workflows like JoyDemo’s offer a compelling case study. By integrating AI coding directly into the production runtime while maintaining strict isolation and review processes, teams may find that the traditional barriers between development and production are no longer necessary. The result is a faster, more reliable development cycle that aligns closely with the reality of the deployed product.