Works on my machine, breaks in production: three true stories and the cure

“It works on my machine” is as old as programming, and it still happens every day, because a developer’s machine never matches production: libraries installed long ago and forgotten, different versions, a different operating system. This article tells three real bugs from Forge, the 3D-printing design site we run, where people build adjustable models in the browser. All three share one root and one cure.

Forge is written in Python 3.11 with trimesh, shapely and manifold3d for geometry, and runs in a python:3.11-slim Docker image.

Story 1: a library only on my machine

Several trimesh functions quietly rely on optional libraries. Slicing a shape and capping the cut (slice_plane with cap=True), for example, needs a library called rtree behind the scenes.

The development machine had rtree from some earlier job. Every test passed. The production image did not have it. Once live, the word-shaped box design failed for every user:

ModuleNotFoundError: No module named 'rtree'

Nobody saw it in testing, because the tests ran on a machine that had everything.

The fix. We stopped using any function that depends on rtree in code that runs live, cutting shapes by subtracting a box (a manifold3d boolean) instead, and wrote the list of forbidden functions into the project handbook so every person and every AI helper sees it each time.

Story 2: a library that crept in and silently changed results

This one is scarier, because there was no error at all.

trimesh can split polygons into triangles in several ways. If a library called mapbox_earcut is installed, it picks that one by itself, which behaves differently on some shapes. For us, the keyboard-key generator returned nothing and the system fell back to a basic shape. Every key came out as a flat slab instead of the designed form.

No exception, no warning in the log. Just a wrongly shaped part.

Why was the library there at all? We never asked for it. It arrived with another library as an optional extra (a transitive dependency). That means simply rebuilding the image on a day another library updates can change what the site produces, without anyone touching a line of code.

The fix. Pin every library that affects geometry:

numpy==2.4.6
scipy==1.17.1
shapely==2.1.2
trimesh==5.0.0
manifold3d==3.5.2

Then write a test that confirms what must be absent really is absent:

import importlib.util

def test_forbidden_libraries_absent():
    for name in ["rtree", "embree", "mapbox_earcut"]:
        assert importlib.util.find_spec(name) is None, f"{name} must not be installed"

This test runs in the production image. If a forbidden library ever creeps in, the test fails before any user sees it.

Story 3: baselines recorded on the wrong machine

To catch changed shapes, we keep a “fingerprint” for each design, such as its volume to 6 decimals, and compare it after every change.

One day all 11 rows failed. It looked as though every shape had changed, but nothing had. That set of baselines had been recorded in an AI helper’s own virtualenv, with library versions different from the real image, so the numbers never matched to begin with.

The lesson: baselines must come from the same environment you check against. Otherwise tests warn about things that never happened, until people stop trusting the tests, which is worse than having none.

Bonus: the font that is “already there”

Forge can put text on a part. The default font comes from files bundled with matplotlib. If an image build ever strips matplotlib’s data files, or a future matplotlib stops bundling the font, every letter turns into a box.

So our test does not trust a version number; it measures a real glyph, rendering one letter and checking its area is what it should be.

The principle: test the thing that runs

If tests run in one place and the code runs in another, passing tests only tell you it works in the first place.

From the three stories, the practice is:

1. One image for testing and for production. Build once, run the tests in that image, and if they pass, ship that very image. No rebuild in between.

   docker build -f apps/forge/Dockerfile -t forge:candidate kura3d/
   docker run --rm forge:candidate python -m pytest tests/

2. Pin whatever affects results, libraries and base images alike. Use versioned tags such as python:3.11-slim, not latest, which can change under you at any time

3. Test the environment, not just the code. Check that what must be absent is absent, and that what must be present is there and works

4. Record baselines from the real image. When versions change, re-record from the new image, not from someone’s machine

5. Check after deploy with a real request. After going live, build one real part through the live site, not just confirm the page returns 200

Differences that are not libraries: the operating system

The same thing happens with operating systems. Another project of ours controls 3D printers over MQTT on Windows. The MQTT library needs one kind of Python event loop, while the server needs another to run subprocesses, and on Windows the two cannot live in one program. The connection timed out every time, even though credentials and network were fine. The same code on Linux has no such problem (the full story is in the article on connecting 3D printers with MQTT).

Common mistakes

• Running tests in a virtualenv on your own machine and calling it a pass
• Rebuilding the image after the tests pass, then shipping the new, untested image
• Using the latest tag for a base image or a database image
• Not pinning versions because “it works right now”
• Checking after deploy only that the site loads, never trying the main feature once