Migration Testimony - Daniel Mats Niklas Bengtsson

Which project or component did you migrate away from Eventlet?

I worked on the oslo.service library, a core component of the OpenStack ecosystem.

It provides the foundation for managing long-running services, multi-process workers, and periodic tasks used across many OpenStack projects like Nova and Neutron.

Historically, oslo.service relied entirely on Eventlet for concurrency and networking — it was the only available backend.

I designed and implemented a new Threading backend to provide a modern, more compatible, and future-proof alternative.

How deeply was Eventlet integrated into your codebase?

Eventlet was deeply embedded in oslo.service’s architecture:

It powered green threads and cooperative multitasking.
It relied on eventlet.monkey_patch() to make standard libraries non-blocking.
Core abstractions like ThreadGroup, LoopingCall, ServiceLauncher, and ProcessLauncher were all built around Eventlet’s concurrency model.

Because of this, replacing Eventlet required re-engineering significant internal logic while ensuring we kept the same public API for consumers.

Which framework or alternative did you choose to replace Eventlet, and why?

I chose a combination of threading, futurist, and cotyledon:

threading: for native Python concurrency and better long-term support.
futurist: provides thread pools, futures, and task management for safe async execution.
cotyledon: handles process management, graceful shutdowns, and worker lifecycle.

The main reason for this choice was simplicity, maintainability, and forward compatibility.
Native threads are better supported in modern Python and avoid the issues caused by Eventlet’s monkey-patching and green-thread model.

What motivated you to start this migration?

Several factors drove this migration:

Python 3.12+ compatibility: Eventlet had serious issues with recent Python versions.
Stability: Monkey-patching caused subtle, unpredictable bugs.
Performance & maintainability: A threading-based solution is easier to reason about and debug.
Community alignment: Many OpenStack components are progressively moving away from Eventlet; this was the natural next step.

Did you have any concerns or doubts before starting?

Absolutely. My main concern was backward compatibility.
Since oslo.service is used by many OpenStack services, any behavioral change could cause major breakages.
That’s why I focused on ensuring the new Threading backend behaved identically to the Eventlet backend from the perspective of external consumers.

How did the migration process go? Where did you start?

I started by deep-diving into the Eventlet backend, analyzing how ProcessLauncher, ServiceLauncher, ThreadGroup, and LoopingCall interacted.

Then, I implemented a new backend using threading, futurist, and cotyledon — while keeping the same public interface.

Finally, I refactored the functional tests so that both Eventlet and Threading backends could be tested side by side to validate identical behavior.

What tools or strategies helped you the most?

Futurist: simplified thread pools and async task management.
Cotyledon: provided robust multi-process worker handling.
Functional test parity: I reused the same test suite for both backends to guarantee API compatibility.

Were there any particularly tricky or painful parts?

Definitely:

Removing Eventlet’s monkey-patching without breaking existing services.
Handling signal propagation and worker restarts via Cotyledon.
Ensuring LoopingCall and ThreadGroup behaved identically across both backends.
Managing Python 3.12+ “spawn” default start method compatibility, since Eventlet depended heavily on os.fork().

Roughly how long did the migration take?

From initial design to a fully working implementation, it took around three months, including testing, reviews, and refinements.

Were you able to migrate incrementally? If so, how?

Yes. We introduced the Threading backend as an alternative, not a replacement.

Users can now choose between Eventlet and Threading via a configuration flag.

This incremental approach minimized risk and allowed us to collect real-world feedback before considering Eventlet deprecation.

What concrete benefits have you seen after migrating?

Compatibility: Fully works with Python 3.12+.
Stability: Fewer unexpected bugs due to removing monkey-patching.
Testability: Much easier to debug and maintain.
Maintainability: The codebase is simpler, modern, and easier for new contributors to understand.

How did your team react to the change?

Feedback has been very positive.
Reviewers appreciated the incremental approach and especially the fact that we preserved API parity.

What advice would you give to a team that's hesitant to migrate?

Start small: Break down the migration into isolated components.
Preserve the API: Keeps downstream consumers safe.
Write tests early: Use the same suite to validate both old and new implementations.
Leverage mature libraries: futurist and cotyledon saved significant time and effort.

Is there anything you would do differently next time?

I would prepare a migration strategy document earlier, align it with the community, and validate the plan upfront.
That would have saved some back-and-forth during reviews.

Have you faced blockers?

Yes, especially regarding Python 3.12+ and multiprocessing.set_start_method("spawn").
Cotyledon required some adjustments to manage process spawning cleanly and avoid incompatibilities.

Would you like to share a link to a patch, repo, or documentation?

Sure, here are the main Gerrit reviews for this migration:

Is there anything else you'd like to share with the community about your experience?

This migration marks an important step forward for oslo.service and the OpenStack ecosystem.
By moving away from Eventlet, we’re making the library more compatible, easier to maintain, and friendlier for new contributors.
It also opens the door for future optimizations and better long-term performance, without being tied to an outdated concurrency model.

Migration Testimony: Daniel Bengtsson's Experience

Daniel Bengtsson

Technical Context