Eventlet Removal Logo
Eventlet Removal

Eventlet Removal PTG Retrospective - Gazpacho to Hibiscus

Report created by Hervé Beraud on April 27, 2026

This report presents a comprehensive analysis of the eventlet removal office hours session held during the April 2026 Project Teams Gathering (PTG), covering the retrospective from the Gazpacho cycle and outlining the call to action for Hibiscus.

Session Recording

Resources: Session Etherpad

Gazpacho Retrospective 🔗

The Gazpacho cycle marked a period of significant progress in the eventlet removal initiative, with foundational work completed in core libraries and substantial advances across multiple OpenStack projects. The session began with updates from the eventlet library itself and Oslo services, which provide critical infrastructure for the broader migration effort.

Core Libraries Progress

The core libraries that underpin OpenStack's concurrency infrastructure made important strides during Gazpacho, particularly in Python version compatibility and multiprocessing support:

Eventlet Library

  • ✅ Upgraded from Python 3.14 alpha to stable
  • ✅ Dropped Python 3.9 support (now 3.10-3.14)
  • 🔄 Monitoring Python 3.15 (waiting for stable)
  • ✅ Strengthened asyncio integration
  • ✅ Fixed Python 3.14 compatibility with asyncio
  • ✅ Refactored patching logic for API stability

Oslo Services

  • ✅ Introduced multiprocessing spawn support
  • ✅ Reduced fork instability in multi-threaded environments
  • ✅ Two patches merged
  • 🔄 One patch pending (requires POC)
  • Primary benefit: Manila stability improvements

Project-by-Project Progress 🔗

The Gazpacho cycle saw remarkable progress across the OpenStack project landscape, with three projects achieving complete migration away from eventlet and many others making substantial advances toward threading-based concurrency. The migration efforts varied in approach, with some projects opting for gradual service-by-service transitions while others pursued comprehensive migrations. This diversity in strategies has provided valuable lessons for the community.

✅ Completed Migrations

Three projects reached the significant milestone of completing their eventlet removal during the Gazpacho cycle, joining the ranks of earlier pioneers like Octavia, Mistral, and Ironic. These successes demonstrate that full migration is achievable across different types of OpenStack services:

Designate

Status: COMPLETE

  • Full migration finished
  • Major milestone achieved in Gazpacho

Neutron

Status: COMPLETE

  • Moved Oslo services to threading
  • Removed all eventlet imports
  • Fully operational

Cyborg

Status: EVENTLET-FREE

  • Removed monkey patch
  • Fixed DB deadlock
  • Native WSGI server removed

🟢 Significant Progress

Several major OpenStack projects have made substantial progress in their migrations, with many services already running in threading mode by default. These projects represent the bulk of the community's migration effort and are demonstrating various successful approaches to the transition. Nova's gradual service-by-service strategy and Manila's performance validation framework stand out as particularly valuable contributions:

Nova

Status: MOSTLY COMPLETE

Gazpacho Defaults to Threading:

  • ✅ API service
  • ✅ Metadata API
  • ✅ Scheduler

Hibiscus Merged Defaults:

  • ✅ Compute
  • ✅ Conductor

Pending:

  • Console proxies (noVNC, spice, serial)
  • CLI gaps
  • Functional tests conversion

Manila

Status: ADVANCED

  • ✅ Environment option for threading
  • ✅ Share manager migrated
  • ✅ Rally performance framework developed
  • ✅ No significant performance regression
  • 🔄 Planning functional tests for Hibiscus
  • 🔄 Considering eventlet default switch

Cinder

Status: GOOD SHAPE

  • ✅ Volume and scheduler operational
  • ✅ Aligned with Oslo services
  • 🔄 Backup service in progress
  • 🔄 Multiple patches targeting Hibiscus

Glance

Status: GOOD SHAPE

  • ✅ Removed eventlet from utils
  • ✅ Unit tests use native threading
  • 🔄 Client migration pending (waiting for consumers)

Watcher

Status: ADVANCED

  • ✅ All components migrated (API, Decision Engine, Applier)
  • Switched to threading by default in early Hibiscus
  • ✅ CI passing with eventlet code deleted
  • 🔄 Considering full removal in 2027 or end of cycle

🟡 In Progress

Swift continues its migration journey with impressive test coverage already achieved. The project faces unique technical challenges related to erasure coding that require careful consideration of concurrency behavior. The team is working through these challenges systematically while maintaining the high reliability standards Swift operators expect:

Swift

Status: IN PROGRESS

  • ✅ 100% functional tests passing
  • ✅ 99% unit tests passing
  • ⚠️ Remaining 1% contains hardest problems
  • 🔧 Erasure coding predictability challenges
  • 🔧 Evaluating test assumptions vs. real requirements

Call to Action for Hibiscus 🔗

Building on the momentum established during Gazpacho, the Hibiscus cycle represents a critical phase in the eventlet removal initiative. The focus shifts from initial migration work to defaulting projects to threading mode, expanding test coverage to include Python 3.14, and addressing remaining technical blockers. The community has identified five primary goals that will guide the work throughout this cycle, along with a clear migration timeline strategy that balances progress with stability.

Primary Goals

The Hibiscus cycle work is organized around five interconnected objectives that address technical migration, testing infrastructure, library cleanup, and operator guidance:

1. Continue Migration Momentum

  • Projects not yet migrated maintain active development
  • Default to threading as soon as technically feasible

2. Gradual Default Switching

  • Release N: Both modes (eventlet default)
  • Release N+1: Both modes (threading default)
  • Release N+2: Remove eventlet support

3. Testing Infrastructure

  • Convert functional tests to threading mode
  • Expand Python 3.14 testing coverage
  • Prepare for Ubuntu 26.04 (release: April 29, 2026)
  • Rally performance comparison jobs

4. Library Cleanup

  • Address unconditional eventlet imports
  • Make eventlet dependencies optional
  • Clean up transitive dependencies

5. Documentation

  • Thread pool sizing guidance for operators
  • Performance tuning recommendations
  • Migration paths for deployments

Example Migration Timeline

The recommended approach to migration follows a three-release pattern that allows for gradual adoption while maintaining stability for operators. This strategy provides a fallback mechanism during the transition period and gives operators time to adapt their deployments. Here's how Nova is applying this pattern to their console proxy services:

Nova Console Proxies:

  • Hibiscus: Support threading (eventlet default)
  • Next cycle: Switch to threading default
  • Following cycle: Remove eventlet support

Concerns and Noticed Problems 🔗

While significant progress has been made, the PTG session identified several technical challenges that require attention during the Hibiscus cycle and beyond. These issues range from critical blockers that could affect Python version compatibility to medium-priority concerns about library dependencies and testing infrastructure. Understanding and addressing these challenges is essential for maintaining the migration timeline and ensuring stable deployments.

🔴 Critical Issues

Three critical issues emerged that require immediate attention. These problems have the potential to block progress on Python version upgrades or affect multiple projects simultaneously. The community is actively working on solutions, but these challenges represent the highest priority concerns for the Hibiscus cycle:

1. Picklable Problem (Multiprocessing Spawn)

Impact: High | Affected: Neutron, Nova, Multiple projects

Issue:

  • Objects passed to multiprocessing spawn must be picklable
  • Config objects from Oslo.config are not picklable
  • Python 3.14 changes default context to fork_server
  • fork context being deprecated (removal timeline unknown)
  • Warning system implemented but fixes pending

Actions Needed:

  • Investigate common patterns for making objects picklable
  • File bugs in Oslo.config if root cause confirmed
  • Test spawn method compatibility across projects
  • Determine if fork_server is acceptable fallback

2. Root Wrap Compatibility

Impact: High | Affected: Nova, Cinder (via OS-brick)

Issue:

  • Root wrap may not work properly in threaded mode
  • OS-brick uses root wrap and is consumed by multiple services
  • Services can be eventlet-free but blocked by library dependencies
  • Proof uses pickle for transport over Unix sockets (related to picklable issue)

Actions Needed:

  • Test root wrap in threading mode
  • Identify workarounds or alternatives
  • Coordinate with OS-brick maintainers

3. Heartbeat Performance Regression

Impact: Medium | Status: Solution identified

Issue:

  • Half-second sleep in while-true loop causes thread blocking
  • Only affects heartbeat_in_pthread=true deployments
  • Feature being deprecated/removed anyway

Actions Agreed:

  • Revert change in master
  • Handle stable branch backports carefully
  • Investigate TCP keepalive as alternative (already working for Tooz/OBS)

🟡 Medium Priority Issues

Beyond the critical blockers, several medium-priority issues were identified that affect code quality, test coverage, and operator experience. While these don't immediately block the migration, they represent technical debt that should be addressed to ensure a smooth transition and maintain long-term code maintainability:

4. Library Eventlet Dependencies

Impact: Medium

  • Oslo.service still depends on eventlet as default
  • Oslo.log unconditionally imports eventlet (pipe mutex fix)
  • Oslo.concurrency checks eventlet at import time
  • Multiple libraries have unnecessary imports

Actions: Audit libraries, convert to conditional imports, make eventlet optional

5. Long Synchronous API Calls

Impact: Medium | Affected: Nova, potentially others

  • Eventlet: blocks one greenthread (cheap)
  • External WSGI + threading: blocks entire process (expensive)
  • Nova fixed volume attach/detach
  • Interface detach still pending

Actions: Audit APIs, convert to async, document operator impacts

6. Python 3.14 Testing Coverage

Impact: Medium

  • Most CI on Python 3.12 (Ubuntu 24.04)
  • Nova testing on 3.13 (Debian) for eventlet-free jobs
  • Limited 3.14 testing despite compatibility work
  • Ubuntu 26.04 releases April 29, 2026

Actions: Enable 3.14 in devstack, test threading-default services

Known issue: sudo-rs (Rust sudo) doesn't launch properly - use GNU version

7. Glance Client Migration Timing

Impact: Low

  • Has eventlet checks that appear unnecessary
  • Imports explicit eventlet classes if patcher applied
  • Likely not needed (patching happens automatically)
  • Waiting for consuming services first

Actions: Test removal in tips job, coordinate with Nova/Cinder

🟣 Architectural Concerns

Looking beyond immediate technical issues, the community discussed longer-term architectural questions about the future of concurrency in OpenStack. These concerns don't require immediate action but represent important strategic considerations that will shape the project's evolution after eventlet is fully removed:

8. Async.io Pressure from Libraries

Impact: Low (future concern)

  • Some libraries forcing async.io only (Pyro2, Blazer)
  • Pyro2 impacts Swift and Nova
  • SQLAlchemy and Oslo.db support async.io (nothing using it yet)

Community Position:

Avoid three runtimes (eventlet + threading + async.io). Complete eventlet removal first, then evaluate async.io adoption.

9. Timeline Alignment Challenges

Impact: Low (process concern)

  • Projects moved at different paces
  • Some eventlet-free for years (Placement, Keystone, Ironic)
  • Can't enforce uniform "all support eventlet in 2027.1"

Agreed Timeline:

  • 2026.1: Threading potentially supported everywhere (eventlet still works)
  • 2027.1: Threading supported everywhere (ideally default, eventlet may work)
  • 2027.2: Eventlet removal from Oslo and dependencies

Performance Testing Insights 🔗

One of the most significant achievements during the Gazpacho cycle was the development of performance testing frameworks that provide concrete data about the impact of migrating from eventlet to native threading. These insights have been crucial in building confidence that the migration can be completed without significant performance degradation. Different projects have encountered varying challenges based on their specific workload characteristics and architectural patterns.

Manila Rally Framework

  • ✅ Developed reusable job template (Grenade-based pattern)
  • ✅ Runs: stable (eventlet) → upgrade → new (threading)
  • ✅ Generates comparative reports
  • Finding: No significant performance differences
  • ✅ Framework adaptable for other projects

Key Result: Performance parity achieved

Nova Considerations

  • Thread pool sizing affects memory usage significantly
  • Native threads more expensive than greenthreads
  • Working to tune defaults for similar performance + memory
  • Developing operator tuning documentation

Focus: Tuning thread pools for production workloads

Swift Challenges

  • Erasure coding requires predictable request ordering
  • Fragment reconstruction depends on query sequencing
  • Eventlet co-routine switching is deterministic
  • Threading introduces non-determinism
  • Evaluating where predictability is actually required vs. test artifacts

Challenge: Distinguishing real requirements from test assumptions

Notable Achievements 🔗

The Gazpacho cycle delivered several significant milestones that demonstrate the viability and momentum of the eventlet removal initiative. These achievements span complete project migrations, successful default switching to threading mode, development of reusable testing infrastructure, and validation of performance parity. The successes provide both concrete progress and valuable lessons that benefit the entire community.

✅ Complete Migrations

  • Designate
  • Neutron
  • Cyborg

🔄 Default Threading

  • Nova (5/7 services)
  • Watcher (all components)

🧪 Testing Infrastructure

  • Rally comparison framework
  • Performance parity validated

🐍 Python Support

  • Eventlet 3.14 compatible
  • Monitoring 3.15

📊 Performance

  • No major regressions
  • Manila shows parity

📚 Knowledge Sharing

  • Reusable patterns
  • Cross-project collaboration

Next Steps Summary 🔗

The eventlet removal initiative follows a phased timeline that balances aggressive progress with operational stability. The roadmap is organized into three time horizons: immediate actions for the Hibiscus cycle, medium-term objectives targeting the 2027.1 release, and long-term goals for 2027.2 and beyond. This structured approach ensures that the community maintains momentum while giving operators adequate time to adapt their deployments and providing fallback options during the transition.

Immediate (Hibiscus Cycle)

  • ☐ Nova: Complete console proxy migration
  • ☐ Address picklable warnings (file bugs, develop patterns)
  • ☐ Revert heartbeat sleep patch
  • ☐ Enable 3.14 testing when Ubuntu 26.04 releases
  • ☐ Convert more functional tests to threading mode
  • ☐ More projects default to threading

Medium Term (2027.1)

  • ☐ All projects support threading mode
  • ☐ Threading is default for most/all projects
  • ☐ Eventlet still available but not recommended
  • ☐ Comprehensive operator documentation

Long Term (2027.2+)

  • ☐ Remove eventlet from Oslo libraries
  • ☐ Remove transitive eventlet dependencies
  • ☐ Evaluate async.io adoption where beneficial
  • ☐ Consider OpenStack SDK migration for clients

Resources

The eventlet removal initiative maintains several resources to support contributors and track progress. These include session recordings and notes from PTG discussions, official documentation and governance materials, and development tracking tools. The community encourages all stakeholders to engage through these channels to stay informed and contribute to the effort.

Conclusion

The Gazpacho cycle demonstrated significant progress across the OpenStack ecosystem in removing eventlet dependencies. With three projects fully migrated (Designate, Neutron, and Cyborg) and major advances in Nova, Manila, Watcher, and others, the community has established clear momentum. The completion of these migrations proves that the initiative is technically feasible and provides concrete examples for other projects to follow.

The development of reusable testing infrastructure, particularly Manila's Rally comparison framework showing performance parity, provides confidence that this transition can be achieved without significant performance degradation. Nova's gradual service-by-service approach and Watcher's successful switch to threading by default in Hibiscus demonstrate multiple viable migration strategies.

Key challenges remain, particularly around picklable objects for multiprocessing spawn, root wrap compatibility, and library dependency cleanup. However, with Ubuntu 26.04 bringing Python 3.14 by default and the deprecation timeline for eventlet becoming clearer, the urgency and path forward are well understood.

The Hibiscus cycle represents a critical phase where the focus shifts from initial migration to defaulting to threading mode, expanding test coverage, and addressing the remaining technical blockers. The community's collaborative approach through dedicated office hours sessions and the #openstack-eventlet-removal channel continues to facilitate knowledge sharing and coordinated progress toward a threading-based future for OpenStack.