History of the question
This topic originates from Python's evolution from pure reference counting to a hybrid garbage collection model introduced in Python 2.0. The core issue emerged when developers used finalizer methods (__del__) to manage external resources like file handles or network sockets. When objects with finalizers formed circular references, Python could not determine a safe destruction order, potentially causing crashes or resource leaks. This limitation led to the implementation of the cyclic garbage collector module (gc) and the special handling of "uncollectable" garbage.
The problem
When a group of objects forms a reference cycle and at least one defines a custom __del__ method, Python faces a deterministic destruction dilemma. The interpreter cannot decide which object to finalize first because the cycle implies mutual dependency, and destroying one might leave others in an invalid state. Consequently, Python moves these objects to the gc.garbage list rather than freeing their memory. This behavior persists in modern versions when finalizers prevent safe collection, leading to gradual memory leaks in long-running applications.
The solution
The definitive solution involves avoiding __del__ methods entirely in favor of context managers (with statements) or weakref callbacks for resource cleanup. If finalizers are unavoidable, explicitly break reference cycles before objects become unreachable by setting instance variables to None in cleanup methods. Starting with Python 3.4, the garbage collector can collect cycles with finalizers in many cases by carefully ordering finalization, but explicit resource management remains the most reliable pattern.
import gc class Resource: def __init__(self, name): self.name = name self.peer = None def __del__(self): print(f"Cleaning up {self.name}") # Creating a cycle with finalizers a = Resource("A") b = Resource("B") a.peer = b b.peer = a # Remove external references del a, b gc.collect() print(f"Uncollectable: {gc.garbage}") # May contain objects in complex scenarios
We maintained a high-throughput data processing pipeline where Node objects represented computational steps in a graph. Each node held references to its neighbors and contained a __del__ method to release GPU memory handles. During intensive workloads, we observed monotonic memory growth despite no apparent memory leaks in profiling. Investigation revealed that complex graph topologies created reference cycles between nodes, and the presence of __del__ methods prevented the cyclic GC from reclaiming these objects, causing them to accumulate in gc.garbage until process termination.
Solution 1: Refactor to context managers
We considered replacing __del__ with explicit acquire() and release() methods called via context managers. This approach would eliminate the finalizer barrier to garbage collection entirely and provide deterministic resource cleanup. However, this required modifying thousands of lines of graph construction code and risked resource leaks if developers forgot to wrap node usage in with blocks, especially in legacy callback-based components.
Solution 2: Implement weak references for graph edges
We explored changing all neighbor references to weakref.ref objects, which would allow nodes to be collected immediately when no external references remained regardless of graph connectivity. While elegant, this introduced significant complexity because the graph traversal algorithms needed to constantly check for dead weak references and handle transient "ghost" nodes during iteration. This approach substantially degraded performance for our use case and required extensive refactoring of the graph traversal logic.
Solution 3: Explicit cycle breaking via cleanup protocol
We implemented a destroy() method that explicitly set self.neighbors = [] and self.gpu_handle = None before removing nodes from the graph. This broke cycles deterministically while keeping the existing API surface intact. We chose this solution because it localized changes to the node removal logic rather than spreading concerns across the entire codebase, and it maintained backward compatibility with existing graph algorithms.
Result
After implementing the explicit cleanup protocol and adding assertions to verify gc.garbage remained empty during CI testing, memory usage stabilized at a constant baseline. The service ran for weeks without the previous gradual memory accumulation. We also documented the pattern to ensure future developers understood the interaction between finalizers and cyclic references.
Why does gc.garbage still contain objects in Python 3.4+ even when finalizers are present in cycles?
While Python 3.4 significantly improved the cyclic GC to handle finalizers by invoking them in a safe order and clearing references afterward, objects may still appear in gc.garbage under specific conditions. If a __del__ method resurrects the object by storing it in a global variable, the GC cannot safely collect the cycle and moves it to gc.garbage to prevent infinite loops. Additionally, C extension objects with custom tp_dealloc slots that do not properly support the cyclic GC protocol may be treated as uncollectable to avoid crashes in native code.
How does weakref.ref with a callback interact with the cyclic garbage collector when the referent is part of an uncollectable cycle?
Candidates often incorrectly assume that weak reference callbacks fire immediately when an object becomes unreachable. In reality, the callback fires when the object is actually destroyed and its memory deallocated. If an object participates in a reference cycle containing finalizers that the GC cannot break, the object remains allocated in gc.garbage and the weak reference callback never executes. This distinction is crucial for designing resource cleanup systems that rely on weak reference callbacks for notification of object destruction.
What is the "resurrection" problem in __del__ methods and how does it prevent garbage collection of circular references?
Resurrection occurs when a finalizer method assigns the dying instance to a global variable or inserts it into a persistent container, effectively reviving it after the GC has marked it for destruction. In a circular reference scenario, if one object's __del__ resurrects any object in the cycle, the entire cycle becomes reachable again. Python's garbage collector detects this anomaly and moves the entire cycle to gc.garbage rather than attempting to resolve the potentially infinite loop of destruction and resurrection, leaving the memory unreclaimed until process termination.