Skip to content

Commit 10c7614

Browse files
HyukjinKwondongjoon-hyun
authored andcommitted
[SPARK-52055][PYTHON][CONNECT] Suppress the exception in ExecutePlanResponseReattachableIterator.__del__
### What changes were proposed in this pull request? This PR proposes to suppress the exception in `ExecutePlanResponseReattachableIterator.__del__`. ### Why are the changes needed? It tries its best effort to close it. When it fails, we should just ignore it. Now when you run `./bin/pyspark --remote local` and `exit(0)` directly without doing anything, it shows a warning as below: ``` Exception ignored in: <function ExecutePlanResponseReattachableIterator.__del__ at 0x12229f7e0> Traceback (most recent call last): File "/.../python/pyspark/sql/connect/client/reattach.py", line 347, in __del__ return self.close() ^^^^^^^^^^^^ File "/.../python/pyspark/sql/connect/client/reattach.py", line 343, in close self._release_all() File "/.../python/pyspark/sql/connect/client/reattach.py", line 245, in _release_all thread_pool.submit(target) File "/opt/miniconda3/envs/python3.11/lib/python3.11/concurrent/futures/thread.py", line 169, in submit raise RuntimeError('cannot schedule new futures after ' RuntimeError: cannot schedule new futures after interpreter shutdown ``` ### Does this PR introduce _any_ user-facing change? The warning will not be shown. ### How was this patch tested? Manually via `./bin/pyspark --remote local`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50847 from HyukjinKwon/SPARK-52055. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 20e94a5 commit 10c7614

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

python/pyspark/sql/connect/client/reattach.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -344,4 +344,7 @@ def close(self) -> None:
344344
return super().close()
345345

346346
def __del__(self) -> None:
347-
return self.close()
347+
try:
348+
return self.close()
349+
except Exception:
350+
pass

0 commit comments

Comments
 (0)