Why pypdfium2 cause python3 process coredump...

[xsank]

version: pypdfium2==4.24.0
traceback:

(gdb) bt
#0  0x00007f0576a1c9d6 in CPDF_Color::~CPDF_Color() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#1  0x00007f0576a21cab in CPDF_ColorState::ColorData::~ColorData() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#2  0x00007f0576a33a5e in CPDF_GraphicStates::~CPDF_GraphicStates() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#3  0x00007f0576a523fe in CPDF_TextObject::~CPDF_TextObject() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#4  0x00007f0576a3ed05 in std::__Cr::deque<std::__Cr::unique_ptr<CPDF_ObjectWalker::SubobjectIterator, std::__Cr::default_delete<CPDF_ObjectWalker::SubobjectIterator> >, std::__Cr::allocator<std::__Cr::unique_ptr<CPDF_ObjectWalker::SubobjectIterator, std::__Cr::default_delete<CPDF_ObjectWalker::SubobjectIterator> > > >::~deque() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#5  0x00007f0576a3eb13 in CPDF_PageObjectHolder::~CPDF_PageObjectHolder() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#6  0x00007f0576a31603 in CPDF_FormObject::~CPDF_FormObject() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#7  0x00007f0576a3ed05 in std::__Cr::deque<std::__Cr::unique_ptr<CPDF_ObjectWalker::SubobjectIterator, std::__Cr::default_delete<CPDF_ObjectWalker::SubobjectIterator> >, std::__Cr::allocator<std::__Cr::unique_ptr<CPDF_ObjectWalker::SubobjectIterator, std::__Cr::default_delete<CPDF_ObjectWalker::SubobjectIterator> > > >::~deque() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#8  0x00007f0576a3eb13 in CPDF_PageObjectHolder::~CPDF_PageObjectHolder() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#9  0x00007f0576a3af0e in CPDF_Page::~CPDF_Page() ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/site-packages/pypdfium2_raw/libpdfium.so
#10 0x00007f066bd90052 in ffi_call_unix64 ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/lib-dynload/../../libffi.so.8
#11 0x00007f066bd8e925 in ffi_call_int ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/lib-dynload/../../libffi.so.8
#12 0x00007f066bd8f06e in ffi_call ()
   from /home/admin/hippo/worker/slave/drogo-share_papermage_L40S_server_pre.papermage_L40S_server_pre_11_51/binary/papermage/lib/python3.9/lib-dynload/../../libffi.so.8
#13 0x00007f066bda01e0 in _call_function_pointer (argtypecount=<optimized out>, argcount=1,
    resmem=0x7ffc89500950, restype=<optimized out>, atypes=<optimized out>, avalues=<optimized out>,
    pProc=0x7f0576b45c70 <FPDF_ClosePage>, flags=4353)
    at /usr/local/src/conda/python-3.9.18/Modules/_ctypes/callproc.c:920
#14 _ctypes_callproc (pProc=0x7f0576b45c70 <FPDF_ClosePage>,
    argtuple=(<LP_struct_fpdf_page_t__ at remote 0x7f04db6fabc0>,), flags=4353, argtypes=<optimized out>,
    restype=None, checker=0x0) at /usr/local/src/conda/python-3.9.18/Modules/_ctypes/callproc.c:1263
#15 0x00007f066bda96ee in PyCFuncPtr_call (self=<optimized out>, inargs=<optimized out>, kwds=0x0)
    at /usr/local/src/conda/python-3.9.18/Modules/_ctypes/_ctypes.c:4201
#16 0x00000000004f073c in _PyObject_MakeTpCall (tstate=0x771440,
    callable=<_FuncPtr(__name__='FPDF_ClosePage') at remote 0x7f05790eed00>, args=<optimized out>,
    nargs=<optimized out>, keywords=0x0) at /usr/local/src/conda/python-3.9.18/Objects/call.c:191
#17 0x00000000004ecc93 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x2bb65088,
    callable=<_FuncPtr(__name__='FPDF_ClosePage') at remote 0x7f05790eed00>, tstate=<optimized out>)
    at /usr/local/src/conda/python-3.9.18/Include/cpython/abstract.h:116
#18 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x2bb65088,
    callable=<_FuncPtr(__name__='FPDF_ClosePage') at remote 0x7f05790eed00>, tstate=<optimized out>)
    at /usr/local/src/conda/python-3.9.18/Include/cpython/abstract.h:103
#19 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x2bb65088,
    callable=<_FuncPtr(__name__='FPDF_ClosePage') at remote 0x7f05790eed00>)
    at /usr/local/src/conda/python-3.9.18/Include/cpython/abstract.h:127
#20 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x771440)
    at /usr/local/src/conda/python-3.9.18/Python/ceval.c:5077
#21 _PyEval_EvalFrameDefault (tstate=<optimized out>,

python traceback:

(gdb) py-bt
Traceback (most recent call first):
  File "/home/admin/test_parse/lib/python3.9/site-packages/pypdfium2/_helpers/page.py", line 38, in _close_impl
    pdfium_c.FPDF_ClosePage(raw)
  File "/home/admin/test_parse/lib/python3.9/site-packages/pypdfium2/internal/bases.py", line 47, in _close_template
    close_func(raw, *args, **kwargs)
  File "/home/admin/test_parse/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  Garbage-collecting
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/psparser.py", line 590, in end_type
    objs = [obj for (_, obj) in self.curstack]
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/psparser.py", line 627, in nextobject
    (pos, objs) = self.end_type("d")
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/pdfdocument.py", line 840, in _getobj_parse
    (_, obj) = self._parser.nextobject()
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/pdfdocument.py", line 866, in getobj
    obj = self._getobj_parse(index, objid)
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/pdftypes.py", line 106, in resolve
    return self.doc.getobj(self.objid)
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/pdftypes.py", line 118, in resolve1
    x = x.resolve(default=default)
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/pdfpage.py", line 60, in __init__
    self.resources: Dict[object, object] = resolve1(
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfminer/pdfpage.py", line 124, in create_pages
    yield cls(document, objid, tree, next(page_labels))
  File "/home/admin/test_parse/lib/python3.9/site-packages/pdfplumber/pdf.py", line 133, in pages
    for i, page in enumerate(PDFPage.create_pages(self.doc)):
  File "/home/admin/test_parse/core/parsers/pdf/image_parser.py", line 41, in parse_image_by_pdfplumber
    for i, (page, mpage) in enumerate(zip(doc.pages, [mdoc[i] for i in range(mdoc.page_count)])):
  File "/home/admin/test_parse/core/parsers/pdf/image_parser.py", line 178, in parse
    self.parse_image_by_pdfplumber(context, entities)

code:

[mara004]

pypdfium2==4.24.0

Please first update to 4.28.0 and see if the problem persists. We don't support non-latest versions.

Also, the traceback on its own doesn't tell me much. Can you paste the calling code? Is it constantly reproducible, or a random crash? Does it happen only with specific documents (can you share a sample)?

[mara004]

Also, which version of pdfplumber is this?

[xsank]

multi-process running pdfplumber.pages would cause this problem. The larger num of processes, the problem more likely to occur.

[xsank]

pdfplumber is 0.10.3

[mara004]

multi-process running pdfplumber.pages would cause this problem. The larger num of processes, the problem more likely to occur.

Reminds me of the random stability issues I had with the fork multiprocessing start method.
AFAIK, this is a python/OS issue and the default will change in python 3.14.
You can try to explicitly use spawn and see if that fixes the issue.

[xsank]

I have used the spawn type:

multiprocessing.set_start_method("spawn")

and i do not use the process executor pool, use the process singly.

[xsank]

it seems that when the auto close page was gc, some ~ CPDF_Color is not thread safe?

[mara004]

Nothing in pdfium is thread-safe, see #303 (comment). This is intentional / WAE.
However, our weakref finalizers are supposed to be executed safely one after another in reverse order to creation (quite unlike __del__).

[xsank]

I will try to write a simple demo to reocurr the problem 100%

[xsank]

Hi，i have solved the problem...

the old version pdfplumber do not close pdfium_doc manully...
thanks again btw.

[mara004]

Hmm, but that should be a somewhat unrelated matter.
It maybe changes runtime circumstances, but this is not a fix. There must have been some other causing factor here, which I'd like to identify (although I don't think this will be a pypdfium2 issue, rather a usage mistake (threading?), or maybe an already-fixed bug in an older version of pypdfium2).
Again, I'd like to see a piece of caller code and sample document that triggers this issue.

Also, this pdfplumber comment is misleading. We talked about that yesterday, see jsvine/pdfplumber#1089 (comment) jsvine/pdfplumber#1072 (comment)