Using a custom sysimage with PythonCall causes error during juliacall initialization

[brian-dellabetta]

Affects: Both

Describe the bug
I would like to use a custom sysimage of our julia package which has PythonCall.jl as a dependency. The image builds fine, and is useable with julia --sysimage=/path/to/myimage.so, but when I try to use it as the custom sysimage with juliacall, I hit the following initialization error:

PYTHON_JULIACALL_SYSIMAGE=/path/to/myimage.so python
>>> from juliacall import Main as jl
fatal: error thrown and no exception handler available.
InitError(mod=:C, error=AssertionError(msg="CTX.which == :PyCall"))
#35 at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:144
with_gil at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/gil.jl:10 [inlined]
with_gil at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/gil.jl:9 [inlined]
init_context at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/context.jl:141
__init__ at /home/xxx/.julia/packages/PythonCall/wXfah/src/cpython/CPython.jl:21
jfptr___init___72284 at /home/xxx/.julia/sysimages/myimage.so (unknown line)
_jl_invoke at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/julia.h:1880 [inlined]
jl_module_run_initializer at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:75
_finish_julia_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/init.c:855
julia_init at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/init.c:804
ijl_init_with_image at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:66 [inlined]
ijl_init_with_image at /cache/build/builder-amdci4-4/julialang/julia-release-1-dot-9/src/jlapi.c:55
unknown function (ip: 0x7f29a0a18d1c)
unknown function (ip: 0x7f29a0a18288)
unknown function (ip: 0x7f29a0a2bab9)
unknown function (ip: 0x7f29a0a2c1b2)
_PyObject_MakeTpCall at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a13df561)
PyEval_EvalCode at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a1447fbd)
unknown function (ip: 0x7f29a14304f7)
_PyEval_EvalFrameDefault at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a13df561)
unknown function (ip: 0x7f29a13f45bc)
PyObject_CallMethodObjArgs at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
PyImport_ImportModuleLevelObject at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a13df561)
PyEval_EvalCode at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a145f1fc)
unknown function (ip: 0x7f29a145f195)
unknown function (ip: 0x7f29a13c6eb6)
_PyRun_InteractiveLoopObject at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a1311c71)
PyRun_AnyFileExFlags at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
unknown function (ip: 0x7f29a130e572)
Py_BytesMain at /usr/local/bin/../lib/libpython3.11.so.1.0 (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at python (unknown line)

I tried a sysimage that didn't have PythonCall imported or as a dependency, and juliacall initalized as expected. Maybe this isn't possible, but the Assertion error is occuring at this line, and I'm not using PyCall anywhere, so perhaps it is a bug in init_context?

Steps to reproduce

In a fresh directory, create

1. build.jl:

using Pkg

Pkg.add("PythonCall")

using PythonCall

Pkg.add("PackageCompiler")
using PackageCompiler

create_sysimage(:PythonCall, sysimage_path="/home/xxx/.julia/sysimages/myimage.so", precompile_execution_file="precompile.jl")

2. precompile.jl:

using PythonCall

call julia --project=. build.jl (preferably in a fresh project) and then PYTHON_JULIACALL_SYSIMAGE=/home/xxx/.julia/sysimages/myimage.so python -c 'from juliacall import Main as jl'

Your system
Please provide detailed information about your system:

• Debian 11 (Docker container on MacOS 14.2, sysimage builds on host were failing for me)
• Julia 1.10, Python 3.11, PythonCall 0.9.15, juliacall 0.9.15
  -Pkg.status() after running build.jl:

Status `~/.julia/environments/v1.10/Project.toml`
  [9b87118b] PackageCompiler v2.1.16
  [6099a3de] PythonCall v0.9.15

• juliapkg.status().

/home/xxx/julia_depot/environments/pyjuliapkg/pyjuliapkg/juliapkg.json
Julia 1.10.0 @ /home/xxx/julia/bin/julia
Packages:
  <ourpackage>: {'uuid': 'xxx', 'dev': True, 'path': '/path/to/ourpackage.jl'}

Additional context
Add any other context about the problem here.

[cjdoris]

Thanks for the report. I suspect the issue is that some state is being saved in the sysimage, i.e. __init__ sets some variables somewhere, and these values are what are saved in the sysimage, which is confusing __init__ because they are not their default values. So probably __init__ needs to explicitly set these to the default.

[brian-dellabetta]

Thanks @cjdoris for the reply. I will try setting the default values in __init__ later this week, hopefully it's a quick fix

[brian-dellabetta]

I added some printlns to PythonCall.jl (see diff here) to get some more information. I thought it would be a matter of setting CTX, but the issue is more subtle than that.

Running from juliacall import Main as jl without sysimage, CTX.is_embedded is true.
Running the same with PYTHON_JULIACALL_SYSIMAGE set, it is false and continues down the function until it fails on this line.

This makes sense, the flag gets set to true normally because __PythonCall_libptr is set in the startup script before the using PythonCall line, but it's false when running with a sysimage because the PythonCall __init__ gets called at julia startup, before any other commands are run.

I'm not sure what the best solution is here. Maybe if the sysimage env var is set, consider it embedded and exit out of the init?

CTX.is_embedded = haskey(ENV, "JULIA_PYTHONCALL_SYSIMAGE") ? true : hasproperty(Base.Main, :__PythonCall_libptr)

There might be edge cases where this isn't desired.