diff options
author | Linus Groh <mail@linusgroh.de> | 2021-01-18 20:26:26 +0100 |
---|---|---|
committer | Andreas Kling <kling@serenityos.org> | 2021-01-18 22:28:56 +0100 |
commit | 39af1f8519d4a33a90ae79e5b3277652aac85270 (patch) | |
tree | a6243fb69b1c10c580e97514d44ad0dd08f83b7a | |
parent | c46056122a49aa5789c44aca9567e729d175cbbe (diff) | |
download | serenity-39af1f8519d4a33a90ae79e5b3277652aac85270.zip |
Ports: Add Python 3.9
The current version of our Python port (3.6.0) is over four years old by
now and has (or had, I haven't actually tried it in a while) some
limitations - time for an upgrade! The latest Python release is 3.9.1,
so I used that version. It's a from-scratch port, no patches are taken
from the previous port to ensure the smallest possible amount of code is
patched. The BuildPython.sh script is useful so I kept it, with some
tweaks. I added a short document explaining each patch to ease judging
their underlying problem and necessity in the future.
Compared to the old Python port, this one does support both the time
module as well as threading (at least _thread) just fine. Importing
modules written in C (everything in /usr/local/lib/python3.9/lib-dynload)
currently asserts in Serenity's dynamic loader, which is unfortunate but
probably solvable. Possibly related to #4642. I didn't try building
Python statically, which might be one possibility to circumvent this
issue.
I also renamed the directory to just "python3", which is analogous to
the Python 3.x package most Linux distributions provide. That implicitly
means that we likely will not support multiple versions of the Python
port at any given time, but again, neither do many other systems by
default. Recent versions are usually backwards compatible anyway though,
so having the latest shouldn't be a problem.
On the other hand bumping the version should now be be as simple as
updating the variables in version.sh, given that no new patches are
required.
These core modules to currently not build - I chose to ignore that for
now rather than adding more patches to make them work somehow, which
means they're fully unavailable. This should probably be fixed in
Serenity itself.
_ctypes, _decimal, _socket, mmap, resource, termios
These optional modules requiring 3rd-party dependencies do currently not
build (even with depends="ncurses openssl zlib"). Especially the absence
of a readline port makes the REPL a bit painful to use. :^)
_bz2, _curses, _curses_panel, _dbm, _gdbm, _hashlib, _lzma, _sqlite3,
_ssl, _tkinter, _uuid, nis, ossaudiodev, readline, spwd, zlib
I did some work on LibC and LibM beforehand to add at least stubs of
missing required functions, it still encounters an ASSERT_NOT_REACHED()
/ TODO() every now and then, notably frexp() (implementations of that
can be found online easily if you want to get that working right now).
But then again that's our fault and not this port's. :^)
20 files changed, 277 insertions, 603 deletions
diff --git a/Ports/python-3.6/README.md b/Ports/python-3.6/README.md deleted file mode 100644 index 84c889c273..0000000000 --- a/Ports/python-3.6/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Python 3.6 Port - -This port is highly experimental. Python binary can be started with `python3`, but many -functionality is expected to not work. - -## Why this version is used - -Python 2.7 will not be supported in future, see e.g. [pythonclock.org]([https://link](https://pythonclock.org/)). -Python 3 is a good candidate for porting. Until Python 3.6 it is easily possible to disable -multi-threading API via `--without-threads` option. This is needed until SerenityOS provides the -pthread APIs. - -## How to improve - -Run the Python test suite via `python3 -m test` to see what fails and start working on that. -If functionality in LibC/LibM/Kernel/... is updated, recompile Python with `./package.sh build`. - -## Known limitations - -* No locale support, default locale encoding set to utf-8 - -* Instead of `/dev/urandom`, `/dev/random` is being used - -* No multi-threading - -* time module not working due to missing time related functions in LibC/Kernel diff --git a/Ports/python-3.6/config.site b/Ports/python-3.6/config.site deleted file mode 100644 index c273024063..0000000000 --- a/Ports/python-3.6/config.site +++ /dev/null @@ -1,2 +0,0 @@ -ac_cv_file__dev_ptmx=no -ac_cv_file__dev_ptc=no diff --git a/Ports/python-3.6/package.sh b/Ports/python-3.6/package.sh deleted file mode 100755 index d09bb42ae0..0000000000 --- a/Ports/python-3.6/package.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash ../.port_include.sh - -source version.sh - -port=python-3.6 -version=3.6 -workdir=Python-3.6.0 -useconfigure=true -configopts="--build=i686 --without-threads --enable-optimizations --without-ensurepip" -makeopts="-j$(nproc) build_all" -installopts="-j$(nproc) build_all" -files="${PYTHON_URL} ${PYTHON_ARCHIVE}" - -export CONFIG_SITE=$(pwd)/config.site - -if [ -x "$(command -v python3)" ]; then - # check if major and minor version of python3 are matching - if python3 -c "import sys;sys.exit('.'.join(str(n) for n in sys.version_info[:2]) in '$PYTHON_VERSION')"; then - echo 'Error: python3 version does not match needed version to build:' $PYTHON_VERSION >&2 - echo 'Please build python3.6 with Toolchain/BuildPython.sh !' >&2 - exit 1 - fi -else - echo 'Error: python3 is not installed, please build python3.6 with Toolchain/BuildPython.sh !' >&2 - exit 1 -fi diff --git a/Ports/python-3.6/patches/0001-Adapt-configuration.patch b/Ports/python-3.6/patches/0001-Adapt-configuration.patch deleted file mode 100644 index 3552324eeb..0000000000 --- a/Ports/python-3.6/patches/0001-Adapt-configuration.patch +++ /dev/null @@ -1,146 +0,0 @@ -diff --git a/Makefile.pre.in b/Makefile.pre.in -index cd7d33d..7949077 100644 ---- a/Makefile.pre.in -+++ b/Makefile.pre.in -@@ -470,7 +470,7 @@ DTRACE_DEPS = \ - # Rules - - # Default target --all: @DEF_MAKE_ALL_RULE@ -+all: build_all - build_all: $(BUILDPYTHON) oldsharedmods sharedmods gdbhooks Programs/_testembed python-config - - # Compile a binary with profile guided optimization. -diff --git a/Modules/Setup.dist b/Modules/Setup.dist -index 8b87fc8..5d6f29f 100644 ---- a/Modules/Setup.dist -+++ b/Modules/Setup.dist -@@ -122,7 +122,7 @@ _stat _stat.c # stat.h interface - time timemodule.c # -lm # time operations and variables - - # access to ISO C locale support --_locale _localemodule.c # -lintl -+#_locale _localemodule.c # -lintl - - # Standard I/O baseline - _io -I$(srcdir)/Modules/_io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c -diff --git a/config.sub b/config.sub -index d654d03..ea26de9 100755 ---- a/config.sub -+++ b/config.sub -@@ -1512,6 +1512,8 @@ case $os in - ;; - -nacl*) - ;; -+ -serenity*) -+ ;; - -none) - ;; - *) -diff --git a/configure b/configure -index cf95b27..4e7241c 100755 ---- a/configure -+++ b/configure -@@ -784,6 +784,7 @@ infodir - docdir - oldincludedir - includedir -+runstatedir - localstatedir - sharedstatedir - sysconfdir -@@ -894,6 +895,7 @@ datadir='${datarootdir}' - sysconfdir='${prefix}/etc' - sharedstatedir='${prefix}/com' - localstatedir='${prefix}/var' -+runstatedir='${localstatedir}/run' - includedir='${prefix}/include' - oldincludedir='/usr/include' - docdir='${datarootdir}/doc/${PACKAGE_TARNAME}' -@@ -1146,6 +1148,15 @@ do - | -silent | --silent | --silen | --sile | --sil) - silent=yes ;; - -+ -runstatedir | --runstatedir | --runstatedi | --runstated \ -+ | --runstate | --runstat | --runsta | --runst | --runs \ -+ | --run | --ru | --r) -+ ac_prev=runstatedir ;; -+ -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \ -+ | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \ -+ | --run=* | --ru=* | --r=*) -+ runstatedir=$ac_optarg ;; -+ - -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb) - ac_prev=sbindir ;; - -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \ -@@ -1283,7 +1294,7 @@ fi - for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \ - datadir sysconfdir sharedstatedir localstatedir includedir \ - oldincludedir docdir infodir htmldir dvidir pdfdir psdir \ -- libdir localedir mandir -+ libdir localedir mandir runstatedir - do - eval ac_val=\$$ac_var - # Remove trailing slashes. -@@ -1436,6 +1447,7 @@ Fine tuning of the installation directories: - --sysconfdir=DIR read-only single-machine data [PREFIX/etc] - --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] - --localstatedir=DIR modifiable single-machine data [PREFIX/var] -+ --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run] - --libdir=DIR object code libraries [EPREFIX/lib] - --includedir=DIR C header files [PREFIX/include] - --oldincludedir=DIR C header files for non-gcc [/usr/include] -@@ -3250,7 +3261,10 @@ then - *-*-linux*) - ac_sys_system=Linux - ;; -- *-*-cygwin*) -+ *-*-serenity*) -+ ac_sys_system=Serenity -+ ;; -+ *-*-cygwin*) - ac_sys_system=Cygwin - ;; - *) -@@ -3295,6 +3309,9 @@ if test "$cross_compiling" = yes; then - _host_cpu=$host_cpu - esac - ;; -+ *-*-serenity*) -+ _host_cpu=$host_cpu -+ ;; - *-*-cygwin*) - _host_cpu= - ;; -@@ -7806,7 +7806,7 @@ - sys/param.h sys/select.h sys/sendfile.h sys/socket.h sys/statvfs.h \ - sys/stat.h sys/syscall.h sys/sys_domain.h sys/termio.h sys/time.h \ - sys/times.h sys/types.h sys/uio.h sys/un.h sys/utsname.h sys/wait.h pty.h \ --libutil.h sys/resource.h netpacket/packet.h sysexits.h bluetooth.h \ -+libutil.h netpacket/packet.h sysexits.h bluetooth.h \ - linux/tipc.h linux/random.h spawn.h util.h alloca.h endian.h \ - sys/endian.h - do : -index 1d63813..79bd3eb 100644 ---- a/configure.ac -+++ b/configure.ac -@@ -382,6 +382,9 @@ then - *-*-linux*) - ac_sys_system=Linux - ;; -+ *-*-serenity*) -+ ac_sys_system=Serenity -+ ;; - *-*-cygwin*) - ac_sys_system=Cygwin - ;; -@@ -427,6 +430,9 @@ if test "$cross_compiling" = yes; then - _host_cpu=$host_cpu - esac - ;; -+ *-*-serenity*) -+ _host_cpu=$host_cpu -+ ;; - *-*-cygwin*) - _host_cpu= - ;; diff --git a/Ports/python-3.6/patches/0002-Remove-not-compiling-asserts.patch b/Ports/python-3.6/patches/0002-Remove-not-compiling-asserts.patch deleted file mode 100644 index fe474d0efe..0000000000 --- a/Ports/python-3.6/patches/0002-Remove-not-compiling-asserts.patch +++ /dev/null @@ -1,238 +0,0 @@ -diff --git a/Include/bytearrayobject.h b/Include/bytearrayobject.h -index a757b88..e4bf703 100644 ---- a/Include/bytearrayobject.h -+++ b/Include/bytearrayobject.h -@@ -49,9 +49,8 @@ PyAPI_FUNC(int) PyByteArray_Resize(PyObject *, Py_ssize_t); - /* Macros, trading safety for speed */ - #ifndef Py_LIMITED_API - #define PyByteArray_AS_STRING(self) \ -- (assert(PyByteArray_Check(self)), \ -- Py_SIZE(self) ? ((PyByteArrayObject *)(self))->ob_start : _PyByteArray_empty_string) --#define PyByteArray_GET_SIZE(self) (assert(PyByteArray_Check(self)), Py_SIZE(self)) -+ (Py_SIZE(self) ? ((PyByteArrayObject*)(self))->ob_start : _PyByteArray_empty_string) -+#define PyByteArray_GET_SIZE(self) (Py_SIZE(self)) - - PyAPI_DATA(char) _PyByteArray_empty_string[]; - #endif -diff --git a/Include/bytesobject.h b/Include/bytesobject.h -index 98e29b6..7432bcd 100644 ---- a/Include/bytesobject.h -+++ b/Include/bytesobject.h -@@ -82,9 +82,8 @@ PyAPI_FUNC(PyObject *) _PyBytes_DecodeEscape(const char *, Py_ssize_t, - - /* Macro, trading safety for speed */ - #ifndef Py_LIMITED_API --#define PyBytes_AS_STRING(op) (assert(PyBytes_Check(op)), \ -- (((PyBytesObject *)(op))->ob_sval)) --#define PyBytes_GET_SIZE(op) (assert(PyBytes_Check(op)),Py_SIZE(op)) -+#define PyBytes_AS_STRING(op) ((((PyBytesObject*)(op))->ob_sval)) -+#define PyBytes_GET_SIZE(op) (Py_SIZE(op)) - #endif - - /* _PyBytes_Join(sep, x) is like sep.join(x). sep must be PyBytesObject*, -diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h -index 8103a63..f0f3e1a 100644 ---- a/Include/unicodeobject.h -+++ b/Include/unicodeobject.h -@@ -381,11 +381,9 @@ PyAPI_DATA(PyTypeObject) PyUnicodeIter_Type; - on request. Use PyUnicode_GET_LENGTH() for the length in code points. */ - - #define PyUnicode_GET_SIZE(op) \ -- (assert(PyUnicode_Check(op)), \ -- (((PyASCIIObject *)(op))->wstr) ? \ -+ ((((PyASCIIObject *)(op))->wstr) ? \ - PyUnicode_WSTR_LENGTH(op) : \ - ((void)PyUnicode_AsUnicode((PyObject *)(op)), \ -- assert(((PyASCIIObject *)(op))->wstr), \ - PyUnicode_WSTR_LENGTH(op))) - - #define PyUnicode_GET_DATA_SIZE(op) \ -@@ -397,8 +395,7 @@ PyAPI_DATA(PyTypeObject) PyUnicodeIter_Type; - use PyUnicode_WRITE() and PyUnicode_READ(). */ - - #define PyUnicode_AS_UNICODE(op) \ -- (assert(PyUnicode_Check(op)), \ -- (((PyASCIIObject *)(op))->wstr) ? (((PyASCIIObject *)(op))->wstr) : \ -+ ((((PyASCIIObject *)(op))->wstr) ? (((PyASCIIObject *)(op))->wstr) : \ - PyUnicode_AsUnicode((PyObject *)(op))) - - #define PyUnicode_AS_DATA(op) \ -@@ -418,9 +415,7 @@ PyAPI_DATA(PyTypeObject) PyUnicodeIter_Type; - string may be compact (PyUnicode_IS_COMPACT_ASCII) or not, but must be - ready. */ - #define PyUnicode_IS_ASCII(op) \ -- (assert(PyUnicode_Check(op)), \ -- assert(PyUnicode_IS_READY(op)), \ -- ((PyASCIIObject*)op)->state.ascii) -+ (((PyASCIIObject*)op)->state.ascii) - - /* Return true if the string is compact or 0 if not. - No type checks or Ready calls are performed. */ -@@ -454,9 +449,7 @@ enum PyUnicode_Kind { - - /* Return one of the PyUnicode_*_KIND values defined above. */ - #define PyUnicode_KIND(op) \ -- (assert(PyUnicode_Check(op)), \ -- assert(PyUnicode_IS_READY(op)), \ -- ((PyASCIIObject *)(op))->state.kind) -+ (((PyASCIIObject *)(op))->state.kind) - - /* Return a void pointer to the raw unicode buffer. */ - #define _PyUnicode_COMPACT_DATA(op) \ -@@ -465,12 +458,10 @@ enum PyUnicode_Kind { - ((void*)((PyCompactUnicodeObject*)(op) + 1))) - - #define _PyUnicode_NONCOMPACT_DATA(op) \ -- (assert(((PyUnicodeObject*)(op))->data.any), \ -- ((((PyUnicodeObject *)(op))->data.any))) -+ (((((PyUnicodeObject *)(op))->data.any))) - - #define PyUnicode_DATA(op) \ -- (assert(PyUnicode_Check(op)), \ -- PyUnicode_IS_COMPACT(op) ? _PyUnicode_COMPACT_DATA(op) : \ -+ (PyUnicode_IS_COMPACT(op) ? _PyUnicode_COMPACT_DATA(op) : \ - _PyUnicode_NONCOMPACT_DATA(op)) - - /* In the access macros below, "kind" may be evaluated more than once. -@@ -517,9 +508,7 @@ enum PyUnicode_Kind { - PyUnicode_READ_CHAR, for multiple consecutive reads callers should - cache kind and use PyUnicode_READ instead. */ - #define PyUnicode_READ_CHAR(unicode, index) \ -- (assert(PyUnicode_Check(unicode)), \ -- assert(PyUnicode_IS_READY(unicode)), \ -- (Py_UCS4) \ -+ ((Py_UCS4) \ - (PyUnicode_KIND((unicode)) == PyUnicode_1BYTE_KIND ? \ - ((const Py_UCS1 *)(PyUnicode_DATA((unicode))))[(index)] : \ - (PyUnicode_KIND((unicode)) == PyUnicode_2BYTE_KIND ? \ -@@ -531,10 +520,8 @@ enum PyUnicode_Kind { - /* Returns the length of the unicode string. The caller has to make sure that - the string has it's canonical representation set before calling - this macro. Call PyUnicode_(FAST_)Ready to ensure that. */ --#define PyUnicode_GET_LENGTH(op) \ -- (assert(PyUnicode_Check(op)), \ -- assert(PyUnicode_IS_READY(op)), \ -- ((PyASCIIObject *)(op))->length) -+#define PyUnicode_GET_LENGTH(op) \ -+ (((PyASCIIObject*)(op))->length) - - - /* Fast check to determine whether an object is ready. Equivalent to -@@ -547,16 +534,14 @@ enum PyUnicode_Kind { - _PyUnicode_Ready(). - Returns 0 on success and -1 on errors. */ - #define PyUnicode_READY(op) \ -- (assert(PyUnicode_Check(op)), \ -- (PyUnicode_IS_READY(op) ? \ -+ ((PyUnicode_IS_READY(op) ? \ - 0 : _PyUnicode_Ready((PyObject *)(op)))) - - /* Return a maximum character value which is suitable for creating another - string based on op. This is always an approximation but more efficient - than iterating over the string. */ - #define PyUnicode_MAX_CHAR_VALUE(op) \ -- (assert(PyUnicode_IS_READY(op)), \ -- (PyUnicode_IS_ASCII(op) ? \ -+ ((PyUnicode_IS_ASCII(op) ? \ - (0x7f) : \ - (PyUnicode_KIND(op) == PyUnicode_1BYTE_KIND ? \ - (0xffU) : \ -@@ -924,8 +909,7 @@ _PyUnicodeWriter_PrepareInternal(_PyUnicodeWriter *writer, - - Return 0 on success, raise an exception and return -1 on error. */ - #define _PyUnicodeWriter_PrepareKind(WRITER, KIND) \ -- (assert((KIND) != PyUnicode_WCHAR_KIND), \ -- (KIND) <= (WRITER)->kind \ -+ ((KIND) <= (WRITER)->kind \ - ? 0 \ - : _PyUnicodeWriter_PrepareKindInternal((WRITER), (KIND))) - -diff --git a/Objects/longobject.c b/Objects/longobject.c -index ad239ce..678cc7c 100644 ---- a/Objects/longobject.c -+++ b/Objects/longobject.c -@@ -17,8 +17,7 @@ - #endif - - /* convert a PyLong of size 1, 0 or -1 to an sdigit */ --#define MEDIUM_VALUE(x) (assert(-1 <= Py_SIZE(x) && Py_SIZE(x) <= 1), \ -- Py_SIZE(x) < 0 ? -(sdigit)(x)->ob_digit[0] : \ -+#define MEDIUM_VALUE(x) (Py_SIZE(x) < 0 ? -(sdigit)(x)->ob_digit[0] : \ - (Py_SIZE(x) == 0 ? (sdigit)0 : \ - (sdigit)(x)->ob_digit[0])) - -diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c -index 9c998f7..25e36bc 100644 ---- a/Objects/unicodeobject.c -+++ b/Objects/unicodeobject.c -@@ -78,17 +78,13 @@ extern "C" { - #define _PyUnicode_UTF8(op) \ - (((PyCompactUnicodeObject*)(op))->utf8) - #define PyUnicode_UTF8(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- assert(PyUnicode_IS_READY(op)), \ -- PyUnicode_IS_COMPACT_ASCII(op) ? \ -+ (PyUnicode_IS_COMPACT_ASCII(op) ? \ - ((char*)((PyASCIIObject*)(op) + 1)) : \ - _PyUnicode_UTF8(op)) - #define _PyUnicode_UTF8_LENGTH(op) \ - (((PyCompactUnicodeObject*)(op))->utf8_length) - #define PyUnicode_UTF8_LENGTH(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- assert(PyUnicode_IS_READY(op)), \ -- PyUnicode_IS_COMPACT_ASCII(op) ? \ -+ (PyUnicode_IS_COMPACT_ASCII(op) ? \ - ((PyASCIIObject*)(op))->length : \ - _PyUnicode_UTF8_LENGTH(op)) - #define _PyUnicode_WSTR(op) \ -@@ -102,28 +98,22 @@ extern "C" { - #define _PyUnicode_HASH(op) \ - (((PyASCIIObject *)(op))->hash) - #define _PyUnicode_KIND(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- ((PyASCIIObject *)(op))->state.kind) -+ (((PyASCIIObject *)(op))->state.kind) - #define _PyUnicode_GET_LENGTH(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- ((PyASCIIObject *)(op))->length) -+ (((PyASCIIObject *)(op))->length) - #define _PyUnicode_DATA_ANY(op) \ - (((PyUnicodeObject*)(op))->data.any) - - #undef PyUnicode_READY - #define PyUnicode_READY(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- (PyUnicode_IS_READY(op) ? \ -+ ((PyUnicode_IS_READY(op) ? \ - 0 : \ - _PyUnicode_Ready(op))) - - #define _PyUnicode_SHARE_UTF8(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- assert(!PyUnicode_IS_COMPACT_ASCII(op)), \ -- (_PyUnicode_UTF8(op) == PyUnicode_DATA(op))) -+ ((_PyUnicode_UTF8(op) == PyUnicode_DATA(op))) - #define _PyUnicode_SHARE_WSTR(op) \ -- (assert(_PyUnicode_CHECK(op)), \ -- (_PyUnicode_WSTR(unicode) == PyUnicode_DATA(op))) -+ ((_PyUnicode_WSTR(unicode) == PyUnicode_DATA(op))) - - /* true if the Unicode object has an allocated UTF-8 memory block - (not shared with other data) */ -diff --git a/Python/pytime.c b/Python/pytime.c -index 3015a6b..07335d4 100644 ---- a/Python/pytime.c -+++ b/Python/pytime.c -@@ -8,8 +8,7 @@ - #endif - - #define _PyTime_check_mul_overflow(a, b) \ -- (assert(b > 0), \ -- (_PyTime_t)(a) < _PyTime_MIN / (_PyTime_t)(b) \ -+ ((_PyTime_t)(a) < _PyTime_MIN / (_PyTime_t)(b) \ - || _PyTime_MAX / (_PyTime_t)(b) < (_PyTime_t)(a)) - - /* To millisecond (10^-3) */ --- -2.17.1 - diff --git a/Ports/python-3.6/patches/0003-Remove-locale-setup-and-use-utf-8-as-default.patch b/Ports/python-3.6/patches/0003-Remove-locale-setup-and-use-utf-8-as-default.patch deleted file mode 100644 index 0bb649a365..0000000000 --- a/Ports/python-3.6/patches/0003-Remove-locale-setup-and-use-utf-8-as-default.patch +++ /dev/null @@ -1,82 +0,0 @@ -diff --git a/Modules/main.c b/Modules/main.c -index d75f64a..143a452 100644 ---- a/Modules/main.c -+++ b/Modules/main.c -@@ -557,8 +557,8 @@ Py_Main(int argc, wchar_t **argv) - Py_FatalError( - "not enough memory to copy PYTHONWARNINGS"); - strcpy(buf, p); -- oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL)); -- setlocale(LC_ALL, ""); -+ //oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL)); -+ //setlocale(LC_ALL, ""); - for (p = strtok(buf, ","); p != NULL; p = strtok(NULL, ",")) { - #ifdef __APPLE__ - /* Use utf-8 on Mac OS X */ -@@ -574,8 +574,8 @@ Py_Main(int argc, wchar_t **argv) - PySys_AddWarnOptionUnicode(unicode); - Py_DECREF(unicode); - } -- setlocale(LC_ALL, oldloc); -- PyMem_RawFree(oldloc); -+ //setlocale(LC_ALL, oldloc); -+ //PyMem_RawFree(oldloc); - PyMem_RawFree(buf); - } - #endif -diff --git a/Programs/python.c b/Programs/python.c -index a7afbc7..07fcbe0 100644 ---- a/Programs/python.c -+++ b/Programs/python.c -@@ -43,17 +43,17 @@ main(int argc, char **argv) - fedisableexcept(FE_OVERFLOW); - #endif - -- oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL)); -- if (!oldloc) { -- fprintf(stderr, "out of memory\n"); -- return 1; -- } -+ // oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL)); -+ // if (!oldloc) { -+ // fprintf(stderr, "out of memory\n"); -+ // return 1; -+ // } - -- setlocale(LC_ALL, ""); -+ // setlocale(LC_ALL, ""); - for (i = 0; i < argc; i++) { - argv_copy[i] = Py_DecodeLocale(argv[i], NULL); - if (!argv_copy[i]) { -- PyMem_RawFree(oldloc); -+ //PyMem_RawFree(oldloc); - fprintf(stderr, "Fatal Python error: " - "unable to decode the command line argument #%i\n", - i + 1); -@@ -63,8 +63,8 @@ main(int argc, char **argv) - } - argv_copy2[argc] = argv_copy[argc] = NULL; - -- setlocale(LC_ALL, oldloc); -- PyMem_RawFree(oldloc); -+ //setlocale(LC_ALL, oldloc); -+ //PyMem_RawFree(oldloc); - - res = Py_Main(argc, argv_copy); - -diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c -index a4f7f82..823372f 100644 ---- a/Python/pylifecycle.c -+++ b/Python/pylifecycle.c -@@ -238,6 +238,8 @@ get_locale_encoding(void) - return get_codec_name(codeset); - #elif defined(__ANDROID__) - return get_codec_name("UTF-8"); -+#elif defined(__serenity__) -+ return get_codec_name("UTF-8"); - #else - PyErr_SetNone(PyExc_NotImplementedError); - return NULL; --- -2.17.1 - diff --git a/Ports/python-3.6/patches/0004-Remove-wrong-ifdef.patch b/Ports/python-3.6/patches/0004-Remove-wrong-ifdef.patch deleted file mode 100644 index 37e00e0685..0000000000 --- a/Ports/python-3.6/patches/0004-Remove-wrong-ifdef.patch +++ /dev/null @@ -1,26 +0,0 @@ -diff --git a/Python/pytime.c b/Python/pytime.c -index 07335d4..50d676a 100644 ---- a/Python/pytime.c -+++ b/Python/pytime.c -@@ -225,7 +225,7 @@ _PyTime_FromNanoseconds(long long ns) - return t; - } - --#ifdef HAVE_CLOCK_GETTIME -+ - static int - _PyTime_FromTimespec(_PyTime_t *tp, struct timespec *ts, int raise) - { -@@ -247,7 +247,8 @@ _PyTime_FromTimespec(_PyTime_t *tp, struct timespec *ts, int raise) - *tp = t; - return res; - } --#elif !defined(MS_WINDOWS) -+ -+#if !defined(MS_WINDOWS) - static int - _PyTime_FromTimeval(_PyTime_t *tp, struct timeval *tv, int raise) - { --- -2.17.1 - diff --git a/Ports/python-3.6/patches/0005-Use-dev-random-instead-of-dev-urandom.patch b/Ports/python-3.6/patches/0005-Use-dev-random-instead-of-dev-urandom.patch deleted file mode 100644 index a03642528d..0000000000 --- a/Ports/python-3.6/patches/0005-Use-dev-random-instead-of-dev-urandom.patch +++ /dev/null @@ -1,16 +0,0 @@ -diff --git a/Python/random.c b/Python/random.c -index 46e3bb5..a4a7136 100644 ---- a/Python/random.c -+++ b/Python/random.c -@@ -316,7 +316,7 @@ dev_urandom(char *buffer, Py_ssize_t size, int blocking, int raise) - } while (0 < size); - } - else { -- fd = _Py_open_noraise("/dev/urandom", O_RDONLY); -+ fd = _Py_open_noraise("/dev/random", O_RDONLY); - if (fd < 0) { - return -1; - } --- -2.17.1 - diff --git a/Ports/python-3.6/version.sh b/Ports/python-3.6/version.sh deleted file mode 100644 index a3efc436fb..0000000000 --- a/Ports/python-3.6/version.sh +++ /dev/null @@ -1,4 +0,0 @@ -PYTHON_VERSION="3.6.0" -PYTHON_MD5SUM="82b143ebbf4514d7e05876bed7a6b1f5" -PYTHON_ARCHIVE="Python-$PYTHON_VERSION.tar.xz" -PYTHON_URL="https://www.python.org/ftp/python/$PYTHON_VERSION/${PYTHON_ARCHIVE}" diff --git a/Ports/python3/package.sh b/Ports/python3/package.sh new file mode 100755 index 0000000000..db5cda4677 --- /dev/null +++ b/Ports/python3/package.sh @@ -0,0 +1,38 @@ +#!/bin/bash ../.port_include.sh + +source version.sh + +port=python3 +version="${PYTHON_VERSION}" +workdir="Python-${version}" +useconfigure="true" +files="${PYTHON_ARCHIVE_URL} ${PYTHON_ARCHIVE} +https://www.python.org/ftp/python/${version}/Python-${version}.tar.xz.asc Python-${version}.tar.xz.asc" +auth_type="sig" +auth_import_key="E3FF2839C048B25C084DEBE9B26995E310250568" +auth_opts="Python-${version}.tar.xz.asc Python-${version}.tar.xz" + +# We could say depends="ncurses openssl zlib" here, but neither of the _curses, _ssl, and zlib modules +# build at the moment even with those available, so it's pointless. + +# FIXME: the --build value is detected correctly by the configure script (via config.guess in the Python source root), +# but still needs to be set explicitly when cross compiling. Figure out how to not hardcode this. +BUILD="x86_64-pc-linux-gnu" + +# FIXME: --enable-optimizations results in lots of __gcov_* linker errors +configopts="--build=${BUILD} --without-ensurepip ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no" + +export BLDSHARED="${CC} -shared" + +if [ -x "$(command -v python3)" ]; then + # Check if major and minor version of python3 are matching + if ! python3 -c "import sys; major, minor, _ = map(int, '${PYTHON_VERSION}'.split('.')); sys.exit(not (sys.version_info.major == major and sys.version_info.minor == minor))"; then + echo "Error: python3 version does not match needed version to build ${PYTHON_VERSION}" >&2 + echo "Build this Python version on your host using Toolchain/BuildPython.sh or install it otherwise and try again." >&2 + exit 1 + fi +else + echo "Error: python3 is not installed but is required to build ${PYTHON_VERSION}" >&2 + echo "Build this Python version on your host using Toolchain/BuildPython.sh or install it otherwise and try again." >&2 + exit 1 +fi diff --git a/Ports/python3/patches/README.md b/Ports/python3/patches/README.md new file mode 100644 index 0000000000..df488fa60f --- /dev/null +++ b/Ports/python3/patches/README.md @@ -0,0 +1,39 @@ +# Patches for Python 3.9 on SerenityOS + +## `define-have-sigset-t.patch` + +Ensures `HAVE_SIGSET_T` is defined, as we *do* have `sigset_t` but it's not detected properly due to some related functions being missing. + +## `define-py-force-utf8-locale.patch` + +Enforce UTF-8 as encoding by defining `_Py_FORCE_UTF8_LOCALE`. + +## `disable-setrlimit.patch` + +Disables check for `RLIMIT_CORE` and subsequent `setrlimit()` call. Would be enabled otherwise as we *do* have `<sys/resource.h>` and therefore `HAVE_SYS_RESOURCE_H`. + +## `fix-autoconf.patch` + +As usual, make the `configure` script recognize Serenity. + +## `fix-hidden-symbol-referenced-by-dso.patch` + +Fix a weird build issue of `python` and other provided binaries by marking the `main()` functions `Py_EXPORTED_SYMBOL`. + +```text +hidden symbol `main' in Programs/python.o is referenced by DSO +``` + +Not sure what the proper fix for this is, but it works fine. + +## `remove-setlocale-from-preconfig.patch` + +Our stub implementation of `setlocale()` always returns `nullptr`, which the interpreter considers critical enough to exit right away. + +## `tweak-unsupported-printf-format-specifiers.patch` + +Replace uses of `%.Ns` with `%s` as the former is not supported by our `printf` implementation yet and would result in empty strings. It uses `snprintf` already, so this is safe. + +## `use-rtld-lazy-for-dlopenflags.patch` + +We have `RTLD_NOW` defined but don't actually support it, so use the provided `RTLD_LAZY` fallback. Doesn't help the dynamic library module import assertion though. diff --git a/Ports/python3/patches/define-have-sigset-t.patch b/Ports/python3/patches/define-have-sigset-t.patch new file mode 100644 index 0000000000..b7447ccae7 --- /dev/null +++ b/Ports/python3/patches/define-have-sigset-t.patch @@ -0,0 +1,12 @@ +--- Python-3.9.1/Modules/posixmodule.h 2021-01-17 20:56:14.590000000 +0100 ++++ Python-3.9.1/Modules/posixmodule.h 2021-01-17 20:56:34.207894812 +0100 +@@ -19,7 +19,8 @@ + #endif /* MS_WINDOWS */ + + #if defined(PYPTHREAD_SIGMASK) || defined(HAVE_SIGWAIT) || \ +- defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT) ++ defined(HAVE_SIGWAITINFO) || defined(HAVE_SIGTIMEDWAIT) || \ ++ defined(__serenity__) + # define HAVE_SIGSET_T + #endif + diff --git a/Ports/python3/patches/define-py-force-utf8-locale.patch b/Ports/python3/patches/define-py-force-utf8-locale.patch new file mode 100644 index 0000000000..d8e6ea1b4b --- /dev/null +++ b/Ports/python3/patches/define-py-force-utf8-locale.patch @@ -0,0 +1,11 @@ +--- Python-3.9.1/Include/pyport.h 2021-01-17 20:45:44.417000000 +0100 ++++ Python-3.9.1/Include/pyport.h 2021-01-17 20:46:07.865663659 +0100 +@@ -838,7 +838,7 @@ + # error "Py_TRACE_REFS ABI is not compatible with release and debug ABI" + #endif + +-#if defined(__ANDROID__) || defined(__VXWORKS__) ++#if defined(__ANDROID__) || defined(__VXWORKS__) || defined(__serenity__) + /* Ignore the locale encoding: force UTF-8 */ + # define _Py_FORCE_UTF8_LOCALE + #endif diff --git a/Ports/python3/patches/disable-setrlimit.patch b/Ports/python3/patches/disable-setrlimit.patch new file mode 100644 index 0000000000..621dba6e74 --- /dev/null +++ b/Ports/python3/patches/disable-setrlimit.patch @@ -0,0 +1,11 @@ +--- Python-3.9.1/Modules/faulthandler.c 2021-01-17 20:45:32.878000000 +0100 ++++ Python-3.9.1/Modules/faulthandler.c 2021-01-17 20:45:33.006210297 +0100 +@@ -993,7 +993,7 @@ + SetErrorMode(mode | SEM_NOGPFAULTERRORBOX); + #endif + +-#ifdef HAVE_SYS_RESOURCE_H ++#if 0 + struct rlimit rl; + + /* Disable creation of core dump */ diff --git a/Ports/python3/patches/fix-autoconf.patch b/Ports/python3/patches/fix-autoconf.patch new file mode 100644 index 0000000000..e25173313c --- /dev/null +++ b/Ports/python3/patches/fix-autoconf.patch @@ -0,0 +1,55 @@ +--- Python-3.9.1/config.sub 2021-01-17 20:15:56.796000000 +0100 ++++ Python-3.9.1/config.sub 2021-01-17 20:21:04.324828217 +0100 +@@ -1485,6 +1485,8 @@ + -oss*) + os=-sysv3 + ;; ++ -serenity*) ++ ;; + -svr4*) + os=-sysv4 + ;; +--- Python-3.9.1/configure.ac 2021-01-17 20:33:50.524295313 +0100 ++++ Python-3.9.1/configure.ac 2021-01-17 20:34:24.631127320 +0100 +@@ -382,6 +382,9 @@ + # a lot of different things including 'define_xopen_source' + # in the case statement below. + case "$host" in ++ *-*-serenity*) ++ ac_sys_system=Serenity ++ ;; + *-*-linux-android*) + ac_sys_system=Linux-android + ;; +@@ -428,6 +431,9 @@ + AC_SUBST(_PYTHON_HOST_PLATFORM) + if test "$cross_compiling" = yes; then + case "$host" in ++ *-*-serenity*) ++ _host_cpu=$host_cpu ++ ;; + *-*-linux*) + case "$host_cpu" in + arm*) +--- Python-3.9.1/configure 2021-01-17 20:35:39.813757019 +0100 ++++ Python-3.9.1/configure 2021-01-17 20:36:00.538654942 +0100 +@@ -3292,6 +3292,9 @@ + # a lot of different things including 'define_xopen_source' + # in the case statement below. + case "$host" in ++ *-*-serenity*) ++ ac_sys_system=Serenity ++ ;; + *-*-linux-android*) + ac_sys_system=Linux-android + ;; +@@ -3339,6 +3342,9 @@ + + if test "$cross_compiling" = yes; then + case "$host" in ++ *-*-serenity*) ++ _host_cpu=$host_cpu ++ ;; + *-*-linux*) + case "$host_cpu" in + arm*) diff --git a/Ports/python3/patches/fix-hidden-symbol-referenced-by-dso.patch b/Ports/python3/patches/fix-hidden-symbol-referenced-by-dso.patch new file mode 100644 index 0000000000..4353a2bf97 --- /dev/null +++ b/Ports/python3/patches/fix-hidden-symbol-referenced-by-dso.patch @@ -0,0 +1,22 @@ +--- Python-3.9.1/Programs/python.c 2021-01-18 08:25:32.203494270 +0100 ++++ Python-3.9.1/Programs/python.c 2021-01-18 08:25:49.711418585 +0100 +@@ -9,7 +9,7 @@ + return Py_Main(argc, argv); + } + #else +-int ++Py_EXPORTED_SYMBOL int + main(int argc, char **argv) + { + return Py_BytesMain(argc, argv); +--- Python-3.9.1/Programs/_testembed.c 2021-01-18 08:22:35.085000000 +0100 ++++ Python-3.9.1/Programs/_testembed.c 2021-01-18 08:23:16.036082910 +0100 +@@ -1711,7 +1711,7 @@ + {NULL, NULL} + }; + +-int main(int argc, char *argv[]) ++Py_EXPORTED_SYMBOL int main(int argc, char *argv[]) + { + if (argc > 1) { + for (struct TestCase *tc = TestCases; tc && tc->name; tc++) { diff --git a/Ports/python3/patches/remove-setlocale-from-preconfig.patch b/Ports/python3/patches/remove-setlocale-from-preconfig.patch new file mode 100644 index 0000000000..f3824c556f --- /dev/null +++ b/Ports/python3/patches/remove-setlocale-from-preconfig.patch @@ -0,0 +1,30 @@ +--- Python-3.9.1/Python/preconfig.c 2021-01-17 21:03:08.698000000 +0100 ++++ Python-3.9.1/Python/preconfig.c 2021-01-17 21:03:47.828031544 +0100 +@@ -790,16 +790,6 @@ + + preconfig_get_global_vars(config); + +- /* Copy LC_CTYPE locale, since it's modified later */ +- const char *loc = setlocale(LC_CTYPE, NULL); +- if (loc == NULL) { +- return _PyStatus_ERR("failed to LC_CTYPE locale"); +- } +- char *init_ctype_locale = _PyMem_RawStrdup(loc); +- if (init_ctype_locale == NULL) { +- return _PyStatus_NO_MEMORY(); +- } +- + /* Save the config to be able to restore it if encodings change */ + PyPreConfig save_config; + +@@ -899,10 +889,6 @@ + status = _PyStatus_OK(); + + done: +- if (init_ctype_locale != NULL) { +- setlocale(LC_CTYPE, init_ctype_locale); +- PyMem_RawFree(init_ctype_locale); +- } + Py_UTF8Mode = init_utf8_mode ; + #ifdef MS_WINDOWS + Py_LegacyWindowsFSEncodingFlag = init_legacy_encoding; diff --git a/Ports/python3/patches/tweak-unsupported-printf-format-specifiers.patch b/Ports/python3/patches/tweak-unsupported-printf-format-specifiers.patch new file mode 100644 index 0000000000..6b262d07d5 --- /dev/null +++ b/Ports/python3/patches/tweak-unsupported-printf-format-specifiers.patch @@ -0,0 +1,22 @@ +--- Python-3.9.1/Python/getversion.c 2021-01-18 08:31:52.780000000 +0100 ++++ Python-3.9.1/Python/getversion.c 2021-01-18 08:32:14.176848948 +0100 +@@ -9,7 +9,7 @@ + Py_GetVersion(void) + { + static char version[250]; +- PyOS_snprintf(version, sizeof(version), "%.80s (%.80s) %.80s", ++ PyOS_snprintf(version, sizeof(version), "%s (%s) %s", + PY_VERSION, Py_GetBuildInfo(), Py_GetCompiler()); + return version; + } +--- Python-3.9.1/Modules/getbuildinfo.c 2021-01-18 08:54:23.766207240 +0100 ++++ Python-3.9.1/Modules/getbuildinfo.c 2021-01-18 08:54:09.757000000 +0100 +@@ -43,7 +43,7 @@ + if (!(*gitid)) + gitid = "default"; + PyOS_snprintf(buildinfo, sizeof(buildinfo), +- "%s%s%s, %.20s, %.9s", gitid, sep, revision, ++ "%s%s%s, %s, %s", gitid, sep, revision, + DATE, TIME); + return buildinfo; + } diff --git a/Ports/python3/patches/use-rtld-lazy-for-dlopenflags.patch b/Ports/python3/patches/use-rtld-lazy-for-dlopenflags.patch new file mode 100644 index 0000000000..dbcc54e664 --- /dev/null +++ b/Ports/python3/patches/use-rtld-lazy-for-dlopenflags.patch @@ -0,0 +1,11 @@ +--- Python-3.9.1/Python/pystate.c 2021-01-18 18:33:06.021000000 +0100 ++++ Python-3.9.1/Python/pystate.c 2021-01-18 18:33:50.274359610 +0100 +@@ -223,7 +223,7 @@ + + interp->eval_frame = _PyEval_EvalFrameDefault; + #ifdef HAVE_DLOPEN +-#if HAVE_DECL_RTLD_NOW ++#if defined(HAVE_DECL_RTLD_NOW) && !defined(__serenity__) + interp->dlopenflags = RTLD_NOW; + #else + interp->dlopenflags = RTLD_LAZY; diff --git a/Toolchain/BuildPython.sh b/Toolchain/BuildPython.sh index 71c89babc9..b8a2029a84 100755 --- a/Toolchain/BuildPython.sh +++ b/Toolchain/BuildPython.sh @@ -1,62 +1,51 @@ -#!/bin/bash +#!/usr/bin/env bash + set -e # This file will need to be run in bash, for now. DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -echo "$DIR" - ARCH=${ARCH:-"i686"} -TARGET="$ARCH-pc-serenity" -PREFIX="$DIR/Local/$ARCH" -BUILD=$(realpath "$DIR/../Build") -SYSROOT="$BUILD/Root" - -source "$DIR/../Ports/python-3.6/version.sh" +PREFIX_DIR="$DIR/Local/$ARCH" +BUILD_DIR="$DIR/Build/$ARCH" +TARBALLS_DIR="$DIR/Tarballs" -echo PYTHON_VERSION is "$PYTHON_VERSION" -echo PYTHON_URL is "$PYTHON_URL" +source "$DIR/../Ports/python3/version.sh" -echo PREFIX is "$PREFIX" -echo SYSROOT is "$SYSROOT" +mkdir -p "${TARBALLS_DIR}" -mkdir -p "$DIR/Tarballs" - -pushd "$DIR/Tarballs" - if [ ! -e "$PYTHON_ARCHIVE" ]; then - curl -O "$PYTHON_URL" +pushd "${TARBALLS_DIR}" + if [ ! -e "${PYTHON_ARCHIVE}" ]; then + echo "Downloading Python from ${PYTHON_ARCHIVE_URL}..." + curl -O "${PYTHON_ARCHIVE_URL}" else - echo "Skipped downloading Python-$PYTHON_VERSION" + echo "${PYTHON_ARCHIVE} already exists, not downloading archive" fi - md5="$(md5sum $PYTHON_ARCHIVE | cut -f1 -d' ')" - echo "python md5='$md5'" - if [ "$md5" != "$PYTHON_MD5SUM" ] ; then - echo "python md5 sum mismatching, please run script again." - rm $PYTHON_ARCHIVE + if ! md5sum --status -c <(echo "${PYTHON_ARCHIVE_MD5SUM}" "${PYTHON_ARCHIVE}"); then + echo "Python archive MD5 sum mismatch, please run script again" + rm -f "${PYTHON_ARCHIVE}" exit 1 fi - if [ ! -d "Python-$PYTHON_VERSION" ]; then - echo "Extracting python..." - tar -xf "$PYTHON_ARCHIVE" + if [ ! -d "Python-${PYTHON_VERSION}" ]; then + echo "Extracting ${PYTHON_ARCHIVE}..." + tar -xf "${PYTHON_ARCHIVE}" else - echo "Skipped extracting python" + echo "Python-${PYTHON_VERSION} already exists, not extracting archive" fi popd -mkdir -p "$PREFIX" -mkdir -p "$DIR/Build/$ARCH/python" - if [ -z "$MAKEJOBS" ]; then MAKEJOBS=$(nproc) fi -pushd "$DIR/Build/$ARCH" - pushd python - "$DIR"/Tarballs/Python-$PYTHON_VERSION/configure --prefix="$PREFIX" || exit 1 - make -j "$MAKEJOBS" || exit 1 - make install || exit 1 - popd +mkdir -p "${PREFIX_DIR}" +mkdir -p "${BUILD_DIR}/python" + +pushd "${BUILD_DIR}/python" + "${TARBALLS_DIR}"/Python-"${PYTHON_VERSION}"/configure --prefix="${PREFIX_DIR}" + make -j "${MAKEJOBS}" + make install popd |