mac arm build and CI addition (#259)

* resolving arch flag issues for mac

* adding mac build to test suite

* adding macos-12 to actions

* x86 test name

* cleanup and python venv setup

* venv update

* build matrix

* adding venv source to test block

* lower numpy version for testing

* setup-python action

* pip3 additions

* numpy version for x86 build

* typo

* python 3.11 for x86

* numpy version reduction

* disable fork safety for test on x86

* set python_requires

just a guess, but the code uses f-strings, which were introduced in 3.6

* update URLs

* restructure as matrix

* install cython

* disable configs that can not work, install filelock package for ray test

* macos-13 build fixes

---------

Co-authored-by: ritvikrao <rsrao2@illinois.edu>
Co-authored-by: Matthias Diener <mdiener@illinois.edu>

Author: 3 people

.github/workflows/charm4py.yml

@@ -6,25 +6,49 @@ on:
       - master
   pull_request:
   schedule:
-      - cron: '5 0 * * *'  # Runs at 00:05 UTC every day.
+    - cron: "5 0 * * *" # Runs at 00:05 UTC every day.
 
 jobs:
-  build:
-    runs-on: ubuntu-latest
+  auto_test:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        python-version: ["3.7", "3.12"]
+        # macos-13 is x86_64, macos-14 is arm64
+        os: [ubuntu-latest, macos-13, macos-14]
+        exclude:
+          - os: macos-14  # Python 3.7 does not exist for macos-14
+            python-version: "3.7"
+      fail-fast: false
 
     steps:
-    - uses: actions/checkout@v3
-    - name: build-charm4py
-      run: |
-        git fetch --unshallow # Need full repo for 'git describe' used by setup.py
-        pip3 install setuptools cython cffi greenlet numpy torch torchvision
-        git clone https://github.com/UIUC-PPL/charm charm_src/charm
-        export CHARM_EXTRA_BUILD_OPTS="--enable-error-checking"
-        export CHARM_BUILD_PROCESSES=2
-        export CHARM4PY_BUILD_CFFI=1
-        python3 setup.py build_ext --inplace
-    - name: test-charm4py
-      run: |
-        export PYTHONPATH="$PWD"
-        export CHARM4PY_TEST_NUM_PROCESSES=2
-        python3 auto_test.py
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Need full repo for 'git describe' used by setup.py
+      - name: setup-python
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          pip install setuptools cython cffi greenlet numpy torch torchvision filelock
+          if [ ${{ matrix.os }} == 'macos-13' ]; then
+            # pypi only distributes torch packages w/ numpy v1 for macos-x86_64
+            pip install 'numpy<2'
+          fi
+      - name: Build Charm++/Charm4py
+        run: |
+          git clone https://github.com/charmplusplus/charm charm_src/charm
+          export CHARM_EXTRA_BUILD_OPTS="--enable-error-checking"
+          export CHARM_BUILD_PROCESSES=2
+          export CHARM4PY_BUILD_CFFI=1
+          python setup.py build_ext --inplace
+      - name: Run auto_test.py
+        run: |
+          export PYTHONPATH="$PWD"
+          export CHARM4PY_TEST_NUM_PROCESSES=2
+          if [ ${{ matrix.os }} == 'macos-13' ]; then
+            # Crashes in ray test otherwise
+            export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
+          fi
+          python auto_test.py

README.rst

@@ -3,8 +3,8 @@ Charm4py
 ========
 
 
-.. image:: https://github.com/UIUC-PPL/charm4py/actions/workflows/charm4py.yml/badge.svg?event=push
-       :target: https://github.com/UIUC-PPL/charm4py/actions/workflows/charm4py.yml
+.. image:: https://github.com/charmplusplus/charm4py/actions/workflows/charm4py.yml/badge.svg?event=push
+       :target: https://github.com/charmplusplus/charm4py/actions/workflows/charm4py.yml
 
 .. image:: http://readthedocs.org/projects/charm4py/badge/?version=latest
        :target: https://charm4py.readthedocs.io/
@@ -83,8 +83,8 @@ or email us at <charm@cs.illinois.edu>.
 Main author at <jjgalvez@illinois.edu>
 
 
-.. _Charm++: https://github.com/UIUC-PPL/charm
+.. _Charm++: https://github.com/charmplusplus/charm
 
 .. _Documentation: https://charm4py.readthedocs.io
 
-.. _discussion page: https://github.com/UIUC-PPL/charm4py/discussions
+.. _discussion page: https://github.com/charmplusplus/charm4py/discussions

docs/contact.rst

@@ -5,7 +5,7 @@ Contact
 You can contact us on the `discussion page`_ for discussion of any topics related to
 Charm4py or Charm++, or in our mailing list at <charm@cs.illinois.edu>
 
-.. _discussion page: https://github.com/UIUC-PPL/charm4py/discussions
+.. _discussion page: https://github.com/charmplusplus/charm4py/discussions
 
 
-Please use the GitHub page to report and track issues (https://github.com/UIUC-PPL/charm4py)
+Please use the GitHub page to report and track issues (https://github.com/charmplusplus/charm4py)

docs/examples.rst

@@ -3,7 +3,7 @@ Examples
 ========
 
 There are several examples in the source code repository, with documentation
-and comments: https://github.com/UIUC-PPL/charm4py/tree/master/examples
+and comments: https://github.com/charmplusplus/charm4py/tree/master/examples
 
 These include:
 

setup.py

@@ -23,9 +23,14 @@
 def get_build_machine():
     machine = platform.machine()
     if machine == 'arm64' or machine == 'aarch64':
-        return 'arm64'
+        return 'arm8'
     return machine
 
+def get_archflag_machine():
+    machine = platform.machine()
+    if machine == 'arm64' or machine == 'aarch64':
+        return 'arm64'
+    return machine
 
 def get_build_os():
     os = platform.system()
@@ -53,7 +58,7 @@ def get_build_triple(build_mpi):
     libcharm_filename2 = 'charm.lib'
     charmrun_filename = 'charmrun.exe'
 elif system == 'darwin':
-    os.environ['ARCHFLAGS'] = f'-arch {machine}'
+    os.environ['ARCHFLAGS'] = f'-arch {get_archflag_machine()}'
     libcharm_filename = 'libcharm.dylib'
     charmrun_filename = 'charmrun'
     if 'CPPFLAGS' in os.environ:
@@ -169,7 +174,7 @@ def build_libcharm(charm_src_dir, build_dir):
             raise DistutilsSetupError('An error occured while building charm library')
 
         if system == 'darwin':
-            old_file_path = os.path.join(charm_src_dir, 'charm', 'lib', 'libcharm.so')
+            old_file_path = os.path.join(charm_src_dir, 'charm', 'lib', 'libcharm.dylib')
             new_file_path = os.path.join(charm_src_dir, 'charm', 'lib', libcharm_filename)
             shutil.move(old_file_path, new_file_path)
             cmd = ['install_name_tool', '-id', '@rpath/../.libs/' + libcharm_filename, new_file_path]
@@ -192,7 +197,7 @@ def build_libcharm(charm_src_dir, build_dir):
         for output_dir in lib_output_dirs:
             log.info('copying ' + os.path.relpath(lib_src_path) + ' to ' + os.path.relpath(output_dir))
             shutil.copy(lib_src_path, output_dir)
-    
+
 
     # ---- copy charmrun ----
     charmrun_src_path = os.path.join(charm_src_dir, 'charm', 'bin', charmrun_filename)
@@ -268,7 +273,7 @@ def run(self):
 class _renameInstalled(_install_lib):
     def __init__(self, *args, **kwargs):
         _install_lib.__init__(self, *args, **kwargs)
-    
+
     def install(self):
         log.info("Renaming libraries")
         outfiles = _install_lib.install(self)
@@ -324,7 +329,7 @@ def install(self):
                 extra_link_args=["-Wl,-rpath,@loader_path/../.libs"]
             else:
                 extra_link_args=["-Wl,-rpath,$ORIGIN/../.libs"]
-        
+
         cobject_extra_args = []
         log.info("Extra object args for object store")
         if os.name != 'nt':
@@ -341,7 +346,7 @@ def install(self):
                               extra_compile_args=['-g0', '-O3'],
                               extra_link_args=extra_link_args,
                               ), compile_time_env={'HAVE_NUMPY': haveNumpy}))
-        
+
         extensions.extend(cythonize(setuptools.Extension('charm4py.c_object_store',
                               sources=['charm4py/c_object_store.pyx'],
                               include_dirs=['charm_src/charm/include'] + my_include_dirs,
@@ -375,7 +380,7 @@ def install(self):
     author_email='jjgalvez@illinois.edu',
     description='Charm4py Parallel Programming Framework',
     long_description=long_description,
-    url='https://github.com/UIUC-PPL/charm4py',
+    url='https://github.com/charmplusplus/charm4py',
     keywords='parallel parallel-programming distributed distributed-computing hpc HPC runtime',
     packages=setuptools.find_packages(),
     package_data={
@@ -387,7 +392,7 @@ def install(self):
         ],
     },
     install_requires=['numpy>=1.10.0', 'greenlet>=3.0.0', 'cython>=3.0.0', 'cmake'],
-    #python_requires='>=2.7, ~=3.4',
+    python_requires='~=3.6',
     classifiers=[
         'Intended Audience :: Developers',
         'License :: Free for non-commercial use',
@@ -396,10 +401,7 @@ def install(self):
         'Operating System :: POSIX :: Linux',
         'Operating System :: Microsoft :: Windows',
         'Programming Language :: Python',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3',
         'Topic :: System :: Distributed Computing',
         'Topic :: System :: Clustering',
     ],