Can't import numpy

[supern8ent]

If I clone gae-init master and simply add numpy==1.6.1 to requirements.txt and import numpy to main.py (after import task), it raises:

ERROR    2020-03-12 00:11:31,162 wsgi.py:269]
Traceback (most recent call last):
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 311, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/Users/nathan.lloyd/dev/gae-init/main/main.py", line 38, in <module>
    import numpy as np
  File "lib/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "lib/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "lib/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "lib/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "lib/numpy/core/__init__.py", line 5, in <module>
    import multiarray
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/python/runtime/sandbox.py", line 1114, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named numpy.core.multiarray

If I also add

  - name: numpy
    version: latest

to app.yaml it raises:

ERROR    2020-03-11 23:37:05,688 wsgi.py:269]
Traceback (most recent call last):
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 311, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Users/nathan.lloyd/google-cloud-sdk/platform/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/Users/nathan.lloyd/dev/gae-init/main/main.py", line 38, in <module>
    import numpy as np
  File "lib/numpy/__init__.py", line 155, in <module>
    import ctypeslib
  File "lib/numpy/ctypeslib.py", line 60, in <module>
    import ctypes
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 29, in <module>
    if int(_os.uname()[2].split('.')[0]) < 8:
ValueError: invalid literal for int() with base 10: ''

I ran npm install and gulp.

Note that I'm having problems in my own application, which was working previously, but for
simplicity's sake I demonstrated in vanilla gae-init. I wonder if this is a gcloud version error...

Python 2.7.17
OS X Mojave 10.14.6
Google Cloud SDK 284.0.0
app-engine-python 1.9.88
app-engine-python-extras 1.9.87
beta 2019.05.17
cloud-datastore-emulator 2.1.0
core 2020.03.06

[supern8ent]

Oh, I also took had to take out grpcio as mentioned here: #1392

[mdxs]

Not using numpy myself, but noting that a special version is provided via "Built-in Third-party Libraries" for Google Appengine, I would guess it might be worth trying to only use the app.yaml approach and not the requirements.txt. As the requirements approach, even if it is the same version; might pull in a numpy version that is incompatible (due to binaries not allowed on Appengine, which is why there is a special built-in variant provided by Google).

@supern8ent could you try that and report back?

[supern8ent]

Unfortunately, you have to install it locally. From https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27#local_development

The following built-in libraries must be installed locally before you can use them with the local development server:

lxml
matplotlib
mysqldb
numpy
PIL
crcmod
pycrypto
grpcio
protobuf

When I tried removing it from requriements.txt I got ImportError: No module named numpy.

Oh, hey, maybe that's what's happening with #1392.

[mdxs]

Perhaps the newer main/requirements-dev.txt approach may help in this regard... not sure, just guessing (and noting that this approach was introduced in December 2020).

[rbubley]

That was the intent of requirements-dev: for packages to be installed and available in the local dev environment but ignored in the upload to production. So in theory you should add it into app.yaml and requirements-dev.

[mdxs]

Thanks for the confirmation @rbubley, as noted in #1720 it may still upload the local dev environment on deployment, but that shouldn't be a showstopper (it isn't used in GAE when deployed, the main/libdev folder is only used locally).

[mdxs]

This issue could probably be closed by @lipis soon as it should be resolved now, unless @supern8ent reports back in a few weeks that it isn't working.