Skip to content
This repository has been archived by the owner on Jul 7, 2023. It is now read-only.

Unable to instantiate problem instance when calling use_vocab_from_other_problem #1913

Open
colmantse opened this issue Jul 14, 2022 · 0 comments
Open

Unable to instantiate problem instance when calling use_vocab_from_other_problem #1913

colmantse opened this issue Jul 14, 2022 · 0 comments

Comments

@colmantse
Copy link

Description

use_vocab_from_other_problem seems to return a function instead of a problem instance
I could directly change vocab_filename, but since this is the recommended approach, guess i better ask.
...

Environment information

OS: <ubuntu-16.14>

$ pip freeze | grep tensor
# graphsurgeon @ file:///opt/anaconda/conda-bld/graphsurgeon_1600268700975/work/tensorrt/graphsurgeon/graphsurgeon-0.4.1-py2.py3-none-any.whl
mesh-tensorflow==0.0.5
tensor2tensor==1.15.7
tensorboard @ file:///opt/anaconda3/conda-bld/tensorboard_1602323996819/work/tmp_pip_dir
tensorboard-plugin-profile==2.8.0
tensorflow==1.15.2
tensorflow-addons==0.13.0
tensorflow-datasets==1.0.1
tensorflow-estimator @ file:///opt/anaconda/conda-bld/tensorflow-estimator_1602324006015/work/tensorflow_estimator_pkg/tensorflow_estimator-1.15.2-py2.py3-none-any.whl
tensorflow-gan==2.0.0
tensorflow-hub==0.7.0
tensorflow-metadata==0.13.0
tensorflow-probability==0.7.0
tensorflow-tensorboard==1.5.1
tensorrt @ file:///opt/anaconda/conda-bld/tensorrt_1600268909119/work/tensorrt/python/tensorrt-6.0.1.5-cp36-none-linux_x86_64.whl
uff @ file:///opt/anaconda/conda-bld/uff_1600268848689/work/tensorrt/uff/uff-0.6.5-py2.py3-none-any.whl


$ python -V
# import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
# installed zipimport hook
# /home/colman/anaconda3/lib/python3.6/encodings/__pycache__/__init__.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/encodings/__init__.py
# code object from '/home/colman/anaconda3/lib/python3.6/encodings/__pycache__/__init__.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/codecs.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/codecs.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/codecs.cpython-36.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4ef128>
# /home/colman/anaconda3/lib/python3.6/encodings/__pycache__/aliases.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/encodings/aliases.py
# code object from '/home/colman/anaconda3/lib/python3.6/encodings/__pycache__/aliases.cpython-36.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4fdac8>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf563c50>
# /home/colman/anaconda3/lib/python3.6/encodings/__pycache__/utf_8.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/encodings/utf_8.py
# code object from '/home/colman/anaconda3/lib/python3.6/encodings/__pycache__/utf_8.cpython-36.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf50a8d0>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# /home/colman/anaconda3/lib/python3.6/encodings/__pycache__/latin_1.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/encodings/latin_1.py
# code object from '/home/colman/anaconda3/lib/python3.6/encodings/__pycache__/latin_1.cpython-36.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf50f400>
# /home/colman/anaconda3/lib/python3.6/__pycache__/io.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/io.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/io.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/abc.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/abc.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/abc.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/_weakrefset.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/_weakrefset.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/_weakrefset.cpython-36.pyc'
import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf515390>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf50f9e8>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf50f630>
# /home/colman/anaconda3/lib/python3.6/__pycache__/site.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/site.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/site.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/os.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/os.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/os.cpython-36.pyc'
import 'errno' # <class '_frozen_importlib.BuiltinImporter'>
# /home/colman/anaconda3/lib/python3.6/__pycache__/stat.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/stat.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/stat.cpython-36.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4ba668>
# /home/colman/anaconda3/lib/python3.6/__pycache__/posixpath.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/posixpath.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/posixpath.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/genericpath.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/genericpath.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/genericpath.cpython-36.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4c5080>
import 'posixpath' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4bad30>
# /home/colman/anaconda3/lib/python3.6/__pycache__/_collections_abc.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/_collections_abc.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/_collections_abc.cpython-36.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4c56a0>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4a9f60>
# /home/colman/anaconda3/lib/python3.6/__pycache__/_sitebuiltins.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/_sitebuiltins.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/_sitebuiltins.cpython-36.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4ac390>
# /home/colman/anaconda3/lib/python3.6/__pycache__/sysconfig.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/sysconfig.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/sysconfig.cpython-36.pyc'
import 'sysconfig' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf479b00>
# /home/colman/anaconda3/lib/python3.6/__pycache__/_sysconfigdata_m_linux_x86_64-linux-gnu.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/_sysconfigdata_m_linux_x86_64-linux-gnu.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/_sysconfigdata_m_linux_x86_64-linux-gnu.cpython-36.pyc'
import '_sysconfigdata_m_linux_x86_64-linux-gnu' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf48f3c8>
# /home/colman/anaconda3/lib/python3.6/__pycache__/_bootlocale.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/_bootlocale.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/_bootlocale.cpython-36.pyc'
import '_locale' # <class '_frozen_importlib.BuiltinImporter'>
import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf49b3c8>
# /home/colman/anaconda3/lib/python3.6/__pycache__/types.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/types.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/types.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/functools.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/functools.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/functools.cpython-36.pyc'
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
# /home/colman/anaconda3/lib/python3.6/collections/__pycache__/__init__.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/collections/__init__.py
# code object from '/home/colman/anaconda3/lib/python3.6/collections/__pycache__/__init__.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/operator.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/operator.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/operator.cpython-36.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce208208>
# /home/colman/anaconda3/lib/python3.6/__pycache__/keyword.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/keyword.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/keyword.cpython-36.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce212470>
# /home/colman/anaconda3/lib/python3.6/__pycache__/heapq.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/heapq.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/heapq.cpython-36.pyc'
# extension module '_heapq' loaded from '/home/colman/anaconda3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so'
# extension module '_heapq' executed from '/home/colman/anaconda3/lib/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so'
import '_heapq' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f68ce19a1d0>
import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce212c88>
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# /home/colman/anaconda3/lib/python3.6/__pycache__/reprlib.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/reprlib.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/reprlib.cpython-36.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce20c860>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1e88d0>
# /home/colman/anaconda3/lib/python3.6/__pycache__/weakref.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/weakref.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/weakref.cpython-36.pyc'
import 'weakref' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1e8cc0>
import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1dd160>
# /home/colman/anaconda3/lib/python3.6/collections/__pycache__/abc.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/collections/abc.py
# code object from '/home/colman/anaconda3/lib/python3.6/collections/__pycache__/abc.cpython-36.pyc'
import 'collections.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1ddeb8>
import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4a14a8>
# /home/colman/anaconda3/lib/python3.6/importlib/__pycache__/__init__.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/importlib/__init__.py
# code object from '/home/colman/anaconda3/lib/python3.6/importlib/__pycache__/__init__.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/__pycache__/warnings.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/warnings.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/warnings.cpython-36.pyc'
import 'warnings' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1e4c88>
import 'importlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf4a15c0>
# /home/colman/anaconda3/lib/python3.6/importlib/__pycache__/util.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/importlib/util.py
# code object from '/home/colman/anaconda3/lib/python3.6/importlib/__pycache__/util.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/importlib/__pycache__/abc.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/importlib/abc.py
# code object from '/home/colman/anaconda3/lib/python3.6/importlib/__pycache__/abc.cpython-36.pyc'
# /home/colman/anaconda3/lib/python3.6/importlib/__pycache__/machinery.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/importlib/machinery.py
# code object from '/home/colman/anaconda3/lib/python3.6/importlib/__pycache__/machinery.cpython-36.pyc'
import 'importlib.machinery' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1c95f8>
import 'importlib.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1bdfd0>
# /home/colman/anaconda3/lib/python3.6/__pycache__/contextlib.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/contextlib.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/contextlib.cpython-36.pyc'
import 'contextlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1d7128>
import 'importlib.util' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce1bd978>
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google
# destroy google
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google
# destroy google
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google/logging
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/mpl_toolkits
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google
# destroy google
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google
# destroy google
# possible namespace for /home/colman/anaconda3/lib/python3.6/site-packages/google
# destroy google
# destroy sphinxcontrib
# destroy sphinxcontrib
# destroy sphinxcontrib
# destroy sphinxcontrib
# destroy sphinxcontrib
# destroy sphinxcontrib
# destroy zope
# /home/colman/anaconda3/lib/python3.6/encodings/__pycache__/cp437.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/encodings/cp437.py
# code object from '/home/colman/anaconda3/lib/python3.6/encodings/__pycache__/cp437.cpython-36.pyc'
import 'encodings.cp437' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce176198>
# zipimport: found 15 names in '/home/colman/anaconda3/lib/python3.6/site-packages/tflms-2.0.2-py3.6.egg'
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68cf51acc0>
Python 3.6.12 |Anaconda custom (64-bit)| (default, Sep  8 2020, 23:10:56) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
# extension module 'readline' loaded from '/home/colman/anaconda3/lib/python3.6/lib-dynload/readline.cpython-36m-x86_64-linux-gnu.so'
# extension module 'readline' executed from '/home/colman/anaconda3/lib/python3.6/lib-dynload/readline.cpython-36m-x86_64-linux-gnu.so'
import 'readline' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7f68ce176400>
import 'atexit' # <class '_frozen_importlib.BuiltinImporter'>
# /home/colman/anaconda3/lib/python3.6/__pycache__/rlcompleter.cpython-36.pyc matches /home/colman/anaconda3/lib/python3.6/rlcompleter.py
# code object from '/home/colman/anaconda3/lib/python3.6/__pycache__/rlcompleter.cpython-36.pyc'
import 'rlcompleter' # <_frozen_importlib_external.SourceFileLoader object at 0x7f68ce176518>

For bugs: reproduction and error logs

# Steps to reproduce:
Defined a custom problem which has no problem running.
Then try to use vocab from said custom problem on a new custom problem as follow.

def use_vocab_from_other_problem(self):
return WmtReorder32k()

# Error logs:
...Traceback (most recent call last):
  File "/home/colman/anaconda3/bin/t2t-trainer", line 33, in <module>
    tf.app.run(main)
  File "/home/colman/anaconda3/lib/python3.6/site-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/colman/anaconda3/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/colman/anaconda3/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/colman/anaconda3/bin/t2t-trainer", line 28, in main
    t2t_trainer.main(argv)
  File "/home/colman/anaconda3/lib/python3.6/site-packages/tensor2tensor/bin/t2t_trainer.py", line 409, in main
    generate_data()
  File "/home/colman/anaconda3/lib/python3.6/site-packages/tensor2tensor/bin/t2t_trainer.py", line 302, in generate_data
    registry.problem(problem_name).generate_data(data_dir, tmp_dir)
  File "/home/colman/anaconda3/lib/python3.6/site-packages/tensor2tensor/data_generators/text_problems.py", line 363, in generate_data
    self.generate_encoded_samples(data_dir, tmp_dir, split), paths)
  File "/media/colman/testmount/projects/trans/usr/t2t_usr/word_order.py", line 2602, in generate_encoded_samples
    encoder = self.get_or_create_vocab(data_dir, tmp_dir)
  File "/home/colman/anaconda3/lib/python3.6/site-packages/tensor2tensor/data_generators/text_problems.py", line 249, in get_or_create_vocab
    return other_problem.get_or_create_vocab(data_dir, tmp_dir, force_get)
AttributeError: 'function' object has no attribute 'get_or_create_vocab'
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@colmantse