[MRG] Data/model storage. Fix 1453 #1705
Conversation
gensim/downloader.py
Outdated
|
|
||
|
|
||
| def load(name, return_path=False): | ||
| """Download (if needed) dataset/model and load it to memory (managed by return_path). |
There was a problem hiding this comment.
"managed by return_path" doesn't sound correct. Do you mean "unless return_path is set"?
There was a problem hiding this comment.
Yes, that's true.
gensim/downloader.py
Outdated
| _create_base_dir() | ||
| file_name = _get_filename(name) | ||
| if file_name is None: | ||
| raise Exception( |
There was a problem hiding this comment.
Exception too generic. This looks like a ValueError.
gensim/downloader.py
Outdated
| folder_dir = os.path.join(base_dir, name) | ||
| path = os.path.join(folder_dir, file_name) | ||
| if not os.path.exists(folder_dir): | ||
| _download(name) |
There was a problem hiding this comment.
Any way to "force" a download? (fix previously broken / partial / deleted downloads etc)
There was a problem hiding this comment.
This situation doesn't happen (because data moved from /tmp/.. to ~/gensim-data after downloading needed files + checking md5). For this reason, we don't add this key (if something happens - run load again.
| else: | ||
| sys.path.insert(0, base_dir) | ||
| module = __import__(name) | ||
| return module.load_data() |
There was a problem hiding this comment.
Where do I find this load_data function? For example for Wikipedia (json-line format from segment_wiki), and for text8?
I got a bit lost in the import magic, and there are no comments.
|
Need to add the last feature:
The process must look like this:
|
* added download and catalogue functions * added link and info * modeified link and info functions * Updated download function * Added logging * Added load function * Removed unused imports * added check for installed models * updated download function * Improved help for terminal * load returns model path * added jupyter notebook and merged code * alternate names for load * corrected formatting * added checksum after download * refactored code * removed log file code * added progressbar * fixed pep8 * added tests * added download for >2gb data * add test for multipart * fixed pep8 * remove tar.gz, use only .gz for all * fix codestyle/docstrings[1] * add module docstring * add downloader to apiref * Fix CLI + more documentation * documentation for load * renaming * fix tests * fix tests[2] * add test for info * reduce logging * Add return_path=True example to docstring * fix * update & rename notebook * Fix docstring + use ValueError when name is incorrect * move list to global var
Based on #1453.
What's done:
--catalogue(instead of this used--info, similar to python API)