Removing System Calls

[glenn-jocher]

@kalenmike this thread is about removing system calls like you suggested. Following removal of unzip calls in #4919 the main remaining system calls are here:

subprocess.check_output()

We use this to return values

os.system()

Used when only a success/failure output is required

Added to our TODO list.

[github-actions]

👋 Hello @glenn-jocher, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

$ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@kalenmike most of the calls are either pip, curl, gsutil or git commands. The state of these is:

• pip: from https://stackoverflow.com/a/50255019

The officially recommended way to install packages from a script is by calling pip's command-line interface via a subprocess. Most other answers presented here are not supported by pip. Furthermore since pip v10, all code has been moved to pip._internal precisely in order to make it clear to users that programmatic use of pip is not allowed.

Use sys.executable to ensure that you will call the same pip associated with the current runtime.

import subprocess
import sys

def install(package):
    subprocess.check_call([sys.executable, "-m", "pip", "install", package])

• curl: curl commands are useful for special download handling not available in other packages (when cookies are required, for automatic retry and resume on fail (important for large dataset downloads with shaky connections like Objects365)), and as backup download commands to the default torch download function:
  

  yolov5/utils/downloads.py

  Lines 24 to 41 in 39c17ce

  	def safe_download(file, url, url2=None, min_bytes=1E0, error_msg=''):
  	# Attempts to download file from url or url2, checks and removes incomplete downloads < min_bytes
  	file = Path(file)
  	assert_msg = f"Downloaded file '{file}' does not exist or size is < min_bytes={min_bytes}"
  	try: # url1
  	print(f'Downloading {url} to {file}...')
  	torch.hub.download_url_to_file(url, str(file))
  	assert file.exists() and file.stat().st_size > min_bytes, assert_msg # check
  	except Exception as e: # url2
  	file.unlink(missing_ok=True) # remove partial downloads
  	print(f'ERROR: {e}\nRe-attempting {url2 or url} to {file}...')
  	os.system(f"curl -L '{url2 or url}' -o '{file}' --retry 3 -C -") # curl download, retry and resume on fail
  	finally:
  	if not file.exists() or file.stat().st_size < min_bytes: # check
  	file.unlink(missing_ok=True) # remove partial downloads
  	print(f"ERROR: {assert_msg}\n{error_msg}")
  	print('')

• gsutil: mostly used for undocumented GCP functionality during evolution. gsutil is available as a pip install, but the directions (https://cloud.google.com/storage/docs/gsutil_install#debian-and-ubuntu) have a big "Experimental" warning and look pretty complicated, and also vary by OS.

• git: used to check git status and advise user to update repo (hopefully addressing our huge issue of users not aware of updated code). GitPython package is available as a pip install (https://github.com/gitpython-developers/GitPython) but seems to be a 3rd party tool and no longer under active development (but is maintained).

[thecaffeinedev]

I can work on creating a PR, replacing the system calls with Python libs. If it's okay ? @glenn-jocher

[glenn-jocher]

@thecaffeinedev we were looking at this a few weeks ago to try to improve compatibility across environments. Unfortunately most of the remaining system calls are not easy to remove as shown in #4922 (comment) above. If you have any ideas though then yes, please submit a PR. Thanks!

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!