Skip to content

New building system blueprint

Jump to bottom Edit New page
Nintendofan885 edited this page Sep 9, 2020 · 29 revisions

A new build system for rg3/youtube-dl

Characteristics and features

  • Central point: how do we want the user to be able to install? (pip, make, download...) should be easy to guess, or accomodate every way
  • Platforms: zipfile, exe, linux packaging, git checkout ?
  • One code base for Python 2 and Python 3
  • Building from Makefile or setup.py? unified or keep both ?
  • What about the other generated files ? (README.md, bash completion, man page) How to make them easy to install ? How to still provide enough information about the depot ?
  • IMHO (FiloSottile) we should remove them from the tree and let the user build them from README.md with make docs and install with make install
  • gcmalloc: maybe just keeping the README to know what the depot is about. Create a minimal readme ?
  • Yup, I thought about keeping only the README, updated at every interested commit, and generating on demand all the rest. This is also quite git philosophy, not blocking simple merges.
  • GPG sign tags and downloads
  • Slightly unrelated: setup.py ready for PyPI (for pip and easy_install)
  • A real one-shoot release.sh script, with docs update

Cleaning

  • Modularity of the IE ? later ? (currently working on something related! @FiloSottile)

Testing

  • Travis-CI (@rg3)
  • Periodic builds
  • mailing hook for travis.
  • testing the update ? cloning an old version then trying to run it should update it.
  • modularize, putting the test videos data in a json, and make a metatest, that tests for fields in info_dict and for their encodings (UNICODE) and skips if not _WORKING

Infrastructure

  • Win: @fraca7 buildbot (Thanks!)
  • GH Downloads section (?) using the GH api
  • I can provide a wm for youtube-dl at my local hackerspace (https://wiki.hackerspaces.org/FIXME) (@gcmalloc)

Transition

  • How to switch from one download location to the new ?
  • Do we have the right to upload to the rg3 depot ?
  • what about the old version ? Do we take them from the repository ?

Related bugs

  • Help OpenBSD packagers #574
  • Remove exe builds - and general discussion #548
  • Remove binary files from repo #423
  • setup.py and packaging #412 and #460
  • GPG signing #479
  • Travis hook #490
  • Update fail when GH in maintainance mode #399
  • Please use gpg signed release tags #479

Where to do it ?

Comunication

Relevant IRC logs

10:47 FiloSottile: ------> First
10:48 FiloSottile: a single, downloadable, executable standalone file
10:48 FiloSottile: multi-platform, that updates itself with -U
10:48 FiloSottile: >>> The zipfile or exe serves as that
10:49 FiloSottile: ------> Second
10:49 FiloSottile: pip install youtube-dl
10:50 FiloSottile: should install the package *and the various man/bash files*
10:50 gcmalloc: I think that if you have a proper setup.py you can then build the packages from there
10:50 gcmalloc: that's what py2exe uses right ?
10:51 FiloSottile: and all those files should be updateable from pip
10:51 FiloSottile: and finally, youtube-dl should appear in $PATH
10:51 FiloSottile: >>> That's where setup.py enters the game
10:52 FiloSottile: gcmalloc: umh... seems like over-engineering to me, as it's as simple as
10:52 FiloSottile: zip --quiet youtube-dl __main__.py youtube_dl/*.py
10:52 FiloSottile: and adding a shebang
10:52 FiloSottile: ------> Third
10:52 FiloSottile: the building, developer and packager flow
10:53 FiloSottile: should be easy and intuitive, both to use and maintain
10:54 FiloSottile: >>> Here's the Makefile/setup.py question, and if build_exe.py should stay stand-alone
10:55 FiloSottile: gcmalloc: The Makefile continues to seems the easier method, also thinking for example to building a deb file, but it can be a bias from myself
10:56 gcmalloc: but you can build a deb file from a proper python package, refering to http://ghantoos.org/2008/10/19/creating-a-deb-package-from-a-python-setuppy/ 
10:58 gcmalloc: this part is really the central point of a python package, even for scripts. From it, you can build big binary files. But they are a way, just a result built by side script based on the setup.py
10:58 gcmalloc: at least that's how I do it
11:04 FiloSottile: gcmalloc: oh, OK you convinced me! :D
11:06 FiloSottile: gcmalloc: but, look here http://ghantoos.org/2008/10/19/creating-a-deb-package-from-a-python-setuppy/#Makefile
11:06 gcmalloc: waiiit
11:06 gcmalloc: I'm not sure myself
11:06 gcmalloc: the makefile is really much simpler to write, so easier to maintain 
11:07 gcmalloc: maybe there is a place for it.
11:07 FiloSottile: We can have both!
11:07 FiloSottile: look at the lik above
11:07 FiloSottile: and here http://lshell.cvs.sourceforge.net/viewvc/lshell/lshell/setup.py?view=markup
11:07 gcmalloc: xD
11:07 gcmalloc: I was on the same link 
11:07 gcmalloc: oh
11:07 gcmalloc: that settle the issue
11:08 FiloSottile: That's how I always wanted it! A clean, pythonic, without hacks setup.py that makes pip work
11:08 FiloSottile: and a Makefile that builds stuff that depends on it
11:08 FiloSottile: I think that also build_exe.py should be separate AND depend on setup.py
11:09 gcmalloc: ahh
11:09 gcmalloc: why ?
11:10 FiloSottile: why the build_exe? well, look at it, it's full of hacks, I don't think it can work as a setup.py
11:11 FiloSottile: no, wait, I;m wrong
11:12 FiloSottile: simply, we should ignore or delete the build folder from somewhere else
11:13 FiloSottile: but they can be merged!
11:13 gcmalloc: there is also this : http://cx-freeze.sourceforge.net/
11:13 gcmalloc: merged and cleaned, that's worth the try
11:14 FiloSottile: gcmalloc: Note: Binary packages must be compatible with the Python installation on your system
11:14 FiloSottile: I don't get it
11:15 gcmalloc: if you have python2.7 on your system, you must have a 2.7 binary

Windows build

My (fraca7) first idea was to provide a Buildbot slave but since it would be used only to build the .exe and youtube-dl already uses Travis for CI, this would be a bit overkill IMO. Another approach would be a simple build system on the XP machine, which would triggered by an HTTP get and cache the resulting binary. From the client side this would look like:

# Trigger build and display log (transfer encoding is chunked)
wget -O - 'http://jeromelaheurte.net:8042/youtube-dl/build?repo=<repo name>&rev=<revision>' || exit 1
# Get binary (cached)
wget 'http://jeromelaheurte.net:8042/youtube-dl/download?repo=<repo name>&rev=<revision>'

How is this ?

Add a custom footer
Add a custom sidebar
Clone this wiki locally