Written by Adam Kafka
Every day, NASA updates the Astronomy Picture of the Day (APOD), with a new beautiful image or video. This program provides a simple command line interface to download media within a date range from their site. The script uses the NASA API to query the service. In order to use the script, you need an API key which you can obtain for free by filling out the form on their website.
Indiviuals can configure the script by creating a 'config.json' file. A skeleton configuration is included as 'config.json.default'. Users should execute cp config.json.default config.json, then edit 'config.json' in the text editor of their choice. The default file includes descriptions for the fields.
- Download both images and video featured on the APOD site
- Support batch downloads by the use of date ranges as command line arguments
- Detect already downloaded media and skip that date
- Support for usage in a task scheduler such as cron with the 'catch-up' mode
- Easily customizable options in the 'config.json' file
- Can create a hard link in a seperate directory (screensaver_dir)
- Useful if you have one directory used as a source for a screensaver, desktop background, etc.
Scala 2.12.2 and sbt 0.13.15 were used during development. More recent versions of these software should work as well. Please submit an issue if you face one.
This script has only been tested on a macOS computer, but it should work on any unix machine, so try it and let me know how it works!
To download videos as well, the python CLI tool youtube-dl is needed.
To begin, run sbt in the cloned directory. From here, we should be able to test the script and its configuration by simply typing run and pressing enter. This is a good first step to confirm that the program is setup succesfully.
Once this is confirmed, you can experiment with date ranges, by executing statements of the form run [start-yyyy-mm-dd] [end-yyyy-mm-dd], with both parameters defaulting to the value of 'today'. Note that these ranges are inclusive, meaning the end date is downloaded as well.
If it is desired to execute the script outside of the sbt tool, one may do so by compiling a complete jar. The easiest way to do this is with the sbt assembly plugin. One should be able to simply execute sbt assembly or assembly in an sbt shell. If you run into problems, see their site and let me know so I can update the code and documentation. Upon success, a jar in the form of target/scala-2.12/NASA APOD grabber-assembly-1.0.jar will be produced. This jar can be executed indepently of sbt by running scala target/scala-2.12/NASA\ APOD\ grabber-assembly-1.0.jar from within the project root.
-m, --mode [MODE] Specify mode of operation. Valid options are:
'normal' : Download all media between start
and end date (inclusive). This is
the default mode of operation.
'catch-up' : Use the last succesfully downloaded
media date as a start date. This is
useful if the script is run regularly,
say everyday via a cron job. If days
are missed, the script will simply
"pick up where it left off last".
Start and end dates are ignored in
this mode.
-s, --start [DATE] Specify start date in yyyy-mm-dd form (default to today)
-e, --end [DATE] Specify end date in yyyy-mm-dd form (default to today)
-h, --help Display usage and help text
run -s 2016-01-01 -e 2016-12-31Download all APOD media in the year 2016run --start 2017-02-01Download all media from February 1st, 2017 up to and including today.run -m catch-upDiscover the start date by finding the most recent item in 'pod_dir', then continue until today.
As of now, the NASA API has a rate limit of 'Hourly Limit: 1,000 requests per hour' (See their documentation for more details). If you exceed this rate, the script will start to fail, returning an HTTP error code. If this happens, simply wait one hour then re-execute the command. It will pick up where it left off.
The daily nature of the task sets it up as a perfect fit for a cron job. The example below demonstrates a cron task (created using crontab -e) to check hourly between 10am and 9pm, everyday. The example below uses the standalone version compiled with sbt assembly above. The script will only download the media once per day, because uses 'catch-up' mode.
SHELL=/bin/bash #If you use a different shell, change it here
PATH=/usr/local/bin:/usr/local/sbin:... #Make sure 'scala' and 'youtube-dl' are on the path
# Min Hour Day Month DayWeek Command
# (0-59) (0-23) (1-31) (1-12 or Jan-Dec) (0-6 or Sun-Sat)
# Nasa Pic of Day
0 10-21 * * * cd /PATH/TO/REPO; scala target/scala-2.12/NASA\ APOD\ grabber-assembly-1.0.jar -m catch-up >> /PATH/TO/LOG 2>&1
This project was created as an introduction into the Scala programming langauge. Therefore, it is a good example of some of the simple mechanisms that Scala provides to its users.
Options are Scala's cleaner way of dealing with failure in functions that are expected to return an object upon success. For example, MediaGrabber.FetchJson returns Option[Response]. Subsequently, MediaGrabber.GrabMedia uses match in the calling code to deal with successs and failure cleanly.
Try is the functional way of dealing with exceptions. parseDate returns a Try[LocalDate], and all calling code uses the other side of this pattern. A good resource to Try can be found in Part 6 of the Neophytes guide to Scala
For more information, see the blogpost on Alvin Alexander's site. This is used when downloading the image val exit_status = new URL(resp.hdurl) #> new File(out_full) ! and downloading the video with the python program youtube-dl.
Case classes are used to make JSON parsing easy, using the json4s package. This occurs when querying the NASA API (see Response case class) and when parsing the config file (see class Config_opts in the Config singleton object).
Throughout the project, Java code is called alongside Scala. One example of this is date parsing, formatting, and iteration.
The project is composed of three singleton objects: Grab, Config, MediaGrabber. These objects help seperate the logical tasks of the program and avoid polluting the global namesapce.