Deal with truncation

[sckott]

https://developer.github.com/v3/gists/#truncation

Applies for GET requests. if files too large, they are truncated, and only part of the file given back, and a field truncated: TRUE. Here's the bit on truncation:

The Gist API provides up to one megabyte of content for each file in the gist. Every call to retrieve a gist through the API has a key called truncated. If truncated is true, the file is too large and only a portion of the contents were returned in content. If you need the full contents of the file, you can make a GET request to the URL specified by raw_url. Be aware that for files larger than ten megabytes, you’ll need to clone the gist via the URL provided by git_pull_url.

[sckott]

for larger files, we could automatically grab files if truncated=TRUE in the response body, but we won't know how large these files are, so i think we'd better not grab these automatically, but just message users that

hey, this is the truncated version, to get the full version, do xyz

[sckott]

above seem reasonable @jennybc @cpsievert

[jennybc]

I don't have anything to say here. Have never run up against this, I guess because I use Gists for truly small things.

[sckott]

thanks @jennybc - just pinged you in case you've dealt with this 😄

[cpsievert]

Sounds good to me! I assume you'll suggest git2r? ;)

[sckott]

i'm soon going to test out using git2r, see these two issues https://github.com/ropensci/gistr/labels/git

following lead of mike bostock's node.js command line client that I believe just uses git instead of the http api -

[karthik]

I don't have anything to say here. Have never run up against this, I guess because I use Gists for truly small things.

I'm with @jennybc here. My use case for gists is also mostly small, reproducible examples to share and get feedback on and I suspect that is the norm. Even a power developer like Hadley (https://gist.github.com/hadley) has only small examples.

Is there a strong reason to spend so much energy on an edge case? @cpsievert, do you have some special use case that requires loading large files into gists?

I'm happy to be wrong here if there is a strong reason to pursue this.

[cpsievert]

Hey @karthik, we've run across this via animint::animint2gist() because some animint visualizations require a lot of data (here's an example from a GOSC applicant that was too big.)

So, this isn't a huge use case, but it'd certainly be nice to have an informative message/warning if things are truncated.

[cpsievert]

BTW, animint::animint2gist() is really useful for us since users can automatically share their viz via http://bl.ocks.org/

[jennybc]

My vague impression is that the main reason to get this working is so that the upload of binary files into Gists works. One can't do that via browser but can via API. Did I imagine this? At least, that's what I was thinking...

[jennybc]

So I thought the reason to push stuff into a Gist via the GitHub API was so one could upload, e.g. a PNG into a Gist. Whenever I want to put up an example with a figure, I have to stash the figure on imgur or some such, which is kind of annoying. If you could get gistr talking to the GitHub API, then this would work. So for me it's not about file size as much as about file type.

[sckott]

AFAIK, truncation (in the context of the github http api) only applies to GET requests, not when POSTing a new gist. For creating gists with larger files, max is 1 MB. I think we'll have to go git route for those.

I think the git route is what we want for both large files, and for binary files (including png files). Got it in draft on a diff. branch, see #39 - as long as git setup is right, it's pretty seamless switching btw http api and git