Support URLs

[timholy]

Right now you can't supply a URL for a filename; you have to download and then load. What do people think about having that be more transparent?

If folks like that: since I'm pretty net-naive, what would be the best thing to search for? Should we require that the string begins with "http://" or "https://"?

CC @alanedelman

[StefanKarpinski]

I strongly object to supporting this with plain strings – it's a huge security hole and means that you can't, based on type analysis, know if code is going to try to hit the network or not! I would be in favor of using a URL type for this instead so you would do URL("https://example.com/data.csv"). That's almost as concise and allows us to use dispatch to handle the case of fetching data.

[Keno]

https://github.com/JuliaWeb/URIParser.jl would be the standard for that.

[timholy]

Thanks, folks---great tips.

[nlw0]

I am trying to load an image from the web. Is the recommended way right now to actually save the data to a file and then load from the file? I would like to decode straight from the file contents in memory. Is there any plan to support something like this with ImageMagick, or is it a better idea to just go for a wrapper to the C libraries?

[timholy]

Perhaps we should add URIParser as a Requires dependency and have this download the file to a temporary location. Thoughts? Care to tackle it @nlw0?

[nlw0]

Eventually someone pointed me out to the "secret" ImageMagick.load_ function, and I've been a happy user ever since...

[oxinabox]

It is practically really not as simple as it might seem for FileIO.jl to handle this with just the URI.
Because the file-extension often matters for how FileIO works.

And getting the filename (and thus extension) is more complex than one might think.
The first half (about 70lines) of this file is dedicated to using HTTP.jl's systems to work it out.
And it still has edge cases it gets wrong (e.g. you are allowed to use fairly arbitrary character set encodings for some of the fields)
https://github.com/JuliaWeb/HTTP.jl/blob/master/src/download.jl

[EricForgy]

It is practically really not as simple as it might seem for FileIO.jl to handle this with just the URI.

How about a data URI?

https://en.wikipedia.org/wiki/Data_URI_scheme

I'm currently having some trouble because I receive an image from the browser as a data URI, e.g.

"data:image/jpeg;base64,long-base-64-encoded-string"

and I want to load this into Images.jl preferably without saving it to a file.

The Julia type could be something like:

struct DataURI
    media
    attributes
    binary
    data
end

I'm kind of stuck on this 😔

[timholy]

For common image formats we can use magic bytes. And any format that doesn't have magic bytes is devil-spawn 😈. There are only so many 3-letter combinations out there, I am amazed that people don't always make use of magic bytes.