Check issue with remote resources.Get and Hugo server

[onedrawingperday]

What version of Hugo are you using (hugo version)?

$ Hugo version

hugo v0.90.0-DEV+extended darwin/arm64 BuildDate=unknown

Does this issue reproduce with the latest release?

No

I just built Hugo from source and tested commit 0eaaa8f that introduces XML support in Hugo.

Using the provided example:

{{ with resources.Get "https://example.com/rss.xml" | transform.Unmarshal }}
     {{ range .channel.item }}
         <strong>{{ .title | plainify | htmlUnescape }}</strong><br />
         <p>{{ .description | plainify | htmlUnescape }}</p>
         {{ $link := .link | plainify | htmlUnescape }}
         <a href="{{ $link }}">{{ $link }}</a><br />
         <hr>
     {{ end }}
 {{ end }}

Upon first execution of the hugo server command I am always getting the following error:
<transform.Unmarshal>: error calling Unmarshal: MIME "application/rss+xml" not supported

However if I execute hugo server first and then paste the above code and hit save, then the remote RSS output is displayed as expected.

Furthermore I am unable to render the contents of the RSS image element from a feed e.g.

  <image>
    <url>https://example.com/image/example.gif</url>
    <title>Example</title>
    <link>https://example.com</link>
  </image>

I tried {{ .image.url }}, {{ .image.title }}, {{ .image.link }}.
They all return empty. Is rendering of this element supported?

Anyway I thought you may want to know.

cc: @bep @vanbroup

[vanbroup]

Thanks for your feedback, do you have a link to the RSS feed or a complete example RSS file that is showing them problem?

Have you tried dumping the results for debugging using:

{{ with resources.Get "https://example.com/rss.xml" | transform.Unmarshal }}
     {{ . }}
 {{ end }}

[onedrawingperday]

By using

{{ with resources.Get "https://example.com/rss.xml" | transform.Unmarshal }}
{{ . }}
{{ end }}

I am getting a map of the feed.

Regarding the image issue I mentioned above, I just realised that most feeds, embed the image inside CDATA typically within the <description> element.

I tried rendering it with: <p>{{ .description | safeHTML }}</p> and I am getting HTML output but I am still unable to render the images locally.

Here are is a sample feed I tested:

https://hyperallergic.com/feed/

Of course all of the above is displayed after the project is served as I mentioned above.

Anyway, thank you for all your hard work in implementing this feature.
It does seems quite complicated though as I've seen all sorts of different feed setups.

P.S. The unmarshall error that I mentioned above still happens. I only get output if I already have the server running and then fetch the remote feed.

[bep]

Thanks for the heads up, that sounds like a issue that needs looking into...

[onedrawingperday]

@vanbroup

I just figured it out and I am able to render remote images locally with {{ .description | safeHTML }}.

The problem I encountered earlier was CORS related, due to the project's server configuration.

Anyway sorry to bother.
Everything seems to work just fine with the new XML support.
Now Hugo can be used as a RSS reader!

Again thanks so much for this!

The transform.Unmarshal error on first server load remains, as mentioned above.

[vanbroup]

I just figured it out and I am able to render remote images locally with {{ .description | safeHTML }}.

The problem I encountered earlier was CORS related, due to the project's server configuration.

This is why the example is using:

{{ .description | plainify | htmlUnescape }}

The embedding of third party HTML comes with significant security risk and should only be used if you completely trust the source.

[vanbroup]

The transform.Unmarshal error on first server load remains, as mentioned above.

I will try to replicate this issue later today.

What is the command you are using to start the server?

[onedrawingperday]

Simply hugo server

[bep]

Info: I'm planning on doing a Hugo release on Monday.

/cc @jmooring

[vanbroup]

@vanbroup

I just figured it out and I am able to render remote images locally with {{ .description | safeHTML }}.

You can also use this method, which is safer than just including third party HTML on your side.

{{ with resources.Get "https://example.com/feed/" | transform.Unmarshal }}
     {{ range .channel.item }}
        <div style="display: inline-block;">
            {{ $desc := (.description | transform.Unmarshal) }}
            {{ with resources.Get (index $desc.img "-src") }}
            {{ $img := .Fill "200x200 smart" }}
            <img src="{{ $img.Permalink }}" style="float: left; margin-right: 10px;"/>
            {{ end }}
            <div>
                <strong>{{ .title | plainify | htmlUnescape }}</strong><br />
                <p>{{ .description | plainify | htmlUnescape }}</p>
                {{ $link := .link | plainify | htmlUnescape }}
                <a href="{{ $link }}">{{ $link }}</a><br />
            </div>
        </div>
        <hr />
     {{ end }}
 {{ end }}

Please note that this downloads and processes images locally, this prevents any CORS issues but keep in mind that images might hold copyrights. You probably want to consider usage and include a copyright notification.

[vanbroup]

Simply hugo server

I have not yet been able to replicate your issue, until now, what operating system are you using?

Can you also test with a new hugo site, using the default config? Just to make sure that this is not triggered by any of the config settings.

[onedrawingperday]

You probably want to consider usage and include a copyright notification.

Thank you very much for the method you posted above.
My use case is the following:
I would like to mirror my Instagram posts on my website.
Currently I am using a third party service to generate the Instagram RSS, up until now I was rendering what I needed with a Python tool.

Your method will be simplifying my workflow.

I already sort of anticipated that there was a method to get the img -src embedded in a RSS description element but I hadn't had the time to explore it just yet.

This feature is way cooler than I thought.

[onedrawingperday]

what operating system are you using?
I am on the latest macOS 12.0.1

Just checked with another skeleton Hugo project, unfortunately again the error is there:

Start building sites … 
hugo v0.90.0-DEV+extended darwin/arm64 BuildDate=unknown
ERROR 2021/12/03 11:07:25 render of "section" failed: "/Users/alex/Downloads/test/layouts/_default/list.html:2:91": execute of template failed: template: _default/list.html:2:91: executing "main" at <transform.Unmarshal>: error calling Unmarshal: MIME "application/rss+xml" not supported
ERROR 2021/12/03 11:07:25 render of "section" failed: "/Users/alex/Downloads/test/layouts/_default/list.html:2:91": execute of template failed: template: _default/list.html:2:91: executing "main" at <transform.Unmarshal>: error calling Unmarshal: MIME "application/rss+xml" not supported
ERROR 2021/12/03 11:07:25 render of "section" failed: "/Users/alex/Downloads/test/layouts/_default/list.html:2:91": execute of template failed: template: _default/list.html:2:91: executing "main" at <transform.Unmarshal>: error calling Unmarshal: MIME "application/rss+xml" not supported
ERROR 2021/12/03 11:07:25 render of "section" failed: "/Users/alex/Downloads/test/layouts/_default/list.html:2:91": execute of template failed: template: _default/list.html:2:91: executing "main" at <transform.Unmarshal>: error calling Unmarshal: MIME "application/rss+xml" not supported
Error: Error building site: failed to render pages: render of "section" failed: "/Users/alex/Downloads/test/layouts/_default/list.html:2:91": execute of template failed: template: _default/list.html:2:91: executing "main" at <transform.Unmarshal>: error calling Unmarshal: MIME "application/rss+xml" not supported

[onedrawingperday]

I just tested the above method:

{{ with resources.Get "https://example.com/feed/" | transform.Unmarshal }}
     {{ range .channel.item }}
        <div style="display: inline-block;">
            {{ $desc := (.description | transform.Unmarshal) }}
            {{ with resources.Get (index $desc.img "-src") }}
            {{ $img := .Fill "200x200 smart" }}
            <img src="{{ $img.Permalink }}" style="float: left; margin-right: 10px;"/>
            {{ end }}
            <div>
                <strong>{{ .title | plainify | htmlUnescape }}</strong><br />
                <p>{{ .description | plainify | htmlUnescape }}</p>
                {{ $link := .link | plainify | htmlUnescape }}
                <a href="{{ $link }}">{{ $link }}</a><br />
            </div>
        </div>
        <hr />
     {{ end }}
 {{ end }}

Unfortunately I am getting another error.

ERROR 2021/12/03 11:24:04 render of "section" failed: "/Users/alex/Downloads/test/layouts/_default/list.html:5:57": execute of template failed: template: _default/list.html:5:57: executing "main" at <transform.Unmarshal>: error calling Unmarshal: interface conversion: interface {} is string, not map[string]interface {}

Apparently transform.Unmarshal has a problem with the input:
{{ $desc := (.description | transform.Unmarshal) }}

error calling Unmarshal: interface conversion: interface {} is string, not map[string]interface {}

so (just for fun) I tried: {{ $desc := (dict (slice "a" "b" "c") .description | transform.Unmarshal) }}

just see what might happen and then I got:

error calling Unmarshal: type map[string]interface {} not supported

[vanbroup]

I think that could happen if the description doesn't contain any HTML, it might need a with or a string content check before trying to unmarshal.

[onedrawingperday]

Ok, I will try later today and report back.

[vanbroup]

I was able to replicate the MIME "application/rss+xml" not supported problem on darwin/amd64, @bep any idea why this works on Linux and Windows but not on MacOS?

[bep]

any idea why this works on Linux and Windows but not on MacOS?

I have no idea ... I'm on MacOS so I could eventually look at it.

[onedrawingperday]

I forgot to mention that I am on macOS darwin/arm64

[vanbroup]

I was able to solve the problem by adding the rss file extension to my config file:

mediaTypes:
  application/rss+xml:
    suffixes:
    - xml
    - rss

This doesn't explain why is works after the server is started and why it works on Linux and on Windows.

Adding the rss extension in the default config (here: https://github.com/gohugoio/hugo/blob/master/media/mediaType.go#L171) solves the problem.

I will create a pull request to get this sorted.

[onedrawingperday]

@bep
The above fix by @vanbroup also works on my end.

Also I just tested the method posted above for downloading and processing images with a feed that contains HTML in the `description element and everything works flawlessly.

The Instagram feed I'm checking has a different setup. I suppose that I will figure it out eventually.

Thank you both so much. This new feature offers plenty of new possibilities.

[onedrawingperday]

Just in case someone else tries to render an image in Hugo from a remote XML feed and the above method throws the aforementioned error because the input element is perceived as a string, I was able to render the remote image as follows.

Remote input sample:

Template code:

    {{ $img := findRE `src="([^"]+)"` .description 1 | print }}
    {{ $img = $img | replaceRE `\[src="([^"]+)"\]` `${1}` | urls.Parse }}
 {{ with resources.Get $img.Path }}
            {{ $img := .Fill "200x200 smart" }}
            <img src="{{ $img.Permalink }}" style="float: left; margin-right: 10px;"/>
            {{ end }}>

Notes

• The print function after findRE is required because otherwise Hugo throws a cryptic error:
  unable to cast []string ... of type []string to string

• urls.Parse is required to decode the remote image link

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.