15.4.0 - Most feeds no longer sync

[DavidHaltinner]

IMPORTANT

Read and tick the following checkbox after you have created the issue or place an x inside the brackets ;)

• I have read the CONTRIBUTING.md and followed the provided tips
• I accept that the issue will be closed without comment if I do not check here
• I accept that the issue will be closed without comment if I do not fill out all items in the issue template.

Explain the Problem

What problem did you encounter?
After updating to 15.4.0, the majority of my feeds no longer sync. Manually going to various feeds in a web browser shows new content. Debug level logs also dont mention an attempt to sync the rss feeds of those ones that are not being synced. It lists ones that do work, and a few random 403s and 404s and what not, but the ones that dont sync at all, dont show in the logs.

Steps to Reproduce

Explain what you did to encounter the issue

1. Update to 15.4.0
2. Sync feeds via cron numerous tmes over the last day
3. Realize that feeds were not updating.

System Information

• News app version: 15.4.0
• Nextcloud version: 21.0.1.1
• Cron type: (system cron/python updater/...): cron
• PHP version: 7.4.18RC1
• Database and version: MariaDB 15.1
• Browser and version: Firefox 88
• OS and version: Fedora 34

Contents of nextcloud/data/nextcloud.log

news.log

Contents of Browser Error Console

Read http://ggnome.com/wiki/Using_The_Browser_Error_Console if you are unsure what to put here

Sync issue, browser not involved.

[SMillerDev]

Can you try toggling full text mode for those?

[DavidHaltinner]

Can you try toggling full text mode for those?

No change in syncing unfortunately. (My sync interval is 900 seconds, i verified that it had ran and that it sync'ed others that were working too).

[pjlinden]

Unfortunately I have the same problems. Within Nextcloud itself and all my rss readers. It seems it doesn't update the newsfeeds at all anymore. Blank screen...

[SMillerDev]

No change in syncing unfortunately. (My sync interval is 900 seconds, i verified that it had ran and that it sync'ed others that were working too).

Do you have some feeds you can share where this happens?

[DavidHaltinner]

No change in syncing unfortunately. (My sync interval is 900 seconds, i verified that it had ran and that it sync'ed others that were working too).

Do you have some feeds you can share where this happens?

Here are a few, I can provide more if needed:

https://io9.gizmodo.com/rss
https://blogmickey.com/feed/
https://trekmovie.com/feed/

[DavidHaltinner]

Of note, to test it out, I deleted one feed, ran the cron so it could clean up, and then re-added the feed, and that first immediate sync worked on it. I will watch it for if the sync updates next time the source feed is updated.

But even if it does work, it is not ideal to have to wipe out all the feeds and put them back. While that's easy enough for me, a single user self-hosted system, those that have many users would not have the ease to do that.

[anoymouserver]

Sounds like one of the fetched feeds has an error, breaking the fetch loop for every following feed.

Possible candidates with fetch errors accoring to the log, are (none of these URLs worked for me):

• http://feeds.newscientist.com/section/news/feed/content
• https://news.nationalgeographic.com/rss/index.rss
• https://www.linuxinsider.com/perl/syndication/rssfull.pl
• https://rhelblog.redhat.com/feed/
• https://owncloud.com/rss-feed/

If you have database access and know how to handle it, you could set the prevent_update field fot he feed to 1 otherwise I would recommend to remove these feeds.

Removing and adding it again bypasses the feed fetcher once but doesn't solve it.

[DavidHaltinner]

i did a 'update oc_news_feeds set prevent_update='1' where update_error_count>1;' since they all had an error count more than 1. It appears the other feeds did sync now.

So i guess the question is why the loop breaks when a single feed errors. I would expect it to simply skip it and keep going, which is I believe the way it handled it before.

[SMillerDev]

I believe the last time the fetch code changed was 14.0. And even before that I don't think it kept going on errors. I do think this should be improved though.

[smoebody]

I have the same problem since news 15.4

In my case the cause seems to be that I use the docker-image (20-fpm) which lacks support for NET_URL2:

$ ./occ news:updater:update-feed smoe 1
An unhandled exception has been thrown:
Error: Class 'Net_URL2' not found in /var/www/html/custom_apps/news/lib/Fetcher/FeedFetcher.php:114
Stack trace:
#0 /var/www/html/custom_apps/news/lib/Service/FeedServiceV2.php(265): OCA\News\Fetcher\FeedFetcher->fetch('https://xkcd.co...', false, NULL, NULL)
#1 /var/www/html/custom_apps/news/lib/Command/Updater/UpdateFeed.php(58): OCA\News\Service\FeedServiceV2->fetch(Object(OCA\News\Db\Feed))
#2 /var/www/html/custom_apps/bookmarks/vendor/symfony/console/Command/Command.php(255): OCA\News\Command\Updater\UpdateFeed->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 /var/www/html/custom_apps/bookmarks/vendor/symfony/console/Application.php(1009): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#4 /var/www/html/custom_apps/bookmarks/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OCA\News\Command\Updater\UpdateFeed), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#5 /var/www/html/custom_apps/bookmarks/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#6 /var/www/html/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/html/console.php(100): OC\Console\Application->run()
#8 /var/www/html/occ(11): require_once('/var/www/html/c...')

[anoymouserver]

I'm not sure how Net_URL2 could be missing since it's bundled with the app inside vendor/pear/net_url2. Did you install News manually or using the app store?
To me it looks like there is something mixed up (but I don't know anything about the NC Docker image) ... the apps are installed under custom_apps and somehow Symfony is used from the bookmarks app instead of from the NC core.

[utack]

I've had a refresh stuck for approx 10h despite cron running just fine
Then it started syncing again
Will browse in the log when I have time, but one feed is getting errors
Maybe it is just hung up for a reaaaly long time on that feed and recovers later?

[smoebody]

i could fix it by removing and fresh installing the app

[maxolasersquad]

I'm running into this as well. I run the following command to update all of my feeds. You will need to supply the $USERNAME you want to update the feeds for or assign the variable the correct value, and replace 191 with at least the highest feed id in your database.

for ((i=1;i<=191;i++)); do sudo -u www-data -H ./occ news:updater:update-feed $USERNAME $i; done;

[skruehoetl]

Not sure if it's the same issue but since my feeds are not updating (anymore) and I also don't seem to get an error message, I'd thought I'd join in.
My issue started after moving my Nextcloud (Docker) from one server to a new one. The transfer went fine and everything started synching - except News.

• Removing and reinstalling didn't help
• Deleting the feeds and re-importing them from OPML was unsuccessful - I got a warning like "feed already exists" for each feed I imported. (Same behaviour for manual import.) --> obviously, deleting the feeds from the web frontend was not complete
• I removed the app, purged the database using this set of instructions (btw: first items, then feeds, then folders), and reinstalled/reimported (didn't help, but I could re-import)
• I removed the app, purged the database, re-installed and added each feed manually watching the logs (they seemed happy)

What I noticed is, that for some reason, the "unread" does no longer sort by newest article date, but seems to sort by "latest added feed"...

Any ideas? Different issue?

(News 15.4.5, PHP 7.4.19, latest Docker image (apache) with Nextcloud 21.0.2, added smb capabilities but no issue on old server with the same config)

[SMillerDev]

What I noticed is, that for some reason, the "unread" does no longer sort by newest article date, but seems to sort by "latest added feed"...

That's because it's sorting on the newest articles for your server. If you have a feed your server didn't know about before those are the newest to the server.

New articles after this will behave as you expect.

[Eothred]

Not sure if this is related to the same issue, but I get the error message attached from my log. The particular feed mentioned in the error message is xmlUrl="https://arseblog.com/feed/" htmlUrl="https://arseblog.com"
nextcloud_news_error.txt

[deadwax]

I ran into a similar issue which was caused by nextcloud/vm#2039. Resolved that and news updated again. Interesting that I only noticed cron had completely broken because my news feed stopped updating.

[Eothred]

Following up on my previous comment, when I removed one offending feed, another blocked the sync (showed up in the log). After removing the ones below it started to work again. Perhaps you can find a pattern in these feeds, or perhaps it could be related to a combination of having these plus some others.

I any case, I think there is a more significant problem in that the sync should continue for the other feeds, even if there is a faulty one. Or maybe I misunderstand what is happening.

The offending feeds were

https://kryptografen.no/feed/
https://blog.revolut.com/rss/
https://coincentral.com/feed/
https://arseblog.com/feed/

After removing these, sync has worked fine for about a week now.

The fact I can see it in the admin log makes me wonder if I am talking about a different issue than David perhaps.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[danielrheinbay]

I any case, I think there is a more significant problem in that the sync should continue for the other feeds, even if there is a faulty one.

I second this, and it's not resolved.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Eothred]

I believe this is still not fixed, please do not close.

[Grotax]

I don't think there is currently anything that can be done, I read through this issue and can't really see a reason for why the update does not work.

There is a known issue that the Nextcloud job system can't handle interrupts, so if PHP, the Webserver, the DB,.. get restarted during the job execution it will hang for at least 12h.

That can't be fixed within news.

If your feed throws an error during update, you need to provide the feed and the error. Then maybe someone would be willing to investigate.

Therefore I think this can be closed as the title and the content don't really help an nobody wanted to take care of it for many weeks.

[Eothred]

Not sure if this is related to the same issue, but I get the error message attached from my log. The particular feed mentioned in the error message is xmlUrl="https://arseblog.com/feed/" htmlUrl="https://arseblog.com" nextcloud_news_error.txt

The error mentioned in this comment of mine was caused by https://arseblog.com/feed/ but I am unfortunately not enough of an expert to understand if it is useful enough.

[Grotax]

I checked this feed with our feed library, no error and I added it to my nextcloud also no issue.
I checked the error text you attatched to this issue, the error is that your DB could not process this particular Item.

This is well known and documented in our faq -> https://github.com/nextcloud/news/blob/d9cb3433085ac2a43cd34b4232350f1578ce8201/docs/faq.md#missing-4-byte-support-sqlstate22007-invalid-datetime-format-1366-incorrect-string-value-

[Eothred]

Aha, great, thanks! Sorry for not finding it myself.

I am not the original reporter, but for me this goes a long way to help me avoid the issue. I have seen the warning about 4 byte support in nextcloud admin but never found the time to fix it.