Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor inactive-blog-filter.js to use Redis vs. files #405

Closed
humphd opened this issue Dec 2, 2019 · 9 comments
Closed

Refactor inactive-blog-filter.js to use Redis vs. files #405

humphd opened this issue Dec 2, 2019 · 9 comments
Assignees
Labels
area: back-end area: redis Redis Database related

Comments

@humphd
Copy link
Contributor

humphd commented Dec 2, 2019

We have some logic that is using files to deal with state, and we need to move that into Redis, so it can be distributed across nodes in production.

The various files that are being used in inactive-blog-filter.js should be refactored to use Sets in Redis, which we should be able to use to do set operations (unions, diffs, intersections, etc), and thereby keep our list of feeds up to date.

See the code in storage.js for some ideas on how to do this. We'll need to add more code there for this use case.

@humphd humphd added area: back-end area: redis Redis Database related labels Dec 2, 2019
@Silvyre
Copy link
Contributor

Silvyre commented Dec 2, 2019

Please allow me!

@Silvyre Silvyre self-assigned this Dec 2, 2019
@Silvyre
Copy link
Contributor

Silvyre commented Dec 2, 2019

Stepping aside from this to assist with #406. Anyone is welcome to take my place.

@Silvyre Silvyre removed their assignment Dec 2, 2019
@haichuan0424
Copy link
Contributor

Karla and I are going to do this.

@haichuan0424
Copy link
Contributor

Going to take #270 instead. Feel free to take this.

@c3ho
Copy link
Contributor

c3ho commented Dec 5, 2019

If someone is not taking this, I think I can work on it too, since there is some dependency from #462 on this.

@c3ho c3ho self-assigned this Dec 7, 2019
@c3ho
Copy link
Contributor

c3ho commented Dec 16, 2019

This depends on changes from #462

c3ho added a commit to c3ho/telescope that referenced this issue Jan 29, 2020
@c3ho
Copy link
Contributor

c3ho commented Jan 31, 2020

async function check(feedUrl, redlistPath = 'feeds-redlist.json') {
  // Read redlist file
  try {
    const redListRaw = await fsPromises.readFile(redlistPath, 'utf-8');
    const redList = [].concat(JSON.parse(redListRaw));
    return redList.some(redItem => isRedlisted(feedUrl, redItem));
  } catch (error) {
    logger.error({ error }, `failed to read ${redlistPath}`);
    throw error;
  }
}

@humphd Most of the code in in inactive-blog-filter is using a feedUrl, did you want to change to using a feedId here instead?

@humphd
Copy link
Contributor Author

humphd commented Feb 1, 2020

I want to get rid of this code completely, and use the the invalid key concept with a feed id like I mentioned in the other comments I left.

@c3ho
Copy link
Contributor

c3ho commented Feb 9, 2020

Closing this as it was addressed by #630

@c3ho c3ho closed this as completed Feb 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: back-end area: redis Redis Database related
Projects
None yet
Development

No branches or pull requests

5 participants