Post parsing/sync microservice #1828

c3ho · 2021-02-25T22:46:22Z

Issue This PR Addresses

Type of Change

Bugfix: Change which fixes an issue
New Feature: Change which adds functionality
Documentation Update: Change which improves documentation
UI: Change which improves UI

Description

Still a WIP. Creates a post parsing microservice from our current code. There's a ton of files and I just want to get this up here for now. I opted to use the JS version of bull-queue, whether that is the correct choice or not I have no idea, but it shouldn't be too hard to make the switch to TS version in a future PR.

I started a small change in processAllFeeds() within the index.js file which will make a GET all user feeds request (that are not flagged) to the User ms.. I'll slowly improve this to do it in a pagination way.

I checked the Post api, there's no PUT routes yet to save all the Post data as per the most recent discussion.

As satellite is waiting for Redis PR to be put in, this container will crash since I can't call Redis() yet. I'll also remove the test route I set up at parserURL/

Lists of tasks for this PR. I'll mark them off (maybe with a commit #) to track progress.

Actually test this
Write tests
Create readme
Create Dockerfile
Update Feeds to store less info in Redis (need to doublecheck what we need and don't need)

Checklist

Quality: This PR builds and passes our npm test and works locally
Tests: This PR includes thorough tests or an explanation of why it does not
Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not (if applicable)
Documentation: This PR includes updated/added documentation to user exposed functionality or configuration variables are added/changed or an explanation of why it does not(if applicable)

manekenpix

You probably need to rebase. docker-compose-api-production.yml doesn't exist anymore. Now it's production.yml.
Also, make sure you use the env files for port number or anything the microservice needs.

humphd · 2021-02-28T23:14:11Z

We need to consider the role of this service within the larger system we're building. We have:

User service (@chrispinkney in Add a User service, with database for user, feed info, drop planet feed list #1642): keeps track of users and their feeds
Post service (@HyperTHD in Creating a Post microservice #1735): deals with managing posts in Redis and Elastic, and has cross-over with this

This service should essentially manage the scheduling and processing of Feeds from to Posts. A bunch of the code is already written, but it needs to be evolved a bit. I think we need

A Job Queue on top of Redis (we have this)
A Job Worker process for extracting parsing jobs from the Job Queue, doing them, and sending results to the Posts service (we have most of this, but it needs to be taught to talk HTTP)
A Job Scheduler, something that figures how when to go get Feeds and put them into the Job Queue to be processed again. I'm not 100% clear on the best way to do this. It could be time based, could be event based, ...

We used to use the Wiki as a source of truth, and that's being replaced by the User service.

@c3ho I see that you're using your favourite "Draft" style PR for this, but I think I'm going to need this to be done more collaboratively with the authors of the other services so that we can figure out our boundaries. Probably this one is easiest to write last (after Users and Posts are merged). That should happen for the 1.8 release in a few weeks.

Until then, maybe you can help review those other services with us as they come in, so we are all on the same page.

rjayroso · 2021-03-09T15:17:27Z

Based on our triage today regarding the user service and post-service it would make sense that we could land this for 1.9

HyperTHD · 2021-03-15T03:43:15Z

Posts service has landed, so you can continue working on this service, and move out any Feed related stuff to your service, @c3ho

c3ho · 2021-03-16T22:12:11Z

@humphd @HyperTHD I don't think I should be managing any Post info here right?

How I think the service should work:

Send a request to User ms to get all feed urls.
Create queue and add feed urls to job list to process feed urls
Process feed urls and send the processed Post info to the Post ms to index in ElasticSearch/Redis
Cache processed feeds in Redis
Rinse and repeat perpetually

In this case, I shouldn't care about how Post is implemented aside from sending parsed blog posts to the Post ms to store/index right? So in this case I am guessing I make a change to articlesToPosts function in processor.js to make
a POST request to the Post ms instead of what it is doing currently?

humphd · 2021-03-18T00:30:31Z

@c3ho, I'm still not 100% sure of the proper design, but here are my thoughts:

The major thing this service does is maintain a job queue, and co-ordinate the creation of jobs. How we do that is a bit of an open question. We could poll, like we do now. We could use repeatable jobs (but we'd still have to maintain sync with the users/feeds in Firebase). Somehow we have to get the feeds into this queue and process them, and repeat that. So the easiest first method would be to get all the feeds from Firebase, yes. It's probably not smart to "download all" like we do with the wiki, so maybe we iterate through the pages and add the jobs. There might be a more "Firebase" way to do this, not sure.

Next we process the feeds like we do now, essentially, and when we're done, I think we put them directly into Redis/Elastic from this service, vs. involving the Posts service (it's read-only).

We need to keep Redis (cache) and Elastic (search) in sync with what's in Firebase. Our feeds won't live in Redis anymore, we'll just cache posts there (with feed details that a post needs). One way to do this would be to add one more key to Redis to track the current ordered set we're using to index posts: we can be showing the old set in the timeline while we build the new one, then delete the old one and continue the process.

We can iterate on this once the service exists. If it mostly works like we have now, with User service vs. wiki feed list, it will be good.

c3ho · 2021-03-18T01:01:02Z

@humphd Thanks for the clarification. You're right, let's get the service working first then we'll reiterate / refactor.

HyperTHD · 2021-03-23T14:36:53Z

@c3ho Are you still interested in working on this PR? We're gonna need this in for 2.0 so someone else can take this issue if you're busy with something else

c3ho · 2021-03-23T14:40:47Z

I’ll have a reviewable PR for Friday

humphd · 2021-03-26T15:22:38Z

config/env.development

+################################################################################
+
+# Parser Service Port (default is 8888)
+PARSER_PORT=8888


We have a lot of things running on 8***, let's pick something else, 10000?

config/env.development

humphd · 2021-03-26T15:23:26Z

config/env.production

+PARSER_PORT=8888
+
+# Parser Service URL
+PARSER_URL=https://localhost/v1/parser


https://api.telescope.cdot.systems/v1/parser

humphd · 2021-03-26T15:23:45Z

config/env.staging

+PARSER_PORT=8888
+
+# Parser Service URL
+PARSER_URL=https://localhost/v1/parser


https://dev.api.telescope.cdot.systems/v1/parser

humphd · 2021-03-26T15:24:07Z

docker/docker-compose.yml

+      - traefik
+      - redis
+      - users
+      - posts


I don't think you'll need posts.

humphd · 2021-03-26T15:27:01Z

src/api/parser/jest.setup.js

@@ -0,0 +1,10 @@
+const path = require('path');
+const result = require('dotenv').config({


I think this has been done in the base config (cc @HyperTHD).

Yup, this file is not needed anymore and can be removed

humphd · 2021-03-26T15:27:29Z

src/api/parser/package.json

+  },
+  "homepage": "https://github.com/Seneca-CDOT/telescope#readme",
+  "dependencies": {
+    "@senecacdot/satellite": "^1.8.0",


humphd · 2021-03-26T15:29:09Z

src/api/parser/src/index.js

@@ -0,0 +1,111 @@
+const { Satellite } = require('@senecacdot/satellite');
+const { router } = require('bull-board');
+// May need in next revision


What does this mean?

This was supposed to have line 4 commented out as I wasn't sure if we'll need other routes aside from the bull-board

humphd · 2021-03-26T15:30:00Z

src/api/parser/src/index.js

+ * @param {Object} feedData - feed data parsed from the wiki feed list.
+ * Returns Promise<Feed>, with the most appropriate Feed Object to use.
+ */
+async function updateFeed(feedData) {


You're doing too much in this file. You should split things up, so that this file just sets up the microservice with Satellite, and all this extra functionality can come from other modules.

humphd · 2021-03-26T15:38:13Z

src/api/parser/src/routes/parser.js

+const router = Router();
+
+router.get('/', (req, res) => {
+  res.status(200).send('hi');


This seems not worth doing. Should we move what you have at /queues to this instead? That is, going to http://localhost/v1/parser would show me the Bull Queues.

This also implies that we need some authentication/authorization (see Satellite, which has middleware you can use for this). We either need to require admin access, or maybe just authenticated users can see it? I'm not sure how sensitive this is.

Sure, I can disregard the whole parser.js file and just do service.router.use('/', bullRouter) in index.js.

humphd · 2021-04-08T18:57:47Z

docker/development.yml

+
+  parser:
+    environment:
+      # For development and testing, the Auth service needs to contact the users


"the Parser service..."

src/api/parser/jest.config.js

humphd · 2021-04-08T19:00:16Z

src/api/parser/readme.md

+
+# dev mode with automatic restarts
+
+npm run dev


You only have a start script above, better change this to start.

humphd · 2021-04-08T19:01:41Z

src/api/parser/src/data/feed.js

+   * Flags the feed, preventing posts from the feed to be displayed
+   * Returns a Promise<Boolean>
+   */
+  flag() {


This stuff will probably need to be changed later to use what's in the Users data vs. what's in Redis. For now, this is fine, but FYI so we don't forget.

src/api/parser/src/lib/queue.js

humphd · 2021-04-09T18:10:40Z

src/api/parser/package.json

+  "name": "parser",
+  "version": "1.0.0",
+  "description": "A service for parsing posts",
+  "main": "index.js",


This can be removed.

humphd · 2021-04-09T18:16:39Z

src/api/parser/src/index.js

+
+const service = new Satellite();
+
+startParserQueue();


Let's comment this out so that we can land on prod/staging today, and not have it compete with the existing backend that is also running a Bull queue in Redis.

vercel bot deployed to Preview February 25, 2021 22:46 View deployment

vercel bot deployed to Preview February 25, 2021 23:01 View deployment

vercel bot deployed to Preview February 25, 2021 23:06 View deployment

manekenpix requested changes Feb 26, 2021

View reviewed changes

HyperTHD mentioned this pull request Feb 26, 2021

Adds Initial Service to Get Posts #1838

Merged

8 tasks

rjayroso added this to the 1.9 Release milestone Mar 9, 2021

This was referenced Mar 12, 2021

Remove Feed related code from Posts Service #1907

Open

Fix #1706: Add Microservices docker-compose to production startup #1900

Merged

chrispinkney added the area: microservices label Mar 16, 2021

chrispinkney assigned c3ho Mar 16, 2021

chrispinkney mentioned this pull request Mar 16, 2021

Proper logging for invalid feeds #653

Closed

c3ho force-pushed the pps-ms branch from 8d0635d to 28cf4e0 Compare March 25, 2021 23:32

vercel bot deployed to Preview March 25, 2021 23:32 View deployment

humphd mentioned this pull request Mar 26, 2021

Put Redis between Users Service and Firestore #2030

Closed

vercel bot had a problem deploying to Preview March 26, 2021 02:11 Failure

c3ho force-pushed the pps-ms branch from 54cd3cd to f04849c Compare March 26, 2021 03:16

c3ho marked this pull request as ready for review March 26, 2021 03:16

vercel bot deployed to Preview March 26, 2021 03:16 View deployment

vercel bot deployed to Preview March 26, 2021 03:33 View deployment

vercel bot had a problem deploying to Preview March 26, 2021 03:48 Failure

vercel bot deployed to Preview March 26, 2021 08:46 View deployment

humphd requested changes Mar 26, 2021

View reviewed changes

c3ho added 14 commits April 8, 2021 03:12

Post parsing/sync microservice

6837524

docker files + readme

83660eb

updated docker-compose file

7881397

updated production file

675d2d5

config files

f0880df

changes

c4070bb

updating readmes

fe9410d

moved part of server.js code to index.js

8037650

added res.status check and changed logic

74cfdc1

changes according to review

449322b

more changes

dc1a678

updated env.staging

8614d7b

changes

535d53a

Updated to use bearer token and docker env

ffb66a9

c3ho force-pushed the pps-ms branch from 3e9a5f0 to ffb66a9 Compare April 8, 2021 07:21

vercel bot deployed to Preview April 8, 2021 07:21 View deployment

c3ho added 2 commits April 8, 2021 03:23

added start script

48a4697

added missing dependency

7e93136

vercel bot deployed to Preview April 8, 2021 16:32 View deployment

humphd requested changes Apr 8, 2021

View reviewed changes

changes

4514667

vercel bot deployed to Preview April 9, 2021 15:58 View deployment

added playright preset to jest.config.e2e

15ad89e

vercel bot deployed to Preview April 9, 2021 16:02 View deployment

humphd requested changes Apr 9, 2021

View reviewed changes

removed entry point and commented out startParserQueue

fbaa57e

vercel bot deployed to Preview April 9, 2021 18:20 View deployment

humphd approved these changes Apr 9, 2021

View reviewed changes

chrispinkney approved these changes Apr 9, 2021

View reviewed changes

humphd merged commit e7df2e3 into Seneca-CDOT:master Apr 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Post parsing/sync microservice #1828

Post parsing/sync microservice #1828

c3ho commented Feb 25, 2021 •

edited

Loading

manekenpix left a comment

humphd commented Feb 28, 2021

rjayroso commented Mar 9, 2021

HyperTHD commented Mar 15, 2021

c3ho commented Mar 16, 2021 •

edited

Loading

humphd commented Mar 18, 2021

c3ho commented Mar 18, 2021

HyperTHD commented Mar 23, 2021

c3ho commented Mar 23, 2021

humphd Mar 26, 2021

humphd Mar 26, 2021

humphd Mar 26, 2021

humphd Mar 26, 2021

humphd Mar 26, 2021

HyperTHD Mar 26, 2021

humphd Mar 26, 2021

humphd Mar 26, 2021

c3ho Mar 26, 2021

humphd Mar 26, 2021

humphd Mar 26, 2021

c3ho Mar 26, 2021 •

edited

Loading

humphd Apr 8, 2021

humphd Apr 8, 2021

humphd Apr 8, 2021

humphd Apr 9, 2021

humphd Apr 9, 2021

		@@ -0,0 +1,10 @@
		const path = require('path');
		const result = require('dotenv').config({

Post parsing/sync microservice #1828

Post parsing/sync microservice #1828

Conversation

c3ho commented Feb 25, 2021 • edited Loading

Issue This PR Addresses

Type of Change

Description

Checklist

manekenpix left a comment

Choose a reason for hiding this comment

humphd commented Feb 28, 2021

rjayroso commented Mar 9, 2021

HyperTHD commented Mar 15, 2021

c3ho commented Mar 16, 2021 • edited Loading

humphd commented Mar 18, 2021

c3ho commented Mar 18, 2021

HyperTHD commented Mar 23, 2021

c3ho commented Mar 23, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c3ho Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c3ho commented Feb 25, 2021 •

edited

Loading

c3ho commented Mar 16, 2021 •

edited

Loading

c3ho Mar 26, 2021 •

edited

Loading