Write tests and enable Parser service

[humphd]

We landed the Parser service today, but we have it not starting the actual parser, nor do we have any tests. We need to fix both things. @c3ho added Jest configs for both unit and e2e tests, but there aren't any.

You can get some idea of how to work across services in tests by looking at my auth test here.

[chrispinkney]

@c3ho What's the status of this? Should someone else take this on?

[c3ho]

@chrispinkney Hi Chris, sorry work has been killing me... as you can tell by the time of this message. Could someone else take it on?

[humphd]

@c3ho no worries, thanks so much for getting the ball rolling on this. We'll take it from here.

[humphd]

When this gets fixed, make sure to undo what I did in #2441, where I disabled the parser service.

[chrispinkney]

@manekenpix and I have been trying to work on this slowly and would like to keep going, nearing completion I think!

[humphd]

Cool. I know that @TueeNguyen is interested in working on this as well, maybe you can collaborate?

[TueeNguyen]

I did some comparison between backend and parser today, I see that parser doesn't have admin.js, user.js, stat.js, does parser need these files or these functionalities are handled somewhere else? (Later is probably correct) @manekenpix @chrispinkney @humphd

[manekenpix]

@TueeNguyen the parser service doesn't need those files. admin and user are now part of sso, and stat used to display info in the hero banner, but we don't use it anymore (the idea is to make status do that work).

[TueeNguyen]

Thank you, I just want to confirm that those are not needed

[TueeNguyen]

Reopened because I haven't got tests in

[humphd]

Or Docker bits to turn it on in docker-compose.yml

[TueeNguyen]

@humphd, can you give me some idea how to write e2e test for this?

I plan on adding a few valid, invalid, youtube feeds to the queue, run it and after the queue is drained, I check data in redis, queue's stats

[humphd]

An e2e test would probably have you add some feeds to the queue (later we'd inject them into the database), have the feed queue start, wait on the queue to drain, then use the posts service to query the feeds and posts data.

[TueeNguyen]

Great, my initial thought is almost correct. But, how do I use posts service in parser? Should I import it to parser?

[humphd]

You wouldn't use it in the parser, you'd fetch() it in your tests. The test is connecting the two, they know nothing about each other.

[TueeNguyen]

Hi @humphd, How do I run e2e tests? I don't think I can run it locally.

[humphd]

Identical way we do it in CI:

https://github.com/Seneca-CDOT/telescope/blob/master/.github/workflows/e2e-tests-ci.yml#L28-L30

[TueeNguyen]

I did some research on testing bull queue using ioredis-mock, apparently, it only works with real redis connection. Refer to this issue OptimalBits/bull#962.

[humphd]

This is why we need to write e2e vs. mock unit tests (i.e., so we have a real redis).