Gitpod cloud development evironment #2439

DukeManh · 2021-11-02T17:07:15Z

Issue This PR Addresses

I think Gitpod is now fully working as intended.

The cool thing is we don't need gitpod.env anymore. Gitpod now uses env.development as a base and updates the URLs every time a workspace starts.

gitpod-io · 2021-11-02T17:07:18Z

DukeManh · 2021-11-02T17:26:55Z

Hmm, it looks like the frontend is fetching from old workspace urls.
I did try to rebuild nginx when a workspace is restarted but that didn't seem to help.

humphd · 2021-11-02T17:29:52Z

.gitpod.yml

-      npm run start
+      docker-compose --env-file .env up -d --no-dep nginx
+      docker-compose --env-file .env up -d
+      gp await-port 9200 && npm run start


Add a comment here that we're waiting on Elasticsearch, so it's clear why you're doing this. Having said that, I'm not clear if we need this. @manekenpix has code in our Elasticsearch client that retries connections.

On some occasions, the npm run start script will cancel itself before Elasticsearch has a chance to start and be connected to. I don't know if the retries are based on a certain number of attempts or after a certain amount of time. For some reason, we usually need to wait longer for Elasticsearch

We should probably file an issue to bump the number of retries we do, see

telescope/src/backend/utils/indexer.js

Lines 127 to 167 in 29951b7

/**

* Checks elasticsearch's connectivity

*/

const checkConnection = () => client.cluster.health();

const waitOnReady = async () => {

/**

* Elasticsearch needs time after deployment for setting up all its components.

* Here we set a timer using 'setTimeout' and check for connectivity during the countdown so elasticsearch

* has time to be fully prepared to start indexing posts.

*/

const DELAY = process.env.ELASTIC_DELAY_MS || 10000;

let intervalId;

let timerId;

const timer = new Promise((resolve, reject) => {

timerId = setTimeout(() => {

reject(

new Error(

'Unable to connect to Elasticsearch. Use `MOCK_ELASTIC=1` in your `.env` to mock Elasticsearch, or install (see https://github.com/Seneca-CDOT/telescope/blob/master/docs/environment-setup.md)'

)

);

}, DELAY);

});

const connectivity = new Promise((resolve) => {

intervalId = setIntervalAsync(async () => {

try {

await checkConnection();

resolve();

} catch (error) {

logger.info('Attempting to connect to elasticsearch...');

}

}, 500);

});

await Promise.race([timer, connectivity]);

await clearIntervalAsync(intervalId);

clearTimeout(timerId);

};

The back-end uses this to determine for how long it should try to connect to ES (it's time, not number of attempts). If it can't connect, the back-end exits, BUT in production/staging we run the back-end in a container with unless-stopped, so it tries again after shutting down.

humphd · 2021-11-02T17:33:58Z

.gitpod.yml

    command: |
-      docker-compose --env-file config/env.gitpod up -d
-      npm run start
+      docker-compose --env-file .env up -d --no-dep nginx


Can you explain why you're running docker-compose multiple times here?

What if we use npm run dev instead of building the nginx-based frontend?

What we really need to do is rework our microservices so that we're building and putting in a container registry from CI vs. always building them from scratch. cc @raygervais for any ideas on how to approach this in a follow-up.

What if we use npm run dev instead of building the nginx-based frontend?

That's a very good idea. I didn't think of that.

What we really need to do is rework our microservices so that we're building and putting in a container registry from CI.

Current build time is somewhere around 20 minutes, which is unbearable.

20 min!? Yeah, we need to pull cached builds vs. creating our own.

@humphd, is there any way I can start all services except the frontend? Or I'd have to list all services by hand.

I'm not aware of a way to do this without saying what you want. What you did is how I'd do it.

humphd · 2021-11-02T17:35:54Z

.gitpod.yml

+      eval $(gp env -e WEB_URL=$(gp url 8000))
+      eval $(gp env -e API_HOST=$(gp url 8443))
+
+      sed -r \


cindyorangis · 2021-11-02T17:37:39Z

I'm testing it out on my machine too but I've been hitting this CORS error all day. Even in the AWS issues #2398 #2390 that are kinda similar to this PR here. No idea if it's on my end or what.

In your sed replacement command, there's one line that remained 'localhost'

ALLOWED_APP_ORIGINS=https://8000-lavender-lemur-niijfpmt.ws-us17.gitpod.io http://localhost:8888

DukeManh · 2021-11-02T17:48:34Z

@cindyledev, notice the urls different, they are ...lavender.. gitpod.io vs ..apricot..gitpod.io. So I think the problem is our Next is using the previous URLs passed to it, not the latest ones because workspace URLs are not stable.

In your sed replacement command, there's one line that remained 'localhost'

I wonder if this would affect anything? I'm currently replacing localhost:3000/8000/8443 only.

cindyorangis · 2021-11-02T17:53:46Z

@cindyledev, notice the urls different, they are ...lavender.. gitpod.io vs ..apricot..gitpod.io. So I think the problem is our Next is using the previous URLs passed to it, not the latest ones because workspace URLs are not stable.

In your sed replacement command, there's one line that remained 'localhost'

I wonder if this would affect anything? I'm currently replacing localhost:3000/8000/8443 only.

I'm able to hit

https://3000-lavender-lemur-niijfpmt.ws-us17.gitpod.io

and

https://8443-lavender-lemur-niijfpmt.ws-us17.gitpod.io/v1/posts

but for some reason, the Next.js frontend is not fetching any data. I wonder where it's trying to get data from...

humphd · 2021-11-02T17:57:06Z

It should be hitting https://8443-lavender-lemur-niijfpmt.ws-us17.gitpod.io/v1/posts (works for me here).

curl -I https://8443-lavender-lemur-niijfpmt.ws-us17.gitpod.io/v1/posts

HTTP/2 200
access-control-allow-origin: *
content-security-policy: default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests
content-type: application/json; charset=utf-8
date: Tue, 02 Nov 2021 17:56:32 GMT
etag: W/"c13-qHxwJ9fAjsNjhhlmoCXTbr0teJc"
expect-ct: max-age=0
link: </posts?per_page=30&page=2>; rel="next", </posts?per_page=30&page=147>; rel="prev", </posts?per_page=30&page=1>; rel="first", </posts?per_page=30&page=147>; rel="last"
referrer-policy: no-referrer
strict-transport-security: max-age=15552000; includeSubDomains
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-permitted-cross-domain-policies: none
x-total-count: 4411
x-xss-protection: 0
content-length: 3091

This has the proper CORS header.

cindyorangis · 2021-11-02T18:01:30Z

It should be hitting https://8443-lavender-lemur-niijfpmt.ws-us17.gitpod.io/v1/posts (works for me here).

curl -I https://8443-lavender-lemur-niijfpmt.ws-us17.gitpod.io/v1/posts

HTTP/2 200
access-control-allow-origin: *
content-security-policy: default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;frame-ancestors 'self';img-src 'self' data:;object-src 'none';script-src 'self';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests
content-type: application/json; charset=utf-8
date: Tue, 02 Nov 2021 17:56:32 GMT
etag: W/"c13-qHxwJ9fAjsNjhhlmoCXTbr0teJc"
expect-ct: max-age=0
link: </posts?per_page=30&page=2>; rel="next", </posts?per_page=30&page=147>; rel="prev", </posts?per_page=30&page=1>; rel="first", </posts?per_page=30&page=147>; rel="last"
referrer-policy: no-referrer
strict-transport-security: max-age=15552000; includeSubDomains
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-permitted-cross-domain-policies: none
x-total-count: 4411
x-xss-protection: 0
content-length: 3091

This has the proper CORS header.

Silly question. Would Next.js need access-control-allow-origin: * as well?

DukeManh · 2021-11-02T18:04:28Z

My working server with npm run dev here, it's hitting the corrects urls.
https://8000-tan-cuckoo-unufc9u0.ws-us17.gitpod.io/

cindyorangis · 2021-11-02T18:05:21Z

My working server with npm run dev here, it's hitting the corrects urls. https://8000-tan-cuckoo-unufc9u0.ws-us17.gitpod.io/

I see it!! Wow!

DukeManh · 2021-11-02T18:09:27Z

Oops, CI failing.

.gitpod.yml

humphd · 2021-11-03T00:17:48Z

.gitpod.yml

+      eval $(gp env -e API_URL=$(gp url 3000))
+      eval $(gp env -e WEB_URL=$(gp url 8000))
+      eval $(gp env -e API_HOST=$(gp url 8443))
+      eval $(gp env -e SERVICES="status image auth search posts feed-discovery users parser planet redis elasticsearch login firebase")


If we want to try and trim this down a bit we could remove planet. It's not needed in development (unless you're specifically working with it), which almost no one will be.

I trimmed planet.

humphd · 2021-11-03T02:32:10Z

Can you rebase to pick up the changes I had to make for the parser being disabled?

DukeManh · 2021-11-03T02:38:06Z

Let me do that again, I didn't disable the parser.

humphd

This looks right to me.

humphd · 2021-11-03T03:05:16Z

I got this working (my first time using GitPod, so cool)!:

I ran into a few things:

React warnings Fix React warning on page load #2443
CORS issues on Satellite health checks [Satellite] Add CORS to health checks #2444
Had to manually restart the backend due to elasticsearch not responding in time:

[ 2021-11-03 02:55:55.062 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Loading all feeds into feed queue for processing
[ 2021-11-03 02:55:55.070 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Telescope listening on port 3000
[ 2021-11-03 02:55:55.582 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:56.076 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:56.580 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:57.081 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:57.582 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:58.084 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:58.587 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:59.095 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:55:59.594 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:00.115 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:00.601 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:01.105 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:01.605 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:02.109 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:02.609 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:03.116 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:03.615 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:04.114 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:04.616 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Attempting to connect to elasticsearch...
[ 2021-11-03 02:56:05.070 ] ERROR (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Unable to connect to Elasticsearch. Use `MOCK_ELASTIC=1` in your `.env` to mock Elasticsearch, or install (see https://github.com/Seneca-CDOT/telescope/blob/master/docs/environment-setup.md)
    Error: Unable to connect to Elasticsearch. Use `MOCK_ELASTIC=1` in your `.env` to mock Elasticsearch, or install (see https://github.com/Seneca-CDOT/telescope/blob/master/docs/environment-setup.md)
        at Timeout._onTimeout (/workspace/telescope/src/backend/utils/indexer.js:145:9)
        at listOnTimeout (node:internal/timers:557:17)
        at processTimers (node:internal/timers:500:7)
[ 2021-11-03 02:56:05.071 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Received UNHANDLED REJECTION, starting shut down
[ 2021-11-03 02:56:05.071 ] ERROR (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01):
    error: {}
[ 2021-11-03 02:56:05.073 ] DEBUG (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Queue feed-queue paused.
[ 2021-11-03 02:56:05.073 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Feed queue shut down.
[ 2021-11-03 02:56:05.074 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Web server shut down.
[ 2021-11-03 02:56:05.074 ] INFO (7775 on ws-f40fc609-55dc-4058-ab0b-8ebddc633b01): Completing shut down.

It would be good to solve this, as most new devs won't have any idea how to solve this.

Also, it's slow to start. We need to improve. But it will work!

cindyorangis

Wow, this is finally working on my end!!!

DukeManh · 2021-11-03T13:46:04Z

Merged!

vercel bot deployed to Preview November 2, 2021 17:07 View deployment

DukeManh requested review from humphd and manekenpix November 2, 2021 17:07

humphd requested changes Nov 2, 2021

View reviewed changes

vercel bot deployed to Preview November 2, 2021 19:02 View deployment

manekenpix reviewed Nov 2, 2021

View reviewed changes

.gitpod.yml Show resolved Hide resolved

humphd requested changes Nov 3, 2021

View reviewed changes

vercel bot deployed to Preview November 3, 2021 01:23 View deployment

DukeManh requested review from manekenpix, cindyorangis and humphd November 3, 2021 01:43

Generate custom .env file

c69b068

DukeManh force-pushed the gitpod branch from bea3fd2 to b507c5b Compare November 3, 2021 02:36

vercel bot deployed to Preview November 3, 2021 02:37 View deployment

Reconfigure Gitpod

5a7d462

DukeManh force-pushed the gitpod branch from b507c5b to 5a7d462 Compare November 3, 2021 02:41

vercel bot deployed to Preview November 3, 2021 02:41 View deployment

humphd approved these changes Nov 3, 2021

View reviewed changes

cindyorangis approved these changes Nov 3, 2021

View reviewed changes

humphd merged commit 555f497 into Seneca-CDOT:master Nov 3, 2021

humphd mentioned this pull request Nov 3, 2021

Add documentation on how to use GitPod to develop/run the code #2429

Closed

cindyorangis mentioned this pull request Nov 4, 2021

Fixes #2381 Update AWS Cloud9 documentation #2390

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gitpod cloud development evironment #2439

Gitpod cloud development evironment #2439

DukeManh commented Nov 2, 2021 •

edited by gitpod-io bot

Loading

gitpod-io bot commented Nov 2, 2021

DukeManh commented Nov 2, 2021 •

edited

Loading

humphd Nov 2, 2021

cindyorangis Nov 2, 2021

humphd Nov 2, 2021

manekenpix Nov 2, 2021

humphd Nov 2, 2021

humphd Nov 2, 2021

humphd Nov 2, 2021

DukeManh Nov 2, 2021

DukeManh Nov 2, 2021

humphd Nov 2, 2021

DukeManh Nov 2, 2021

humphd Nov 3, 2021

humphd Nov 2, 2021

cindyorangis commented Nov 2, 2021

DukeManh commented Nov 2, 2021

cindyorangis commented Nov 2, 2021

humphd commented Nov 2, 2021

cindyorangis commented Nov 2, 2021 •

edited

Loading

DukeManh commented Nov 2, 2021

cindyorangis commented Nov 2, 2021

DukeManh commented Nov 2, 2021

humphd Nov 3, 2021

DukeManh Nov 3, 2021

humphd commented Nov 3, 2021

DukeManh commented Nov 3, 2021

humphd left a comment

humphd commented Nov 3, 2021

cindyorangis left a comment

DukeManh commented Nov 3, 2021

	/**
	* Checks elasticsearch's connectivity
	*/
	const checkConnection = () => client.cluster.health();

	const waitOnReady = async () => {
	/**
	* Elasticsearch needs time after deployment for setting up all its components.
	* Here we set a timer using 'setTimeout' and check for connectivity during the countdown so elasticsearch
	* has time to be fully prepared to start indexing posts.
	*/
	const DELAY = process.env.ELASTIC_DELAY_MS \|\| 10000;
	let intervalId;
	let timerId;

	const timer = new Promise((resolve, reject) => {
	timerId = setTimeout(() => {
	reject(
	new Error(
	'Unable to connect to Elasticsearch. Use `MOCK_ELASTIC=1` in your `.env` to mock Elasticsearch, or install (see https://github.com/Seneca-CDOT/telescope/blob/master/docs/environment-setup.md)'
	)
	);
	}, DELAY);
	});

	const connectivity = new Promise((resolve) => {
	intervalId = setIntervalAsync(async () => {
	try {
	await checkConnection();
	resolve();
	} catch (error) {
	logger.info('Attempting to connect to elasticsearch...');
	}
	}, 500);
	});

	await Promise.race([timer, connectivity]);

	await clearIntervalAsync(intervalId);
	clearTimeout(timerId);
	};

Gitpod cloud development evironment #2439

Gitpod cloud development evironment #2439

Conversation

DukeManh commented Nov 2, 2021 • edited by gitpod-io bot Loading

Issue This PR Addresses

gitpod-io bot commented Nov 2, 2021

DukeManh commented Nov 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cindyorangis commented Nov 2, 2021

DukeManh commented Nov 2, 2021

cindyorangis commented Nov 2, 2021

humphd commented Nov 2, 2021

cindyorangis commented Nov 2, 2021 • edited Loading

DukeManh commented Nov 2, 2021

cindyorangis commented Nov 2, 2021

DukeManh commented Nov 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humphd commented Nov 3, 2021

DukeManh commented Nov 3, 2021

humphd left a comment

Choose a reason for hiding this comment

humphd commented Nov 3, 2021

cindyorangis left a comment

Choose a reason for hiding this comment

DukeManh commented Nov 3, 2021

DukeManh commented Nov 2, 2021 •

edited by gitpod-io bot

Loading

DukeManh commented Nov 2, 2021 •

edited

Loading

cindyorangis commented Nov 2, 2021 •

edited

Loading