Update parser to fetch feeds from Supabase

[Kevan-Y]

What would you like to be added:

Currently, we have a parser that fetches feed from CDOT Feed List.
We want to update our parser to fetch feed from Supabase once #2860 is implemented.

[TueeNguyen]

After discussing. We decided to to

• Write a seed script to migrate our wiki feeds to Supabase leveraging the /tools/migrate/migrate.js.
• After seeding, find a way to persist the data (repo level like redis-data for development, and higher repo level for production like redis in production.yml

  telescope/docker/production.yml

  Lines 173 to 177 in bc01b29

  	redis:
  	restart: unless-stopped
  	volumes:
  	# TODO: confirm this path...
  	- ../../redis-data:/data

• Modify this

  telescope/src/api/parser/src/parser.js

  Line 62 in bc01b29

  const [all, wiki] = await Promise.all([Feed.all(), getWikiFeeds()]);

  to something like

// Supabase may not allow us to return all data at once, if it's the case => do it in portions
const feeds = await getWiki() // getWiki() queries feeds from Supabase

• For development, seeding all the feeds into Supabase may be too much, we thought of adding a small number of feeds but this is for later

cc @humphd @DukeManh

[TueeNguyen]

Currently waiting for #3373 to merge previous test changes before I can continue to change the test to reflect on the parser's change

[humphd]

OK, #3373 is in, what's next?

[JerryHue]

This issue is related to #1907, and #3355.

[JerryHue]

I would like to point out the things we need to also target for the parser service:

• The parser service needs to pull the feeds from Supabase instead of redis and the wiki (done in Update parser to fetch feeds from Supabase #2861).
• The parser service needs to invalidate the feeds on Supabase instead of redis.
• The parser service needs to flag feeds on Supabase instead of redis.
• The parser service does not need to updateFeeds anymore. Refer to:
  

  telescope/src/api/parser/src/parser.js

  Line 48 in d56306a

  const currentFeed = await updateFeed(feed);

  
  

  telescope/src/api/parser/src/parser.js

  Lines 12 to 28 in d56306a

  	const updateFeed = async (feedData) => {
  	let currentFeed;
  	// If we have an existing feed in the database for this URL, prefer that,
  	// since it might have updated cache info (e.g., etag).
  	const existingFeed = await Feed.byUrl(feedData.url);
  	if (existingFeed) {
  	// We have a version of this feed in the database already, prefer that
  	currentFeed = existingFeed;
  	} else {
  	// First time we're seeing this feed, add it to the database
  	const id = await Feed.create(feedData);
  	currentFeed = await Feed.byId(id);
  	}
  	return currentFeed;
  	};

We may target the previous concerns in a PR separate from #2861.

A few clarifications:

• Redis may keep the etag and lastModified fields, since they would not be that ergonomic in Postgres.
• Delayed feeds can be kept in Redis, since they would not be as ergonomic in Postgres.