Code

Backend Scraper#87

Code

Merged

leventov merged 24 commits intomasterclimatescape/climatescape.org:masterfrom add-backendclimatescape/climatescape.org:add-backendCopy head branch name to clipboardMar 21, 2020

+10,016

Commits

Commits on Feb 24, 2020

• Revert "Revert "Add backend, part of #40""

  Show description for 29879c6
  bloudermilk

  committed
  29879c6

Commits on Feb 25, 2020

• Fix backend readme to mention yarn 1.x, not 2 install, and link to the discussion from 5-use-yarn.md

  leventov

  committed
  56c3e7d

• More links from decision docs to discussions.

  leventov

  committed
  d6ccd83

• Add comment to isProduction

  leventov

  committed
  4f70864

Commits on Feb 26, 2020

• backend: add airtable.js

  leventov

  committed
  6b3a569

• backend: reformat pg.js according to the project's style

  leventov

  committed
  14c7c1b

• backend: reformat setupScraping

  leventov

  committed
  3ca9d7c

• backend: reformat worker.js

  leventov

  committed
  34a99c9

• backend: migrate to sequelize

  leventov

  committed
  b68663e

Commits on Feb 27, 2020

• backend: add backupAirtable.js, setupAirtableBackup.js

  leventov

  committed
  91c6899

• backend: update yarn instructions in README

  leventov

  committed
  7166a87

• backend: in backupAirtable.js, write orgs to Postgres

  leventov

  committed
  6fb93a4

Commits on Mar 2, 2020

• Update decision docs

  leventov

  committed
  003bbc8

Commits on Mar 5, 2020

• Migrate to Knex, add backupAirtable.test.js

  leventov

  committed
  9563e73

• Add decision doc about pushing data to Airtable from backend

  leventov

  committed
  ae3615e

Commits on Mar 6, 2020

• Add determineOrgsToScrapeFirstTime function

  leventov

  committed
  e5369a1

Commits on Mar 7, 2020

• Add addFirstTimeScrapingJobs() function

  leventov

  committed
  53f1866

Commits on Mar 8, 2020

• Connect to Twitter API

  leventov

  committed
  337cf32

Commits on Mar 9, 2020

• Refactor twitter.js and addFirstTimeScrapingJobs.js

  leventov

  committed
  127f3d7

• Complete first-time scraping of Twitter followers

  leventov

  committed
  450614e

Commits on Mar 10, 2020

• backend: refactoring: store just Twitter followers -> store the whole Twitter user objects in scraping_results table

  leventov

  committed
  651595b

• backend: refactor: extract twitterUserObjectScraping.js from worker.js

  leventov

  committed
  1a4d346

• backend: refactor: extract firstTimeScraping.js from addFirstTimeScrapingJobs.js, add setupPgBossQueue() function and pass pgBossQueue around explicitly

  leventov

  committed
  e87151c

• Fix bug in getTwitterScreenName()

  leventov

  committed
  de6bd42