Skip to content

felipetio/github_api

Repository files navigation

GitHub API

Small service that scrapes User and Repository data from GitHub and stores the data in a database. This service expose a query API that accepts parameters relevant to the data and returns a well-formed API object.

Installation

This steps are necessary to get the application up and running.

It requires Ruby 2.6.3 and Mongo 4.0 previous installed to run. Check this gist to have the pre requisites installed on MacOSX.

$ cd github_api
$ bundle install
$ rake db:setup
$ rake db:create_indexes
$ rails s

Scraping github data

The followings steps are necessary to connect to GitHub’s API to scrape and store Repository and User data.

Fetching current user data

$ cd github_api
$ rake "github:fetch_me" GITHUB_ACCESS_TOKEN=your_top_secret_access_token

Fetching others public user data

$ cd github_api
$ rake "github:fetch[felipetio,fabiofleitas]" GITHUB_ACCESS_TOKEN=your_top_secret_access_token

Cronjobs

To schedule cronjobs to scrape more data you can add a .env file with your token to protected it. GITHUB_ACCESS_TOKEN=your_top_secret_access_token

Now you can add the following comand in you corntab: cd path/to/github_api && bundle exec rake github:fetch_me

Query Format

The query format is designed to be simple and readable in the URL, it follows these rules:

  • All query parameters are encapsulate in a dict, keys are assumed to be field names
  • Special query parameters sort, limit and offset are treated specially, any other query parameter is ignored.

Operators

In the most simplistic form, eq[<field>]=<value> in the query translates to an equals filter, for example name=foo translates to a {name: 'foo'} query filter specification.

In the previous example the eq operator was used. To use other operators change it to supported operators: gt, gte, lt, lte

Values

Values are auto-detected and converted to an data-type, but would be important to have a way explicitly specified in the field spec.

If multiple values are given for the same field, the value is considered an array, regardless of the operator type.

Special Parameters

offset

Specifies the offset into the documents list.

limit

Specifies the number of documents to limit the result.

sort

Field to sort the result on. By default, the sort is in an ascending order.

Examples

Basic search

Search with operators

Sorting results

Pagination

Testing

This steps are necessary to execute the test suite:

$ cd github_api
$ rspec

About

Backend Engineering Take-Home Project

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published