Support for Bitfinex Importing (take 2) #1145

cmroche · 2017-09-21T01:10:03Z

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

(feature) Support for importing trade data from Bitfinex

What is the current behavior? (You can also link to an open issue here)

Issue #918

What is the new behavior (if this is a feature change)?

Bitfinex import is functional though due to the high volume nature of this exchange it should be noted that those imports can be very slow and large.

The importer uses a 2.5 second delay between requests to prevent coming up against the API rate limit. While the Bitfinex API documentation says that you can either do 60 or 90 requests per second (depending on the source in their website) in practice I'm finding it to be a little bit lower than that.

Since it is only possible to import historical data using the v2 api, I've update the bitfinex-api-node library and made minor changes for compatibility to the initialization code. The regular exchange functions still use the v1 API as v2 is considered beta and a bit of a pain to change over.

Other information:

Bitfinex's new API for trades uses a time range for the import, start and end must be specified for the query to work, and results are limited to a max of 1000 per query. However if the time range exceeds the record limit the API will return results from the end point first heading towards start, that is will return more recent trades first. Because of this approach it is necessary to do the import by iterating backwards, however since this doesn't work with many modules the system will substep by breaking the import into sequential time ranges, then reverse iterate to fill them before sending to the dispatcher.

Update bitfinex-api-node to latest

…ration

streetbits · 2017-09-26T01:05:47Z

@cmroche Receiving this error when trying to import data from bitfinex

/home/gekko3/importers/exchanges/bitfinex.js:24
    this.bitfinex.makePublicRequest(path, (err, data) => {
                  ^

TypeError: this.bitfinex.makePublicRequest is not a function
    at Trader.Fetcher.getTrades (/home/gekko3/importers/exchanges/bitfinex.js:24:19)
    at Trader.bound [as getTrades] (/home/gekko3/node_modules/lodash/dist/lodash.js:729:21)
    at Object.fetch (/home/gekko3/importers/exchanges/bitfinex.js:71:17)
    at Market.get (/home/gekko3/core/markets/importer.js:95:16)
    at Market.bound [as get] (/home/gekko3/node_modules/lodash/dist/lodash.js:729:21)
    at new Market (/home/gekko3/core/markets/importer.js:80:8)
    at /home/gekko3/core/pipeline.js:187:20
    at /home/gekko3/node_modules/async/dist/async.js:3853:9
    at /home/gekko3/node_modules/async/dist/async.js:484:16
    at replenish (/home/gekko3/node_modules/async/dist/async.js:1025:25)
RECEIVED ERROR IN IMPORT 9603528732250863
Child process has died.

cmroche · 2017-09-26T01:07:27Z

@streetbits You probably need to npm install as the bitfinex-node-api package was updated to support this.

streetbits · 2017-09-27T20:24:10Z

Worked like a charm, thanks man you're the best

This is noticeable when grabing BCH before August.

PhatDoan · 2017-09-30T01:31:08Z

@streetbits Could you please show me the step how you got it import? I tried but it still shows this error. Gekko can use Bitfinex exchange in watch mode.

Do I need to do anything else? Thank you.

$ node gekko --config myconfig.js --import

2017-09-30 08:29:42 (INFO):	Setting up Gekko in importer mode
2017-09-30 08:29:42 (INFO):	
2017-09-30 08:29:42 (WARN):	The plugin Trading Advisor does not support the mode importer. It has been disabled.
2017-09-30 08:29:42 (WARN):	The plugin Paper Trader does not support the mode importer. It has been disabled.
2017-09-30 08:29:42 (WARN):	The plugin Performance Analyzer does not support the mode importer. It has been disabled.
2017-09-30 08:29:42 (DEBUG):	No end date specified for importing, setting to 2017-09-30 01:29:42

 ERROR: The exchange Bitfinex does not provide full history (or Gekko doesn't support importing it)

cmroche · 2017-09-30T01:34:42Z

@PhatDoan It looks like you didn't apply the changes correctly. The exchanges/bitfinex.js file should have an update with providesFullHistory: true which can be seen in the changes for the pull request.

PhatDoan · 2017-09-30T02:42:02Z

Ah my bad. Thank you!

* Fix issues with the 'since' property and add filtering to kraken getTrades * Add currency prefix hack for all applicable markets on kraken exchange

…ct in getPortfolio.

askmike · 2017-09-30T17:38:04Z

Great stuff, Though I am gettting rate limited when I run this (see below). Would you mind adding a delay between fetches?

Here is the error (I logged from your monkey patched getTrades):

2017-09-30 18:34:29 (DEBUG):  Processing 2512 new trades. From 2017-08-16 20:59:57 UTC to 2017-08-16 22:59:46 UTC. (2 hours)
2017-09-30 18:34:33 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502845200000&end=1502931600000
2017-09-30 18:34:36 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502843136000&end=1502929536000
2017-09-30 18:34:38 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502842349000&end=1502928749000
2017-09-30 18:34:41 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502841683000&end=1502928083000
2017-09-30 18:34:44 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502840311000&end=1502926711000
2017-09-30 18:34:46 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502839841000&end=1502926241000
2017-09-30 18:34:49 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502838137000&end=1502924537000
2017-09-30 18:34:50 (DEBUG):  Processing 3116 new trades. From 2017-08-16 23:00:04 UTC to 2017-08-17 00:59:46 UTC. (2 hours)
2017-09-30 18:34:53 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502852400000&end=1502938800000
2017-09-30 18:34:56 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502849793000&end=1502936193000
2017-09-30 18:34:59 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502848264000&end=1502934664000
2017-09-30 18:34:59 (DEBUG):  ERROR Error: StatusCodeError: 429 - ["error",11010,"ratelimit: error"]
    at rp.then.catch (/Users/pear/projects/gekko/node_modules/bitfinex-api-node/rest2.js:78:26)
    at tryCatcher (/Users/pear/projects/gekko/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/Users/pear/projects/gekko/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/pear/projects/gekko/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/Users/pear/projects/gekko/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:800:20)
    at tryOnImmediate (timers.js:762:5)
    at processImmediate [as _immediateCallback] (timers.js:733:5)
2017-09-30 18:34:59 (DEBUG):  Bitfinex returned an error, retrying..
2017-09-30 18:35:09 (DEBUG):  Querying trades with: trades/tBTCUSD/hist?limit=1000&start=1502848264000&end=1502934664000
2017-09-30 18:35:09 (DEBUG):  ERROR Error: StatusCodeError: 429 - ["error",11010,"ratelimit: error"]
    at rp.then.catch (/Users/pear/projects/gekko/node_modules/bitfinex-api-node/rest2.js:78:26)
    at tryCatcher (/Users/pear/projects/gekko/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/Users/pear/projects/gekko/node_modules/bluebird/js/release/promise.js:689:18)
    at Async._drainQueue (/Users/pear/projects/gekko/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/Users/pear/projects/gekko/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/Users/pear/projects/gekko/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:800:20)
    at tryOnImmediate (timers.js:762:5)
    at processImmediate [as _immediateCallback] (timers.js:733:5)
2017-09-30 18:35:09 (DEBUG):  Bitfinex returned an error, retrying..

cmroche · 2017-09-30T22:59:36Z

@askmike There already is a rate limiter, I set it to 2.5 seconds based on testing my local system (which admittedly is an older and slower computer). I bumped it up a second so that should fix it.

Update bitfinex-api-node to latest

…ration

This is noticeable when grabing BCH before August.

…mport

cmroche · 2017-09-30T23:08:12Z

FYI this PR now showing a lot of commits since I rebased develop onto this branch instead of merging, sorry in advance.

askmike · 2017-10-01T12:54:45Z

Yes that seems to work fine, if someone runs into this I will just bump the timer some more. Thanks a lot :)

askmike · 2017-10-01T13:30:22Z

Thanks a lot :) I wasn't able to merge automatically so have done it manually. It's now all in the latest develop!

cmroche and others added 12 commits September 20, 2017 21:07

Add support for import and history to Bitfinex

4b9e76c

Update bitfinex-api-node to latest

Moved getTrades function into importer, bitfinex requires reverse ite…

a90fa7a

…ration

New approach to bitfinex import uses sub-step iteration

d8fae18

Fix end detection

d658473

Really fix end detection this time.

d3306f8

added tests for SMA indicator

d09aad7

O(1) implementation of SMA indicator

265ce30

Add ETP to bitfinex

67f97ae

Fix ETP pairs

44ad2c5

use this.result instead of this.rsi

52314c6

change strat to use correct indicator output

391a929

fix rsi test

b1671eb

update ppo.js for equality

d2acf4f

Handle empty result sets better

464992e

This is noticeable when grabing BCH before August.

generalectric and others added 10 commits September 30, 2017 16:37

execute end function after backtest complete

81a62a5

remove example from custom.js

8ac17de

add function to strat doc

ab46743

* Implement kraken as an import source

1a29cd7

* Fix issues with the 'since' property and add filtering to kraken getTrades * Add currency prefix hack for all applicable markets on kraken exchange

Fix up kraken exchange to enable historical getTrades

67ccb42

Fix iteration for Kraken import, looks to be working

cbea00e

Update import documentation, kraken now supports import

9196efd

Clean up comments

0c94e28

Add "BCH" to crypto list

c4c7535

Fix outstanding issues with prefixless assets not being queried corre…

2a1ce4c

…ct in getPortfolio.

cmroche and others added 7 commits September 30, 2017 18:17

Code cleanup

3c7cead

Kraken supports ETH is a backing currency for some asset pairs.

b21ecfa

Bug fix for live market watch, use moment instead of raw ts for since.

8916334

Change to getPortfolio was wrong, fix it.

46e089b

Fixing getPortfolio issue with prefixless asset pairs

b981d57

More incorrect and duplicated currency listings removed

a5fb116

initialize price and ensure amount > 0 to trade

d75b71f

krieghof and others added 5 commits September 30, 2017 20:18

changed btc-e to wex.nz and added all new pairs

3945dcc

update supported exchanges documentation

841e7c5

Fixed token pairs

8923f07

update node-wex dependency

b804827

Increase rate limit to 3.5 seconds

ad71cae

cmroche added 9 commits September 30, 2017 19:02

Add support for import and history to Bitfinex

229a639

Update bitfinex-api-node to latest

Moved getTrades function into importer, bitfinex requires reverse ite…

3c73304

…ration

New approach to bitfinex import uses sub-step iteration

0dc3cc3

Fix end detection

6bcdd10

Really fix end detection this time.

c39d5a8

Handle empty result sets better

d2efee5

This is noticeable when grabing BCH before August.

Increase rate limit to 3.5 seconds

c38677c

Merge remote-tracking branch 'origin/bitfinex_import' into bitfinex_i…

7698411

…mport

Fix supported exchanges after rebase.

5e930a4

Fixing Bitstamp in supported exchanges doc after merge error

84667ef

askmike closed this Oct 1, 2017

cmroche deleted the bitfinex_import branch October 1, 2017 13:47

snyk-bot mentioned this pull request Mar 31, 2020

[Snyk] Upgrade koa from 1.4.0 to 1.7.0 dmitriz/gekko#5

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Bitfinex Importing (take 2) #1145

Support for Bitfinex Importing (take 2) #1145

cmroche commented Sep 21, 2017

streetbits commented Sep 26, 2017

cmroche commented Sep 26, 2017 •

edited

Loading

streetbits commented Sep 27, 2017

PhatDoan commented Sep 30, 2017 •

edited

Loading

cmroche commented Sep 30, 2017

PhatDoan commented Sep 30, 2017

askmike commented Sep 30, 2017 •

edited

Loading

cmroche commented Sep 30, 2017

cmroche commented Sep 30, 2017

askmike commented Oct 1, 2017

askmike commented Oct 1, 2017

Support for Bitfinex Importing (take 2) #1145

Support for Bitfinex Importing (take 2) #1145

Conversation

cmroche commented Sep 21, 2017

streetbits commented Sep 26, 2017

cmroche commented Sep 26, 2017 • edited Loading

streetbits commented Sep 27, 2017

PhatDoan commented Sep 30, 2017 • edited Loading

cmroche commented Sep 30, 2017

PhatDoan commented Sep 30, 2017

askmike commented Sep 30, 2017 • edited Loading

cmroche commented Sep 30, 2017

cmroche commented Sep 30, 2017

askmike commented Oct 1, 2017

askmike commented Oct 1, 2017

cmroche commented Sep 26, 2017 •

edited

Loading

PhatDoan commented Sep 30, 2017 •

edited

Loading

askmike commented Sep 30, 2017 •

edited

Loading