In this lesson, we download historical data using Zorro. Zorro supports several data vendors out of the box, such as FXCM, Oanda, AlphaVantage, and Quandl, but today, we will investigate how to download data from any documented REST API. To demonstrate this, we will connect to the REST API by Darwinex. (This API is in beta, so it is subject to change.)
HTTP in Brief
It is helpful to know a little about how HTTP works. In most cases, it is as simple as this:
- A client (such as a browser) sends a request to an HTTP server.
- The server sends a response. The response can be anything from a web page, to a csv file, to a json file.
And that’s it.
For example, suppose you want to download historical data, for symbol EUR/USD, in 1-minute bar format, for the months of August through December of 2018, and you have a REST API that can give you this data. Here’s how you do it:
- Get API access. (For simplicity, let’s assume we only need to supply a single key or password.)
- Read the API documentation. It will tell you how to format the requests and the responses you should expect.
- Construct your HTTP request, using your parameters (symbol, format, begin date, end date).
- Send the HTTP request.
- Wait for the response.
- Receive the response.
- Parse, use, and/or save the data.
So let’s get some data!
The Code
Here’s a complete Zorro script that can download JSON data from the Darwinex Beta API. It uses the language Lite-C, which is very similar to C.
// Reads Zorro.ini file for "DarwinexKey" value.
// Returns a string with that value.
const char* can_get_darwinex_key();
// Downloads historical data var a Darwin product in JSON format.
// Returns a **temporary** JSON string.
const char* can_get_darwinex_json(const char* productName, const int bar_period, const int start, const int end);
// Converts int yyyymmdd into the unix epoch value, int.
int epoch(int yyyymmdd){return utm(dmy(yyyymmdd));}
int main(){
Verbose = 7;
if(!is(INITRUN))
return 0;
const char* key = can_get_darwinex_key();
if(key){
printf("\nkey: %s",key);
} else {
printf("\nno key!");
return 1;
}
printf("\n%d",epoch(20190113));
const char* jsonstr = can_get_darwinex_json("SCS", 1440, 20180101, 20190113);
const char* filename = ".\\History\\scs_1d.json";
file_write(filename,jsonstr,strlen(jsonstr));
printf("\nsaved file to \"%s\"",filename);
printf("\nAll done.");
return 0;
}
const char* can_get_darwinex_key(){
static char* out = 0;
if(out) return out; // no need to repeat
// read zorro.ini file
char* ini = file_content("Zorro.ini");
// parse for darwin key
char* output = strtext(ini, "DarwinexKey", "");
// check for bad outcome, if so return false
if(!strlen(output)) return NULL;
// output darwin key return true
out = malloc(strlen(output)+1);
if(!out) return NULL;
strcpy(out,output);
return out;
}
const char* can_get_darwinex_json(const char* productName, const int bar_period, const int start, const int end){
static char* out = 0;
if(out) {
free(out);
out = 0;
}
const char* apikey = can_get_darwinex_key();
if(!apikey){
printf("\nCannot load Darwin Key! Check Zorro.ini");
return NULL;
}
char resolution[4];
memset(resolution,0,4);
switch(bar_period){
case 1: strcpy(resolution,"1m"); break;
case 5: strcpy(resolution,"5m"); break;
case 15: strcpy(resolution,"15m"); break;
case 30: strcpy(resolution,"30m"); break;
case 60: strcpy(resolution,"1h"); break;
case 240: strcpy(resolution,"4h"); break;
case 1440: strcpy(resolution,"1d"); break;
default: // leave blank
}
char url[512];
sprintf(
url,
"https://api.darwinex.com/darwininfo/products/%s/candles?from=%d&to=%d",
productName,
epoch(start),
epoch(end)
);
if(strlen(resolution)){
strcat(url,strf("&resolution=%s",resolution));
}
char header[512];
sprintf(
header,
"Authorization: Bearer %s",
can_get_darwinex_key()
);
int h = http_send(url,0,header);
int bytes = 0;
for(;;){
bytes = http_status(h);
if(bytes==0) // still in progress
{
if(!wait(100)) // pause 100ms, abort if necessary
{
http_free(h);
return NULL;
}
}
else if ((bytes == -1) || (bytes == -2)) // failed or invalid
{
http_free(h);
return NULL;
}
else // transfer completed
break;
}
printf("\nbytes: %d",bytes);
//get the string
out = malloc(bytes);
memset(out,0,bytes);
http_result(h,out,bytes);
//free the handle
http_free(h);
return out;
}So let’s walk through what’s happening here.
First, you need an API key. For security reasons, you should try not to hard-code it into your script. The function can_get_darwinex_key() checks the Zorro.ini file for a line that looks like this:
DarwinexKey = "4b3-b1f41e71--966428a-190d371698956e"
Note that this line is presently not a default Zorro.ini entry. I have simply added this on my own, for the sake of this script, and it works.
Next, the script confirms that the key exists. In main(), we make a variable const char* key. We check to see if this is a null pointer or a valid string. If it’s invalid, the script ends.
Next, let’s peek at the documentation for /products/{productName}/candles (under the API Console tab). This documentation site allows you to try out live requests on their API – give it a try!
Here, we can learn how to construct a request. You will be needing:
- productName. This is the name of the Darwin asset.
- resolution. This means bar size, such as 1m, 5m, 1h, 1d, etc.
- from. This is the earliest date to check, in the unix epoch format.
- to. This is the last date to check, in the unix epoch format.
- (Not to mention that you need the API key, but that goes in the header.)
The first problem is that we need to calculate epoch dates. The above script takes care of the problem with a one-liner:
int epoch(int yyyymmdd){return utm(dmy(yyyymmdd));}
You input int yyyymmdd. Zorro has a function dmy() which converts it to the Windows DATE format. Next, it gets handed off to Zorro’s utm() function, which conveniently converts from Windows DATE to unix epoch. Let’s do a range of 2018-01-01 through 2019-01-13.
Next, we need a product name. For those unfamiliar with Darwinex, their products are… well, “Darwins”. Here’s a description. Picking one from the list, let’s go with SCS.
Finally, we need a resolution. For convenience (especially for Zorro script writers), we use a period size, in the same format as Zorro’s BarPeriod variable, where 1 is “1m”, 5 is “5m”, 60 is “1h”, 1440 is “1d”, etc.
In the can_get_darwinex_json() function, we construct the request with strings, to be supplied to Zorro’s handy http functions. From the API documentation, we know we need to have a url and a header with the API token; our function constructs both.
http_send() returns a handle, which we need to track in order to receive its resonponse. We learn how many bytes are received, and we allocate memory for it. We consume the memory, and then we free up the handle.
A peculiar thing about network programming is that you need to account for time passage. We use the wait() function to give the user the chance to abort if downloading takes forever. Otherwise, you can have a frozen Zorro instance.
To finish off main(), we simply save the json output to a file.
That’s all for Part 1. In Part 2, we will parse the JSON file and convert it into a useful T6 format for Zorro traders.
Comments are disabled due to spam. For questions, please email me directly. Thank you.
Nice Post, Andrew
Thanks!