Introduction
Welcome to the General Index documentation site. This site contains the information required to submit and extract data from the General Index system, including file formats and API specifications.
For questions regarding integrating GX data into your systems and submitting data, please contact support@general-index.com.
Methodology Framework
GX generates pricing assessments under a standard set of procedures that is mathematical, consistent and fully transparent. This is defined in the GX Methodology Framework, available online.
This documentation does not take precedence over the Methodology Framework or any contractual agreements in place.
Data Notices
GX sends out ad-hoc data notices containing information on new data series, updates to metadata and corrections that apply to data more than 30 days old. We encourage all users of GX data to sign-up to receive these notices. New users can sign-up here:
https://pages.general-index.com/datanotices
Index Modules
Index Series are grouped for licensing and access via Index Modules. Index Modules correspond to trading hubs and/or commodities.
Example Index Modules:
GlobalIndexes
NorthAmericaCrude
EuropeanCrude
EuropeanGasoline
EuropeanJet
All data within a single Index Module is available to licensed users including all GX index series, index types, forward periods and timestamps for that Module.
Individual Index Series may be contained within multiple Index Modules.
Mappings of Index Series to Index Modules are available through the GX API. Please contact GX Support if you would like to do this.
Time zones
Please note: All times and dates associated with Index Seriers are subject to time zones. Time zones are specified in metadata and are as provided in the IANA Time Zone Database (so, for example Europe/London
)
Index Series Format
This section describes how GX Index Series are defined. It outlines standard data fields and file structures that are available across GX systems.
GX Index Codes
GX assigns a unique identifier to each Index Series – each identifier consists of the letters ‘GX’ followed by 7 digits (e.g. GX0000001
), along with a title describing the content of the series (e.g. Global Crude Index (GCX) London Assessments
).
Index Methodology Factsheets
A full definition of each index series is documented within the relevant factsheet. Factsheets include time zone, units of measure, location, pricing basis and calculation methodology.
Factsheet server
PDF versions of the factsheets are available from the factsheet server via this URL
https://factsheets.g-x.co/[GX_CODE].pdf
So, for example, the factsheet for Global Crude Index (GCX) London Assessments can be obtained from https://factsheets.g-x.co/GX0000001.pdf
Index Keys
Each series has several time series associated with it. Individual time series are defined by the following keys. All keys are stored as text fields.
Key | Description | Type | Examples |
---|---|---|---|
Code | Index Series code. GX followed by 7 digits |
Text | GX0000001 |
TimeRef | The timestamp of the assessment, always 4 digits. Note the time zone of the timestamp is defined in the Factsheet and metadata | HHMM | 1630 , 0700 , 0000 |
PeriodType | Granularity of the assessment period. | Text | Month , Prompt |
Period | Name of the period. This field is 1 when Period Type is Prompt . |
Text | 1 , 2 |
Index Types
For a specific key combination (Index code at a particular TimeRef, Period, PeriodType), GX produces values for Index Types as defined by the factsheet, for example to denote Mid, High and Low.
Decimal places and precision of numerical fields are defined in the factsheet.
Index Type | Description | Type |
---|---|---|
Mid | n/a | Double |
High | n/a | Double |
Low | n/a | Double |
Index | This index type can appear | Double |
Close | n/a | Double |
Explanation | Assessment explanation / rationale | Text |
Additional Columns & Metadata
All GX codes carry the following columns:
Column | Description | Type | Example(s) |
---|---|---|---|
FactsheetVersion | Version number of the Factsheet used to generate the assessment value. | Double | 1.0 |
PeriodAbs | Absolute period which the assessment value relates to. | String | 2020-04 |
PeriodRel | Relative period which the assessment value relates to. For most series, this reflects a future delivery period. In some markets, negative values may be published to reflect historic periods The granularity applicable to each series is defined by the relevant Factsheet and available in metadata. | Integer | 1 , 2 |
PeriodStart | The start date of the delivery window. Defined in ISO8601 format with option for time zone to be defined in the column header. | Text | 2020-07-01T00:00:00 |
PeriodEnd | The end date of the delivery window. Defined in ISO8601 format with option for time zone to be defined in the column header. | Text | 2020-07-31T23:59:59 |
RecordStatus | Column depreciated. N in all cases |
Text | N |
In addition to the value columns, metadata from the factsheet is provided to aide the consumption of the Index Series:
Column | Description | Type | Example(s) |
---|---|---|---|
Alias | Text ‘short hand’ descriptor relating to a particular TimeRef, PeriodType and Period | Text | GCXC |
Title | Text description of the data series. | Text | Eurobob Oxy Gasoline NWE FOB Barges |
Timezone | The timezone of the timestamp (TimeRef). Time zones are as defined in the IANA Time Zone Database | Text | Europe/London |
Units | Units of measure | Text | BBL NULL |
Currency | Currency | Text | USD NULL |
HolidayCalendar | The holiday calendar that defines when the Index Series is not published | Text | Holidays_GX_Europe |
Source | This will be GX unless the data is reproduced from a third party |
Text | GX |
DeliveryBasis | Freight and delivery information | Text | FOB Barges |
PricingBasis | Flat or difference to another market | Text | Flat , Diff |
TradingHub | The physical location | Text | NWE , Global |
Commodity | Underlying commodity | Text | Crude |
Frequency | Assessment frequency | Text | Daily , Monthly |
Holiday Calendars
curl "https://api.g-x.co/index?group=Meta_Holiday_Calendars&from=all&metadata=true" \
-H 'Authorization: Bearer [YourGXAccessToken]'
GX maintains a list of holiday calendars for GX Index Series and third-party timeseries. Calendar files have the following format:
Field Name | Description | Type | Example(s) |
---|---|---|---|
Calendar | The name of the calendar. Where relevant this will appear in the Factsheet. | Text | Holidays_GCX Holidays_CME |
Date | Date specified without a time zone. | YYYY-MM-DD | 2020-12-25 |
Value(S) | Holiday name or Yes . |
Text | Christmas |
Holiday calendars can be queried through the API. Please see example. The holiday calendar should be queried in UTC time zone and the time component of the Date
field ignored in the response.
Absolute Period Format
The granularity of absolute assessment periods (and held in PeriodAbs
) is defined by the factsheet. Absolute assessment periods are published in the following format:
Granularity / PeriodType |
Description | Format | Example |
---|---|---|---|
n/a | No delivery period applies | Null | Null |
Prompt | Prompt delivery period | Prompt | Prompt |
12Hour | 12 hour period. Shown as start of period | YYYY-MM-DDTHH:MM | 2020-03-22T12:00 |
Day | Calendar day | YYYY-MM-DD | 2020-04-01 |
Week | ISO week number | YYYY W[00-52] | 2020 W04 |
HalfMonth | First or second half of a calendar month. Shown as start date of period | YYYY-MM-DD | 2020-04-16 |
Month | Calendar month | YYYY-MM | 2020-04 |
Quarter | Quarter of a calendar year | YYYY Q[1-4] | 2020 Q1 |
Season | Calendar season | YYYY [SUM/WIN] | 2020 WIN |
HalfYear | First or second half of a calendar year | YYYY H[1-2] | 2020 H2 |
Year | Calendar year | YYYY | 2020 |
Cycle | Pipeline cycle number | C[0-9][0-9] | C03 |
API Specification
Authorisation
General Index provides multiple ways to authenticate:
- API key passed as URL parameter (preferred)
- Basic token passed as URL parameter
- JWT bearer access token in Header
- Basic token in Header
Username and password
An invitation email is sent to the address provided. Click on the link to create a password. Contact GX Support to have it reset.
URL authentication
Parameter | Required | Description |
---|---|---|
token | Optional | Either API key or basic token |
API key
To generate a new API key, visit:
https://app.g-x.co/MyAccount#Settings
Scroll to the bottom of the page and click 'Generate New Key'. The name is for your use. Please make a note of the token generated. You will have to generate a new token once you close the window.
Basic token
# Basic token passed in url
# The token used in the example was generated for user@user.com:password
curl "https://api.g-x.co/index?token=dXNlckB1c2VyLmNvbTpwYXNzd29yZA=="
A basic token is generated from the username and password, Base64 encoded separated by a colon.
It can be generated from command line, for example in macOS / Linux:
echo -n "username:password" | base64
Header authentication
Parameter | Required | |
---|---|---|
Authorization | Optional | Bearer yourGXAccessToken or Basic base64(user:password) |
Basic auth token
Basic auth token is Base64-encoded username and password, separated by a colon, as described above.
Log in
This endpoint returns a jwt access token and a refresh token. When access token is expired, refresh token can be used to generate a new one.
curl -u 'login:password'
https://api.g-x.co/auth/login
If successful, the above command returns a status code 200 and token, otherwise please check Errors section.
{
"token": "YourGXAccessToken",
"refreshToken": "your refresh token"
}
HTTP Request
GET https://api.g-x.co/auth/login
Request Body
Parameter | Description |
---|---|
username | username of account |
password | password of account |
Refresh Access Token
This endpoint returns a new access token.
curl -X POST
https://api.g-x.co/auth/refresh
-d '{
"refreshToken": "your refresh token"
}'
If successful, the above command returns a status code 200 and token, otherwise please check Errors section.
{
"token": "YourGXAccessToken",
"refreshToken": "your refresh token"
}
HTTP Request
POST https://api.g-x.co/auth/refresh
Request Body
your refresh token
Log out
Log out the user by invalidating the refresh token.
curl -X POST
https://api.g-x.co/auth/logout
-H "Authorization: Bearer YourGXAccessToken"
-d '{
"refreshToken": "your refresh token"
}'
If successful, the above command returns status code 200, otherwise please check Errors section.
HTTP Request
POST https://api.g-x.co/auth/logout
Data Catalogue
Catalogue Endpoint
curl "https://api.g-x.co/catalogue?module=EuropeanGasoline \
-H 'Authorization: Bearer [YourGXAccessToken]'
The above command returns a CSV response
Code,PeriodType,TimeRef,IndexModule,Alias,Bid,Commodity,Currency,DeliveryBasis,EndDate,FactsheetVersion,Frequency,Group,High,HolidayCalendar,Index,InterAffiliateDataAccepted,LastTradeDateCalendar,Low,MapLocation,Mid,Offer,ParentCodes,PeriodMax,PeriodMin,PermissionLatestStatus,Precision,PricingBasis,PriorityToTransactions,QaSeries,Quantity,SoleSourcedDataAccepted,Source,StartDate,Status,ThirdPartyMappings,TimeRefDetails,TimeZone,Title,TradingHub,Units,Volume
GX0000614,Month,1630,EuropeanGasoline,,,Gasoline,USD,FOB Barges,,1,Daily,Prod_Indexes,,Holidays_GX_Europe,Y,N,,,,,,GX0000002 GX0000002,1,1,PRIMARY,2,Diff,N,ProdIndexes HolidaysGXEurope Daily Index,,N,GX,2021-02-25,Live,LSEG:GX0000614L1630c1 Bloomberg:GIOS0614 Index,London Close,Europe/London,Eurobob Oxy Gasoline NWE FOB Barges M1 vs M6,NWE,MT,
GX0000472,Prompt,1630,EuropeanGasoline,,,Gasoline,USD,CIF Cargoes,,1,Monthly,Prod_Indexes,Y,Holidays_GX_Europe,,N,,Y,,Y,,,1,1,PRIMARY,4,Flat,N,QA ProdIndexes HolidaysGXEurope Monthly Mid,,N,GX,2008-01-31,Live,LSEG:GX0000472L1630P Bloomberg:GIOS0472 Index Platts:AAWZB03,London Close,Europe/London,Gasoline 95 RON MED CIF Cargoes Monthly Average,MED,MT,
HTTP Request
GET https://api.g-x.co/catalogue
The endpoint returns a list of all available Index data series. URL parameters allows filtering data by Module.
URL Parameters
Parameter | Required | Default | Description |
---|---|---|---|
module | Optional | None |
Module name, if not provided all indexes are returned |
token | Optional | None |
If Authorization header is missing, Basic Auth Token can be passed as a parameter |
Data Structure
Indexes are stored as timeseries data with associated symbols, metadata and values
Symbols
Individual time series are accessible through the following keys. A combination of symbols uniquely defines a time series. Symbol fields are case sensitive and are returned as text strings.
Field | explanation | example |
---|---|---|
Code | GX Index Series code | GX0000001 |
TimeRef (nb: Case sensitive) | Timestamp, expressed as 4 digits. Timezone of timestamp is defined in metadata | 1630 0700 |
PeriodType (nb: Case sensitive) | Granularity of the assessment period | Month Prompt |
Period | Name of the Period | 1 |
Metadata
The Fields listed below represent the standard metadata published for all Index Series. Additional Metadata is available for individual Index Series where appropriate. Note Field names are case sensitive.
Field | explanation | example |
---|---|---|
Title | Long-form name | Global Crude Index |
Alias | Short-form name | GCXC |
TimeZone (nb: Case sensitive) | Timezone that the Index Series is published in | Europe/London America/New_York |
Currency | Currency | USD |
Units | Units | BBL |
HolidayCalendar (nb: Case sensitive) | Relevant holiday calendar | Holidays_GCX |
Source | This will be GX unless the data is reproduced from a third party |
Text |
DeliveryBasis | Freight and delivery information | FOB Barges |
PricingBasis | Flat or difference to another market | Flat , Diff |
TradingHub | The physical location | NWE , Global |
Commodity | Underlying commodity | Crude |
Frequency | Assessment frequency | Daily , Monthly |
Values
The Fields listed below represent the standard timeseries values associated with each Index Series. Additional Value series are available where appropriate – in particular reflecting multiple Index Value series. All field names are case sensitive.
Field | explanation | example |
---|---|---|
FactsheetVersion | Integer value of relevant Factsheet methodology | 1 |
PeriodAbs | Long-form name | 2020-07 |
PeriodRel | Short-form name | 1 |
PeriodStart | ISO8601 format. Returned in timezone the Index Series is published in | |
PeriodEnd | ISO8601 format. Returned in timezone the Index Series is published in | |
RecordStatus | Currently unsupported. Default N |
N |
[Index Values] | Values as defined in the Index Types | Index High Volume |
Pull Index Data Series
Index Endpoint
The endpoint retrieves index timeseries data in the provided time range. URL parameters allows filtering data by Code or metadata fields.
HTTP Request
curl "https://api.g-x.co/index?code=GX0000004&metadata=true" \
-H 'Authorization: Bearer [YourGXAccessToken]'
The above command returns a CSV response
Code,Period,PeriodType,TimeRef,Date(UTC),FactsheetVersion,High,Low,Mid,PeriodAbs(S),PeriodEnd(D),PeriodRel(S),PeriodStart(D),RecordStatus(S),Alias(MD-S),Commodity(MD-S),Currency(MD-S),DeliveryBasis(MD-S),Frequency(MD-S),HolidayCalendar(MD-S),PricingBasis(MD-S),Source(MD-S),TimeZone(MD-S),Title(MD-S),TradingHub(MD-S),Units(MD-S)
GX0000004,1,Prompt,1730,2022-11-29T17:30:00,2,774.75,774.25,774.5,Prompt,2022-12-07T23:59:59,1,2022-12-01T00:00:00,N,EUROBOBNONC,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
GX0000004,1,Prompt,1630,2022-11-29T16:30:00,1,774.75,774.25,774.5,Prompt,2022-12-07T23:59:59,1,2022-12-01T00:00:00,N,,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
GX0000004,1,Prompt,1730,2022-11-28T17:30:00,2,732.25,731.75,732,Prompt,2022-12-06T23:59:59,1,2022-11-30T00:00:00,N,EUROBOBNONC,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
GX0000004,1,Prompt,1630,2022-11-28T16:30:00,1,732.25,731.75,732,Prompt,2022-12-06T23:59:59,1,2022-11-30T00:00:00,N,,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
# All indexes published today (by London timezone) with username / password authorization:
curl -u 'username:password' https://api.g-x.co/index?timezone=Europe/London
# All indexes published today (UTC timezone) with metadata:
curl -u 'username:password' https://api.g-x.co/index?metadata=true
# Last 10 days data of code GX0000002:
curl -u 'username:password' https://api.g-x.co/index?code=GX0000002&from=10d
# All Diesel indexes for last 5 days:
curl -u 'username:password' https://api.g-x.co/index?metadata.Commodity=Diesel&from=5d
# Authorization token passed in url
# The token used in the example was generated for user@user.com:password
curl "https://api.g-x.co/index?from=5d&token=dXNlckB1c2VyLmNvbTpwYXNzd29yZA=="
GET https://api.g-x.co/index
URL Parameters
Parameter | Required | Default | Description |
---|---|---|---|
code | Optional | None |
Comma separated list of GX codes. If not provided all indexes are returned |
period | Optional | None |
Comma separated list of period values. If not provided all periods are returned |
periodtype | Optional | None |
Comma separated list of periodtype values. If not provided all periodtypes are returned |
timeref | Optional | None |
Comma separated list of timeref values. If not provided all timerefs are returned |
module | Optional | None |
Index module to return, ex. module=NorthAmericaCrude . If not provided all modules are returned. Cannot be used in conjunction with the code, period, or periodtype parameters |
group | Optional | Prod_Indexes |
Use default value unless advised otherwise by GX Support. Note: This is case sensitive. |
from | Optional | today |
Start date, accepted values:
|
to | Optional | today |
End date, accepted values:
|
metadata.field | Optional | None |
Filter indexes by field value, ex. metadata.Commodity=Diesel |
metadata | Optional | false |
Controls whether returns metadata, accepted values: true / false / only |
delta | Optional | false |
Whether use time range against transaction time or insertion time. Accepted values: true / false . This can be used to run queries showing what has changed in the database in the requested date range. |
timezone | Optional | UTC |
A timezone name for requested time range and returned data. Time zones are as defined in the IANA Time Zone Database. |
token | Optional | None |
If Authorization header is missing, Basic Token can be passed as a parameter. See url authentication above. |
Header Parameters
See header authentication above.
Parameter | Required | |
---|---|---|
Authorization | Optional | Bearer yourGXAccessToken or Basic base64(user:password) |
Accept | Optional | text/csv (default) or application/json |
Submit Data
GX accepts data submissions via file uploads to the GX data lake. File uploading has two steps
- First step is file meta data posting
- Second step is file uploading
File Metadata Posting
First step:
curl -X POST
https://api.g-x.co/file
-d '{
"groupName":"group01",
"fileName":"test.csv",
"fileType":"SOURCE",
"fields":[
{
"name":"String field",
"value":"test"
},
{
"name":"Integer field",
"value":"1"
},
{
"name":"Float field",
"value":"1.1"},
{
"name":"Date field",
"value":"2017-09-05"
}
...
]
}'
-H "Authorization: Bearer yourGXToken"
If successful, the above command returns status code 201, otherwise please check Errors section.
This endpoint stores the file metadata and returns a location to PUT the file content. If the file group requires metadata for the uploaded file, you can override default values and provide values for metadata fields that require values. On creation the call returns a 'location', you must PUT the file content to this location.
HTTP Request
POST https://api.g-x.co/file
Request Body
Data | Notes |
---|---|
groupName | Provided by GX Support and unique to your upload, for example Input_AcmeTrading_PushAPI ) |
fileName | File name should not be blank, must be unique for each file upload |
fileType | type of the file, available types are: SOURCE (Default. Use this unless advised otherwise by GX Support) and NCSV |
fields | Optional Meta data |
* name | field name |
* value | field value |
HTTP Response
Response:
{
"location": "https://fileUploadURL",
"fileId": "file id"
}
A JSON object is returned with an url to upload a file content.
File Uploading
This endpoint saves the file content into the GX Input Data Repository. You should use the 'location' link for the file from the first step response body.
Second step:
curl -X PUT --data-binary @"file-name.csv"
"https://fileUploadURL"
If successful, the above command returns status code 200.
HTTP Request
Use --data-binary body option to select file from your local machine.
PUT https://fileUploadURL
Download PDF Reports
## FILES
curl "https://api.g-x.co/file/search?size=1000&from=0&query=groupName%3DProd_Reports" \
-H 'Authorization: Bearer [yourGXAccessToken]'
The above returns a JSON response
{
"from": 0,
"totalSize": 100,
"items": [
{
"fileName": "GX_MiddleEastCrude_Benchmark_210324.pdf",
"fileType": "SOURCE",
"groupName": "Prod_Reports",
"fid": "c6da572e-d71f-4bdb-8394-a31396cf7c7a",
"fields": {
"SKIP_METADATA": "false"
},
"arrivalTime": "2021-03-24T14:47:08.399",
"owner": "system",
"size": 122903,
"checkSum": "300ffa79f079f365006e9f1c85b3dcae",
"zipped": false,
"shared": false,
"removed": false,
"version": 1,
"rootId": "c6da572e-d71f-4bdb-8394-a31396cf7c7a"
},
{
"fileName": "GX_MiddleEastCrude_Benchmark_210323.pdf",
"fileType": "SOURCE",
"groupName": "Prod_Reports",
"fid": "2b490176-b385-49cc-9453-bf25fe39c969",
"fields": {
"SKIP_METADATA": "false"
},
"arrivalTime": "2021-03-23T14:40:57.861",
"owner": "system",
"size": 134910,
"checkSum": "2cac31e0c971600f336c69b32b399665",
"zipped": false,
"shared": false,
"removed": false,
"version": 1,
"rootId": "2b490176-b385-49cc-9453-bf25fe39c969"
}
]
}
Second step:
curl -v "https://api.g-x.co/file/c6da572e-d71f-4bdb-8394-a31396cf7c7a/download"
-H 'Authorization: Bearer [yourGXAccessToken]'
The above returns the file as the response.
For certain Index Modules, General Index publishes PDFs on a regular (daily or weekly) basis. If you have been permissioned, reports are available using the same credentials.
There are two steps to downloading PDFs:
- First step is retrieve file IDs
- Second step is to retrieve the specific file based on the ID obtained.
File naming
All PDFs follow the same naming convention. Files are named according to the Index Module, report type and publishing date in the following format:
GX_[Index Module]_[Report Type]_yymmdd.pdf
For example: GX_MiddleEastCrude_Benchmark_210324.pdf
The timezone of the date in the file name is according to the location of the Index Module.
Reports that are published as a correction to the original report will have the date of the original report in the filename but have a more recent arrivalTime
.
Retrieve file IDs
HTTP Request
GET https://api.g-x.co/file/search
Header Parameters
Parameter | Required | |
---|---|---|
Authorization | Required | Bearer yourGXAccessToken |
URL Parameters
Parameter | Description |
---|---|
groupName | Prod_Reports unless advised otherwise by GX Support. |
size | The maximum size of returned hits list. Max size is 1000 . Default size is 100 . |
from | The index of the first hit. It allows to do paging easily. Default value is 0 . |
Retrieve specific file
The URL is in the header of the response.
HTTP Request
curl -v "https://api.g-x.co/file/[fid]/download"
Header Parameters
Parameter | Required | |
---|---|---|
Authorization | Required | Bearer yourGXAccessToken |
URL Parameters
Parameter | Description |
---|---|
fid | fileID to retrieve |
Report Holiday Calendars
PDF reports are published daily, Monday to Friday apart from on dates specified in the relevant holiday calendar as specified below:
Report Name | Holiday Calendar |
---|---|
GX_AmericasMarine_Benchmark | Holidays_ICE_Brent |
GX_AsiaFuelOil_Benchmark | Holidays_GX_Singapore |
GX_AsiaGasoil_Benchmark | Holidays_GX_Singapore |
GX_AsiaGasoline_Benchmark | Holidays_GX_Singapore |
GX_AsiaJet_Benchmark | Holidays_GX_Singapore |
GX_AsiaLPG_Benchmark | Holidays_GX_Singapore |
GX_AsiaMarine_Benchmark | Holidays_ICE_Brent |
GX_EuropeanGasoline_Benchmark | Holidays_GX_Europe |
GX_EuropeanJet_Benchmark | Holidays_GX_Europe |
GX_EuropeanLPG_Benchmark | Holidays_GX_Europe |
GX_EuropeanMarine_Benchmark | Holidays_ICE_Brent |
GX_GlobalMarineCarbon_Benchmark | Holidays_ICE_Brent |
GX_MiddleEastCrude_Benchmark | Holidays_GX_Singapore |
GX_NorthAmericaCrude_Benchmark | Holidays_GX_NorthAmerica |
GX_NorthAmericaStorage_Benchmark | Holidays_GX_NorthAmerica |
Please see documentation for instructions to access detailed holiday calendar information.
Errors
The General Index API uses the following status codes:
Status Code | Meaning |
---|---|
400 | Bad Request -- The request you sent is incorrect. |
401 | Unauthorized -- Your API key is wrong. |
403 | Forbidden -- The requested resource is hidden and may be accessed by the administrators only. |
404 | Not Found -- The specified resource could not be found. |
405 | Method Not Allowed -- You tried to access a resource with an invalid method. |
406 | Not Acceptable -- You requested a format that isn't json. |
410 | Gone -- The resource requested has been removed from our servers. |
429 | Too Many Requests -- You're requesting too many times! Slow down! |
500 | Internal Server Error -- We had a problem with our server. Try again later. |
503 | Service Unavailable -- We're temporarily offline for maintenance. Please try again later. |
Data Integration Guide
Approach
The following approach is recommended to receive an accurate and timely replica of GX data.
Regular Data Pull
# All indexes published within a 15 minute time window with username / password authorization:
"https://api.g-x.co/index?from=2022-09-01T17:00:00&to=2022-09-01T17:14:59.999&metadata=true&delta=true&token=[YourGXAccessToken]"
Query the /index api every 15 minutes with the following parameters.
HTTP Request
https://api.g-x.co/index
Parameter | Value | Example |
---|---|---|
from | 16 mins before the current time, expressed as UTC | 2022-09-01T17:00:00 |
to | 1 min before the current time, expressed as UTC | 2022-09-01T17:14:59.999 |
delta | Set to true to get all changes to the database in that time period | true |
metadata | Set to true to include metadata fields | true |
The range for each query should end 1 minute prior to the query time to ensure that all data in range is available via the API.
Data will be returned in UTC by default. Each series can be translated into the publication timezone using the metadata associated with the series.
Only columns updated during the insert range are returned. Where values for one or more columns have been updated during the range selected, unchanged columns will return empty values.
Metadata Request
# Metadata only
"https://api.g-x.co/index?metadata=only"
Metadata can be returned without values by passing a metadata value of 'only'.
Please note that Metadata values may be updated from time to time. Minor updates will not be included in a Data Notice.
Python SDK
SDK Introduction
GX maintains an SDK to allow developers to query & download index series data and upload submissions to our data lake.
Documentation & Installation
Full documentation can be found on the PyPI project page
Example Code
You can download an example script using the SDK.
Excel Data Extract Tool
The General Index data extract tool allows clients to search the data catalogue and download our index data into Excel for further analysis.
Requirements
This tool is designed to run on either Apple or Windows PCs with the Office 365 version of Excel. The tool uses native Excel functionality to query the GX API via "PowerQuery".
On first opening the spreadsheet or refreshing data, a Privacy Levels dialogue window may be displayed. If so, the "Ignore Privacy Levels" checkbox should be selected.
Installation
The latest version of the tool can be found here:
https://www.general-index.com/data-extraction-tool
Getting started
Settings Tab: Enter your GX username and password on the Settings tab.
*** Your Password is CONFIDENTIAL to you. Please remove before sharing and contact GX Support if you would like credentials issued to another user. ***
Catalogue Tab: Leave all fields blank and choose "Refresh All" from the Data section of the Excel Ribbon to display all data series.
Catalogue Tab: Choose one or more series and copy the corresponding Code, PeriodType, Period and Type values. For example:
Code | PeriodType | Period | TimeRef |
---|---|---|---|
GX0000001 |
Month |
1 |
1630 |
Data Query tab: Paste values copied from the "Catalogue" tab into the table rows and delete any additional rows containing series not needed.
Data Output: Choose "Refresh All" from the Data section of the Excel Ribbon to display all data series matching the entries shown on the "Data Query" tab.
Catalogue
This tab allows the full GX Data Catalogue to be queried and returned.
Filters can be applied to restrict results by Title, Alias, Commodity and Module. Partial strings can be passed as filters by adding a *
to the start and/or end. Either a wildcard, or a string must be entered into all fields.
Query results can be updated either by right clicking on the data table and selecting "Refresh", or using the "Refresh All" option on the Data tab.
Data Query / Data Output
The Data Query tab allows data series to be selected for return on the Data Output tab.
To select a data series to return, values should be entered into each of the table columns (Code, PeriodType, Period and Type).
Values can be copied into the table from the Data Query tab, or entered manually.
Table rows can be added or deleted as required
A wildcard *
can be used to return all matching values for a column.
Query results can be updated either by right clicking on the data table and selecting "Refresh", or using the "Refresh All" option on the Data tab.
Please note that it is possible to select large volumes of data via this query which may take a significant amount of time to return.
Settings
Data Size
controls the maximum number of unique DateTime records returned on the Data Output tab.
TimeZone
dropdown list can be used to adjust the timezone in which the DateTime records are returned on the Data Output tab.
Submitting Data
GX accepts data submissions from partners globally.
Source Systems
Where possible, GX has a strong preference for receiving data from formally managed IT systems (e.g. CTRM, confirmation interfaces, Bid/Offer management, nomination and logistics software) which are supported and maintained outside the Front Office. Where this is not possible (for example, with bid/offer data), this should be highlighted.
Submit Data via Email
GX accepts data submissions via a dedicated email address. Email can contain data in the email body as well as attachments. Contact GX Support for a dedicated email address to accept your data submissions.
Submit Data via API
Refer to the API documentation on how to submit data to our data lake once your API credentials have been updated to allow uploads.
Trade Data Format
General Index processes trade data from a number of sources across commodities and geographies. We have developed a flexible data format that allows accurate and efficient processing
GX assessment processes include both trades and bid/offer data. Our data structure applies to both.
File Format
The following describes the data format which can be used for downloading and uploading trade data as CSV files.
Trade Data Keys
The combination of keys must be unique. Transactions with the same keys will be considered corrections and the previous data will be overwritten.
All keys are required:
Key | Type | Req. | Description | Example |
---|---|---|---|---|
Code | Text | Y | GX Trade Series code. GT followed by 7 digits. Codes can be provided by GX Support |
GT1234567 |
Reference | Integer | Y | Unique reference assigned to each deal captured per Code / DateTime. Values used to distinguish between trades with a matching timestamp. | 1 |
TransactionType | Text | Y | Options are Bid , Offer , Trade |
Bid |
DateTime(TimeZone) | ISO-8601 | Y | Timeseries datetime. TimeZone must be specified in the CSV header, so for example DateTime(Europe/London) , and are from the IANA Time Zone Database |
2021-06-15T16:30:00 |
Columns - Core
Most Core colummns are required:
Column | Type | Req. | Description | Example |
---|---|---|---|---|
Buyer(S) | Text | N | Buying company. Blank for Offers. | Counterparty A |
LeadParty(S) | Text | N | Lead party side (market maker). Defined list of Buyer ,Seller , or blank (for markets where this is unavailable) |
Buyer |
Seller(S) | Text | N | Selling company. Blank for Bids. | Counterparty B |
TradingHubDetail(S) | Text | Y | Traded Location. Can match TradingHub (see Metadata), or provide multiple locations depending on data series. | NWE |
Price(DB) | Double | Y | Outright traded price / differential to floating price | 71.75 |
PricingCurrency(S) | Text | Y | ISO-4217 Currency Code, or USC for US Cents, GBX for GB Sterling pennies |
USD |
PricingType(S) | Text | Y | Defined list of Fixed , COD , Laycan , Balmo , Balmo Next Day , EFP , Custom |
COD |
PricingUnit(S) | Text | Y | Unit of measure used for pricing. TOTAL, DAY or otherwise | BBL |
Quantity(DB) | Double | Y | Main traded quantity / volume | 25 |
QuantityUnit(S) | Text | Y | Main unit, including period. /TOTAL, /DAY or otherwise | BBL/DAY |
Columns - Additional
Additional columns that can be accepted:
Column | Type | Req. | Description | Example(s) |
---|---|---|---|---|
CODDays(DB) | Double | N | Completion of discharge days | 5 |
DeliveryBasis(S) | Text | N | Trade delivery terms | FOB , CIF , Delivered |
DeliveryDetails(S) | Text | N | Text description of timing information, e.g. for US pipeline cycle numbers | C5 UNR |
DeliveryEnd(D) | ISO-8601 | N | End of day on which delivery ends | 2021-08-01T00:00:00 |
DeliveryStart(D) | ISO-8601 | N | Start of day on which delivery starts | 2021-08-31T23:59:59 |
Destination(S) | Text | N | Destination location or port | Open |
DiffToSwap(DB) | Double | N | Quoted difference to swap value | 5 |
Excluded(S) | Text | N | Market quoted trade excluded by Source associated with the data series. Defined list of Y ,N ,R |
N |
ExclusionDetails(S) | Text | N | Reason for excluding a market quoted trade. | 'Non Standard Terms' |
Grade(S) | Text | N | NULL if not applicable |
Summer |
OCORef(S) | Text | N | Order cancels order reference | 1 |
Origin(S) | Text | N | Origin location or port | UAE |
PriceMax(DB) | Double | N | Maximum price (e.g for Tenders). Blank if not available | 75.25 |
PricingDetails(S) | Text | N | Free text description of trade terms | MARKET BUYS FROM CPTY 5KMT AT $27.50 (08:28:42) (DEEMED PRICING NOV22) |
PricingMarket(S) | Text | N | Text description of market for floating price deals, or Fixed |
ICE Gasoil , 95 RON GASOLINE MED FOB CARGOES SWAPS , Brent |
PricingPeriod(S) | Text | N | Text description of the pricing period. Dates in YYYY-MM format | 2023-05 |
PricingPeriodEnd(D) | ISO-8601 | N | Final second of last day of pricing period. Blank for fixed price trades | 2021-08-31T23:59:59 |
PricingPeriodStart(D) | ISO-8601 | N | Start of first day of pricing period. Blank for fixed price trades. | 2021-08-01T00:00:00 |
QuantityMax(DB) | Double | N | Maximum quantity tradable as main volume (for Bids / Offers only). Blank if not available | 3 |
SourceDetails(S) | Text | N | Direct provider of the data series. | GX |
TransactionRef(S) | Text | N | 3rd party identifier where available | 123456 |
TransactionStatus(S) | Text | N | Transaction status. Defined list of BidOffer , ReBidOffer , WithdrawsBidOffer , WithdrawnBidOffer , UnwoundBuySell , Deleted , Traded ,Indication |
WithdrawnBidOffer |
TransactionValuation(DB) | Double | N | Valuation of the trade assessed by Source associated with data series | 75.45 |
Vessel(S) | Text | N | Vessel name or type | BP , KPC , TOTAL |
WindowTrade(S) | Text | N | Traded entered into a pricing window platform. Defined list of Y ,N |
N |
Columns - Optional volumes
For optional volumes, the following columns are accepted
Column | Type | Req. | Description | Example(s) |
---|---|---|---|---|
OptCODDays(DB) | Double | N | Optional. Completion of discharge days | 5 |
OptPrice(DB) | Double | N | Outright traded price / differential to floating price for optional leg | 75.15 |
OptPricingCurrency(S) | Text | N | ISO-4217 Currency Code, or USC for US Cents, GBX for GB Sterling pennies |
GBX |
OptPricingMarket(S) | Text | N | Text description of market for floating price deals, or Fixed |
ICE Gasoil |
OptPricingPeriodEnd(D) | ISO-8601 | N | Final second of last day of optional leg pricing period. Blank for fixed price trades. | 2021-08-31T23:59:59 |
OptPricingPeriodStart(D) | ISO-8601 | N | Start of first day of optional leg pricing period. Blank for fixed price trades. | 2021-08-01T00:00:00 |
OptPricingType(S) | Text | N | Defined lisy of Fixed , COD ,Laycan , Balmo , Balmo Next Day , Custom |
COD |
OptPricingUnit(S) | Text | N | Unit of measure used for pricing, including period. /TOTAL, /DAY or otherwise | BBL/TOTAL |
OptQuanityUnit(S) | Text | N | Volume unit, including period. /TOTAL, /DAY or otherwise | BBL/DAY |
OptQuantity(DB) | Text | N | Optional traded quantity / volume | 5 |
QuantityTotal(DB) | Text | N | Total quantity including both Quantity and OptQuantity | '30' |
Trade Metadata
Metadata must be presented in the CSV after any columns. Metadata applies to the set of keys and cannot vary by DateTime.
Metadata | Type | Req. | Description | Example(s) |
---|---|---|---|---|
Commodity(MD-S) | Text | Y | Traded commodity | Gasoline |
Source(MD-S) | Text | Y | Publisher of the data series | GX |
TimeZone(MD-S) | IANA TimeZone | Y | Timezone that all date time fields of this data apply to | Europe/London |
Title(MD-S) | Text | Y | Title of trade data series | Naphtha CIF NWE Cargoes 24KT MOC |
TradingHub(MD-S) | Text | Y | Traded location | NWE |
HolidayCalendar(MD-S) | Text | Y | Data series holiday calendar. Null if not applicable |
Holidays_GX_Europe |
EarlyCloseCalendar(MD-S) | Text | Y | Data series early close calendar. Null if not applicable |
EarlyClose_GX_Europe |