NAV
shell

Introduction

Welcome to the General Index documentation site. This site contains the information required to submit and extract data from the General Index system, including file formats and API specifications.

For questions regarding integrating GX data into your systems and submitting data, please contact support@general-index.com.

Methodology Framework

GX generates pricing assessments under a standard set of procedures that is mathematical, consistent and fully transparent. This is defined in the GX Methodology Framework, available online.

This documentation does not take precedence over the Methodology Framework or any contractual agreements in place.

Data Notices

GX sends out ad-hoc data notices containing information on new data series, updates to metadata and corrections that apply to data more than 30 days old. We encourage all users of GX data to sign-up to receive these notices. New users can sign-up here:

https://pages.general-index.com/datanotices

Index Modules

Index Series are grouped for licensing and access via Index Modules. Index Modules correspond to trading hubs and/or commodities.

Example Index Modules:

GlobalIndexes

NorthAmericaCrude

EuropeanCrude

EuropeanGasoline

EuropeanJet

All data within a single Index Module is available to licensed users including all GX index series, index types, forward periods and timestamps for that Module.

Individual Index Series may be contained within multiple Index Modules.

Mappings of Index Series to Index Modules are available through the GX API. Please contact GX Support if you would like to do this.

Time zones

Please note: All times and dates associated with Index Seriers are subject to time zones. Time zones are specified in metadata and are as provided in the IANA Time Zone Database (so, for example Europe/London)

Index Series Format

This section describes how GX Index Series are defined. It outlines standard data fields and file structures that are available across GX systems.

GX Index Codes

GX assigns a unique identifier to each Index Series – each identifier consists of the letters ‘GX’ followed by 7 digits (e.g. GX0000001), along with a title describing the content of the series (e.g. Global Crude Index (GCX) London Assessments).

Index Methodology Factsheets

A full definition of each index series is documented within the relevant factsheet. Factsheets include time zone, units of measure, location, pricing basis and calculation methodology.

Factsheet server

PDF versions of the factsheets are available from the factsheet server via this URL

https://factsheets.g-x.co/[GX_CODE].pdf

So, for example, the factsheet for Global Crude Index (GCX) London Assessments can be obtained from https://factsheets.g-x.co/GX0000001.pdf

Index Keys

Each series has several time series associated with it. Individual time series are defined by the following keys. All keys are stored as text fields.

Key Description Type Examples
Code Index Series code. GX followed by 7 digits Text GX0000001
TimeRef The timestamp of the assessment, always 4 digits. Note the time zone of the timestamp is defined in the Factsheet and metadata HHMM 1630, 0700, 0000
PeriodType Granularity of the assessment period. Text Month, Prompt
Period Name of the period. This field is 1 when Period Type is Prompt. Text 1, 2

Index Types

For a specific key combination (Index code at a particular TimeRef, Period, PeriodType), GX produces values for Index Types as defined by the factsheet, for example to denote Mid, High and Low.

Decimal places and precision of numerical fields are defined in the factsheet.

Index Type Description Type
Mid n/a Double
High n/a Double
Low n/a Double
Index This index type can appear Double
Close n/a Double
Explanation Assessment explanation / rationale Text

Additional Columns & Metadata

All GX codes carry the following columns:

Column Description Type Example(s)
FactsheetVersion Version number of the Factsheet used to generate the assessment value. Double 1.0
PeriodAbs Absolute period which the assessment value relates to. String 2020-04
PeriodRel Relative period which the assessment value relates to. For most series, this reflects a future delivery period. In some markets, negative values may be published to reflect historic periods The granularity applicable to each series is defined by the relevant Factsheet and available in metadata. Integer 1, 2
PeriodStart The start date of the delivery window. Defined in ISO8601 format with option for time zone to be defined in the column header. Text 2020-07-01T00:00:00
PeriodEnd The end date of the delivery window. Defined in ISO8601 format with option for time zone to be defined in the column header. Text 2020-07-31T23:59:59
RecordStatus Column depreciated. N in all cases Text N

In addition to the value columns, metadata from the factsheet is provided to aide the consumption of the Index Series:

Column Description Type Example(s)
Alias Text ‘short hand’ descriptor relating to a particular TimeRef, PeriodType and Period Text GCXC
Title Text description of the data series. Text Eurobob Oxy Gasoline NWE FOB Barges
Timezone The timezone of the timestamp (TimeRef). Time zones are as defined in the IANA Time Zone Database Text Europe/London
Units Units of measure Text BBL NULL
Currency Currency Text USD NULL
HolidayCalendar The holiday calendar that defines when the Index Series is not published Text Holidays_GX_Europe
Source This will be GX unless the data is reproduced from a third party Text GX
DeliveryBasis Freight and delivery information Text FOB Barges
PricingBasis Flat or difference to another market Text Flat, Diff
TradingHub The physical location Text NWE, Global
Commodity Underlying commodity Text Crude
Frequency Assessment frequency Text Daily, Monthly

Holiday Calendars


curl "https://api.g-x.co/index?group=Meta_Holiday_Calendars&from=all&metadata=true" \
     -H 'Authorization: Bearer [YourGXAccessToken]'

GX maintains a list of holiday calendars for GX Index Series and third-party timeseries. Calendar files have the following format:

Field Name Description Type Example(s)
Calendar The name of the calendar. Where relevant this will appear in the Factsheet. Text Holidays_GCX Holidays_CME
Date Date specified without a time zone. YYYY-MM-DD 2020-12-25
Value(S) Holiday name or Yes. Text Christmas

Holiday calendars can be queried through the API. Please see example. The holiday calendar should be queried in UTC time zone and the time component of the Date field ignored in the response.

Absolute Period Format

The granularity of absolute assessment periods (and held in PeriodAbs) is defined by the factsheet. Absolute assessment periods are published in the following format:

Granularity / PeriodType Description Format Example
n/a No delivery period applies Null Null
Prompt Prompt delivery period Prompt Prompt
12Hour 12 hour period. Shown as start of period YYYY-MM-DDTHH:MM 2020-03-22T12:00
Day Calendar day YYYY-MM-DD 2020-04-01
Week ISO week number YYYY W[00-52] 2020 W04
HalfMonth First or second half of a calendar month. Shown as start date of period YYYY-MM-DD 2020-04-16
Month Calendar month YYYY-MM 2020-04
Quarter Quarter of a calendar year YYYY Q[1-4] 2020 Q1
Season Calendar season YYYY [SUM/WIN] 2020 WIN
HalfYear First or second half of a calendar year YYYY H[1-2] 2020 H2
Year Calendar year YYYY 2020
Cycle Pipeline cycle number C[0-9][0-9] C03

API Specification

Authorisation

General Index provides multiple ways to authenticate:

Username and password

An invitation email is sent to the address provided. Click on the link to create a password. Contact GX Support to have it reset.

URL authentication

Parameter Required Description
token Optional Either API key or basic token

API key

To generate a new API key, visit:

https://app.g-x.co/MyAccount#Settings

Scroll to the bottom of the page and click 'Generate New Key'. The name is for your use. Please make a note of the token generated. You will have to generate a new token once you close the window.

Basic token

# Basic token passed in url
# The token used in the example was generated for user@user.com:password
curl "https://api.g-x.co/index?token=dXNlckB1c2VyLmNvbTpwYXNzd29yZA=="

A basic token is generated from the username and password, Base64 encoded separated by a colon. It can be generated from command line, for example in macOS / Linux:
echo -n "username:password" | base64

Header authentication

Parameter Required
Authorization Optional Bearer yourGXAccessToken or Basic base64(user:password)

Basic auth token

Basic auth token is Base64-encoded username and password, separated by a colon, as described above.

Log in

This endpoint returns a jwt access token and a refresh token. When access token is expired, refresh token can be used to generate a new one.

curl -u 'login:password' 
  https://api.g-x.co/auth/login

If successful, the above command returns a status code 200 and token, otherwise please check Errors section.

{
    "token": "YourGXAccessToken",
    "refreshToken": "your refresh token"
}

HTTP Request

GET https://api.g-x.co/auth/login

Request Body

Parameter Description
username username of account
password password of account

Refresh Access Token

This endpoint returns a new access token.

curl -X POST
  https://api.g-x.co/auth/refresh
  -d '{
    "refreshToken": "your refresh token"
    }'

If successful, the above command returns a status code 200 and token, otherwise please check Errors section.

{
    "token": "YourGXAccessToken",
    "refreshToken": "your refresh token"
}

HTTP Request

POST https://api.g-x.co/auth/refresh

Request Body

your refresh token

Log out

Log out the user by invalidating the refresh token.

curl -X POST
  https://api.g-x.co/auth/logout
  -H "Authorization: Bearer YourGXAccessToken"
  -d '{
    "refreshToken": "your refresh token"
    }'

If successful, the above command returns status code 200, otherwise please check Errors section.

HTTP Request

POST https://api.g-x.co/auth/logout

Data Catalogue

Catalogue Endpoint


curl "https://api.g-x.co/catalogue?module=EuropeanGasoline \
     -H 'Authorization: Bearer [YourGXAccessToken]'

The above command returns a CSV response
Code,PeriodType,TimeRef,IndexModule,Alias,Bid,Commodity,Currency,DeliveryBasis,EndDate,FactsheetVersion,Frequency,Group,High,HolidayCalendar,Index,InterAffiliateDataAccepted,LastTradeDateCalendar,Low,MapLocation,Mid,Offer,ParentCodes,PeriodMax,PeriodMin,PermissionLatestStatus,Precision,PricingBasis,PriorityToTransactions,QaSeries,Quantity,SoleSourcedDataAccepted,Source,StartDate,Status,ThirdPartyMappings,TimeRefDetails,TimeZone,Title,TradingHub,Units,Volume
GX0000614,Month,1630,EuropeanGasoline,,,Gasoline,USD,FOB Barges,,1,Daily,Prod_Indexes,,Holidays_GX_Europe,Y,N,,,,,,GX0000002 GX0000002,1,1,PRIMARY,2,Diff,N,ProdIndexes HolidaysGXEurope Daily Index,,N,GX,2021-02-25,Live,LSEG:GX0000614L1630c1 Bloomberg:GIOS0614 Index,London Close,Europe/London,Eurobob Oxy Gasoline NWE FOB Barges M1 vs M6,NWE,MT,
GX0000472,Prompt,1630,EuropeanGasoline,,,Gasoline,USD,CIF Cargoes,,1,Monthly,Prod_Indexes,Y,Holidays_GX_Europe,,N,,Y,,Y,,,1,1,PRIMARY,4,Flat,N,QA ProdIndexes HolidaysGXEurope Monthly Mid,,N,GX,2008-01-31,Live,LSEG:GX0000472L1630P Bloomberg:GIOS0472 Index Platts:AAWZB03,London Close,Europe/London,Gasoline 95 RON MED CIF Cargoes Monthly Average,MED,MT,

HTTP Request

GET https://api.g-x.co/catalogue

The endpoint returns a list of all available Index data series. URL parameters allows filtering data by Module.

URL Parameters
Parameter Required Default Description
module Optional None Module name, if not provided all indexes are returned
token Optional None If Authorization header is missing, Basic Auth Token can be passed as a parameter

Data Structure

Indexes are stored as timeseries data with associated symbols, metadata and values

Symbols

Individual time series are accessible through the following keys. A combination of symbols uniquely defines a time series. Symbol fields are case sensitive and are returned as text strings.

Field explanation example
Code GX Index Series code GX0000001
TimeRef (nb: Case sensitive) Timestamp, expressed as 4 digits. Timezone of timestamp is defined in metadata 1630 0700
PeriodType (nb: Case sensitive) Granularity of the assessment period Month Prompt
Period Name of the Period 1

Metadata

The Fields listed below represent the standard metadata published for all Index Series. Additional Metadata is available for individual Index Series where appropriate. Note Field names are case sensitive.

Field explanation example
Title Long-form name Global Crude Index
Alias Short-form name GCXC
TimeZone (nb: Case sensitive) Timezone that the Index Series is published in Europe/London America/New_York
Currency Currency USD
Units Units BBL
HolidayCalendar (nb: Case sensitive) Relevant holiday calendar Holidays_GCX
Source This will be GX unless the data is reproduced from a third party Text
DeliveryBasis Freight and delivery information FOB Barges
PricingBasis Flat or difference to another market Flat, Diff
TradingHub The physical location NWE, Global
Commodity Underlying commodity Crude
Frequency Assessment frequency Daily, Monthly

Values

The Fields listed below represent the standard timeseries values associated with each Index Series. Additional Value series are available where appropriate – in particular reflecting multiple Index Value series. All field names are case sensitive.

Field explanation example
FactsheetVersion Integer value of relevant Factsheet methodology 1
PeriodAbs Long-form name 2020-07
PeriodRel Short-form name 1
PeriodStart ISO8601 format. Returned in timezone the Index Series is published in
PeriodEnd ISO8601 format. Returned in timezone the Index Series is published in
RecordStatus Currently unsupported. Default N N
[Index Values] Values as defined in the Index Types Index High Volume

Pull Index Data Series

Index Endpoint

The endpoint retrieves index timeseries data in the provided time range. URL parameters allows filtering data by Code or metadata fields.

HTTP Request

curl "https://api.g-x.co/index?code=GX0000004&metadata=true" \
     -H 'Authorization: Bearer [YourGXAccessToken]'

The above command returns a CSV response
Code,Period,PeriodType,TimeRef,Date(UTC),FactsheetVersion,High,Low,Mid,PeriodAbs(S),PeriodEnd(D),PeriodRel(S),PeriodStart(D),RecordStatus(S),Alias(MD-S),Commodity(MD-S),Currency(MD-S),DeliveryBasis(MD-S),Frequency(MD-S),HolidayCalendar(MD-S),PricingBasis(MD-S),Source(MD-S),TimeZone(MD-S),Title(MD-S),TradingHub(MD-S),Units(MD-S)
GX0000004,1,Prompt,1730,2022-11-29T17:30:00,2,774.75,774.25,774.5,Prompt,2022-12-07T23:59:59,1,2022-12-01T00:00:00,N,EUROBOBNONC,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
GX0000004,1,Prompt,1630,2022-11-29T16:30:00,1,774.75,774.25,774.5,Prompt,2022-12-07T23:59:59,1,2022-12-01T00:00:00,N,,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
GX0000004,1,Prompt,1730,2022-11-28T17:30:00,2,732.25,731.75,732,Prompt,2022-12-06T23:59:59,1,2022-11-30T00:00:00,N,EUROBOBNONC,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
GX0000004,1,Prompt,1630,2022-11-28T16:30:00,1,732.25,731.75,732,Prompt,2022-12-06T23:59:59,1,2022-11-30T00:00:00,N,,Gasoline,USD,FOB Barges,Daily,Holidays_GX_Europe,Flat,GX,Europe/London,Eurobob Non Oxy Gasoline NWE FOB Barges,NWE,MT
# All indexes published today (by London timezone) with username / password authorization: 
curl -u 'username:password' https://api.g-x.co/index?timezone=Europe/London
# All indexes published today (UTC timezone) with metadata: 
curl -u 'username:password' https://api.g-x.co/index?metadata=true
# Last 10 days data of code GX0000002: 
curl -u 'username:password' https://api.g-x.co/index?code=GX0000002&from=10d
# All Diesel indexes for last 5 days: 
curl -u 'username:password' https://api.g-x.co/index?metadata.Commodity=Diesel&from=5d
# Authorization token passed in url
# The token used in the example was generated for user@user.com:password
curl "https://api.g-x.co/index?from=5d&token=dXNlckB1c2VyLmNvbTpwYXNzd29yZA=="

GET https://api.g-x.co/index

URL Parameters
Parameter Required Default Description
code Optional None Comma separated list of GX codes. If not provided all indexes are returned
period Optional None Comma separated list of period values. If not provided all periods are returned
periodtype Optional None Comma separated list of periodtype values. If not provided all periodtypes are returned
timeref Optional None Comma separated list of timeref values. If not provided all timerefs are returned
module Optional None Index module to return, ex. module=NorthAmericaCrude. If not provided all modules are returned. Cannot be used in conjunction with the code, period, or periodtype parameters
group Optional Prod_Indexes Use default value unless advised otherwise by GX Support. Note: This is case sensitive.
from Optional today Start date, accepted values:
  • today - beginning of the current day in provided timezone parameter
  • *v - specific number of values back
  • *d - specific number of days back
  • *m - specific number of months back
  • *y - specific number of years back
  • all - no date restriction
  • ISO date ex. 2022-11-21T10:00:00
to Optional today End date, accepted values:
  • today - end of the current day in provided timezone parameter
  • *v - specific number of values back
  • *d - specific number of days back
  • *m - specific number of months back
  • *y - specific number of years back
  • ISO date ex. 2022-11-21T11:00:00
metadata.field Optional None Filter indexes by field value, ex. metadata.Commodity=Diesel
metadata Optional false Controls whether returns metadata, accepted values: true / false / only
delta Optional false Whether use time range against transaction time or insertion time. Accepted values: true / false. This can be used to run queries showing what has changed in the database in the requested date range.
timezone Optional UTC A timezone name for requested time range and returned data. Time zones are as defined in the IANA Time Zone Database.
token Optional None If Authorization header is missing, Basic Token can be passed as a parameter. See url authentication above.
Header Parameters

See header authentication above.

Parameter Required
Authorization Optional Bearer yourGXAccessToken or Basic base64(user:password)
Accept Optional text/csv (default) or application/json

Submit Data

GX accepts data submissions via file uploads to the GX data lake. File uploading has two steps

File Metadata Posting

First step:

curl -X POST
  https://api.g-x.co/file
  -d '{
        "groupName":"group01",
        "fileName":"test.csv",
        "fileType":"SOURCE",
        "fields":[
                {
                    "name":"String field",
                    "value":"test"
                },
                {
                    "name":"Integer field",
                    "value":"1"
                },
                {
                    "name":"Float field",
                    "value":"1.1"},
                {
                    "name":"Date field",
                    "value":"2017-09-05"
                }
                ...
            ]
     }'
  -H "Authorization: Bearer yourGXToken"

If successful, the above command returns status code 201, otherwise please check Errors section.

This endpoint stores the file metadata and returns a location to PUT the file content. If the file group requires metadata for the uploaded file, you can override default values and provide values for metadata fields that require values. On creation the call returns a 'location', you must PUT the file content to this location.

HTTP Request

POST https://api.g-x.co/file

Request Body

Data Notes
groupName Provided by GX Support and unique to your upload, for example Input_AcmeTrading_PushAPI)
fileName File name should not be blank, must be unique for each file upload
fileType type of the file, available types are: SOURCE (Default. Use this unless advised otherwise by GX Support) and NCSV
fields Optional Meta data
* name field name
* value field value

HTTP Response

Response:

{
    "location": "https://fileUploadURL",
    "fileId": "file id"
}

A JSON object is returned with an url to upload a file content.

File Uploading

This endpoint saves the file content into the GX Input Data Repository. You should use the 'location' link for the file from the first step response body.

Second step:

curl -X PUT --data-binary @"file-name.csv"
   "https://fileUploadURL"

If successful, the above command returns status code 200.

HTTP Request

Use --data-binary body option to select file from your local machine.

PUT https://fileUploadURL

Download PDF Reports


## FILES
curl "https://api.g-x.co/file/search?size=1000&from=0&query=groupName%3DProd_Reports" \
     -H 'Authorization: Bearer [yourGXAccessToken]'


The above returns a JSON response

{
  "from": 0,
  "totalSize": 100,
  "items": [
    {
      "fileName": "GX_MiddleEastCrude_Benchmark_210324.pdf",
      "fileType": "SOURCE",
      "groupName": "Prod_Reports",
      "fid": "c6da572e-d71f-4bdb-8394-a31396cf7c7a",
      "fields": {
        "SKIP_METADATA": "false"
      },
      "arrivalTime": "2021-03-24T14:47:08.399",
      "owner": "system",
      "size": 122903,
      "checkSum": "300ffa79f079f365006e9f1c85b3dcae",
      "zipped": false,
      "shared": false,
      "removed": false,
      "version": 1,
      "rootId": "c6da572e-d71f-4bdb-8394-a31396cf7c7a"
    },
    {
      "fileName": "GX_MiddleEastCrude_Benchmark_210323.pdf",
      "fileType": "SOURCE",
      "groupName": "Prod_Reports",
      "fid": "2b490176-b385-49cc-9453-bf25fe39c969",
      "fields": {
        "SKIP_METADATA": "false"
      },
      "arrivalTime": "2021-03-23T14:40:57.861",
      "owner": "system",
      "size": 134910,
      "checkSum": "2cac31e0c971600f336c69b32b399665",
      "zipped": false,
      "shared": false,
      "removed": false,
      "version": 1,
      "rootId": "2b490176-b385-49cc-9453-bf25fe39c969"
    }
  ]
}

Second step:


curl -v "https://api.g-x.co/file/c6da572e-d71f-4bdb-8394-a31396cf7c7a/download"
     -H 'Authorization: Bearer [yourGXAccessToken]'


The above returns the file as the response.

For certain Index Modules, General Index publishes PDFs on a regular (daily or weekly) basis. If you have been permissioned, reports are available using the same credentials.

There are two steps to downloading PDFs:

  1. First step is retrieve file IDs
  2. Second step is to retrieve the specific file based on the ID obtained.

File naming

All PDFs follow the same naming convention. Files are named according to the Index Module, report type and publishing date in the following format:

GX_[Index Module]_[Report Type]_yymmdd.pdf

For example: GX_MiddleEastCrude_Benchmark_210324.pdf

The timezone of the date in the file name is according to the location of the Index Module.

Reports that are published as a correction to the original report will have the date of the original report in the filename but have a more recent arrivalTime.

Retrieve file IDs

HTTP Request

GET https://api.g-x.co/file/search

Header Parameters
Parameter Required
Authorization Required Bearer yourGXAccessToken
URL Parameters
Parameter Description
groupName Prod_Reports unless advised otherwise by GX Support.
size The maximum size of returned hits list. Max size is 1000. Default size is 100.
from The index of the first hit. It allows to do paging easily. Default value is 0.

Retrieve specific file

The URL is in the header of the response.

HTTP Request

curl -v "https://api.g-x.co/file/[fid]/download"

Header Parameters
Parameter Required
Authorization Required Bearer yourGXAccessToken
URL Parameters
Parameter Description
fid fileID to retrieve

Report Holiday Calendars

PDF reports are published daily, Monday to Friday apart from on dates specified in the relevant holiday calendar as specified below:

Report Name Holiday Calendar
GX_AmericasMarine_Benchmark Holidays_ICE_Brent
GX_AsiaFuelOil_Benchmark Holidays_GX_Singapore
GX_AsiaGasoil_Benchmark Holidays_GX_Singapore
GX_AsiaGasoline_Benchmark Holidays_GX_Singapore
GX_AsiaJet_Benchmark Holidays_GX_Singapore
GX_AsiaLPG_Benchmark Holidays_GX_Singapore
GX_AsiaMarine_Benchmark Holidays_ICE_Brent
GX_EuropeanGasoline_Benchmark Holidays_GX_Europe
GX_EuropeanJet_Benchmark Holidays_GX_Europe
GX_EuropeanLPG_Benchmark Holidays_GX_Europe
GX_EuropeanMarine_Benchmark Holidays_ICE_Brent
GX_GlobalMarineCarbon_Benchmark Holidays_ICE_Brent
GX_MiddleEastCrude_Benchmark Holidays_GX_Singapore
GX_NorthAmericaCrude_Benchmark Holidays_GX_NorthAmerica
GX_NorthAmericaStorage_Benchmark Holidays_GX_NorthAmerica

Please see documentation for instructions to access detailed holiday calendar information.

Errors

The General Index API uses the following status codes:

Status Code Meaning
400 Bad Request -- The request you sent is incorrect.
401 Unauthorized -- Your API key is wrong.
403 Forbidden -- The requested resource is hidden and may be accessed by the administrators only.
404 Not Found -- The specified resource could not be found.
405 Method Not Allowed -- You tried to access a resource with an invalid method.
406 Not Acceptable -- You requested a format that isn't json.
410 Gone -- The resource requested has been removed from our servers.
429 Too Many Requests -- You're requesting too many times! Slow down!
500 Internal Server Error -- We had a problem with our server. Try again later.
503 Service Unavailable -- We're temporarily offline for maintenance. Please try again later.

Data Integration Guide

Approach

The following approach is recommended to receive an accurate and timely replica of GX data.

Regular Data Pull

# All indexes published within a 15 minute time window with username / password authorization: 
"https://api.g-x.co/index?from=2022-09-01T17:00:00&to=2022-09-01T17:14:59.999&metadata=true&delta=true&token=[YourGXAccessToken]"

Query the /index api every 15 minutes with the following parameters.

HTTP Request

https://api.g-x.co/index

Parameter Value Example
from 16 mins before the current time, expressed as UTC 2022-09-01T17:00:00
to 1 min before the current time, expressed as UTC 2022-09-01T17:14:59.999
delta Set to true to get all changes to the database in that time period true
metadata Set to true to include metadata fields true

The range for each query should end 1 minute prior to the query time to ensure that all data in range is available via the API.

Data will be returned in UTC by default. Each series can be translated into the publication timezone using the metadata associated with the series.

Only columns updated during the insert range are returned. Where values for one or more columns have been updated during the range selected, unchanged columns will return empty values.

Metadata Request

# Metadata only
"https://api.g-x.co/index?metadata=only"

Metadata can be returned without values by passing a metadata value of 'only'.

Please note that Metadata values may be updated from time to time. Minor updates will not be included in a Data Notice.

Python SDK

SDK Introduction

GX maintains an SDK to allow developers to query & download index series data and upload submissions to our data lake.

Documentation & Installation

Full documentation can be found on the PyPI project page

Example Code

You can download an example script using the SDK.

Excel Data Extract Tool

The General Index data extract tool allows clients to search the data catalogue and download our index data into Excel for further analysis.

Requirements

This tool is designed to run on either Apple or Windows PCs with the Office 365 version of Excel. The tool uses native Excel functionality to query the GX API via "PowerQuery".

On first opening the spreadsheet or refreshing data, a Privacy Levels dialogue window may be displayed. If so, the "Ignore Privacy Levels" checkbox should be selected.

Installation

The latest version of the tool can be found here:

https://www.general-index.com/data-extraction-tool

Getting started

Settings Tab: Enter your GX username and password on the Settings tab.

*** Your Password is CONFIDENTIAL to you. Please remove before sharing and contact GX Support if you would like credentials issued to another user. ***

Catalogue Tab: Leave all fields blank and choose "Refresh All" from the Data section of the Excel Ribbon to display all data series.

Catalogue Tab: Choose one or more series and copy the corresponding Code, PeriodType, Period and Type values. For example:

Code PeriodType Period TimeRef
GX0000001 Month 1 1630

Data Query tab: Paste values copied from the "Catalogue" tab into the table rows and delete any additional rows containing series not needed.

Data Output: Choose "Refresh All" from the Data section of the Excel Ribbon to display all data series matching the entries shown on the "Data Query" tab.

Catalogue

This tab allows the full GX Data Catalogue to be queried and returned.

Filters can be applied to restrict results by Title, Alias, Commodity and Module. Partial strings can be passed as filters by adding a * to the start and/or end. Either a wildcard, or a string must be entered into all fields.

Query results can be updated either by right clicking on the data table and selecting "Refresh", or using the "Refresh All" option on the Data tab.

Data Query / Data Output

The Data Query tab allows data series to be selected for return on the Data Output tab.

To select a data series to return, values should be entered into each of the table columns (Code, PeriodType, Period and Type).

Values can be copied into the table from the Data Query tab, or entered manually.

Table rows can be added or deleted as required

A wildcard * can be used to return all matching values for a column.

Query results can be updated either by right clicking on the data table and selecting "Refresh", or using the "Refresh All" option on the Data tab.

Please note that it is possible to select large volumes of data via this query which may take a significant amount of time to return.

Settings

Data Size controls the maximum number of unique DateTime records returned on the Data Output tab.

TimeZone dropdown list can be used to adjust the timezone in which the DateTime records are returned on the Data Output tab.

Submitting Data

GX accepts data submissions from partners globally.

Source Systems

Where possible, GX has a strong preference for receiving data from formally managed IT systems (e.g. CTRM, confirmation interfaces, Bid/Offer management, nomination and logistics software) which are supported and maintained outside the Front Office. Where this is not possible (for example, with bid/offer data), this should be highlighted.

Submit Data via Email

GX accepts data submissions via a dedicated email address. Email can contain data in the email body as well as attachments. Contact GX Support for a dedicated email address to accept your data submissions.

Submit Data via API

Refer to the API documentation on how to submit data to our data lake once your API credentials have been updated to allow uploads.

Trade Data Format

General Index processes trade data from a number of sources across commodities and geographies. We have developed a flexible data format that allows accurate and efficient processing

GX assessment processes include both trades and bid/offer data. Our data structure applies to both.

File Format

The following describes the data format which can be used for downloading and uploading trade data as CSV files.

Trade Data Keys

The combination of keys must be unique. Transactions with the same keys will be considered corrections and the previous data will be overwritten.

All keys are required:

Key Type Req. Description Example
Code Text Y GX Trade Series code. GT followed by 7 digits. Codes can be provided by GX Support GT1234567
Reference Integer Y Unique reference assigned to each deal captured per Code / DateTime. Values used to distinguish between trades with a matching timestamp. 1
TransactionType Text Y Options are Bid, Offer, Trade Bid
DateTime(TimeZone) ISO-8601 Y Timeseries datetime. TimeZone must be specified in the CSV header, so for example DateTime(Europe/London), and are from the IANA Time Zone Database 2021-06-15T16:30:00

Columns - Core

Most Core colummns are required:

Column Type Req. Description Example
Buyer(S) Text N Buying company. Blank for Offers. Counterparty A
LeadParty(S) Text N Lead party side (market maker). Defined list of Buyer,Seller, or blank (for markets where this is unavailable) Buyer
Seller(S) Text N Selling company. Blank for Bids. Counterparty B
TradingHubDetail(S) Text Y Traded Location. Can match TradingHub (see Metadata), or provide multiple locations depending on data series. NWE
Price(DB) Double Y Outright traded price / differential to floating price 71.75
PricingCurrency(S) Text Y ISO-4217 Currency Code, or USC for US Cents, GBX for GB Sterling pennies USD
PricingType(S) Text Y Defined list of Fixed, COD, Laycan, Balmo, Balmo Next Day, EFP, Custom COD
PricingUnit(S) Text Y Unit of measure used for pricing. TOTAL, DAY or otherwise BBL
Quantity(DB) Double Y Main traded quantity / volume 25
QuantityUnit(S) Text Y Main unit, including period. /TOTAL, /DAY or otherwise BBL/DAY

Columns - Additional

Additional columns that can be accepted:

Column Type Req. Description Example(s)
CODDays(DB) Double N Completion of discharge days 5
DeliveryBasis(S) Text N Trade delivery terms FOB, CIF, Delivered
DeliveryDetails(S) Text N Text description of timing information, e.g. for US pipeline cycle numbers C5 UNR
DeliveryEnd(D) ISO-8601 N End of day on which delivery ends 2021-08-01T00:00:00
DeliveryStart(D) ISO-8601 N Start of day on which delivery starts 2021-08-31T23:59:59
Destination(S) Text N Destination location or port Open
DiffToSwap(DB) Double N Quoted difference to swap value 5
Excluded(S) Text N Market quoted trade excluded by Source associated with the data series. Defined list of Y,N,R N
ExclusionDetails(S) Text N Reason for excluding a market quoted trade. 'Non Standard Terms'
Grade(S) Text N NULL if not applicable Summer
OCORef(S) Text N Order cancels order reference 1
Origin(S) Text N Origin location or port UAE
PriceMax(DB) Double N Maximum price (e.g for Tenders). Blank if not available 75.25
PricingDetails(S) Text N Free text description of trade terms MARKET BUYS FROM CPTY 5KMT AT $27.50 (08:28:42) (DEEMED PRICING NOV22)
PricingMarket(S) Text N Text description of market for floating price deals, or Fixed ICE Gasoil, 95 RON GASOLINE MED FOB CARGOES SWAPS, Brent
PricingPeriod(S) Text N Text description of the pricing period. Dates in YYYY-MM format 2023-05
PricingPeriodEnd(D) ISO-8601 N Final second of last day of pricing period. Blank for fixed price trades 2021-08-31T23:59:59
PricingPeriodStart(D) ISO-8601 N Start of first day of pricing period. Blank for fixed price trades. 2021-08-01T00:00:00
QuantityMax(DB) Double N Maximum quantity tradable as main volume (for Bids / Offers only). Blank if not available 3
SourceDetails(S) Text N Direct provider of the data series. GX
TransactionRef(S) Text N 3rd party identifier where available 123456
TransactionStatus(S) Text N Transaction status. Defined list of BidOffer, ReBidOffer, WithdrawsBidOffer, WithdrawnBidOffer, UnwoundBuySell, Deleted, Traded,Indication WithdrawnBidOffer
TransactionValuation(DB) Double N Valuation of the trade assessed by Source associated with data series 75.45
Vessel(S) Text N Vessel name or type BP, KPC, TOTAL
WindowTrade(S) Text N Traded entered into a pricing window platform. Defined list of Y,N N

Columns - Optional volumes

For optional volumes, the following columns are accepted

Column Type Req. Description Example(s)
OptCODDays(DB) Double N Optional. Completion of discharge days 5
OptPrice(DB) Double N Outright traded price / differential to floating price for optional leg 75.15
OptPricingCurrency(S) Text N ISO-4217 Currency Code, or USC for US Cents, GBX for GB Sterling pennies GBX
OptPricingMarket(S) Text N Text description of market for floating price deals, or Fixed ICE Gasoil
OptPricingPeriodEnd(D) ISO-8601 N Final second of last day of optional leg pricing period. Blank for fixed price trades. 2021-08-31T23:59:59
OptPricingPeriodStart(D) ISO-8601 N Start of first day of optional leg pricing period. Blank for fixed price trades. 2021-08-01T00:00:00
OptPricingType(S) Text N Defined lisy of Fixed, COD,Laycan, Balmo, Balmo Next Day, Custom COD
OptPricingUnit(S) Text N Unit of measure used for pricing, including period. /TOTAL, /DAY or otherwise BBL/TOTAL
OptQuanityUnit(S) Text N Volume unit, including period. /TOTAL, /DAY or otherwise BBL/DAY
OptQuantity(DB) Text N Optional traded quantity / volume 5
QuantityTotal(DB) Text N Total quantity including both Quantity and OptQuantity '30'

Trade Metadata

Metadata must be presented in the CSV after any columns. Metadata applies to the set of keys and cannot vary by DateTime.

Metadata Type Req. Description Example(s)
Commodity(MD-S) Text Y Traded commodity Gasoline
Source(MD-S) Text Y Publisher of the data series GX
TimeZone(MD-S) IANA TimeZone Y Timezone that all date time fields of this data apply to Europe/London
Title(MD-S) Text Y Title of trade data series Naphtha CIF NWE Cargoes 24KT MOC
TradingHub(MD-S) Text Y Traded location NWE
HolidayCalendar(MD-S) Text Y Data series holiday calendar. Null if not applicable Holidays_GX_Europe
EarlyCloseCalendar(MD-S) Text Y Data series early close calendar. Null if not applicable EarlyClose_GX_Europe