NAV Navbar
shell
  • Introduction
  • Index Series Format
  • sFTP Specification
  • API Specification
  • Email Specification
  • Trade Data Submission
  • Introduction

    Welcome to the General Index documentation site. This site contains the information required to submit and extract data from the General Index system, including file formats, API and FTP specifications.

    For questions regarding integrating GX data into your systems and submitting data, please contact support@general-index.com.

    Index Modules

    Index Series are grouped for licensing and access via Index Modules. Index Modules correspond to trading hubs and/or commodities.

    Example Index Modules:

    GlobalIndexes

    NorthAmericanCrude

    EuropeanCrude

    EuropeanGasoline

    EuropeanJet

    All data within a single Index Module is available to licensed users including all GX index series, index types, forward periods and timestamps for that Module.

    Individual Index Series may be contained within multiple Index Modules.

    Mappings of Index Series to Index Modules are available through the GX API. Please contact GX Support if you would like to do this.

    Timezones

    Please note: All times and dates associated with Index Seriers are subject to timezones. Timezones are specified in metadata and are as provided in the IANA Time Zone Database (so, for example Europe/London)

    Index Series Format

    This section describes how GX Index Series are defined. It outlines standard data fields and file structures that are available across GX systems.

    GX Index Codes

    GX assigns a unique identifier to each Index Series – each identifier consists of the letters ‘GX’ followed by 7 digits (e.g. GX0000001), along with a title describing the content of the series (e.g. Global Crude Index (GCX) London Assessments).

    Index Factsheets

    A full definition of each index series is documented within the relevant factsheet. Factsheets are available on general-index.com and also via API calls. Factsheets include time zone, units of measure location and pricing basis and calculation methodology.

    Index Keys

    Each series has several time series associated with it. Individual time series are defined by the following keys. All keys are stored as text fields.

    Key Description Type Examples
    Code Index Series code. GX followed by 7 digits Text GX0000001
    TimeRef The timestamp of the assessment. Note the time zone of the timestamp is defined in the Factsheet and metadata HHMM 1630
    PeriodType Granularity of the assessment period. Text Month, Prompt
    Period Name of the period. This field is 1 when Period Type is Prompt. Text 1, 2

    Index Types

    For a specific key combination (Index code at a particular TimeRef, Period, PeriodType), GX produces values for Index Types as defined by the factsheet, for example to denote Mid, High and Low.

    Decimal places and precision of numerical fields are defined in the factsheet.

    Index Type Description Type
    Mid n/a Double
    High n/a Double
    Low n/a Double
    Index This index type can appear Double
    Close n/a Double
    Explanation Assessment explanation / rationale Text

    Additional Columns & Metadata

    All GX codes carry the following columns:

    Column Description Type Example(s)
    FactsheetVersion Version number of the Factsheet used to generate the assessment value. Double 1.0
    PeriodAbs Absolute period which the assessment value relates to. String 2020-04
    PeriodRel Relative period which the assessment value relates to. For most series, this reflects a future delivery period. In some markets, negative values may be published to reflect historic periods The granularity applicable to each series is defined by the relevant Factsheet and available in metadata. Integer 1, 2
    PeriodStart The start date of the delivery window. Defined in ISO8601 format with option for time zone to be defined in the column header. Text 2020-07-01T00:00:00
    PeriodEnd The end date of the delivery window. Defined in ISO8601 format with option for time zone to be defined in the column header. Text 2020-07-31T23:59:59
    RecordStatus A single character lookup value representing the status of the assessment. Statuses used are New N, Correction C and Deleted D N

    In addition to the value columns, metadata from the factsheet is provided to aide the consumption of the Index Series:

    Column Description Type Example(s)
    Alias Text ‘short hand’ descriptor relating to a particular TimeRef, PeriodType and Period Text GCXC
    Title Text description of the data series. Text Eurobob Oxy Gasoline NWE FOB Barges
    Timezone The timezone of the timestamp (TimeRef). Time zones are as defined in the IANA Time Zone Database Text Europe/London
    Units Units of measure Text BBL
    Currency Currency Text USD NULL
    HolidayCalendar The holiday calendar that defines when the Index Series is not published Text Holidays_GX_Europe
    Source This will be GX unless the data is reproduced from a third part Text GX
    DeliveryBasis Freight and delivery information Text FOB Barges
    PricingBasis Flat or difference to another market Text Flat, Diff
    TradingHub The physical location Text NWE, Global
    Commodity Underlying commodity Text Crude
    Frequency Assessment frequency Text Daily, Monthly

    Holiday Calendars

    curl -X "POST" "https://api.g-x.co/ts?size=5000" \
         -H 'Authorization: Bearer [yourGXAccessToken]' \
         -H 'Accept: text/json' \
         -H 'Content-Type: application/json; charset=utf-8' \
         -d $'{
      "timeZone": "UTC",
      "formatType": "STANDARD",
      "metadata": true,
      "keys": [
        {
          "pattern": true,
          "symbols": {
            "Calendar": "*"
          },
          "groupName": "Meta_Holiday_Calendars"
        }
      ],
      "corrections": false
    }'
    

    GX maintains a list of holiday calendars for GX Index Series and third-party timeseries. Calendar files have the following format:

    Field Name Description Type Example(s)
    Calendar The name of the calendar. Where relevant this will appear in the Factsheet. Text Holidays_GCX Holidays_CME
    Date Date specified without a time zone. YYYY-MM-DD 2020-12-25
    Value(S) Holiday name or Yes. Text Christmas

    Holiday calendars can be queried through the API. Please see example. The holiday calendar should be queried in UTC timezone and the time component of the Date field ignored in the response.

    Absolute Period Format

    The granularity of absolute assessment periods (and held in PeriodAbs) is defined by the factsheet. Absolute assessment periods are published in the following format:

    Granularity Format Example
    Sub Daily YYYY-MM-DD HH[24]MM 2020-04-01 1630
    Daily YYYY-MM-DD 2020-04-01
    Weekly YYYY W[00-52] 2020 W04
    Half Monthly YYYY-MM-DD 2020-04-01
    Monthly YYYY-MM 2020-04
    Quarterly YYYY Q[1-4] 2020 Q1
    Seasonal YYYY [SUM/WIN] 2020 WIN

    sFTP Specification

    GX maintains a sFTP site to enable clients to pull data. For login credentials, please contact GX Support.

    Connection Information

    Info Details
    Protocol sFTP
    Endpoint sftp.g-x.co
    Port 22
    Authentication Username, SSH key

    Folder Structure

    Data files are arranged within a hierarchical folder structure on the sFTP site. Each client can only see the folders relevant to their licensing permissions.

    Sub-folders are arranged according to the following structure:

       Index Module
          Publication Year (YYYY)
             Publication Month (two digit code in the format MM)

    File Naming

    Index data is produced in comma delimited files containing a single header row followed by multiple rows containing individual assessment values for all Index Series published at a given point in time. Files may include indexes for dates other than the publication date (e.g. historic corrections).

    Files are named according to the Index Module and publishing time in the following format:

    GX_[Index Module]_[Publication Time (ISO8601 basic format)].csv

    For example: GX_GlobalIndexes_20200411T100103Z.csv

    The timezone of the time in the file name is always UTC regardless of the timezone of the Index Series. Please note that the publication time may vary from day to day and should be used for sequential processessing and unique file identification purposes only.

    File Structure

    Data is presented in columns. For detailed explanations of the fields, please refer to the Index Series File Format section above.

    Note The order of the columns may vary between CSVs but always follow the basic structure of Index keys, then the Date field, then value fields (including index values) and then metadata columns.

    Note, as different Index Series have different Index Types, the number of columns presented in a particular CSV may vary.

    Field Description
    Code GX Index Series code
    TimeRef Time stamp
    PeriodType Granularity
    Period Period name
    Date followed by timezone (e.g. Date(Europe/London)) Date and time presented in IS8601 format
    [Index Types] Index Types relevant to the index presented as separate columns. Please see above for Index Types available
    FactsheetVersion Double version number
    RecordStatus(S) Currently unsuppoorted. Default N
    PeriodAbs(S) Absolute Period
    PeriodRel(S) Relative Period
    PeriodStart(D) Delivery start date presented in ISO8601 format
    PeriodEnd(D) Delivery end date presented in ISO8601 format
    Alias(MD-S) Short form alias
    Timezone(MD-S) The timezone of the underlying Index Series and of the TimeRef as defined in IANA time zone database. Note it is technically possible for this timezone to be different to the date/time information presented in the Date column
    Units(MD-S) Units of the Index Series
    Currency(MD-S) Currency of the Index Series
    HolidayCalendar(MD-S) Relevant holiday calendar
    Source(MD-S) This will be GX unless the data is reproduced from a third party
    DeliveryBasis(MD-S) Freight and delivery information
    PricingBasis(MD-S) Flat or difference to another market
    TradingHub(MD-S) The physical location
    Commodity(MD-S) Underlying commodity
    Frequency(MD-S) Assessment frequency

    Example File

    An example CSV file is available for download

    Code,Period,PeriodType,TimeRef,Date(Asia/Singapore),FactsheetVersion,Index,PeriodAbs(S),PeriodEnd(D),PeriodRel(S),PeriodStart(D),RecordStatus(S),TimeZone(MD-S),Alias(MD-S),Title(MD-S),Units(MD-S),Currency(MD-S),HolidayCalendar(MD-S),Source(MD-S),DeliveryBasis(MD-S),PricingBasis(MD-S),TradingHub(MD-S),Commodity(MD-S),Frequency(MD-S)
    GX0000097,1,Month,1630,2020-09-28T16:30:00,1,41.375,2020-11,2020-11-30T23:59:59,1,2020-11-01T00:00:00,N,Asia/Singapore,DPART,Crude Oil Dubai FOB Partial Cargoes,BBL,USD,Holidays_GX_Singapore,GX,FOB Partial Cargoes,Flat,Middle East,Crude,Daily
    GX0000098,1,Month,1630,2020-09-28T16:30:00,1,41.375,2020-11,2020-11-30T23:59:59,1,2020-11-01T00:00:00,N,Asia/Singapore,,Crude Oil Oman FOB Partial Cargoes,BBL,USD,Holidays_GX_Singapore,GX,FOB Partial Cargoes,Flat,Middle East,Crude,Daily
    

    API Specification

    Authorization

    General Index API uses API keys to allow access to the API. General Index API expects the API key to be included in a header of all API requests, except when logging in to the server.

    To authorize, use Token or Basic Authentication:

    curl https://api.g-x.co/[api_endpoint_here]
      -H "Authorization: Bearer yourGXAccessToken"
    
    curl https://api.g-x.co/[api_endpoint_here]
      -u username:password ...
    

    Log in

    This endpoint returns a jwt access token and a refresh token. When access token is expired, refresh token can be used to generate a new one. Access tokens are used to validate requests to General Index API.

    curl -u 'login:password' 
      https://api.g-x.co/auth/login
    

    If successful, the above command returns a status code 200 and token, otherwise please check Errors section.

    {
        "token": "your access token",
        "refreshToken": "your refresh token"
    }
    

    HTTP Request

    GET https://api.g-x.co/auth/login

    Request Body

    Parameter Description
    username username of account
    password password of account

    Refresh Access Token

    This endpoint returns a new access token.

    curl -X POST
      https://api.g-x.co/auth/refresh
      -d '{
        "refreshToken": "your refresh token"
        }'
    

    If successful, the above command returns a status code 200 and token, otherwise please check Errors section.

    {
        "token": "your access token",
        "refreshToken": "your refresh token"
    }
    

    HTTP Request

    POST https://api.g-x.co/auth/refresh

    Request Body

    your refresh token

    Log out

    Log out the user by invalidating the refresh token.

    curl -X POST
      https://api.g-x.co/auth/logout
      -H "Authorization: Bearer yourAccessToken"
      -d '{
        "refreshToken": "your refresh token"
        }'
    

    If successful, the above command returns status code 200, otherwise please check Errors section.

    HTTP Request

    POST https://api.g-x.co/auth/logout

    Submit Data

    GX accepts data submissions via file uploads to the GX data lake. File uploading has two steps

    File Metadata Posting

    First step:

    curl -X POST
      https://api.g-x.co/file
      -d '{
            "groupName":"group01",
            "fileName":"test.csv",
            "fileType":"SOURCE",
            "fields":[
                    {
                        "name":"String field",
                        "value":"test"
                    },
                    {
                        "name":"Integer field",
                        "value":"1"
                    },
                    {
                        "name":"Float field",
                        "value":"1.1"},
                    {
                        "name":"Date field",
                        "value":"2017-09-05"
                    }
                    ...
                ]
         }'
      -H "Authorization: Bearer yourGXToken"
    

    If successful, the above command returns status code 201, otherwise please check Errors section.

    This endpoint stores the file metadata and returns a location to PUT the file content. If the file group requires metadata for the uploaded file, you can override default values and provide values for metadata fields that require values. On creation the call returns a 'location', you must PUT the file content to this location.

    HTTP Request

    POST https://api.g-x.co/file

    Request Body

    Data Notes
    groupName Provided by GX Support and unique to your upload, for example Input_AcmeTrading_PushAPI)
    fileName File name should not be blank, must be unique for each file upload
    fileType type of the file, available types are: SOURCE (Default. Use this unless advised otherwise by GX Support) and NCSV
    fields Optional Meta data
    * name field name
    * value field value

    HTTP Response

    Response:

    {
        "location": "https://fileUploadURL",
        "fileId": "file id"
    }
    

    A JSON object is returned with an url to upload a file content.

    File Uploading

    This endpoint saves the file content into the GX Input Data Repository. You should use the 'location' link for the file from the first step response body.

    Second step:

    curl -X PUT --data-binary @"file-name.csv"
       "https://fileUploadURL"
    
    

    If successful, the above command returns status code 200.

    HTTP Request

    Use --data-binary body option to select file from your local machine.

    PUT https://fileUploadURL

    Download Index Series

    Data Structure

    Indexes are stored as timeseries data with associated symbols, metadata and values

    Symbols

    Individual time series are accessible through the following keys. A combination of symbols uniquely defines a time series. Symbol fields are case sensitive and are returned as text strings.

    Field explanation example
    Code GX Index Series code GX0000001
    TimeRef (nb: Case sensitive) Timestamp. Timezone of timestamp is defined in metadata 1630 1200
    PeriodType (nb: Case sensitive) Granularity of the assessment period Month Prompt
    Period Name of the Period 1

    Metadata

    The Fields listed below represent the standard metadata published for all Index Series. Additional Metadata is available for individual Index Series where appropriate. Note Field names are cae sensitive.

    Field explanation example
    Title Long-form name Global Crude Index
    Alias Short-form name GCXC
    TimeZone (nb: Case sensitive) Timezone that the Index Series is published in Europe/London America/New_York
    Currency Currency USD
    Units Units BBL
    HolidayCalendar (nb: Case sensitive) Relevant holiday calendar Holidays_GCX
    Source This will be GX unless the data is reproduced from a third party Text
    DeliveryBasis Freight and delivery information FOB Barges
    PricingBasis Flat or difference to another market Flat, Diff
    TradingHub The physical location NWE, Global
    Commodity Underlying commodity Crude
    Frequency Assessment frequency Daily, Monthly

    Values

    The Fields listed below represent the standard timeseries values associated with each Index Series. Additional Value series are available where appropriate – in particular reflecting multiple Index Value series. All field names are case sensitive.

    Field explanation example
    FactsheetVersion Integer value of relevant Factsheet methodology 1
    PeriodAbs Long-form name 2020-07
    PeriodRel Short-form name 1
    PeriodStart ISO8601 format. Returned in timezone the Index Series is published in
    PeriodEnd ISO8601 format. Returned in timezone the Index Series is published in
    RecordStatus Currently unsupported. Default N N
    [Index Values] Values as defined in the Index Types Index High Volume

    Retrieve Symbol Variants

    curl -X "POST" "https://api.g-x.co/ts/symbol/values" \
         -H 'Authorization: Bearer [yourGXAccessToken]' \
         -d $'{
      "groupName": "Prod_Indexes",
      "key": "Code"
    }'
    
    

    The above returns a JSON response

    {
      "from": 0,
      "items": [
        "GX0000001",
        "GX0000003",
        "GX0000209",
        "GX0000210",
        "GX0000211",
        "GX0000212"
      ],
      "totalSize": 6
    }
    

    This endpoint retrieves the symbols available and variants.

    HTTP Request

    POST https://api.g-x.co/ts/symbol/values

    Header Parameters
    Parameter Required
    Authorization Required Bearer yourGXAccessToken
    URL Parameters
    Parameter Description
    from Index of first element to return. 0 unless retrieving multiple pages.
    limit Number of variants to return
    sort asc / desc
    Request Body
    Parameter Description
    groupName Prod_Indexes unless advised otherwise by GX Support. Note this is case sensitive.
    key Code, TimeRef, PeriodType, Period to retrieve symbol variants available.

    Search Index Series Data

    curl "https://api.g-x.co/ts/symbol/search?query=groupName=Dev_Indexes%26symbols.Code=GX0000001%26symbols.PeriodType=month" \
         -H 'Authorization: Bearer [yourGXAccessToken]'
    

    The above command returns JSON or CSV structured response. It depends on the header value

    {
      "from": 0,
      "totalSize": 6,
      "items": [
        {
          "groupName": "Prod_Indexes",
          "symbols": {
            "TimeRef": "1230",
            "PeriodType": "Month",
            "Period": "1",
            "Code": "GX0000001"
          },
          "metadata": {
            "TimeZone": "Europe/London",
            "Alias": "GCXN",
            "Currency": "USD",
            "Title": "Global Crude Index",
            "HolidayCalendar": "Holidays_GCX",
            "Units": "BBL"
          },
          "values": [
            "PeriodAbs",
            "PeriodStart",
            "FactsheetVersion",
            "Index",
            "PeriodEnd",
            "PeriodRel",
            "RecordStatus"
          ]
        },
        {
          "groupName": "Prod_Indexes",
          "symbols": {
            "TimeRef": "1230",
            "PeriodType": "Month",
            "Period": "2020-07",
            "Code": "GX0000001"
          },
          "metadata": {
            "TimeZone": "Europe/London",
            "Alias": "",
            "Currency": "USD",
            "Title": "Global Crude Index",
            "HolidayCalendar": "Holidays_GCX",
            "Units": "BBL"
          },
          "values": [
            "PeriodAbs",
            "PeriodStart",
            "FactsheetVersion",
            "Index",
            "PeriodEnd",
            "PeriodRel",
            "RecordStatus"
          ]
        }
        ...
      ]
    }
    

    This endpoint returns the symbols and metadata for index timeseries data.

    HTTP Request

    GET https://api.g-x.co/ts/symbol/search?query=

    Header Parameters
    Parameter Required
    Authorization Required Bearer yourGXAccessToken
    Accept Optional text/csv or text/json
    URL Parameters
    Parameter Required Description
    groupName Required Prod_Indexes unless advised otherwise by GX Support. Note: This is case sensitive.
    symbols.[symbolName] Optional Filter for one or more symbols
    size Optional The maximum size of returned hits list. Max size is 1000. Default size is 100.
    from Optional The index of the first hit, to facilitate paging easily. Default value is 0.
    tokenExpiration Optional Duration of keeping alive context with nextPageToken. Example values 10s, 10m.
    nextPageToken Optional It can be passed to get next page of data. Every request with nextPageToken must also specify tokenExpiration.

    Retrieve Symbols For Alias

    
    curl "https://api.g-x.co/ts/symbol/search?query=Prod_Indexes%26metadata.Alias%3DGCXC" \
         -H 'Authorization: Bearer [YourGXAccessToken]'
    
    

    Returns the following JSON

    {
      "from": 0,
      "totalSize": 1,
      "items": [
        {
          "groupName": "Prod_Indexes",
          "symbols": {
            "TimeRef": "1630",
            "PeriodType": "Month",
            "Period": "1",
            "Code": "GX0000001"
          },
          "metadata": {
            "TimeZone": "Europe/London",
            "Alias": "GCXC",
            "Currency": "USD",
            "Title": "Global Crude Index",
            "HolidayCalendar": "Holidays_GCX",
            "Units": "BBL"
          },
          "values": [
            "PeriodAbs",
            "PeriodStart",
            "FactsheetVersion",
            "Index",
            "PeriodEnd",
            "PeriodRel",
            "RecordStatus"
          ]
        }
      ]
    }
    
    

    The endpoint can also be used to return the symbols for a particular GX Alias.

    HTTP Request

    GET https://api.g-x.co/ts/symbol/search?query=

    Header Parameters
    Parameter Required
    Authorization Required Bearer yourGXAccessToken
    Accept Optional text/csv or text/json
    URL Parameters
    Parameter Required Description
    groupName Required Prod_Indexes unless advised otherwise by GX Support. Note: This is case sensitive.
    metadata.Alias Required The Alias required

    Retrieve Single Index Timeseries

    curl -X "POST" "https://api.g-x.co/ts?size=2" \
         -H 'Authorization: Bearer [yourGXAccessToken]' \
         -d $'{
      "timeZone": "Europe/London",
      "formatType": "STANDARD",
      "startDate": "2020-07-01T10:01:03.999",
      "endDate": null,
      "corrections": true,
      "keys": [
        {
          "columns": [
            "Index"
          ],
          "symbols": {
            "PeriodType": "Month",
            "Period": "1",
            "Code": "GX0000001",
            "TimeRef": "1630"
          },
          "groupName": "Prod_Indexes"
        }
      ]
    }'
    
    

    The above command returns JSON or CSV structured response, depending on the header value:

    {
      "hasNextPage": true,
      "nextPageToken": "1593707400000",
      "columns": [
        {
          "symbols": {
            "Code": "GX0000001",
            "TimeRef": "1630",
            "PeriodType": "Month",
            "Period": "1"
          },
          "groupId": 45,
          "valueName": "Index",
          "offset": 0,
          "valueType": "S",
          "zoneId": "Europe/London",
          "fillType": "NONE"
        }
      ],
      "values": [
        {
          "time": "2020-07-01T16:30:00.000",
          "insertTime": "2020-07-27T19:23:16.173",
          "values": [
            "41.7534"
          ]
        },
        {
          "time": "2020-07-02T16:30:00.000",
          "insertTime": "2020-07-27T19:23:16.173",
          "values": [
            "42.1287"
          ]
        }
      ]
    }
    

    This endpoint retreives index timeseries data for specific symbols in the provided time range. If the symbol does not exist, column is not displayed.

    HTTP Request

    POST https://api.g-x.co/ts

    Header Parameters
    Parameter Required
    Authorization Required Bearer yourGXAccessToken
    Accept Optional text/csv or text/json
    URL Parameters
    Parameter Required Description
    nextPageToken Optional Token returned by last response to get next page of data.
    size Optional Number of rows per page. Max size is 10000. Default size is 10000.
    Request Body
    Parameter Required Description
    startDate Optional Date from (ISO 8601)
    endDate Optional Date to (ISO 8601)
    timeZone Optional Timezone to output data to, converted from the time zone of the underlying Index Series. Note: This does not change the TimeRef of the output data which remains in the time zone of the Index Series
    corrections Optional TRUE provides corrections separately in output file. FALSE (default) provides data with corrections applied.
    onDateTime Optional Show results as it was at a given time (ISO 8601)
    formatType Optional STANDARD (default), PANDAS (optimised for additional data science), NCSV (provides data in more digestible format with metadata on each row.
    metadata Optional TRUE provides the metadata at the start of the response. FALSE (default) no metadat provided.
    keys Required See below

    An alternative to specifying absolute start and end dates is to use the range parameter. The following call retrieves the past 24 hours, for example:

    Parameter Required Description
    range Required (if using lastType call) LAST
    lastType Required (if using lastType call) DAY
    lastTypeAmount Required (if using lastType call) 1
    Request Body - keys
    Parameter Required Description
    groupName Required Prod_Indexes unless advised otherwise by GX Support. Note case sensitive.
    symbols Required Code, PeriodType, Period and TimeRef
    columns Optional List of column names (described in the values section in Data Structure above). If missing, the API returns all columns.

    Retrieve Multiple Timeseries for an Index Series

    curl -X "POST" "https://api.g-x.co/ts?size=5" \
         -H 'Authorization: Bearer [yourGXAccessToken]' \
         -H 'Accept: text/csv' \
         -d $'{
      "formatType": "STANDARD",
      "timeZone": "Europe/London",
      "endDate": null,
      "startDate": "2020-07-01T10:01:03.999",
      "keys": [
        {
          "pattern": true,
          "symbols": {
            "Code": "GX0000001",
            "Period": "1",
            "PeriodType": "Month"
          },
          "columns": [
            "Index"
          ],
          "groupName": "Prod_Indexes"
        }
      ]
    }'
    

    The above command returns JSON or CSV structured response. It depends on the header value.

    time"{TimeRef=1230, PeriodType=Month, Period=1, Code=GX0000001}|Index""{TimeRef=1630, PeriodType=Month, Period=1, Code=GX0000001}|Index"2020-07-01T12:30:00.00041.82562020-07-01T16:30:00.00041.75342020-07-02T12:30:00.00041.93512020-07-02T16:30:00.00042.12872020-07-03T12:30:00.00041.9906
    

    This endpoint retreives multiple timeseries (for example multiple timestamps denoted by different TimeRef) for a particular Index Series.

    Header Parameters
    Parameter Required
    Authorization Required Bearer yourGXAccessToken
    Accept Optional text/csv or text/json
    URL Parameters
    Parameter Required Description
    nextPageToken Optional Token returned by last response to get next page of data.
    size Optional Number of rows per page. Max size is 10000. Default size is 10000.
    Request Body
    Parameter Required Description
    startDate Optional Date from (ISO 8601)
    endDate Optional Date to (ISO 8601)
    timeZone Optional Time Zone to output data to, converted from the time zone of the underlying Index Series. Note: this does not change the TimeRef of the output data which remains in the time zone of the Index Series.
    formatType Optional STANDARD (default), PANDAS (optimised for additional data science), NCSV (provides data in more digestible format with metadata on each row
    keys Required See below

    An alternative to specifying absolute start and end dates is to use the range parameter. The following call retrieves the past 24 hours, for example:

    Parameter Required Description
    range Required (if using lastType call) LAST
    lastType Required (if using lastType call) DAY
    lastTypeAmount Required (if using lastType call) 1
    Request Body - keys
    Parameter Required Description
    pattern Required TRUE
    groupName Required Prod_Indexes unless advised otherwise by GX Support. Note case sensitive.
    symbols Required Code is required. PeriodType, Period and TimeRef can be left out to retrieve all variants.
    columns Optional List of column names (described in the values section in Data Structure above). If missing, the API returns all columns

    Download PDF Reports

    
    ## FILES
    curl "https://api.g-x.co/file/search?size=1000&from=0&query=groupName%3DProd_Reports" \
         -H 'Authorization: Bearer [yourGXAccessToken]'
    
    
    

    The above returns a JSON response

    {
      "from": 0,
      "totalSize": 100,
      "items": [
        {
          "fileName": "GX_MiddleEastCrude_Benchmark_210324.pdf",
          "fileType": "SOURCE",
          "groupName": "Prod_Reports",
          "fid": "c6da572e-d71f-4bdb-8394-a31396cf7c7a",
          "fields": {
            "SKIP_METADATA": "false"
          },
          "arrivalTime": "2021-03-24T14:47:08.399",
          "owner": "system",
          "size": 122903,
          "checkSum": "300ffa79f079f365006e9f1c85b3dcae",
          "zipped": false,
          "shared": false,
          "removed": false,
          "version": 1,
          "rootId": "c6da572e-d71f-4bdb-8394-a31396cf7c7a"
        },
        {
          "fileName": "GX_MiddleEastCrude_Benchmark_210323.pdf",
          "fileType": "SOURCE",
          "groupName": "Prod_Reports",
          "fid": "2b490176-b385-49cc-9453-bf25fe39c969",
          "fields": {
            "SKIP_METADATA": "false"
          },
          "arrivalTime": "2021-03-23T14:40:57.861",
          "owner": "system",
          "size": 134910,
          "checkSum": "2cac31e0c971600f336c69b32b399665",
          "zipped": false,
          "shared": false,
          "removed": false,
          "version": 1,
          "rootId": "2b490176-b385-49cc-9453-bf25fe39c969"
        }
      ]
    }
    
    

    Second step:

    
    curl -v "https://api.g-x.co/file/c6da572e-d71f-4bdb-8394-a31396cf7c7a"
         -H 'Authorization: Bearer [yourGXAccessToken]'
    
    
    

    The above returns a response

    *   Trying 35.177.128.118...
    * TCP_NODELAY set
    * Connected to api.g-x.co (35.177.128.118) port 443 (#0)
    * ALPN, offering h2
    * ALPN, offering http/1.1
    * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
    * successfully set certificate verify locations:
    *   CAfile: /etc/ssl/cert.pem
      CApath: none
    * TLSv1.2 (OUT), TLS handshake, Client hello (1):
    * TLSv1.2 (IN), TLS handshake, Server hello (2):
    * TLSv1.2 (IN), TLS handshake, Certificate (11):
    * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
    * TLSv1.2 (IN), TLS handshake, Server finished (14):
    * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
    * TLSv1.2 (OUT), TLS change cipher, Client hello (1):
    * TLSv1.2 (OUT), TLS handshake, Finished (20):
    * TLSv1.2 (IN), TLS change cipher, Client hello (1):
    * TLSv1.2 (IN), TLS handshake, Finished (20):
    * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
    * ALPN, server did not agree to a protocol
    * Server certificate:
    *  subject: CN=api.g-x.co
    *  start date: Jun 23 00:00:00 2020 GMT
    *  expire date: Jul 23 12:00:00 2021 GMT
    *  subjectAltName: host "api.g-x.co" matched cert's "api.g-x.co"
    *  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
    *  SSL certificate verify ok.
    > GET /file/c6da572e-d71f-4bdb-8394-a31396cf7c7 HTTP/1.1
    > Host: api.g-x.co
    > User-Agent: curl/7.54.0
    > Accept: */*
    > Authorization: Bearer [yourGXAccessToken]
    > 
    < HTTP/1.1 200 
    < Server: nginx/1.17.8
    < Date: Wed, 24 Mar 2021 22:18:20 GMT
    < Content-Type: application/json
    < Content-Length: 404
    < Connection: keep-alive
    < Vary: Accept-Encoding
    < Access-Control-Expose-Headers: Location, X-KH-JOB-ID
    < Access-Control-Allow-Methods: POST, GET, OPTIONS, PATCH, PUT, DELETE
    < Access-Control-Allow-Credentials: true
    < X-KH-JOB-ID: 7ed474bf-dbec-431f-a47e-3c9b2dd9b6d8
    < Access-Control-Max-Age: 86400
    < Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With, X-KH-JOB-ID
    < Location: https://datalake-gx-ng.s3.eu-west-2.amazonaws.com/c6da572e-d71f-4bdb-8394-a31396cf7c7a?response-content-disposition=attachment%3Bfilename%3DGX_MiddleEastCrude_Benchmark_210324.pdf&X-Amz-Security-Token=[AWS Security Tokens]
    < 
    * Connection #0 to host api.g-x.co left intact
    
    

    Third step: \

    
    https://datalake-gx-ng.s3.eu-west-2.amazonaws.com/c6da572e-d71f-4bdb-8394-a31396cf7c7a?response-content-disposition=attachment%3Bfilename%3DGX_MiddleEastCrude_Benchmark_210324.pdf&X-Amz-Security-Token=[AWS Security Tokens]
    
    

    For certain Index Modules, General Index publishes PDFs on a regular (daily or weekly) basis. If you have been permissioned, reports are available using the same credentials.

    There are three steps to downloading PDFs:

    1. First step is retreive file IDs
    2. Second step is to retreive the specific file URL based on the ID obtained.
    3. Third step is to download the file.

    File naming

    All PDFs follow the same naming convention. Files are named according to the Index Module, report type and publishing date in the following format:

    GX_[Index Module]_[Report Type]_yymmdd.pdf

    For example: GX_MiddleEastCrude_Benchmark_210324.pdf

    The timezone of the date in the file name is according to the location of the Index Module.

    Reports that are published as a correction to the original report have the date of the original report in the filename followed by a C, for example GX_MiddleEastCrude_Benchmark_201127C.pdf

    Retreive file IDs

    HTTP Request

    GET https://api.g-x.co/file/search

    Header Parameters
    Parameter Required
    Authorization Required Bearer yourGXAccessToken
    URL Parameters
    Parameter Description
    groupName Prod_Reports unless advised otherwise by GX Support.
    size The maximum size of returned hits list. Max size is 1000. Default size is 100.
    from The index of the first hit. It allows to do paging easily. Default value is 0.

    Retreive specific file URL based

    The URL is in the header of the response.

    HTTP Request

    curl -v "https://api.g-x.co/file/"

    Header Parameters

    None

    URL Parameters
    Parameter Description
    fileID fileID to retreive

    Download Settings Modules

    HTTP Request

    
    curl -X "POST" "https://api.g-x.co/ts?size=5000" \
         -H 'Authorization: Bearer [YourGXToken]' \
         -H 'Accept: text/csv' \
         -H 'Content-Type: application/json; charset=utf-8' \
         -d $'{
      "formatType": "NCSV",
      "keys": [
        {
          "pattern": true,
          "symbols": {
            "Code": "*"
          },
          "groupName": "Settings_Modules"
        }
      ],
      "lastTypeAmount": "1",
      "corrections": false,
      "lastType": "VALUE",
      "range": "LAST",
      "metadata": true
    }'
    
    

    The above command returns a CSV response

    
    Code,Group,IndexModule,Date(UTC),PermissionStatusHistory(S),PermissionLatestStatus(MD-S),Title(MD-S)
    GX0000315,Prod_Indexes,MiddleEastCrude,2021-03-25T00:00:00,PRIMARY,PRIMARY,ARAMCO OSP Crude Oil NWE Extra Light vs ICE Brent Differential
    GX0000324,Prod_Indexes,MiddleEastCrude,2021-03-25T00:00:00,PRIMARY,PRIMARY,Crude Oil Umm Lulu FOB Full Cargoes
    
    ...
    

    Index Series are grouped for licensing and access via Index Modules. Index Modules correspond to trading hubs and/or commodities.

    The example API query can be used to download the latest mappings between Index Modules and Index Series

    Data Struture

    Field explanation example
    Code The GX code of the Index Series GX0000002
    Group Prod_Indexes for all series Prod_Indexes
    IndexModule The index module MiddleEastCrude
    Date(UTC) In ISO 8601 format 2021-03-25T00:00:00
    PermissionStatusHistory(S) A time series of the history of the status. This can be used to determine whether a particular Index Series was permissioned to a particular Index Module on a particular date PRIMARY, SECONDARY, DEVELOPMENT, DELETED
    PermissionLatestStatus(MD-S) The permission status that is currently valid PRIMARY, SECONDARY, DEVELOPMENT, DELETED
    Title(MD-S) The title of the Index Series

    Errors

    The General Index API uses the following error codes:

    Error Code Meaning
    400 Bad Request -- The request you sent is incorrect.
    401 Unauthorized -- Your API key is wrong.
    403 Forbidden -- The requested resource is hidden and may be accessed by the administrators only.
    404 Not Found -- The specified resource could not be found.
    405 Method Not Allowed -- You tried to access a resource with an invalid method.
    406 Not Acceptable -- You requested a format that isn't json.
    410 Gone -- The resource requested has been removed from our servers.
    429 Too Many Requests -- You're requesting too many times! Slow down!
    500 Internal Server Error -- We had a problem with our server. Try again later.
    503 Service Unavailable -- We're temporarily offline for maintenance. Please try again later.

    Email Specification

    Submit Data via Email

    GX accepts data submissions via a dedicated email address. Email can contain data in the email body as well as attachments. Contact GX Support to set up a dedicated GX email address to accept your data submissions.

    Trade Data Submission

    Introduction

    This section provides a high-level overview of the trading data which GX requires to produce its indexes and how this can operate from a process perspective. It is not intended to be prescriptive, but to support discussion between General Index and its clients as data submitters and to inform the planning and design of formal interfaces.

    GX has a strong preference for robust, consistent interfaces but is otherwise agnostic to the format, presentation and structure of data. Data flows should be structured, but do not need to be normalised or to meet specific external definitions for submission to General Index. GX can pull data from client systems as required (e.g. client API or FTP) or can accept submissions as per the documentation available here.

    Data submissions need to provide a balance between speed, accuracy and deliverability.

    Methodology Framework

    GX generates pricing assessments under a standard set of procedures that is mathematical, consistent and fully transparent. This is defined in the GX Methodology Framework, available online.

    This documentation does not take precedence over the Methodology Framework or any contractual agreements in place.

    Scope

    Data shared with GX should represent a complete view of relevant trading activity during the official daily trading window in order to avoid any risk of distortion in the assessments provided. This does not include options data, or (for example) long-term structured contracts.

    This document focuses on requirements for crude and refined product markets, including barges and cargoes. It is recognised that other commodities may require enhancement, or entirely separate approaches.

    Bid/offer and trade data

    GX assessment processes include both trades and bid/offer data. Data submissions should include bid/offer data, though GX recognises that client system and process limitations may require different solutions to achieve this.

    Process and Timing

    In order to produce timely assessments, market actions should be received as soon as is practical.

    GX welcomes trade data ahead of formal confirmation with counterparties and operates internal processes to match deals with information from other sources.

    GX will accept updates to trade data to reflect verification and confirmation status. This is reliant on receipt of a unique transaction identifier.

    In particular data forming part of the daily assessment window should be submitted within a maximum of 10 minutes after market close to support inclusion.

    To allow additional assessment timings to be offered in the future, GX prefers to receive updated deal information throughout the day and as soon as possible after execution.

    Any corrections to reported market actions should be communicated to GX as soon as possible. Updates will be considered as per the relevant methodology.

    Interface Design

    Recognising that there is no common data model across the industry, this section describes the information GX needs in general terms – these are not intended to reflect individual data fields in operational systems.

    The data groupings and field names used in this section are for convenience and readability purposes. GX can accept this information in any amalgamation of these fields across headings.

    No Requirement For GX-proprietary Format

    Trading data does not need to be submitted in a specific GX format, but should be submitted consistently, so that GX can normalise the data via its load-then-transform process. Information should be presented in as standard a format as possible.

    Data Definition

    Data is presented as a table below to support information gathering and interface definition. GX recommends that data submitters use a spreadsheet version of this information to capture where and how data is held in internal systems ahead of joint discussion with GX to agree the final interface design, including field names and regular content.

    Core Data

    The table lists critical information needed to form assessments (exact needs will vary across markets). Each data item will either need to be included in regular submissions or defined as static values.

    Data Group Details Source System(s) Fields
    Transaction Reference ID
    Transaction date and time (UTC / with time zone)
    Type (e.g. bid/offer, deal, one-cancels-other bid/offer)
    Trade direction
    Counterparty name
    Status of Market on Close activity (e.g. withdrawn or not)
    Governing terms & conditions
    Commodity Underlying commodity
    Grade
    Seasonal grade
    Quality specifications (especially if outside norm)
    Tolerance
    Delivery Location (including ports and terminals)
    Freight basis
    Loading date range (including window range and optionality)
    Delivery date range
    Pricing Price
    Currency
    Type (e.g. fixed-price / differential to index)
    Basis (including forward period)
    Additional pricing formula characteristics (e.g. optional volume)
    Quantity Quantity (fixed and optional volume)
    Unit of measure
    Energy and volumetric conversion factors

    Preferred Data

    Information listed within this section supports deal wider understanding of market activity and, in particular, provides assurance around deal validity and execution. This is an initial listed intended as a starting point for discussion and not a comprehensive list of all information which could be shared.

    Data Group Details Source System(s) Fields
    Preferred Data Counterparty transaction reference ID
    Trade validation and confirmation status
    Invoice and payment terms
    Other delivery terms
    Laytime
    Demurrage
    Vessel information
    Vessel approval companies
    Governing law

    Source Systems

    Where possible, GX has a strong preference for receiving data from formally managed IT systems (e.g. CTRM, confirmation interfaces, Bid/Offer management, nomination and logistics software) which are supported and maintained outside the Front Office. Where this is not possible (for example, with bid/offer data), this should be highlighted.