Connector for Web Data

The Web Data connector allows the retrieval and processing of JSON, XML, delimited Text (such as CSV, TSV), Excel files, HTML tables and Arrow IPC files that are accessible over HTTP/HTTPS.

 

  1. Select the appropriate HTTP Method for the request from the following options:

     

    HTTP Method Description

    GET

    To retrieve data.

    POST

    To add new data.

    PUT

    To replace existing data

     

  2. Enter the absolute URL of the web data including scheme (HTTP/HTTPS) into the URL field.

  3. Click  . A successful connection will result to the following:
    • The Connection Successful status is displayed along with some raw data returned by the server.

    • The connector tries auto discovery of the Data Type (JSON, Text, XML, Excel, HTML, Arrow IPC)

      When loading JSON data, the Record Path can be suggested automatically by pressing the Fetch button. Refer to the section Supported JSON Structures for details about the available options for turning the JSON into a table structure.

      For the Web Data connector, you can also select the HTML data type.

      Property Description

      Table Name

      Click   to fetch HTML table elements then select the HTML table.

      Header Rows

      Select the number of rows in the table to be retrieved as column headings. The default value is 1, but a higher number should be set if the HTML table has column titles that use more than 1 row. If all rows of the table are data and there are no column headers, set it to 0.

       

      NOTE: If no table element is found in the HTML document, an error message displays.

      You can also opt to load or save a copy of the column definition.

  4. Adjust the Authentication Type, if required.
    • Select the Authorization tab.

      • Set the required settings:
        Authentication Type Description

        None

        No authentication needed.

        Basic

        Enter the User ID and Password to connect to the connector’s service. 

        Select the Show Characters box to display the entered characters.

        OAuth

        Some standard OAuth token request parameters come initialized with empty values. You can set these values and also add/remove any/all of the keys.

        Enter or select the following settings:

        • Token URL – The URL to retrieve the access token from.

        • Add Access Token To - The Access token retrieved from the Token URL can be added to headers, URL, or request body, depending on how the endpoint needs the token.

          • Request Headers - A header is automatically added to the REST API request.

          • Request URL - The URL needs to be manually parameterised with a {access_token} parameter, before calling the REST API, the parameter is replaced with the actual token.

          •  Request Body - The Request Body needs to be manually parameterised with a {access_token} parameter, before calling the REST API, the parameter is replaced with the actual token.

          NOTE: The given request parameters key/value pairs are formatted by the connector as application/xwww-form-urlencoded, and posted to Token URL.

        Bearer Token

        If you already have a authentication token, enter the token string into Bearer Token input box. This can be parameterized also.

         

  5. The Headers tab allows you to enter any custom headers required to be passed to the URL endpoint, typically to provide additional metadata. Enter any key/value pairs you need, and the connector will send them along with request.

  6. Set the Body if a POST/PUT request is required.
    • Select the Body tab.

    • Set the required settings: 
      Property Description

      Content Type

      Select or enter content-type based on request body (payload) format.

      NOTE: This property is disabled when the HTTP Method is GET.

      Request Body

      The Request Body for the HTTP POST method.

       

  7. Set the Advanced settings, if needed.
    • Select the Advanced tab.

    • Set the required settings:
      Property Description

      Proxy Server URI

      The HTTP Proxy setting that will allow the connector to reach the endpoint.

      Content Encoding

      Select the Content Encoding with the HTTP Header: None, GZip, Deflate, or GZip and Deflate.

      Timeout

      The length of time to wait for the server response (10 to 300). Default is 10.

      Decimal Separator

      Select either the dot (.) or comma (,) as the decimal separator.

      File Encoding

      Set the character encoding to use in text data.

      • UTF-8

      • UTF-16

      • US-ASCII

      • Windows-1252

       

    • You can opt to define the Show in Timezone and Source Timezone settings.

 

 

 

(c) 2013-2024 Altair Engineering Inc. All Rights Reserved.

Intellectual Property Rights Notice | Technical Support