Class SimpleHttp
java.lang.Object
org.apache.drill.exec.store.http.util.SimpleHttp
- All Implemented Interfaces:
AutoCloseable
Performs the actual HTTP requests for the HTTP Storage Plugin. The core
method is the getInputStream() method which accepts a url and opens an
InputStream with that URL's contents.
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic class
Intercepts requests and adds authentication headers to the requeststatic class
This interceptor is used in pagination situations or elsewhere when APIs have burst throttling.static class
-
Field Summary
Modifier and TypeFieldDescriptionstatic final okhttp3.MediaType
static final okhttp3.MediaType
-
Constructor Summary
ConstructorDescriptionSimpleHttp
(okhttp3.HttpUrl url, File tempDir, HttpProxyConfig proxyConfig, CustomErrorContext errorContext, Paginator paginator, PersistentTokenTable tokenTable, HttpStoragePluginConfig pluginConfig, HttpApiConfig endpointConfig, String connection, Map<String, String> filters, int rateLimit) This constructor does not have an HttpSubScan and can be used outside the context of the HttpStoragePlugin.SimpleHttp
(HttpSubScan scanDefn, okhttp3.HttpUrl url, File tempDir, HttpProxyConfig proxyConfig, CustomErrorContext errorContext, Paginator paginator) -
Method Summary
Modifier and TypeMethodDescriptionstatic void
addProxyInfo
(okhttp3.OkHttpClient.Builder builder, HttpProxyConfig proxyConfig) Applies the proxy configuration to the OkHttp3 builder.static SimpleHttp
apiCall
(HttpStoragePlugin plugin, HttpApiConfig endpointConfig, DrillbitContext context, List<String> args) This function makes an API call and returns a string of the parsed results.static SimpleHttp.SimpleHttpBuilder
builder()
buildParameterList
(NullableVarCharHolder[] inputReaders) Accepts a list of input readers and converts that into an ArrayList of Stringsokhttp3.FormBody.Builder
buildPostBody
(Map<String, String> filters, String postBody) This function is used to push filters down to the post body rather than the URL query string.void
close()
static String
decodedURL
(okhttp3.HttpUrl url) Returns the URL-decoded URL.static String
getDefaultParameterValue
(okhttp3.HttpUrl url, String parameter) This function is used to extract the default parameter supplied in a URL.static HttpApiConfig
getEndpointConfig
(String endpoint, HttpStoragePluginConfig pluginConfig) This function is used to obtain the configuration information for a given API in the HTTP UDF.Returns an InputStream based on the URL and config in the scanSpec.static HttpProxyConfig
getProxySettings
(HttpStoragePluginConfig config, com.typesafe.config.Config drillConfig, okhttp3.HttpUrl url) static InputStream
static String
int
Gets the HTTP response code from the HTTP call.Gets the HTTP response code from the HTTP call.Gets the HTTP response code from the HTTP call.Gets the HTTP response code from the HTTP call.static HttpStoragePlugin
getStoragePlugin
(String pluginName, DrillbitContext context) This function will return aHttpStoragePlugin
for use in the HTTP UDFs.getURLParameters
(okhttp3.HttpUrl url) APIs are sometimes structured with parameters in the URL itself.static boolean
hasURLParameters
(okhttp3.HttpUrl url) Returns true if the url has url parameters, as indicated by the presence of {param} in a url.static okhttp3.ResponseBody
static String
mapPositionalParameters
(String rawUrl, List<String> params) static String
mapURLParameters
(okhttp3.HttpUrl url, Map<String, String> filters) Used for APIs which have parameters in the URL.protected okhttp3.OkHttpClient
Configures the OkHTTP3 server object with configuration info from the user.url()
static boolean
validateUrl
(String url) Validates a URL.
-
Field Details
-
JSON_MEDIA_TYPE
public static final okhttp3.MediaType JSON_MEDIA_TYPE -
XML_MEDIA_TYPE
public static final okhttp3.MediaType XML_MEDIA_TYPE
-
-
Constructor Details
-
SimpleHttp
public SimpleHttp(HttpSubScan scanDefn, okhttp3.HttpUrl url, File tempDir, HttpProxyConfig proxyConfig, CustomErrorContext errorContext, Paginator paginator) -
SimpleHttp
public SimpleHttp(okhttp3.HttpUrl url, File tempDir, HttpProxyConfig proxyConfig, CustomErrorContext errorContext, Paginator paginator, PersistentTokenTable tokenTable, HttpStoragePluginConfig pluginConfig, HttpApiConfig endpointConfig, String connection, Map<String, String> filters, int rateLimit) This constructor does not have an HttpSubScan and can be used outside the context of the HttpStoragePlugin.- Parameters:
url
- The URL for an HTTP requesttempDir
- Temp directory for cachingproxyConfig
- Proxy configuration for making API callserrorContext
- The error context for error messagespaginator
- ThePaginator
object for pagination.tokenTable
- The OAuth token tablepluginConfig
- HttpStoragePlugin configuration. The plugin obtains OAuth and timeout info from this config.endpointConfig
- Theconnection
- The name of the connectionfilters
- A Key/value set of filters and values
-
-
Method Details
-
builder
-
setupHttpClient
protected okhttp3.OkHttpClient setupHttpClient()Configures the OkHTTP3 server object with configuration info from the user.- Returns:
- OkHttpClient configured server
-
url
-
addProxyInfo
Applies the proxy configuration to the OkHttp3 builder. This ensures that proxy configurations will be consistent across HTTP REST connections.- Parameters:
builder
- The input OkHttp3 builderproxyConfig
- The proxy configuration
-
getInputStream
Returns an InputStream based on the URL and config in the scanSpec. If anything goes wrong the method throws a UserException.- Returns:
- An Inputstream of the data from the URL call. The caller is responsible for calling close() on the InputStream.
-
getResultsFromApiCall
-
getProxySettings
public static HttpProxyConfig getProxySettings(HttpStoragePluginConfig config, com.typesafe.config.Config drillConfig, okhttp3.HttpUrl url) -
getResponseCode
public int getResponseCode()Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.- Returns:
- int value of the HTTP response code
-
getResponseMessage
Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.- Returns:
- int of HTTP response code
-
getResponseProtocol
Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.- Returns:
- The HTTP response protocol
-
getResponseURL
Gets the HTTP response code from the HTTP call. Note that this value is only available after the getInputStream() method has been called.- Returns:
- The HTTP response URL
-
buildPostBody
This function is used to push filters down to the post body rather than the URL query string. It will also add the static parameters to the post body as well.- Parameters:
filters
- A HashMap of the filters and valuespostBody
- The post body of static parameters.- Returns:
- The post body builder with the filters and static parameters
-
decodedURL
Returns the URL-decoded URL. If the URL is invalid, return the original URL.- Returns:
- Returns the URL-decoded URL
-
hasURLParameters
public static boolean hasURLParameters(okhttp3.HttpUrl url) Returns true if the url has url parameters, as indicated by the presence of {param} in a url.- Returns:
- True if there are URL params, false if not
-
getURLParameters
APIs are sometimes structured with parameters in the URL itself. For instance, to request a list of an organization's repositories in github, the URL is: https://api.github.com/orgs/{org}/repos, where you can replace the org with the actual organization name.- Returns:
- A list of URL parameters enclosed by curly braces.
-
getDefaultParameterValue
This function is used to extract the default parameter supplied in a URL. For instance, if the supplied URL is http://someapi.com/path/{p1=foo}, the function will return foo. If there is not a matching parameter or no default value, the function will return null.- Parameters:
url
- The URL containing a default parameterparameter
- The parameter for which you need the value- Returns:
- The value for the supplied parameter
-
mapURLParameters
Used for APIs which have parameters in the URL. This function maps the filters pushed down from the query into the URL. For example the API: github.com/orgs/{org}/repos requires a user to specify an organization and replace {org} with an actual organization. The filter is passed down from the query. Note that if a URL contains URL parameters and one is not provided in the filters, Drill will throw a UserException.- Parameters:
url
- The HttpUrl containing URL Parametersfilters
- A CaseInsensitiveMap of filters- Returns:
- A string of the URL with the URL parameters replaced by filter values
-
mapPositionalParameters
-
validateUrl
Validates a URL.- Parameters:
url
- The input URL. Should be a string.- Returns:
- True of the URL is valid, false if not.
-
buildParameterList
Accepts a list of input readers and converts that into an ArrayList of Strings- Parameters:
inputReaders
- The array of FieldReaders- Returns:
- A List of Strings containing the values from the FieldReaders or null to indicate that at least one null is present in the arguments.
-
getEndpointConfig
public static HttpApiConfig getEndpointConfig(String endpoint, HttpStoragePluginConfig pluginConfig) This function is used to obtain the configuration information for a given API in the HTTP UDF. If aliasing is enabled, this function will resolve aliases for connections.- Parameters:
endpoint
- The name of the endpoint. Should be aString
pluginConfig
- TheHttpStoragePluginConfig
the configuration from the plugin- Returns:
- The
HttpApiConfig
corresponding with the endpoint.
-
getStoragePlugin
This function will return aHttpStoragePlugin
for use in the HTTP UDFs. If user or public aliases are used, the function will resolve those aliases.- Parameters:
context
- ADrillbitContext
from the current querypluginName
- AString
of the plugin name. Note that the function will resolve aliases.- Returns:
- A
HttpStoragePlugin
of the plugin.
-
apiCall
public static SimpleHttp apiCall(HttpStoragePlugin plugin, HttpApiConfig endpointConfig, DrillbitContext context, List<String> args) This function makes an API call and returns a string of the parsed results. It is used in the http_get() UDF and retrieves all the configuration parameters contained in the storage plugin and endpoint configuration. The exception is pagination. This does not support pagination.- Parameters:
plugin
- The HTTP storage plugin upon which the API call is based.endpointConfig
- The configuration of the API endpoint upon which the API call is based.context
-DrillbitContext
The context from the current queryargs
- An optional list of parameter arguments which will be included in the URL- Returns:
- A String of the results.
-
getRequestAndStringResponse
-
getRequestAndStreamResponse
- Parameters:
url
-- Returns:
- an input stream which the caller is responsible for closing.
-
makeSimpleGetRequest
- Parameters:
url
-- Returns:
- response body which the caller is responsible for closing.
- Throws:
IOException
-
close
public void close()- Specified by:
close
in interfaceAutoCloseable
-