org.apache.drill.yarn.core

Class DfsFacade

java.lang.Object

org.apache.drill.yarn.core.DfsFacade

public class DfsFacade
extends Object

Facade to the distributed file system (DFS) system that implements Drill-on-YARN related operations. Some operations are used by both the client and AM applications.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DfsFacade.DfsFacadeException
static class	DfsFacade.Localizer

Constructor Summary

Constructors

Constructor and Description
DfsFacade(com.typesafe.config.Config config)

Method Summary

Modifier and Type	Method and Description
void	connect()
boolean	exists(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.Path	getUploadPath(File localArchiveFile)
org.apache.hadoop.fs.Path	getUploadPath(String baseName)
boolean	isLocalized()
org.apache.hadoop.yarn.api.records.LocalResource	makeResource(org.apache.hadoop.fs.Path dfsPath, org.apache.hadoop.fs.FileStatus dfsFileStatus, org.apache.hadoop.yarn.api.records.LocalResourceType type, org.apache.hadoop.yarn.api.records.LocalResourceVisibility visibility) Create a local resource definition for YARN.
void	removeDrillFile(String fileName)
void	uploadArchive(File localArchiveFile, org.apache.hadoop.fs.Path destPath, String label)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DfsFacade

public DfsFacade(com.typesafe.config.Config config)

Method Detail

isLocalized

public boolean isLocalized()

connect

public void connect()
             throws DfsFacade.DfsFacadeException

Throws:

DfsFacade.DfsFacadeException

exists

public boolean exists(org.apache.hadoop.fs.Path path)
               throws IOException

Throws:

IOException

getUploadPath

public org.apache.hadoop.fs.Path getUploadPath(File localArchiveFile)

getUploadPath

public org.apache.hadoop.fs.Path getUploadPath(String baseName)

uploadArchive

public void uploadArchive(File localArchiveFile,
                          org.apache.hadoop.fs.Path destPath,
                          String label)
                   throws DfsFacade.DfsFacadeException

Throws:

DfsFacade.DfsFacadeException

makeResource

public org.apache.hadoop.yarn.api.records.LocalResource makeResource(org.apache.hadoop.fs.Path dfsPath,
                                                                     org.apache.hadoop.fs.FileStatus dfsFileStatus,
                                                                     org.apache.hadoop.yarn.api.records.LocalResourceType type,
                                                                     org.apache.hadoop.yarn.api.records.LocalResourceVisibility visibility)
                                                              throws DfsFacade.DfsFacadeException

Create a local resource definition for YARN. A local resource is one that must be localized onto the remote node prior to running a command on that node.

YARN uses the size and timestamp are used to check if the file has changed on HDFS to check if YARN can use an existing copy, if any.

Resources are made public.

Parameters:

conf - Configuration created from the Hadoop config files, in this case, identifies the target file system.

resourcePath - the path (relative or absolute) to the file on the configured file system (usually HDFS).

Returns:

a YARN local resource records that contains information about path, size, type, resource and so on that YARN requires.

Throws:

IOException - if the resource does not exist on the configured file system

DfsFacade.DfsFacadeException

removeDrillFile

public void removeDrillFile(String fileName)
                     throws DfsFacade.DfsFacadeException

Throws:

DfsFacade.DfsFacadeException