GCLOUD_DATAPROC_JOBS_SUBMIT_SPARK-SQL(1)

NAME

gcloud dataproc jobs submit spark-sql - submit a Spark SQL job to a cluster

SYNOPSIS

gcloud dataproc jobs submit spark-sql --cluster CLUSTER [--async] [--bucket BUCKET] [--driver-log-levels [PACKAGE=LEVEL,...]] [--execute QUERY, -e QUERY] [--file FILE, -f FILE] [--jars [JAR,...]] [--params [PARAM=VALUE,...]] [--properties [PROPERTY=VALUE,...]] [GLOBAL-FLAG ...]

DESCRIPTION
Submit a Spark SQL job to a cluster.

REQUIRED FLAGS
--cluster CLUSTER
    The Dataproc cluster to submit the job to.

OPTIONAL FLAGS
--async
    Does not wait for the job to run.
--bucket BUCKET
    The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.
--driver-log-levels [PACKAGE=LEVEL,...]
    A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO
--execute QUERY, -e QUERY
    A Spark SQL query to execute as part of the job.
--file FILE, -f FILE
    HCFS URI of file containing Spark SQL script to execute as the job.
--jars [JAR,...]
    Comma separated list of jar files to be provided to the Hive and MR. May contain UDFs.
--params [PARAM=VALUE,...]
    A list of key value pairs to set variables in the Hive queries.
--properties [PROPERTY=VALUE,...]
    A list of key value pairs to configure Hive.

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a Spark SQL job with a local script, run:
    $ gcloud dataproc jobs submit spark-sql --cluster my_cluster \
        --file my_queries.ql
To submit a Spark SQL job with inline queries, run:
    $ gcloud dataproc jobs submit spark-sql --cluster my_cluster \
        -e \
        "CREATE EXTERNAL TABLE foo(bar int) LOCATION\
     'gs://my_bucket/'" -e "SELECT * FROM foo WHERE bar > 2"