GCLOUD_DATAPROC_JOBS_SUBMIT_SPARK(1)

NAME

gcloud dataproc jobs submit spark - submit a Java or Scala Spark job to a cluster

SYNOPSIS

gcloud dataproc jobs submit spark --cluster CLUSTER [--archives [ARCHIVE,...]] [--async] [--bucket BUCKET] [--class MAIN_CLASS] [--driver-log-levels [PACKAGE=LEVEL,...]] [--files [FILE,...]] [--jar MAIN_JAR] [--jars [JAR,...]] [--properties [PROPERTY=VALUE,...]] [GLOBAL-FLAG ...] [JOB_ARGS ...]

DESCRIPTION
Submit a Java or Scala Spark job to a cluster.

POSITIONAL ARGUMENTS
[JOB_ARGS ...]
    The arguments to pass to the driver.

REQUIRED FLAGS
--cluster CLUSTER
    The Dataproc cluster to submit the job to.

OPTIONAL FLAGS
--archives [ARCHIVE,...]
    Comma separated list of archives to be provided to the job. must be one of the following file formats: .zip, .tar, .tar.gz, or .tgz.
--async
    Does not wait for the job to run.
--bucket BUCKET
    The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.
--class MAIN_CLASS
    The class containing the main method of the driver. Must be in a provided jar or jar that is already on the classpath
--driver-log-levels [PACKAGE=LEVEL,...]
    A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO
--files [FILE,...]
    Comma separated list of files to be provided to the job.
--jar MAIN_JAR
    The HCFS URI of jar file containing the driver jar.
--jars [JAR,...]
    Comma separated list of jar files to be provided to the Executor and driver classpaths.
--properties [PROPERTY=VALUE,...]
    A list of key value pairs to configure Spark.

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a Spark job that runs the main class of a jar, run:
    $ gcloud dataproc jobs submit spark --cluster my_cluster \
        --jar my_jar.jar arg1 arg2
To submit a Spark job that runs a specific class of a jar, run:
    $ gcloud dataproc jobs submit spark --cluster my_cluster \
        --class org.my.main.Class --jars my_jar1.jar,my_jar2.jar arg1 \
        arg2
To submit a Spark job that runs a jar that is already on the cluster, run:
    $ gcloud dataproc jobs submit spark --cluster my_cluster \
        --class org.apache.spark.examples.SparkPi \
        --jars file:///usr/lib/spark/lib/spark-examples.jar 1000