Create article for "Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory"

[JoshRosen]

We should create an article for "Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory", since that seems to be a common issue.

[JoshRosen]

One cause of this error is network connectivity issues between the master and driver, so maybe we should also add a note to https://github.com/databricks/spark-knowledgebase/blob/master/troubleshooting/connectivity_issues.md

[jkleckner]

Just got bitten by an environment variable set by another person for EXECUTOR_MEMORY to a large value. I was using spark-submit with --executor-memory 3G but the env var took precedence.

Do you think the explicit argument should take precedence?

[JoshRosen]

Hi @jkleckner,

We've deprecated most environment variables in favor of the newer configuration mechanisms, so system properties and SparkSubmit / SparkConf settings are intended to take precedence over environment variables. Which version of Spark are you using? Do you have a simple reproduction for this issue? If so, do you mind filing a JIRA ticket and linking it here? https://issues.apache.org/jira/browse/SPARK

[jkleckner]

Sorry, I found that someone else had explicitly programmed environment vars to override config values....

[JoshRosen]

Sorry, I found that someone else had explicitly programmed environment vars to override config values....

You mean in your own application / user-code, you have code that reads from the environment variable and uses it to set the corresponding SparkConf setting, or something like that?

[jkleckner]

Yes, in our programming someone intentionally made it work that way. Obviously I will be changing that. So to quote Emily Litella, never mind...

[deepujain]

I am facing this error from 4 days and no one seems to be able to figure out a fix for it. Could you please suggest something, I reduced my input data size from 1TB to 1GB to 10 simple records. I still get the same error, making me believe that this error is occurring at request time and not execution time.

[jkleckner]

@deepujain, If you are using YARN, bring up the page NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUNNING Applications by browsing on the master node port 9026 (what AWS EMR uses but can vary) as in http://127.0.0.1:9026/cluster . Examine the nodes and the queues to see if there is an old application zombie around. If so kill it with:

yarn application -list
yarn application -kill <jobid>

Some situations can lead to old jobs hanging around and using up resources.

[rodriguezsergio]

Thanks so much for opening this issue!

I was having issues setting up a spark on mesos dev environment for the last few days and had made zero headway until I set spark.mesos.coarse to true and then lowered the spark.executor.memory below the default 512 value (running on m1.smalls on EC2 here). Couldn't even finish running /bin/run-example SparkPi 10 and was ready to give up until I saw this.

[hokiegeek2]

This is excellent. I actually had a zombie Mesos Spark app, killed that, and now I am back in business--well done, guys!

[jkleckner]

@hokiegeek2 glad you found it.

Recently, I found that spark jobs could hang because exceptions didn't pass up to an exit and added this snippet. Now the testing process doesn't result in strewn bodies in the cluster...

    try {
      Foo.runAnalysis(sc, debug = true)
    } catch {
      case e: Exception => Seq[String]()
      println(e)
      sc.stop
      sys.exit(1)
    }

[gdubicki]

+1

[zheolong]

+1

[xuedihualu]

@rodriguezsergio
now,I have the same issues, spark on mesos,when i Create task

TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/01/22 15:31:25 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

My driver is only on the master, the actuator is also

[vinesinha]

I have a similar problem.

When I run a code on the "spark-shell", it works just fine.

However, a similar code written in eclipse and then deployed to spark master fails (no resources are assigned).

I've posted a stackoverflow post about this.

Thanks