Details
-
Improvement
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.5.5, 4.1.0
Description
I made a serialization mistake when develop a feature for our production enviroment.
However the Exception and Stack trace is Confusion.
We only get the root serialization cause, but the `Serialization stack` is not shown, it is not easy to find the real problem
```
13:38:31.443 WARN org.apache.spark.serializer.SerializationDebugger: Exception in serialization debugger
org.apache.spark.SparkRuntimeException: Cannot get SQLConf inside scheduler event loop thread.
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotGetSQLConfInSchedulerEventLoopThreadError(QueryExecutionErrors.scala:2002)
at org.apache.spark.sql.internal.SQLConf$.get(SQLConf.scala:225)
at org.apache.spark.sql.execution.ScalarSubquery.toString(subquery.scala:69)
at java.lang.String.valueOf(String.java:2994)
at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:203)
at scala.collection.immutable.Stream.addString(Stream.scala:701)
at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:377)
[info] - SPARK-35874: AQE Shuffle should wait for its subqueries to finish before materializing *** FAILED *** (1 second, 660 milliseconds)
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException: org.apache.spark.SimpleFutureAction
[info] at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2865)
[info] at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2800)
[info] at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2799)
[info] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[info] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
```