SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 16/02/11 20:27:09 INFO ADAMMain: ADAM invoked with args: "adam2vcf" "hdfs:///user/amchen/NA12878.mapped.ILLUMINA.bwa.CEU.high_coverage_pcr_free.20130906.bam.unified.raw.SNP.gatk.vcf.adam.filtered" "hdfs:///user/amchen/NA12878.gmm.vcf" "-sort_on_save" 16/02/11 20:27:09 INFO SparkContext: Running Spark version 1.5.0-cdh5.5.1 16/02/11 20:27:10 INFO SecurityManager: Changing view acls to: amchen 16/02/11 20:27:10 INFO SecurityManager: Changing modify acls to: amchen 16/02/11 20:27:10 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(amchen); users with modify permissions: Set(amchen) 16/02/11 20:27:10 INFO Slf4jLogger: Slf4jLogger started 16/02/11 20:27:10 INFO Remoting: Starting remoting 16/02/11 20:27:10 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.247.253:35234] 16/02/11 20:27:10 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@192.168.247.253:35234] 16/02/11 20:27:10 INFO Utils: Successfully started service 'sparkDriver' on port 35234. 16/02/11 20:27:10 INFO SparkEnv: Registering MapOutputTracker 16/02/11 20:27:10 INFO SparkEnv: Registering BlockManagerMaster 16/02/11 20:27:10 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-56191eb4-e8f1-45f3-a691-4ee41c353fc1 16/02/11 20:27:10 INFO MemoryStore: MemoryStore started with capacity 530.0 MB 16/02/11 20:27:11 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c279357b-23d1-47f5-aaf2-70fe449809a2/httpd-9576f7ca-7e2c-4892-af48-067ce0d08aa1 16/02/11 20:27:11 INFO HttpServer: Starting HTTP Server 16/02/11 20:27:11 INFO Server: jetty-8.y.z-SNAPSHOT 16/02/11 20:27:11 INFO AbstractConnector: Started SocketConnector@0.0.0.0:38475 16/02/11 20:27:11 INFO Utils: Successfully started service 'HTTP file server' on port 38475. 16/02/11 20:27:11 INFO SparkEnv: Registering OutputCommitCoordinator 16/02/11 20:27:11 INFO Server: jetty-8.y.z-SNAPSHOT 16/02/11 20:27:11 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.spark-project.jetty.server.Server.doStart(Server.java:293) at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:236) at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:246) at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:246) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1913) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1904) at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:246) at org.apache.spark.ui.WebUI.bind(WebUI.scala:136) at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474) at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474) at scala.Option.foreach(Option.scala:236) at org.apache.spark.SparkContext.(SparkContext.scala:474) at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:51) at org.bdgenomics.adam.cli.ADAM2Vcf.run(ADAM2Vcf.scala:58) at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:136) at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:76) at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/02/11 20:27:11 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@1ec7d8b3: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187) at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316) at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265) at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.spark-project.jetty.server.Server.doStart(Server.java:293) at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:236) at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:246) at org.apache.spark.ui.JettyUtils$$anonfun$3.apply(JettyUtils.scala:246) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1913) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1904) at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:246) at org.apache.spark.ui.WebUI.bind(WebUI.scala:136) at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474) at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:474) at scala.Option.foreach(Option.scala:236) at org.apache.spark.SparkContext.(SparkContext.scala:474) at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:51) at org.bdgenomics.adam.cli.ADAM2Vcf.run(ADAM2Vcf.scala:58) at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:136) at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:76) at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 16/02/11 20:27:11 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 16/02/11 20:27:11 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 16/02/11 20:27:11 INFO Server: jetty-8.y.z-SNAPSHOT 16/02/11 20:27:11 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041 16/02/11 20:27:11 INFO Utils: Successfully started service 'SparkUI' on port 4041. 16/02/11 20:27:11 INFO SparkUI: Started SparkUI at http://192.168.247.253:4041 16/02/11 20:27:11 INFO SparkContext: Added JAR file:/home/eecs/amchen/adam/adam-cli/target/adam-cli_2.10-0.18.3-SNAPSHOT.jar at http://192.168.247.253:38475/jars/adam-cli_2.10-0.18.3-SNAPSHOT.jar with timestamp 1455251231359 16/02/11 20:27:11 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 16/02/11 20:27:11 INFO RMProxy: Connecting to ResourceManager at amp-bdg-master.amp/192.168.247.253:8032 16/02/11 20:27:11 INFO Client: Requesting a new application from cluster with 64 NodeManagers 16/02/11 20:27:11 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (245760 MB per container) 16/02/11 20:27:11 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 16/02/11 20:27:11 INFO Client: Setting up container launch context for our AM 16/02/11 20:27:11 INFO Client: Setting up the launch environment for our AM container 16/02/11 20:27:11 INFO Client: Preparing resources for our AM container 16/02/11 20:27:12 INFO Client: Uploading resource file:/tmp/spark-c279357b-23d1-47f5-aaf2-70fe449809a2/__spark_conf__6718016987273452392.zip -> hdfs://amp-bdg-master.amp:8020/user/amchen/.sparkStaging/application_1455039715601_0066/__spark_conf__6718016987273452392.zip 16/02/11 20:27:12 INFO SecurityManager: Changing view acls to: amchen 16/02/11 20:27:12 INFO SecurityManager: Changing modify acls to: amchen 16/02/11 20:27:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(amchen); users with modify permissions: Set(amchen) 16/02/11 20:27:12 INFO Client: Submitting application 66 to ResourceManager 16/02/11 20:27:12 INFO YarnClientImpl: Submitted application application_1455039715601_0066 16/02/11 20:27:13 INFO Client: Application report for application_1455039715601_0066 (state: ACCEPTED) 16/02/11 20:27:13 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.amchen start time: 1455251232565 final status: UNDEFINED tracking URL: http://amp-bdg-master.amp:8088/proxy/application_1455039715601_0066/ user: amchen 16/02/11 20:27:14 INFO Client: Application report for application_1455039715601_0066 (state: ACCEPTED) 16/02/11 20:27:15 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as AkkaRpcEndpointRef(Actor[akka.tcp://sparkYarnAM@192.168.247.45:43008/user/YarnAM#1544061108]) 16/02/11 20:27:15 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> amp-bdg-master.amp, PROXY_URI_BASES -> http://amp-bdg-master.amp:8088/proxy/application_1455039715601_0066), /proxy/application_1455039715601_0066 16/02/11 20:27:15 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 16/02/11 20:27:15 INFO Client: Application report for application_1455039715601_0066 (state: RUNNING) 16/02/11 20:27:15 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.247.45 ApplicationMaster RPC port: 0 queue: root.amchen start time: 1455251232565 final status: UNDEFINED tracking URL: http://amp-bdg-master.amp:8088/proxy/application_1455039715601_0066/ user: amchen 16/02/11 20:27:15 INFO YarnClientSchedulerBackend: Application application_1455039715601_0066 has started running. 16/02/11 20:27:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53930. 16/02/11 20:27:15 INFO NettyBlockTransferService: Server created on 53930 16/02/11 20:27:15 INFO BlockManager: external shuffle service port = 7337 16/02/11 20:27:15 INFO BlockManagerMaster: Trying to register BlockManager 16/02/11 20:27:15 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.247.253:53930 with 530.0 MB RAM, BlockManagerId(driver, 192.168.247.253, 53930) 16/02/11 20:27:15 INFO BlockManagerMaster: Registered BlockManager 16/02/11 20:27:15 INFO EventLoggingListener: Logging events to hdfs://amp-bdg-master.amp:8020/user/spark/applicationHistory/application_1455039715601_0066 16/02/11 20:27:15 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 16/02/11 20:27:15 INFO ADAM2Vcf: test 16/02/11 20:27:15 INFO ADAMContext: Reading the ADAM file at hdfs:///user/amchen/NA12878.mapped.ILLUMINA.bwa.CEU.high_coverage_pcr_free.20130906.bam.unified.raw.SNP.gatk.vcf.adam.filtered to create RDD 16/02/11 20:27:16 INFO MemoryStore: ensureFreeSpace(269288) called with curMem=0, maxMem=555755765 16/02/11 20:27:16 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 263.0 KB, free 529.8 MB) 16/02/11 20:27:16 INFO MemoryStore: ensureFreeSpace(23031) called with curMem=269288, maxMem=555755765 16/02/11 20:27:16 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 22.5 KB, free 529.7 MB) 16/02/11 20:27:16 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.247.253:53930 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:16 INFO SparkContext: Created broadcast 0 from newAPIHadoopFile at ADAMContext.scala:166 16/02/11 20:27:16 INFO FileInputFormat: Total input paths to process : 32 16/02/11 20:27:16 INFO VariantContextRDDFunctions: Writing VCF file to hdfs:///user/amchen/NA12878.gmm.vcf 16/02/11 20:27:16 INFO SparkContext: Starting job: collect at VariationRDDFunctions.scala:66 16/02/11 20:27:16 INFO DAGScheduler: Registering RDD 2 (keyBy at VariationRDDFunctions.scala:153) 16/02/11 20:27:16 INFO DAGScheduler: Registering RDD 6 (distinct at VariationRDDFunctions.scala:65) 16/02/11 20:27:16 INFO DAGScheduler: Got job 0 (collect at VariationRDDFunctions.scala:66) with 32 output partitions 16/02/11 20:27:16 INFO DAGScheduler: Final stage: ResultStage 2(collect at VariationRDDFunctions.scala:66) 16/02/11 20:27:16 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 16/02/11 20:27:16 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1) 16/02/11 20:27:17 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[2] at keyBy at VariationRDDFunctions.scala:153), which has no missing parents 16/02/11 20:27:17 INFO MemoryStore: ensureFreeSpace(4256) called with curMem=292319, maxMem=555755765 16/02/11 20:27:17 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.2 KB, free 529.7 MB) 16/02/11 20:27:17 INFO MemoryStore: ensureFreeSpace(2302) called with curMem=296575, maxMem=555755765 16/02/11 20:27:17 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 529.7 MB) 16/02/11 20:27:17 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.247.253:53930 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:17 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:861 16/02/11 20:27:17 INFO DAGScheduler: Submitting 32 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[2] at keyBy at VariationRDDFunctions.scala:153) 16/02/11 20:27:17 INFO YarnScheduler: Adding task set 0.0 with 32 tasks 16/02/11 20:27:18 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1) 16/02/11 20:27:19 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 3) 16/02/11 20:27:20 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 7) 16/02/11 20:27:21 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 15) 16/02/11 20:27:21 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-24.amp:60072]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-24.amp:60072] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-24.amp:60072 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:22 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-35.amp:52236]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-35.amp:52236] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-35.amp:52236 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:22 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 31) 16/02/11 20:27:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-24.amp:41672/user/Executor#2108790673]) with ID 1 16/02/11 20:27:22 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-56.amp:35870]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-56.amp:35870] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-56.amp:35870 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:22 INFO ExecutorAllocationManager: New executor 1 has registered (new total is 1) 16/02/11 20:27:22 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, amp-bdg-24.amp, partition 0,NODE_LOCAL, 2371 bytes) 16/02/11 20:27:22 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-03.amp:50557]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-03.amp:50557] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-03.amp:50557 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:22 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-24.amp:36201 with 530.0 MB RAM, BlockManagerId(1, amp-bdg-24.amp, 36201) 16/02/11 20:27:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-56.amp:41787/user/Executor#-614228405]) with ID 3 16/02/11 20:27:22 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 1, amp-bdg-56.amp, partition 11,NODE_LOCAL, 2375 bytes) 16/02/11 20:27:22 INFO ExecutorAllocationManager: New executor 3 has registered (new total is 2) 16/02/11 20:27:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-35.amp:60229/user/Executor#2077998258]) with ID 4 16/02/11 20:27:22 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, amp-bdg-35.amp, partition 2,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:22 INFO ExecutorAllocationManager: New executor 4 has registered (new total is 3) 16/02/11 20:27:22 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-03.amp:46605/user/Executor#-1352869846]) with ID 2 16/02/11 20:27:22 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 3, amp-bdg-03.amp, partition 17,NODE_LOCAL, 2374 bytes) 16/02/11 20:27:22 INFO ExecutorAllocationManager: New executor 2 has registered (new total is 4) 16/02/11 20:27:22 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-56.amp:50396 with 530.0 MB RAM, BlockManagerId(3, amp-bdg-56.amp, 50396) 16/02/11 20:27:22 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-35.amp:44956 with 530.0 MB RAM, BlockManagerId(4, amp-bdg-35.amp, 44956) 16/02/11 20:27:22 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-03.amp:36085 with 530.0 MB RAM, BlockManagerId(2, amp-bdg-03.amp, 36085) 16/02/11 20:27:22 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-15.amp:48300]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-15.amp:48300] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-15.amp:48300 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:22 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-46.amp:45491]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-46.amp:45491] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-46.amp:45491 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:22 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-36.amp:33589]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-36.amp:33589] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-36.amp:33589 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-20.amp:48314]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-20.amp:48314] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-20.amp:48314 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 32) 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-49.amp:36339]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-49.amp:36339] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-49.amp:36339 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-49.amp:48677]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-49.amp:48677] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-49.amp:48677 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-15.amp:35924/user/Executor#1887716808]) with ID 6 16/02/11 20:27:23 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 4, amp-bdg-15.amp, partition 8,NODE_LOCAL, 2373 bytes) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 6 has registered (new total is 5) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-46.amp:59053/user/Executor#1481658015]) with ID 5 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 5 has registered (new total is 6) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-36.amp:48076/user/Executor#-2028908984]) with ID 7 16/02/11 20:27:23 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 5, amp-bdg-36.amp, partition 7,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 7 has registered (new total is 7) 16/02/11 20:27:23 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-24.amp:36201 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-60.amp:38259]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-60.amp:38259] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-60.amp:38259 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-37.amp:40052]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-37.amp:40052] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-37.amp:40052 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-15.amp:46865 with 530.0 MB RAM, BlockManagerId(6, amp-bdg-15.amp, 46865) 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-46.amp:38842 with 530.0 MB RAM, BlockManagerId(5, amp-bdg-46.amp, 38842) 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-22.amp:36642]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-22.amp:36642] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-22.amp:36642 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-04.amp:41844]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-04.amp:41844] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-04.amp:41844 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-36.amp:49131 with 530.0 MB RAM, BlockManagerId(7, amp-bdg-36.amp, 49131) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-20.amp:57148/user/Executor#1759255633]) with ID 9 16/02/11 20:27:23 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 6, amp-bdg-20.amp, partition 3,NODE_LOCAL, 2373 bytes) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 9 has registered (new total is 8) 16/02/11 20:27:23 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-25.amp:53773]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-25.amp:53773] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-25.amp:53773 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-20.amp:58218 with 530.0 MB RAM, BlockManagerId(9, amp-bdg-20.amp, 58218) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-49.amp:51930/user/Executor#-10077425]) with ID 8 16/02/11 20:27:23 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 7, amp-bdg-49.amp, partition 6,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 8 has registered (new total is 9) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-49.amp:60339/user/Executor#1323909427]) with ID 10 16/02/11 20:27:23 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 8, amp-bdg-49.amp, partition 16,NODE_LOCAL, 2370 bytes) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 10 has registered (new total is 10) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-60.amp:55591/user/Executor#-1197637818]) with ID 13 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 13 has registered (new total is 11) 16/02/11 20:27:23 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 9, amp-bdg-60.amp, partition 21,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-04.amp:41693/user/Executor#-7342410]) with ID 11 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-49.amp:40827 with 530.0 MB RAM, BlockManagerId(10, amp-bdg-49.amp, 40827) 16/02/11 20:27:23 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 10, amp-bdg-04.amp, partition 1,NODE_LOCAL, 2373 bytes) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 11 has registered (new total is 12) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-37.amp:43269/user/Executor#-775764186]) with ID 15 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 15 has registered (new total is 13) 16/02/11 20:27:23 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 11, amp-bdg-37.amp, partition 18,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-22.amp:38208/user/Executor#1409014097]) with ID 14 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-49.amp:36486 with 530.0 MB RAM, BlockManagerId(8, amp-bdg-49.amp, 36486) 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 14 has registered (new total is 14) 16/02/11 20:27:23 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 12, amp-bdg-22.amp, partition 4,NODE_LOCAL, 2373 bytes) 16/02/11 20:27:23 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-24.amp:36201 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-60.amp:38207 with 530.0 MB RAM, BlockManagerId(13, amp-bdg-60.amp, 38207) 16/02/11 20:27:23 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-25.amp:44820/user/Executor#-763496790]) with ID 12 16/02/11 20:27:23 INFO ExecutorAllocationManager: New executor 12 has registered (new total is 15) 16/02/11 20:27:23 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 13, amp-bdg-25.amp, partition 5,NODE_LOCAL, 2373 bytes) 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-04.amp:36661 with 530.0 MB RAM, BlockManagerId(11, amp-bdg-04.amp, 36661) 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-37.amp:42720 with 530.0 MB RAM, BlockManagerId(15, amp-bdg-37.amp, 42720) 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-22.amp:34751 with 530.0 MB RAM, BlockManagerId(14, amp-bdg-22.amp, 34751) 16/02/11 20:27:23 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-25.amp:57820 with 530.0 MB RAM, BlockManagerId(12, amp-bdg-25.amp, 57820) 16/02/11 20:27:23 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-35.amp:44956 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-54.amp:33356]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-54.amp:33356] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-54.amp:33356 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-47.amp:40652]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-47.amp:40652] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-47.amp:40652 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-61.amp:33381]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-61.amp:33381] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-61.amp:33381 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-05.amp:48766]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-05.amp:48766] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-05.amp:48766 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-44.amp:50122]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-44.amp:50122] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-44.amp:50122 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-54.amp:34211/user/Executor#800136917]) with ID 16 16/02/11 20:27:24 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 14, amp-bdg-54.amp, partition 10,NODE_LOCAL, 2371 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 16 has registered (new total is 16) 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-10.amp:46549]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-10.amp:46549] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-10.amp:46549 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-35.amp:44956 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-34.amp:58359]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-34.amp:58359] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-34.amp:58359 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-41.amp:48536]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-41.amp:48536] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-41.amp:48536 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-57.amp:60541]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-57.amp:60541] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-57.amp:60541 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-47.amp:56604/user/Executor#-151196730]) with ID 18 16/02/11 20:27:24 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 15, amp-bdg-47.amp, partition 23,NODE_LOCAL, 2375 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 18 has registered (new total is 17) 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-28.amp:46070]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-28.amp:46070] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-28.amp:46070 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-02.amp:54162]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-02.amp:54162] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-02.amp:54162 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-54.amp:34279 with 530.0 MB RAM, BlockManagerId(16, amp-bdg-54.amp, 34279) 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-64.amp:44767]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-64.amp:44767] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-64.amp:44767 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-18.amp:34364]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-18.amp:34364] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-18.amp:34364 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-62.amp:50649]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-62.amp:50649] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-62.amp:50649 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-51.amp:42389]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-51.amp:42389] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-51.amp:42389 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-27.amp:52745]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-27.amp:52745] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-27.amp:52745 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-47.amp:57360 with 530.0 MB RAM, BlockManagerId(18, amp-bdg-47.amp, 57360) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-61.amp:52505/user/Executor#-1585733348]) with ID 19 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 19 has registered (new total is 18) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-44.amp:34888/user/Executor#889262664]) with ID 20 16/02/11 20:27:24 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 16, amp-bdg-44.amp, partition 15,NODE_LOCAL, 2371 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 20 has registered (new total is 19) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-05.amp:53075/user/Executor#-1281353087]) with ID 17 16/02/11 20:27:24 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 17, amp-bdg-05.amp, partition 30,NODE_LOCAL, 2371 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 17 has registered (new total is 20) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-61.amp:41089 with 530.0 MB RAM, BlockManagerId(19, amp-bdg-61.amp, 41089) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-34.amp:51277/user/Executor#-1516247805]) with ID 26 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 26 has registered (new total is 21) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-10.amp:47549/user/Executor#-467189530]) with ID 28 16/02/11 20:27:24 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 18, amp-bdg-10.amp, partition 13,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 28 has registered (new total is 22) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-57.amp:53625/user/Executor#-1097665193]) with ID 21 16/02/11 20:27:24 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 19, amp-bdg-57.amp, partition 20,NODE_LOCAL, 2373 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 21 has registered (new total is 23) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-41.amp:58413/user/Executor#1381118110]) with ID 27 16/02/11 20:27:24 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 20, amp-bdg-41.amp, partition 31,NODE_LOCAL, 2375 bytes) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-62.amp:50643/user/Executor#1505534507]) with ID 30 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 27 has registered (new total is 24) 16/02/11 20:27:24 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 21, amp-bdg-62.amp, partition 12,NODE_LOCAL, 2374 bytes) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-28.amp:33300/user/Executor#1230549257]) with ID 29 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 30 has registered (new total is 25) 16/02/11 20:27:24 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 22, amp-bdg-28.amp, partition 19,NODE_LOCAL, 2374 bytes) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-02.amp:51182/user/Executor#587562765]) with ID 23 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 29 has registered (new total is 26) 16/02/11 20:27:24 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 23, amp-bdg-02.amp, partition 25,NODE_LOCAL, 2375 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 23 has registered (new total is 27) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-44.amp:40528 with 530.0 MB RAM, BlockManagerId(20, amp-bdg-44.amp, 40528) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-34.amp:34029 with 530.0 MB RAM, BlockManagerId(26, amp-bdg-34.amp, 34029) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-05.amp:54334 with 530.0 MB RAM, BlockManagerId(17, amp-bdg-05.amp, 54334) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-10.amp:47502 with 530.0 MB RAM, BlockManagerId(28, amp-bdg-10.amp, 47502) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-57.amp:56334 with 530.0 MB RAM, BlockManagerId(21, amp-bdg-57.amp, 56334) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-27.amp:46972/user/Executor#1262360764]) with ID 25 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 25 has registered (new total is 28) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-62.amp:46717 with 530.0 MB RAM, BlockManagerId(30, amp-bdg-62.amp, 46717) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-28.amp:48326 with 530.0 MB RAM, BlockManagerId(29, amp-bdg-28.amp, 48326) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-41.amp:57788 with 530.0 MB RAM, BlockManagerId(27, amp-bdg-41.amp, 57788) 16/02/11 20:27:24 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-02.amp:49749 with 530.0 MB RAM, BlockManagerId(23, amp-bdg-02.amp, 49749) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-64.amp:52828/user/Executor#156045960]) with ID 22 16/02/11 20:27:24 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 24, amp-bdg-64.amp, partition 29,NODE_LOCAL, 2374 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 22 has registered (new total is 29) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-18.amp:37719/user/Executor#-1371152136]) with ID 31 16/02/11 20:27:24 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 25, amp-bdg-18.amp, partition 24,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 31 has registered (new total is 30) 16/02/11 20:27:24 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-51.amp:34878/user/Executor#1617659177]) with ID 24 16/02/11 20:27:24 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 26, amp-bdg-51.amp, partition 22,NODE_LOCAL, 2374 bytes) 16/02/11 20:27:24 INFO ExecutorAllocationManager: New executor 24 has registered (new total is 31) 16/02/11 20:27:25 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-27.amp:47425 with 530.0 MB RAM, BlockManagerId(25, amp-bdg-27.amp, 47425) 16/02/11 20:27:25 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-18.amp:46764 with 530.0 MB RAM, BlockManagerId(31, amp-bdg-18.amp, 46764) 16/02/11 20:27:25 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-64.amp:32917 with 530.0 MB RAM, BlockManagerId(22, amp-bdg-64.amp, 32917) 16/02/11 20:27:25 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-51.amp:46478 with 530.0 MB RAM, BlockManagerId(24, amp-bdg-51.amp, 46478) 16/02/11 20:27:25 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://driverPropsFetcher@amp-bdg-29.amp:47580]: Error [Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-29.amp:47580] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@amp-bdg-29.amp:47580 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:27:25 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-56.amp:50396 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:25 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-15.amp:46865 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:25 INFO YarnClientSchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@amp-bdg-29.amp:47528/user/Executor#-888554094]) with ID 32 16/02/11 20:27:25 INFO ExecutorAllocationManager: New executor 32 has registered (new total is 32) 16/02/11 20:27:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-56.amp:50396 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:26 INFO BlockManagerMasterEndpoint: Registering block manager amp-bdg-29.amp:34585 with 530.0 MB RAM, BlockManagerId(32, amp-bdg-29.amp, 34585) 16/02/11 20:27:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-15.amp:46865 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:27 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 27, amp-bdg-24.amp, partition 14,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:27 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 5720 ms on amp-bdg-24.amp (1/32) 16/02/11 20:27:28 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-03.amp:36085 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:28 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-25.amp:57820 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:28 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-03.amp:36085 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:28 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 28, amp-bdg-35.amp, partition 26,NODE_LOCAL, 2372 bytes) 16/02/11 20:27:28 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 6087 ms on amp-bdg-35.amp (2/32) 16/02/11 20:27:28 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-25.amp:57820 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:29 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-54.amp:34279 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:29 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-04.amp:36661 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:29 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-36.amp:49131 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:29 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-54.amp:34279 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-04.amp:36661 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-36.amp:49131 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-20.amp:58218 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 29, amp-bdg-56.amp, partition 28,NODE_LOCAL, 2370 bytes) 16/02/11 20:27:30 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 1) in 8081 ms on amp-bdg-56.amp (3/32) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-18.amp:46764 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 4) in 7342 ms on amp-bdg-15.amp (4/32) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-64.amp:32917 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-60.amp:38207 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-47.amp:57360 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-20.amp:58218 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:30 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 27) in 3082 ms on amp-bdg-24.amp (5/32) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-18.amp:46764 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-64.amp:32917 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-60.amp:38207 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-47.amp:57360 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-28.amp:48326 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-49.amp:36486 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 28) in 3065 ms on amp-bdg-35.amp (6/32) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-51.amp:46478 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-28.amp:48326 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-44.amp:40528 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:31 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-41.amp:57788 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-49.amp:40827 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-49.amp:36486 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-51.amp:46478 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-10.amp:47502 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-05.amp:54334 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-62.amp:46717 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-44.amp:40528 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-37.amp:42720 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-41.amp:57788 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-22.amp:34751 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-49.amp:40827 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-02.amp:49749 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-57.amp:56334 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-10.amp:47502 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-05.amp:54334 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-62.amp:46717 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 3) in 10126 ms on amp-bdg-03.amp (7/32) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-37.amp:42720 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-22.amp:34751 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 13) in 9124 ms on amp-bdg-25.amp (8/32) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-02.amp:49749 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:32 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-57.amp:56334 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:33 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 30, amp-bdg-29.amp, partition 27,RACK_LOCAL, 2374 bytes) 16/02/11 20:27:33 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 31, amp-bdg-46.amp, partition 9,RACK_LOCAL, 2374 bytes) 16/02/11 20:27:33 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 29) in 3239 ms on amp-bdg-56.amp (9/32) 16/02/11 20:27:33 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 10) in 10276 ms on amp-bdg-04.amp (10/32) 16/02/11 20:27:34 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 5) in 10798 ms on amp-bdg-36.amp (11/32) 16/02/11 20:27:34 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 14) in 9825 ms on amp-bdg-54.amp (12/32) 16/02/11 20:27:34 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-46.amp:38842 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:34 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on amp-bdg-29.amp:34585 (size: 2.2 KB, free: 530.0 MB) 16/02/11 20:27:34 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 9) in 11282 ms on amp-bdg-60.amp (13/32) 16/02/11 20:27:34 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 6) in 11589 ms on amp-bdg-20.amp (14/32) 16/02/11 20:27:35 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-46.amp:38842 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:35 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 15) in 10679 ms on amp-bdg-47.amp (15/32) 16/02/11 20:27:35 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on amp-bdg-29.amp:34585 (size: 22.5 KB, free: 530.0 MB) 16/02/11 20:27:35 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 24) in 10186 ms on amp-bdg-64.amp (16/32) 16/02/11 20:27:35 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 25) in 10360 ms on amp-bdg-18.amp (17/32) 16/02/11 20:27:35 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 22) in 11036 ms on amp-bdg-28.amp (18/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 26) in 11179 ms on amp-bdg-51.amp (19/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 16) in 11537 ms on amp-bdg-44.amp (20/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 20) in 11588 ms on amp-bdg-41.amp (21/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 18) in 11742 ms on amp-bdg-10.amp (22/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 7) in 13060 ms on amp-bdg-49.amp (23/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 17) in 12023 ms on amp-bdg-05.amp (24/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 23) in 11957 ms on amp-bdg-02.amp (25/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 21) in 11987 ms on amp-bdg-62.amp (26/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 12) in 13259 ms on amp-bdg-22.amp (27/32) 16/02/11 20:27:36 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 11) in 13287 ms on amp-bdg-37.amp (28/32) 16/02/11 20:27:37 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 8) in 13537 ms on amp-bdg-49.amp (29/32) 16/02/11 20:27:37 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 19) in 12372 ms on amp-bdg-57.amp (30/32) 16/02/11 20:27:38 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 31) in 5371 ms on amp-bdg-46.amp (31/32) 16/02/11 20:27:39 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 30) in 5415 ms on amp-bdg-29.amp (32/32) 16/02/11 20:27:39 INFO DAGScheduler: ShuffleMapStage 0 (keyBy at VariationRDDFunctions.scala:153) finished in 20.279 s 16/02/11 20:27:39 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 16/02/11 20:27:39 INFO DAGScheduler: looking for newly runnable stages 16/02/11 20:27:39 INFO DAGScheduler: running: Set() 16/02/11 20:27:39 INFO DAGScheduler: waiting: Set(ShuffleMapStage 1, ResultStage 2) 16/02/11 20:27:39 INFO DAGScheduler: failed: Set() 16/02/11 20:27:39 INFO DAGScheduler: Missing parents for ShuffleMapStage 1: List() 16/02/11 20:27:39 INFO DAGScheduler: Missing parents for ResultStage 2: List(ShuffleMapStage 1) 16/02/11 20:27:39 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[6] at distinct at VariationRDDFunctions.scala:65), which is now runnable 16/02/11 20:27:39 INFO MemoryStore: ensureFreeSpace(5384) called with curMem=298877, maxMem=555755765 16/02/11 20:27:39 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 5.3 KB, free 529.7 MB) 16/02/11 20:27:39 INFO MemoryStore: ensureFreeSpace(2712) called with curMem=304261, maxMem=555755765 16/02/11 20:27:39 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.6 KB, free 529.7 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.247.253:53930 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:861 16/02/11 20:27:39 INFO DAGScheduler: Submitting 32 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[6] at distinct at VariationRDDFunctions.scala:65) 16/02/11 20:27:39 INFO YarnScheduler: Adding task set 1.0 with 32 tasks 16/02/11 20:27:39 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 32, amp-bdg-03.amp, partition 0,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 33, amp-bdg-44.amp, partition 1,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 34, amp-bdg-62.amp, partition 2,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 35, amp-bdg-20.amp, partition 3,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 36, amp-bdg-25.amp, partition 4,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 37, amp-bdg-54.amp, partition 5,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 38, amp-bdg-34.amp, partition 6,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 39, amp-bdg-15.amp, partition 7,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 40, amp-bdg-18.amp, partition 8,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 41, amp-bdg-41.amp, partition 9,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 42, amp-bdg-49.amp, partition 10,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 43, amp-bdg-36.amp, partition 11,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 44, amp-bdg-35.amp, partition 12,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 45, amp-bdg-29.amp, partition 13,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 46, amp-bdg-51.amp, partition 14,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 47, amp-bdg-37.amp, partition 15,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 48, amp-bdg-04.amp, partition 16,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 49, amp-bdg-02.amp, partition 17,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 50, amp-bdg-49.amp, partition 18,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 51, amp-bdg-22.amp, partition 19,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 20.0 in stage 1.0 (TID 52, amp-bdg-24.amp, partition 20,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 21.0 in stage 1.0 (TID 53, amp-bdg-64.amp, partition 21,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 22.0 in stage 1.0 (TID 54, amp-bdg-56.amp, partition 22,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 23.0 in stage 1.0 (TID 55, amp-bdg-61.amp, partition 23,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 24.0 in stage 1.0 (TID 56, amp-bdg-57.amp, partition 24,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 25.0 in stage 1.0 (TID 57, amp-bdg-27.amp, partition 25,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 26.0 in stage 1.0 (TID 58, amp-bdg-05.amp, partition 26,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 27.0 in stage 1.0 (TID 59, amp-bdg-47.amp, partition 27,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 28.0 in stage 1.0 (TID 60, amp-bdg-46.amp, partition 28,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 29.0 in stage 1.0 (TID 61, amp-bdg-60.amp, partition 29,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 30.0 in stage 1.0 (TID 62, amp-bdg-28.amp, partition 30,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO TaskSetManager: Starting task 31.0 in stage 1.0 (TID 63, amp-bdg-10.amp, partition 31,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-03.amp:36085 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-49.amp:40827 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-15.amp:46865 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-64.amp:32917 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-20.amp:58218 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-41.amp:57788 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-56.amp:50396 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-54.amp:34279 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-36.amp:49131 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-18.amp:46764 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-49.amp:36486 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-25.amp:57820 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-51.amp:46478 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-02.amp:49749 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-10.amp:47502 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-44.amp:40528 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-46.amp:38842 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-62.amp:46717 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-35.amp:44956 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-28.amp:48326 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-60.amp:38207 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-29.amp:34585 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-37.amp:42720 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-24.amp:36201 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-05.amp:54334 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-47.amp:57360 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-57.amp:56334 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-04.amp:36661 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-22.amp:34751 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-56.amp:41787 16/02/11 20:27:39 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 447 bytes 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-15.amp:35924 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-49.amp:60339 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-18.amp:37719 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-49.amp:51930 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-03.amp:46605 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-41.amp:58413 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-20.amp:57148 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-54.amp:34211 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-51.amp:34878 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-64.amp:52828 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-10.amp:47549 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-35.amp:60229 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-36.amp:48076 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-46.amp:59053 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-37.amp:43269 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-25.amp:44820 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-02.amp:51182 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-44.amp:34888 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-62.amp:50643 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-57.amp:53625 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-05.amp:53075 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-24.amp:41672 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-29.amp:47528 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-22.amp:38208 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-47.amp:56604 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-60.amp:55591 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-28.amp:33300 16/02/11 20:27:39 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-04.amp:41693 16/02/11 20:27:40 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-34.amp:34029 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:40 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-27.amp:47425 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:40 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on amp-bdg-61.amp:41089 (size: 2.6 KB, free: 530.0 MB) 16/02/11 20:27:40 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-61.amp:52505 16/02/11 20:27:40 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-34.amp:51277 16/02/11 20:27:40 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to amp-bdg-27.amp:46972 16/02/11 20:27:42 INFO BlockManagerInfo: Added rdd_4_8 in memory on amp-bdg-18.amp:46764 (size: 188.7 MB, free: 341.3 MB) 16/02/11 20:27:42 INFO BlockManagerInfo: Added rdd_4_15 in memory on amp-bdg-37.amp:42720 (size: 187.7 MB, free: 342.3 MB) 16/02/11 20:27:42 INFO BlockManagerInfo: Added rdd_4_4 in memory on amp-bdg-25.amp:57820 (size: 187.9 MB, free: 342.1 MB) 16/02/11 20:27:43 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 40) in 3979 ms on amp-bdg-18.amp (1/32) 16/02/11 20:27:43 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 47) in 4032 ms on amp-bdg-37.amp (2/32) 16/02/11 20:27:43 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 36) in 4070 ms on amp-bdg-25.amp (3/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_19 in memory on amp-bdg-22.amp:34751 (size: 190.2 MB, free: 339.8 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_24 in memory on amp-bdg-57.amp:56334 (size: 189.9 MB, free: 340.0 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 51) in 14208 ms on amp-bdg-22.amp (4/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_28 in memory on amp-bdg-46.amp:38842 (size: 189.5 MB, free: 340.5 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_12 in memory on amp-bdg-35.amp:44956 (size: 190.4 MB, free: 339.6 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_3 in memory on amp-bdg-20.amp:58218 (size: 190.6 MB, free: 339.4 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_21 in memory on amp-bdg-64.amp:32917 (size: 189.3 MB, free: 340.7 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_5 in memory on amp-bdg-54.amp:34279 (size: 190.5 MB, free: 339.5 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 24.0 in stage 1.0 (TID 56) in 14454 ms on amp-bdg-57.amp (5/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_2 in memory on amp-bdg-62.amp:46717 (size: 188.9 MB, free: 341.1 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 28.0 in stage 1.0 (TID 60) in 14550 ms on amp-bdg-46.amp (6/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 44) in 14556 ms on amp-bdg-35.amp (7/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_7 in memory on amp-bdg-15.amp:46865 (size: 190.5 MB, free: 339.5 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_11 in memory on amp-bdg-36.amp:49131 (size: 188.9 MB, free: 341.1 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_30 in memory on amp-bdg-28.amp:48326 (size: 190.8 MB, free: 339.2 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_26 in memory on amp-bdg-05.amp:54334 (size: 190.5 MB, free: 339.5 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_22 in memory on amp-bdg-56.amp:50396 (size: 190.2 MB, free: 339.8 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_14 in memory on amp-bdg-51.amp:46478 (size: 189.8 MB, free: 340.1 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 21.0 in stage 1.0 (TID 53) in 14665 ms on amp-bdg-64.amp (8/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 37) in 14673 ms on amp-bdg-54.amp (9/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_1 in memory on amp-bdg-44.amp:40528 (size: 190.3 MB, free: 339.7 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_13 in memory on amp-bdg-29.amp:34585 (size: 190.0 MB, free: 340.0 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_27 in memory on amp-bdg-47.amp:57360 (size: 190.1 MB, free: 339.9 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 34) in 14759 ms on amp-bdg-62.amp (10/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 35) in 14762 ms on amp-bdg-20.amp (11/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_0 in memory on amp-bdg-03.amp:36085 (size: 189.2 MB, free: 340.8 MB) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_20 in memory on amp-bdg-24.amp:36201 (size: 189.9 MB, free: 340.1 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 43) in 14857 ms on amp-bdg-36.amp (12/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 30.0 in stage 1.0 (TID 62) in 14863 ms on amp-bdg-28.amp (13/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 39) in 14873 ms on amp-bdg-15.amp (14/32) 16/02/11 20:27:53 INFO BlockManagerInfo: Added rdd_4_31 in memory on amp-bdg-10.amp:47502 (size: 189.9 MB, free: 340.0 MB) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 22.0 in stage 1.0 (TID 54) in 14885 ms on amp-bdg-56.amp (15/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 46) in 14898 ms on amp-bdg-51.amp (16/32) 16/02/11 20:27:53 INFO TaskSetManager: Finished task 26.0 in stage 1.0 (TID 58) in 14909 ms on amp-bdg-05.amp (17/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 33) in 15017 ms on amp-bdg-44.amp (18/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 20.0 in stage 1.0 (TID 52) in 15030 ms on amp-bdg-24.amp (19/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 32) in 15042 ms on amp-bdg-03.amp (20/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 45) in 15045 ms on amp-bdg-29.amp (21/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 27.0 in stage 1.0 (TID 59) in 15044 ms on amp-bdg-47.amp (22/32) 16/02/11 20:27:54 INFO BlockManagerInfo: Added rdd_4_18 in memory on amp-bdg-49.amp:36486 (size: 189.8 MB, free: 340.2 MB) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 31.0 in stage 1.0 (TID 63) in 15122 ms on amp-bdg-10.amp (23/32) 16/02/11 20:27:54 INFO BlockManagerInfo: Added rdd_4_16 in memory on amp-bdg-04.amp:36661 (size: 190.1 MB, free: 339.9 MB) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 50) in 15391 ms on amp-bdg-49.amp (24/32) 16/02/11 20:27:54 INFO BlockManagerInfo: Added rdd_4_9 in memory on amp-bdg-41.amp:57788 (size: 190.7 MB, free: 339.2 MB) 16/02/11 20:27:54 INFO BlockManagerInfo: Added rdd_4_10 in memory on amp-bdg-49.amp:40827 (size: 191.5 MB, free: 338.5 MB) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 48) in 15552 ms on amp-bdg-04.amp (25/32) 16/02/11 20:27:54 INFO BlockManagerInfo: Added rdd_4_17 in memory on amp-bdg-02.amp:49749 (size: 191.1 MB, free: 338.9 MB) 16/02/11 20:27:54 INFO BlockManagerInfo: Added rdd_4_29 in memory on amp-bdg-60.amp:38207 (size: 191.4 MB, free: 338.5 MB) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 41) in 15736 ms on amp-bdg-41.amp (26/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 42) in 15852 ms on amp-bdg-49.amp (27/32) 16/02/11 20:27:54 INFO TaskSetManager: Finished task 29.0 in stage 1.0 (TID 61) in 15944 ms on amp-bdg-60.amp (28/32) 16/02/11 20:27:55 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 49) in 15955 ms on amp-bdg-02.amp (29/32) 16/02/11 20:27:55 INFO BlockManagerInfo: Added rdd_4_23 in memory on amp-bdg-61.amp:41089 (size: 190.1 MB, free: 339.9 MB) 16/02/11 20:27:55 INFO TaskSetManager: Finished task 23.0 in stage 1.0 (TID 55) in 16620 ms on amp-bdg-61.amp (30/32) 16/02/11 20:27:55 INFO BlockManagerInfo: Added rdd_4_25 in memory on amp-bdg-27.amp:47425 (size: 190.6 MB, free: 339.4 MB) 16/02/11 20:27:55 INFO BlockManagerInfo: Added rdd_4_6 in memory on amp-bdg-34.amp:34029 (size: 190.0 MB, free: 340.0 MB) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 25.0 in stage 1.0 (TID 57) in 16968 ms on amp-bdg-27.amp (31/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 38) in 17239 ms on amp-bdg-34.amp (32/32) 16/02/11 20:27:56 INFO YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 16/02/11 20:27:56 INFO DAGScheduler: ShuffleMapStage 1 (distinct at VariationRDDFunctions.scala:65) finished in 17.243 s 16/02/11 20:27:56 INFO DAGScheduler: looking for newly runnable stages 16/02/11 20:27:56 INFO DAGScheduler: running: Set() 16/02/11 20:27:56 INFO DAGScheduler: waiting: Set(ResultStage 2) 16/02/11 20:27:56 INFO DAGScheduler: failed: Set() 16/02/11 20:27:56 INFO DAGScheduler: Missing parents for ResultStage 2: List() 16/02/11 20:27:56 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[8] at distinct at VariationRDDFunctions.scala:65), which is now runnable 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(2800) called with curMem=306973, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.7 KB, free 529.7 MB) 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(1598) called with curMem=309773, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1598.0 B, free 529.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.247.253:53930 (size: 1598.0 B, free: 530.0 MB) 16/02/11 20:27:56 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:861 16/02/11 20:27:56 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 2 (MapPartitionsRDD[8] at distinct at VariationRDDFunctions.scala:65) 16/02/11 20:27:56 INFO YarnScheduler: Adding task set 2.0 with 32 tasks 16/02/11 20:27:56 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 64, amp-bdg-44.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 65, amp-bdg-02.amp, partition 1,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 66, amp-bdg-27.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 67, amp-bdg-28.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 68, amp-bdg-15.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 69, amp-bdg-05.amp, partition 5,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 70, amp-bdg-56.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 71, amp-bdg-36.amp, partition 7,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 72, amp-bdg-10.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 73, amp-bdg-49.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 74, amp-bdg-41.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 11.0 in stage 2.0 (TID 75, amp-bdg-37.amp, partition 11,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 12.0 in stage 2.0 (TID 76, amp-bdg-46.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 13.0 in stage 2.0 (TID 77, amp-bdg-64.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 14.0 in stage 2.0 (TID 78, amp-bdg-61.amp, partition 14,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 15.0 in stage 2.0 (TID 79, amp-bdg-25.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 16.0 in stage 2.0 (TID 80, amp-bdg-34.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 17.0 in stage 2.0 (TID 81, amp-bdg-62.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 18.0 in stage 2.0 (TID 82, amp-bdg-04.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 19.0 in stage 2.0 (TID 83, amp-bdg-49.amp, partition 19,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 20.0 in stage 2.0 (TID 84, amp-bdg-35.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 21.0 in stage 2.0 (TID 85, amp-bdg-51.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 22.0 in stage 2.0 (TID 86, amp-bdg-57.amp, partition 22,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 23.0 in stage 2.0 (TID 87, amp-bdg-20.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 24.0 in stage 2.0 (TID 88, amp-bdg-54.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 25.0 in stage 2.0 (TID 89, amp-bdg-60.amp, partition 25,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 26.0 in stage 2.0 (TID 90, amp-bdg-29.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 27.0 in stage 2.0 (TID 91, amp-bdg-18.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 28.0 in stage 2.0 (TID 92, amp-bdg-47.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 29.0 in stage 2.0 (TID 93, amp-bdg-03.amp, partition 29,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 30.0 in stage 2.0 (TID 94, amp-bdg-22.amp, partition 30,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 31.0 in stage 2.0 (TID 95, amp-bdg-24.amp, partition 31,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-34.amp:34029 (size: 1598.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-44.amp:40528 (size: 1598.0 B, free: 339.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-18.amp:46764 (size: 1598.0 B, free: 341.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-41.amp:57788 (size: 1598.0 B, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-46.amp:38842 (size: 1598.0 B, free: 340.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-10.amp:47502 (size: 1598.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-64.amp:32917 (size: 1598.0 B, free: 340.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-36.amp:49131 (size: 1598.0 B, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-51.amp:46478 (size: 1598.0 B, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-02.amp:49749 (size: 1598.0 B, free: 338.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-37.amp:42720 (size: 1598.0 B, free: 342.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-05.amp:54334 (size: 1598.0 B, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-62.amp:46717 (size: 1598.0 B, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-35.amp:44956 (size: 1598.0 B, free: 339.6 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-28.amp:48326 (size: 1598.0 B, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-49.amp:36486 (size: 1598.0 B, free: 340.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-24.amp:36201 (size: 1598.0 B, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-20.amp:58218 (size: 1598.0 B, free: 339.4 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-56.amp:50396 (size: 1598.0 B, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-49.amp:40827 (size: 1598.0 B, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-47.amp:57360 (size: 1598.0 B, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-29.amp:34585 (size: 1598.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-15.amp:46865 (size: 1598.0 B, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-27.amp:47425 (size: 1598.0 B, free: 339.4 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-60.amp:38207 (size: 1598.0 B, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-57.amp:56334 (size: 1598.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-04.amp:36661 (size: 1598.0 B, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-54.amp:34279 (size: 1598.0 B, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-03.amp:36085 (size: 1598.0 B, free: 340.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-22.amp:34751 (size: 1598.0 B, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-25.amp:57820 (size: 1598.0 B, free: 342.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on amp-bdg-61.amp:41089 (size: 1598.0 B, free: 339.9 MB) 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-34.amp:51277 16/02/11 20:27:56 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 354 bytes 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-64.amp:52828 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-51.amp:34878 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-46.amp:59053 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-44.amp:34888 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-36.amp:48076 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-54.amp:34211 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-62.amp:50643 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-10.amp:47549 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-56.amp:41787 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-18.amp:37719 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-41.amp:58413 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-05.amp:53075 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-49.amp:51930 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-29.amp:47528 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-57.amp:53625 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-49.amp:60339 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-47.amp:56604 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-35.amp:60229 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-28.amp:33300 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-24.amp:41672 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-25.amp:44820 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-37.amp:43269 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-02.amp:51182 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-04.amp:41693 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-03.amp:46605 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-20.amp:57148 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-60.amp:55591 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-15.amp:35924 16/02/11 20:27:56 INFO TaskSetManager: Finished task 21.0 in stage 2.0 (TID 85) in 46 ms on amp-bdg-51.amp (1/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 16.0 in stage 2.0 (TID 80) in 48 ms on amp-bdg-34.amp (2/32) 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-27.amp:46972 16/02/11 20:27:56 INFO TaskSetManager: Finished task 13.0 in stage 2.0 (TID 77) in 49 ms on amp-bdg-64.amp (3/32) 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-22.amp:38208 16/02/11 20:27:56 INFO TaskSetManager: Finished task 12.0 in stage 2.0 (TID 76) in 49 ms on amp-bdg-46.amp (4/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 24.0 in stage 2.0 (TID 88) in 50 ms on amp-bdg-54.amp (5/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 71) in 53 ms on amp-bdg-36.amp (6/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 64) in 56 ms on amp-bdg-44.amp (7/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 10.0 in stage 2.0 (TID 74) in 53 ms on amp-bdg-41.amp (8/32) 16/02/11 20:27:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to amp-bdg-61.amp:52505 16/02/11 20:27:56 INFO TaskSetManager: Finished task 19.0 in stage 2.0 (TID 83) in 53 ms on amp-bdg-49.amp (9/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 22.0 in stage 2.0 (TID 86) in 55 ms on amp-bdg-57.amp (10/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 72) in 58 ms on amp-bdg-10.amp (11/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 17.0 in stage 2.0 (TID 81) in 57 ms on amp-bdg-62.amp (12/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 73) in 59 ms on amp-bdg-49.amp (13/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 26.0 in stage 2.0 (TID 90) in 56 ms on amp-bdg-29.amp (14/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 70) in 61 ms on amp-bdg-56.amp (15/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 28.0 in stage 2.0 (TID 92) in 58 ms on amp-bdg-47.amp (16/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 67) in 63 ms on amp-bdg-28.amp (17/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 15.0 in stage 2.0 (TID 79) in 60 ms on amp-bdg-25.amp (18/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 27.0 in stage 2.0 (TID 91) in 59 ms on amp-bdg-18.amp (19/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 29.0 in stage 2.0 (TID 93) in 60 ms on amp-bdg-03.amp (20/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 11.0 in stage 2.0 (TID 75) in 63 ms on amp-bdg-37.amp (21/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 20.0 in stage 2.0 (TID 84) in 62 ms on amp-bdg-35.amp (22/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 18.0 in stage 2.0 (TID 82) in 64 ms on amp-bdg-04.amp (23/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 23.0 in stage 2.0 (TID 87) in 64 ms on amp-bdg-20.amp (24/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 65) in 68 ms on amp-bdg-02.amp (25/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 68) in 68 ms on amp-bdg-15.amp (26/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 25.0 in stage 2.0 (TID 89) in 64 ms on amp-bdg-60.amp (27/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 30.0 in stage 2.0 (TID 94) in 63 ms on amp-bdg-22.amp (28/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 31.0 in stage 2.0 (TID 95) in 64 ms on amp-bdg-24.amp (29/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 66) in 73 ms on amp-bdg-27.amp (30/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 14.0 in stage 2.0 (TID 78) in 75 ms on amp-bdg-61.amp (31/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 69) in 87 ms on amp-bdg-05.amp (32/32) 16/02/11 20:27:56 INFO YarnScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool 16/02/11 20:27:56 INFO DAGScheduler: ResultStage 2 (collect at VariationRDDFunctions.scala:66) finished in 0.090 s 16/02/11 20:27:56 INFO DAGScheduler: Job 0 finished: collect at VariationRDDFunctions.scala:66, took 39.543487 s 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(120) called with curMem=311371, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 120.0 B, free 529.7 MB) 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(72) called with curMem=311491, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 72.0 B, free 529.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.247.253:53930 (size: 72.0 B, free: 530.0 MB) 16/02/11 20:27:56 INFO SparkContext: Created broadcast 4 from broadcast at VariationRDDFunctions.scala:93 16/02/11 20:27:56 INFO SparkContext: Starting job: count at VariationRDDFunctions.scala:106 16/02/11 20:27:56 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 447 bytes 16/02/11 20:27:56 INFO DAGScheduler: Got job 1 (count at VariationRDDFunctions.scala:106) with 32 output partitions 16/02/11 20:27:56 INFO DAGScheduler: Final stage: ResultStage 4(count at VariationRDDFunctions.scala:106) 16/02/11 20:27:56 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3) 16/02/11 20:27:56 INFO DAGScheduler: Missing parents: List() 16/02/11 20:27:56 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[9] at mapPartitionsWithIndex at VariationRDDFunctions.scala:94), which has no missing parents 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(5608) called with curMem=311563, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 5.5 KB, free 529.7 MB) 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(2867) called with curMem=317171, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.8 KB, free 529.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.247.253:53930 (size: 2.8 KB, free: 530.0 MB) 16/02/11 20:27:56 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:861 16/02/11 20:27:56 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 4 (MapPartitionsRDD[9] at mapPartitionsWithIndex at VariationRDDFunctions.scala:94) 16/02/11 20:27:56 INFO YarnScheduler: Adding task set 4.0 with 32 tasks 16/02/11 20:27:56 INFO TaskSetManager: Starting task 23.0 in stage 4.0 (TID 96, amp-bdg-61.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 30.0 in stage 4.0 (TID 97, amp-bdg-28.amp, partition 30,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 22.0 in stage 4.0 (TID 98, amp-bdg-56.amp, partition 22,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 4.0 in stage 4.0 (TID 99, amp-bdg-25.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 20.0 in stage 4.0 (TID 100, amp-bdg-24.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 5.0 in stage 4.0 (TID 101, amp-bdg-54.amp, partition 5,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 19.0 in stage 4.0 (TID 102, amp-bdg-22.amp, partition 19,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 27.0 in stage 4.0 (TID 103, amp-bdg-47.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 2.0 in stage 4.0 (TID 104, amp-bdg-62.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 29.0 in stage 4.0 (TID 105, amp-bdg-60.amp, partition 29,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 10.0 in stage 4.0 (TID 106, amp-bdg-49.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 11.0 in stage 4.0 (TID 107, amp-bdg-36.amp, partition 11,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 18.0 in stage 4.0 (TID 108, amp-bdg-49.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 3.0 in stage 4.0 (TID 109, amp-bdg-20.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 24.0 in stage 4.0 (TID 110, amp-bdg-57.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 111, amp-bdg-03.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 25.0 in stage 4.0 (TID 112, amp-bdg-27.amp, partition 25,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 6.0 in stage 4.0 (TID 113, amp-bdg-34.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 12.0 in stage 4.0 (TID 114, amp-bdg-35.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 7.0 in stage 4.0 (TID 115, amp-bdg-15.amp, partition 7,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 31.0 in stage 4.0 (TID 116, amp-bdg-10.amp, partition 31,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 15.0 in stage 4.0 (TID 117, amp-bdg-37.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 13.0 in stage 4.0 (TID 118, amp-bdg-29.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 17.0 in stage 4.0 (TID 119, amp-bdg-02.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 120, amp-bdg-44.amp, partition 1,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 28.0 in stage 4.0 (TID 121, amp-bdg-46.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 26.0 in stage 4.0 (TID 122, amp-bdg-05.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 16.0 in stage 4.0 (TID 123, amp-bdg-04.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 8.0 in stage 4.0 (TID 124, amp-bdg-18.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 9.0 in stage 4.0 (TID 125, amp-bdg-41.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 14.0 in stage 4.0 (TID 126, amp-bdg-51.amp, partition 14,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 21.0 in stage 4.0 (TID 127, amp-bdg-64.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-44.amp:40528 (size: 2.8 KB, free: 339.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-56.amp:50396 (size: 2.8 KB, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-47.amp:57360 (size: 2.8 KB, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-57.amp:56334 (size: 2.8 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-24.amp:36201 (size: 2.8 KB, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-49.amp:40827 (size: 2.8 KB, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-15.amp:46865 (size: 2.8 KB, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-64.amp:32917 (size: 2.8 KB, free: 340.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-03.amp:36085 (size: 2.8 KB, free: 340.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-62.amp:46717 (size: 2.8 KB, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-29.amp:34585 (size: 2.8 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-18.amp:46764 (size: 2.8 KB, free: 341.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-37.amp:42720 (size: 2.8 KB, free: 342.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-02.amp:49749 (size: 2.8 KB, free: 338.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-04.amp:36661 (size: 2.8 KB, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-36.amp:49131 (size: 2.8 KB, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-61.amp:41089 (size: 2.8 KB, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-41.amp:57788 (size: 2.8 KB, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-60.amp:38207 (size: 2.8 KB, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-35.amp:44956 (size: 2.8 KB, free: 339.6 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-22.amp:34751 (size: 2.8 KB, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-25.amp:57820 (size: 2.8 KB, free: 342.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-20.amp:58218 (size: 2.8 KB, free: 339.4 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-54.amp:34279 (size: 2.8 KB, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-51.amp:46478 (size: 2.8 KB, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-34.amp:34029 (size: 2.8 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-28.amp:48326 (size: 2.8 KB, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-27.amp:47425 (size: 2.8 KB, free: 339.4 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-46.amp:38842 (size: 2.8 KB, free: 340.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-05.amp:54334 (size: 2.8 KB, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-49.amp:36486 (size: 2.8 KB, free: 340.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on amp-bdg-10.amp:47502 (size: 2.8 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-44.amp:40528 (size: 72.0 B, free: 339.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-56.amp:50396 (size: 72.0 B, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-47.amp:57360 (size: 72.0 B, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-49.amp:40827 (size: 72.0 B, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-41.amp:57788 (size: 72.0 B, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-18.amp:46764 (size: 72.0 B, free: 341.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-64.amp:32917 (size: 72.0 B, free: 340.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-36.amp:49131 (size: 72.0 B, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-15.amp:46865 (size: 72.0 B, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-37.amp:42720 (size: 72.0 B, free: 342.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-57.amp:56334 (size: 72.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-22.amp:34751 (size: 72.0 B, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-20.amp:58218 (size: 72.0 B, free: 339.4 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-03.amp:36085 (size: 72.0 B, free: 340.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-02.amp:49749 (size: 72.0 B, free: 338.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-24.amp:36201 (size: 72.0 B, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-04.amp:36661 (size: 72.0 B, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-60.amp:38207 (size: 72.0 B, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-51.amp:46478 (size: 72.0 B, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-35.amp:44956 (size: 72.0 B, free: 339.6 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-61.amp:41089 (size: 72.0 B, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-46.amp:38842 (size: 72.0 B, free: 340.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-25.amp:57820 (size: 72.0 B, free: 342.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-54.amp:34279 (size: 72.0 B, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-29.amp:34585 (size: 72.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-49.amp:36486 (size: 72.0 B, free: 340.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-28.amp:48326 (size: 72.0 B, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-10.amp:47502 (size: 72.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-34.amp:34029 (size: 72.0 B, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-05.amp:54334 (size: 72.0 B, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-27.amp:47425 (size: 72.0 B, free: 339.4 MB) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 22.0 in stage 4.0 (TID 98) in 84 ms on amp-bdg-56.amp (1/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 120) in 82 ms on amp-bdg-44.amp (2/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 27.0 in stage 4.0 (TID 103) in 89 ms on amp-bdg-47.amp (3/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 9.0 in stage 4.0 (TID 125) in 86 ms on amp-bdg-41.amp (4/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 10.0 in stage 4.0 (TID 106) in 89 ms on amp-bdg-49.amp (5/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 21.0 in stage 4.0 (TID 127) in 86 ms on amp-bdg-64.amp (6/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 15.0 in stage 4.0 (TID 117) in 89 ms on amp-bdg-37.amp (7/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 7.0 in stage 4.0 (TID 115) in 90 ms on amp-bdg-15.amp (8/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 11.0 in stage 4.0 (TID 107) in 91 ms on amp-bdg-36.amp (9/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 8.0 in stage 4.0 (TID 124) in 88 ms on amp-bdg-18.amp (10/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 19.0 in stage 4.0 (TID 102) in 94 ms on amp-bdg-22.amp (11/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 17.0 in stage 4.0 (TID 119) in 91 ms on amp-bdg-02.amp (12/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 3.0 in stage 4.0 (TID 109) in 95 ms on amp-bdg-20.amp (13/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 29.0 in stage 4.0 (TID 105) in 95 ms on amp-bdg-60.amp (14/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 24.0 in stage 4.0 (TID 110) in 95 ms on amp-bdg-57.amp (15/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 111) in 96 ms on amp-bdg-03.amp (16/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 14.0 in stage 4.0 (TID 126) in 94 ms on amp-bdg-51.amp (17/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 4.0 in stage 4.0 (TID 99) in 99 ms on amp-bdg-25.amp (18/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 28.0 in stage 4.0 (TID 121) in 96 ms on amp-bdg-46.amp (19/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 5.0 in stage 4.0 (TID 101) in 100 ms on amp-bdg-54.amp (20/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 31.0 in stage 4.0 (TID 116) in 97 ms on amp-bdg-10.amp (21/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 12.0 in stage 4.0 (TID 114) in 99 ms on amp-bdg-35.amp (22/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 30.0 in stage 4.0 (TID 97) in 102 ms on amp-bdg-28.amp (23/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 20.0 in stage 4.0 (TID 100) in 102 ms on amp-bdg-24.amp (24/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 16.0 in stage 4.0 (TID 123) in 98 ms on amp-bdg-04.amp (25/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 26.0 in stage 4.0 (TID 122) in 100 ms on amp-bdg-05.amp (26/32) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on amp-bdg-62.amp:46717 (size: 72.0 B, free: 341.1 MB) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 23.0 in stage 4.0 (TID 96) in 105 ms on amp-bdg-61.amp (27/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 13.0 in stage 4.0 (TID 118) in 102 ms on amp-bdg-29.amp (28/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 18.0 in stage 4.0 (TID 108) in 104 ms on amp-bdg-49.amp (29/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 6.0 in stage 4.0 (TID 113) in 106 ms on amp-bdg-34.amp (30/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 25.0 in stage 4.0 (TID 112) in 114 ms on amp-bdg-27.amp (31/32) 16/02/11 20:27:56 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID 104) in 144 ms on amp-bdg-62.amp (32/32) 16/02/11 20:27:56 INFO DAGScheduler: ResultStage 4 (count at VariationRDDFunctions.scala:106) finished in 0.146 s 16/02/11 20:27:56 INFO YarnScheduler: Removed TaskSet 4.0, whose tasks have all completed, from pool 16/02/11 20:27:56 INFO DAGScheduler: Job 1 finished: count at VariationRDDFunctions.scala:106, took 0.161360 s 16/02/11 20:27:56 INFO VariantContextRDDFunctions: Set VCF header on driver 16/02/11 20:27:56 INFO SparkContext: Starting job: sortByKey at VariationRDDFunctions.scala:131 16/02/11 20:27:56 INFO DAGScheduler: Got job 2 (sortByKey at VariationRDDFunctions.scala:131) with 32 output partitions 16/02/11 20:27:56 INFO DAGScheduler: Final stage: ResultStage 6(sortByKey at VariationRDDFunctions.scala:131) 16/02/11 20:27:56 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 5) 16/02/11 20:27:56 INFO DAGScheduler: Missing parents: List() 16/02/11 20:27:56 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[13] at sortByKey at VariationRDDFunctions.scala:131), which has no missing parents 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(6304) called with curMem=320038, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 6.2 KB, free 529.7 MB) 16/02/11 20:27:56 INFO MemoryStore: ensureFreeSpace(3183) called with curMem=326342, maxMem=555755765 16/02/11 20:27:56 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 3.1 KB, free 529.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.247.253:53930 (size: 3.1 KB, free: 530.0 MB) 16/02/11 20:27:56 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:861 16/02/11 20:27:56 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 6 (MapPartitionsRDD[13] at sortByKey at VariationRDDFunctions.scala:131) 16/02/11 20:27:56 INFO YarnScheduler: Adding task set 6.0 with 32 tasks 16/02/11 20:27:56 INFO TaskSetManager: Starting task 23.0 in stage 6.0 (TID 128, amp-bdg-61.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 14.0 in stage 6.0 (TID 129, amp-bdg-51.amp, partition 14,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 30.0 in stage 6.0 (TID 130, amp-bdg-28.amp, partition 30,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 7.0 in stage 6.0 (TID 131, amp-bdg-15.amp, partition 7,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 29.0 in stage 6.0 (TID 132, amp-bdg-60.amp, partition 29,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 133, amp-bdg-03.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 10.0 in stage 6.0 (TID 134, amp-bdg-49.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 16.0 in stage 6.0 (TID 135, amp-bdg-04.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 15.0 in stage 6.0 (TID 136, amp-bdg-37.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 25.0 in stage 6.0 (TID 137, amp-bdg-27.amp, partition 25,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 17.0 in stage 6.0 (TID 138, amp-bdg-02.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 27.0 in stage 6.0 (TID 139, amp-bdg-47.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 5.0 in stage 6.0 (TID 140, amp-bdg-54.amp, partition 5,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 13.0 in stage 6.0 (TID 141, amp-bdg-29.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 8.0 in stage 6.0 (TID 142, amp-bdg-18.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 11.0 in stage 6.0 (TID 143, amp-bdg-36.amp, partition 11,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 21.0 in stage 6.0 (TID 144, amp-bdg-64.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 22.0 in stage 6.0 (TID 145, amp-bdg-56.amp, partition 22,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 6.0 in stage 6.0 (TID 146, amp-bdg-34.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 3.0 in stage 6.0 (TID 147, amp-bdg-20.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 28.0 in stage 6.0 (TID 148, amp-bdg-46.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 20.0 in stage 6.0 (TID 149, amp-bdg-24.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 31.0 in stage 6.0 (TID 150, amp-bdg-10.amp, partition 31,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 1.0 in stage 6.0 (TID 151, amp-bdg-44.amp, partition 1,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 18.0 in stage 6.0 (TID 152, amp-bdg-49.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 9.0 in stage 6.0 (TID 153, amp-bdg-41.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 19.0 in stage 6.0 (TID 154, amp-bdg-22.amp, partition 19,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 26.0 in stage 6.0 (TID 155, amp-bdg-05.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 4.0 in stage 6.0 (TID 156, amp-bdg-25.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 24.0 in stage 6.0 (TID 157, amp-bdg-57.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 12.0 in stage 6.0 (TID 158, amp-bdg-35.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO TaskSetManager: Starting task 2.0 in stage 6.0 (TID 159, amp-bdg-62.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-64.amp:32917 (size: 3.1 KB, free: 340.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-60.amp:38207 (size: 3.1 KB, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-49.amp:36486 (size: 3.1 KB, free: 340.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-36.amp:49131 (size: 3.1 KB, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-51.amp:46478 (size: 3.1 KB, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-49.amp:40827 (size: 3.1 KB, free: 338.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-03.amp:36085 (size: 3.1 KB, free: 340.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-28.amp:48326 (size: 3.1 KB, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-02.amp:49749 (size: 3.1 KB, free: 338.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-04.amp:36661 (size: 3.1 KB, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-57.amp:56334 (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-44.amp:40528 (size: 3.1 KB, free: 339.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-61.amp:41089 (size: 3.1 KB, free: 339.9 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-35.amp:44956 (size: 3.1 KB, free: 339.6 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-54.amp:34279 (size: 3.1 KB, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-41.amp:57788 (size: 3.1 KB, free: 339.2 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-22.amp:34751 (size: 3.1 KB, free: 339.7 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-29.amp:34585 (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-37.amp:42720 (size: 3.1 KB, free: 342.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-47.amp:57360 (size: 3.1 KB, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-18.amp:46764 (size: 3.1 KB, free: 341.3 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-34.amp:34029 (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-46.amp:38842 (size: 3.1 KB, free: 340.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-62.amp:46717 (size: 3.1 KB, free: 341.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-15.amp:46865 (size: 3.1 KB, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-05.amp:54334 (size: 3.1 KB, free: 339.5 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-24.amp:36201 (size: 3.1 KB, free: 340.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-56.amp:50396 (size: 3.1 KB, free: 339.8 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-25.amp:57820 (size: 3.1 KB, free: 342.1 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-27.amp:47425 (size: 3.1 KB, free: 339.4 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-10.amp:47502 (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:27:56 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on amp-bdg-20.amp:58218 (size: 3.1 KB, free: 339.4 MB) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 4.0 in stage 6.0 (TID 156) in 949 ms on amp-bdg-25.amp (1/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 27.0 in stage 6.0 (TID 139) in 953 ms on amp-bdg-47.amp (2/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 17.0 in stage 6.0 (TID 138) in 962 ms on amp-bdg-02.amp (3/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 29.0 in stage 6.0 (TID 132) in 963 ms on amp-bdg-60.amp (4/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 133) in 967 ms on amp-bdg-03.amp (5/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 7.0 in stage 6.0 (TID 131) in 970 ms on amp-bdg-15.amp (6/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 28.0 in stage 6.0 (TID 148) in 969 ms on amp-bdg-46.amp (7/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 6.0 in stage 6.0 (TID 146) in 976 ms on amp-bdg-34.amp (8/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 19.0 in stage 6.0 (TID 154) in 975 ms on amp-bdg-22.amp (9/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 11.0 in stage 6.0 (TID 143) in 977 ms on amp-bdg-36.amp (10/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 2.0 in stage 6.0 (TID 159) in 976 ms on amp-bdg-62.amp (11/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 3.0 in stage 6.0 (TID 147) in 984 ms on amp-bdg-20.amp (12/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 24.0 in stage 6.0 (TID 157) in 986 ms on amp-bdg-57.amp (13/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 14.0 in stage 6.0 (TID 129) in 991 ms on amp-bdg-51.amp (14/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 23.0 in stage 6.0 (TID 128) in 992 ms on amp-bdg-61.amp (15/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 22.0 in stage 6.0 (TID 145) in 989 ms on amp-bdg-56.amp (16/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 5.0 in stage 6.0 (TID 140) in 992 ms on amp-bdg-54.amp (17/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 8.0 in stage 6.0 (TID 142) in 993 ms on amp-bdg-18.amp (18/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 12.0 in stage 6.0 (TID 158) in 991 ms on amp-bdg-35.amp (19/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 26.0 in stage 6.0 (TID 155) in 991 ms on amp-bdg-05.amp (20/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 9.0 in stage 6.0 (TID 153) in 993 ms on amp-bdg-41.amp (21/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 15.0 in stage 6.0 (TID 136) in 1001 ms on amp-bdg-37.amp (22/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 13.0 in stage 6.0 (TID 141) in 1001 ms on amp-bdg-29.amp (23/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 31.0 in stage 6.0 (TID 150) in 1008 ms on amp-bdg-10.amp (24/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 1.0 in stage 6.0 (TID 151) in 1009 ms on amp-bdg-44.amp (25/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 25.0 in stage 6.0 (TID 137) in 1013 ms on amp-bdg-27.amp (26/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 16.0 in stage 6.0 (TID 135) in 1024 ms on amp-bdg-04.amp (27/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 30.0 in stage 6.0 (TID 130) in 1042 ms on amp-bdg-28.amp (28/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 18.0 in stage 6.0 (TID 152) in 1039 ms on amp-bdg-49.amp (29/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 21.0 in stage 6.0 (TID 144) in 1045 ms on amp-bdg-64.amp (30/32) 16/02/11 20:27:57 INFO TaskSetManager: Finished task 10.0 in stage 6.0 (TID 134) in 1088 ms on amp-bdg-49.amp (31/32) 16/02/11 20:27:58 INFO TaskSetManager: Finished task 20.0 in stage 6.0 (TID 149) in 1837 ms on amp-bdg-24.amp (32/32) 16/02/11 20:27:58 INFO YarnScheduler: Removed TaskSet 6.0, whose tasks have all completed, from pool 16/02/11 20:27:58 INFO DAGScheduler: ResultStage 6 (sortByKey at VariationRDDFunctions.scala:131) finished in 1.841 s 16/02/11 20:27:58 INFO DAGScheduler: Job 2 finished: sortByKey at VariationRDDFunctions.scala:131, took 1.855683 s 16/02/11 20:27:58 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 16/02/11 20:27:58 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 16/02/11 20:27:58 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile at VariationRDDFunctions.scala:140 16/02/11 20:27:58 INFO DAGScheduler: Registering RDD 11 (keyBy at VariationRDDFunctions.scala:130) 16/02/11 20:27:58 INFO DAGScheduler: Got job 3 (saveAsNewAPIHadoopFile at VariationRDDFunctions.scala:140) with 32 output partitions 16/02/11 20:27:58 INFO DAGScheduler: Final stage: ResultStage 9(saveAsNewAPIHadoopFile at VariationRDDFunctions.scala:140) 16/02/11 20:27:58 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 8) 16/02/11 20:27:58 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 8) 16/02/11 20:27:58 INFO DAGScheduler: Submitting ShuffleMapStage 8 (MapPartitionsRDD[11] at keyBy at VariationRDDFunctions.scala:130), which has no missing parents 16/02/11 20:27:58 INFO MemoryStore: ensureFreeSpace(6528) called with curMem=329525, maxMem=555755765 16/02/11 20:27:58 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 6.4 KB, free 529.7 MB) 16/02/11 20:27:58 INFO MemoryStore: ensureFreeSpace(3358) called with curMem=336053, maxMem=555755765 16/02/11 20:27:58 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 3.3 KB, free 529.7 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.247.253:53930 (size: 3.3 KB, free: 530.0 MB) 16/02/11 20:27:58 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:861 16/02/11 20:27:58 INFO DAGScheduler: Submitting 32 missing tasks from ShuffleMapStage 8 (MapPartitionsRDD[11] at keyBy at VariationRDDFunctions.scala:130) 16/02/11 20:27:58 INFO YarnScheduler: Adding task set 8.0 with 32 tasks 16/02/11 20:27:58 INFO TaskSetManager: Starting task 23.0 in stage 8.0 (TID 160, amp-bdg-61.amp, partition 23,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 21.0 in stage 8.0 (TID 161, amp-bdg-64.amp, partition 21,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 24.0 in stage 8.0 (TID 162, amp-bdg-57.amp, partition 24,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 18.0 in stage 8.0 (TID 163, amp-bdg-49.amp, partition 18,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 27.0 in stage 8.0 (TID 164, amp-bdg-47.amp, partition 27,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 11.0 in stage 8.0 (TID 165, amp-bdg-36.amp, partition 11,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 10.0 in stage 8.0 (TID 166, amp-bdg-49.amp, partition 10,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 26.0 in stage 8.0 (TID 167, amp-bdg-05.amp, partition 26,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 28.0 in stage 8.0 (TID 168, amp-bdg-46.amp, partition 28,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 17.0 in stage 8.0 (TID 169, amp-bdg-02.amp, partition 17,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 16.0 in stage 8.0 (TID 170, amp-bdg-04.amp, partition 16,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 5.0 in stage 8.0 (TID 171, amp-bdg-54.amp, partition 5,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 172, amp-bdg-44.amp, partition 1,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 29.0 in stage 8.0 (TID 173, amp-bdg-60.amp, partition 29,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 6.0 in stage 8.0 (TID 174, amp-bdg-34.amp, partition 6,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 25.0 in stage 8.0 (TID 175, amp-bdg-27.amp, partition 25,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 176, amp-bdg-03.amp, partition 0,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 7.0 in stage 8.0 (TID 177, amp-bdg-15.amp, partition 7,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 30.0 in stage 8.0 (TID 178, amp-bdg-28.amp, partition 30,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 15.0 in stage 8.0 (TID 179, amp-bdg-37.amp, partition 15,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 4.0 in stage 8.0 (TID 180, amp-bdg-25.amp, partition 4,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 3.0 in stage 8.0 (TID 181, amp-bdg-20.amp, partition 3,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 22.0 in stage 8.0 (TID 182, amp-bdg-56.amp, partition 22,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 12.0 in stage 8.0 (TID 183, amp-bdg-35.amp, partition 12,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 14.0 in stage 8.0 (TID 184, amp-bdg-51.amp, partition 14,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 13.0 in stage 8.0 (TID 185, amp-bdg-29.amp, partition 13,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 31.0 in stage 8.0 (TID 186, amp-bdg-10.amp, partition 31,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 9.0 in stage 8.0 (TID 187, amp-bdg-41.amp, partition 9,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 8.0 in stage 8.0 (TID 188, amp-bdg-18.amp, partition 8,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 20.0 in stage 8.0 (TID 189, amp-bdg-24.amp, partition 20,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 19.0 in stage 8.0 (TID 190, amp-bdg-22.amp, partition 19,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO TaskSetManager: Starting task 2.0 in stage 8.0 (TID 191, amp-bdg-62.amp, partition 2,PROCESS_LOCAL, 1967 bytes) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-62.amp:46717 (size: 3.3 KB, free: 341.1 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-25.amp:57820 (size: 3.3 KB, free: 342.1 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-27.amp:47425 (size: 3.3 KB, free: 339.4 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-46.amp:38842 (size: 3.3 KB, free: 340.5 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-18.amp:46764 (size: 3.3 KB, free: 341.3 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-02.amp:49749 (size: 3.3 KB, free: 338.9 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-15.amp:46865 (size: 3.3 KB, free: 339.5 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-51.amp:46478 (size: 3.3 KB, free: 340.1 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-44.amp:40528 (size: 3.3 KB, free: 339.7 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-54.amp:34279 (size: 3.3 KB, free: 339.5 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-61.amp:41089 (size: 3.3 KB, free: 339.9 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-34.amp:34029 (size: 3.3 KB, free: 340.0 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-28.amp:48326 (size: 3.3 KB, free: 339.2 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-41.amp:57788 (size: 3.3 KB, free: 339.2 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-20.amp:58218 (size: 3.3 KB, free: 339.4 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-64.amp:32917 (size: 3.3 KB, free: 340.7 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-57.amp:56334 (size: 3.3 KB, free: 340.0 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-22.amp:34751 (size: 3.3 KB, free: 339.7 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-03.amp:36085 (size: 3.3 KB, free: 340.8 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-29.amp:34585 (size: 3.3 KB, free: 340.0 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-60.amp:38207 (size: 3.3 KB, free: 338.5 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-36.amp:49131 (size: 3.3 KB, free: 341.1 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-37.amp:42720 (size: 3.3 KB, free: 342.2 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-47.amp:57360 (size: 3.3 KB, free: 339.8 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-10.amp:47502 (size: 3.3 KB, free: 340.0 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-56.amp:50396 (size: 3.3 KB, free: 339.8 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-49.amp:40827 (size: 3.3 KB, free: 338.5 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-04.amp:36661 (size: 3.3 KB, free: 339.9 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-35.amp:44956 (size: 3.3 KB, free: 339.6 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-49.amp:36486 (size: 3.3 KB, free: 340.2 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-24.amp:36201 (size: 3.3 KB, free: 340.1 MB) 16/02/11 20:27:58 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on amp-bdg-05.amp:54334 (size: 3.3 KB, free: 339.5 MB) 16/02/11 20:28:00 INFO TaskSetManager: Finished task 19.0 in stage 8.0 (TID 190) in 2272 ms on amp-bdg-22.amp (1/32) 16/02/11 20:28:00 INFO TaskSetManager: Finished task 2.0 in stage 8.0 (TID 191) in 2287 ms on amp-bdg-62.amp (2/32) 16/02/11 20:28:00 INFO TaskSetManager: Finished task 7.0 in stage 8.0 (TID 177) in 2337 ms on amp-bdg-15.amp (3/32) 16/02/11 20:28:00 INFO TaskSetManager: Finished task 9.0 in stage 8.0 (TID 187) in 2348 ms on amp-bdg-41.amp (4/32) 16/02/11 20:28:00 INFO TaskSetManager: Finished task 23.0 in stage 8.0 (TID 160) in 2352 ms on amp-bdg-61.amp (5/32) 16/02/11 20:28:00 INFO TaskSetManager: Finished task 11.0 in stage 8.0 (TID 165) in 2362 ms on amp-bdg-36.amp (6/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 26.0 in stage 8.0 (TID 167) in 2383 ms on amp-bdg-05.amp (7/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 29.0 in stage 8.0 (TID 173) in 2384 ms on amp-bdg-60.amp (8/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 8.0 in stage 8.0 (TID 188) in 2384 ms on amp-bdg-18.amp (9/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 5.0 in stage 8.0 (TID 171) in 2396 ms on amp-bdg-54.amp (10/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 30.0 in stage 8.0 (TID 178) in 2406 ms on amp-bdg-28.amp (11/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 28.0 in stage 8.0 (TID 168) in 2410 ms on amp-bdg-46.amp (12/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 3.0 in stage 8.0 (TID 181) in 2421 ms on amp-bdg-20.amp (13/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 13.0 in stage 8.0 (TID 185) in 2429 ms on amp-bdg-29.amp (14/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 24.0 in stage 8.0 (TID 162) in 2437 ms on amp-bdg-57.amp (15/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 14.0 in stage 8.0 (TID 184) in 2445 ms on amp-bdg-51.amp (16/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 176) in 2453 ms on amp-bdg-03.amp (17/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 31.0 in stage 8.0 (TID 186) in 2452 ms on amp-bdg-10.amp (18/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 25.0 in stage 8.0 (TID 175) in 2463 ms on amp-bdg-27.amp (19/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 172) in 2481 ms on amp-bdg-44.amp (20/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 21.0 in stage 8.0 (TID 161) in 2491 ms on amp-bdg-64.amp (21/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 17.0 in stage 8.0 (TID 169) in 2510 ms on amp-bdg-02.amp (22/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 12.0 in stage 8.0 (TID 183) in 2510 ms on amp-bdg-35.amp (23/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 15.0 in stage 8.0 (TID 179) in 2513 ms on amp-bdg-37.amp (24/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 16.0 in stage 8.0 (TID 170) in 2520 ms on amp-bdg-04.amp (25/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 6.0 in stage 8.0 (TID 174) in 2531 ms on amp-bdg-34.amp (26/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 22.0 in stage 8.0 (TID 182) in 2531 ms on amp-bdg-56.amp (27/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 10.0 in stage 8.0 (TID 166) in 2608 ms on amp-bdg-49.amp (28/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 18.0 in stage 8.0 (TID 163) in 2609 ms on amp-bdg-49.amp (29/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 4.0 in stage 8.0 (TID 180) in 2701 ms on amp-bdg-25.amp (30/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 20.0 in stage 8.0 (TID 189) in 2742 ms on amp-bdg-24.amp (31/32) 16/02/11 20:28:01 INFO TaskSetManager: Finished task 27.0 in stage 8.0 (TID 164) in 2771 ms on amp-bdg-47.amp (32/32) 16/02/11 20:28:01 INFO YarnScheduler: Removed TaskSet 8.0, whose tasks have all completed, from pool 16/02/11 20:28:01 INFO DAGScheduler: ShuffleMapStage 8 (keyBy at VariationRDDFunctions.scala:130) finished in 2.773 s 16/02/11 20:28:01 INFO DAGScheduler: looking for newly runnable stages 16/02/11 20:28:01 INFO DAGScheduler: running: Set() 16/02/11 20:28:01 INFO DAGScheduler: waiting: Set(ResultStage 9) 16/02/11 20:28:01 INFO DAGScheduler: failed: Set() 16/02/11 20:28:01 INFO DAGScheduler: Missing parents for ResultStage 9: List() 16/02/11 20:28:01 INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[15] at map at VariationRDDFunctions.scala:132), which is now runnable 16/02/11 20:28:01 INFO MemoryStore: ensureFreeSpace(71904) called with curMem=339411, maxMem=555755765 16/02/11 20:28:01 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 70.2 KB, free 529.6 MB) 16/02/11 20:28:01 INFO MemoryStore: ensureFreeSpace(25890) called with curMem=411315, maxMem=555755765 16/02/11 20:28:01 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 25.3 KB, free 529.6 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.247.253:53930 (size: 25.3 KB, free: 529.9 MB) 16/02/11 20:28:01 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:861 16/02/11 20:28:01 INFO DAGScheduler: Submitting 32 missing tasks from ResultStage 9 (MapPartitionsRDD[15] at map at VariationRDDFunctions.scala:132) 16/02/11 20:28:01 INFO YarnScheduler: Adding task set 9.0 with 32 tasks 16/02/11 20:28:01 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 192, amp-bdg-03.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 1.0 in stage 9.0 (TID 193, amp-bdg-64.amp, partition 1,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 2.0 in stage 9.0 (TID 194, amp-bdg-62.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 3.0 in stage 9.0 (TID 195, amp-bdg-28.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 4.0 in stage 9.0 (TID 196, amp-bdg-24.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 5.0 in stage 9.0 (TID 197, amp-bdg-49.amp, partition 5,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 6.0 in stage 9.0 (TID 198, amp-bdg-61.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 7.0 in stage 9.0 (TID 199, amp-bdg-34.amp, partition 7,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 8.0 in stage 9.0 (TID 200, amp-bdg-56.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 9.0 in stage 9.0 (TID 201, amp-bdg-44.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 10.0 in stage 9.0 (TID 202, amp-bdg-57.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 11.0 in stage 9.0 (TID 203, amp-bdg-04.amp, partition 11,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 12.0 in stage 9.0 (TID 204, amp-bdg-35.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 13.0 in stage 9.0 (TID 205, amp-bdg-36.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 14.0 in stage 9.0 (TID 206, amp-bdg-25.amp, partition 14,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 15.0 in stage 9.0 (TID 207, amp-bdg-49.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 16.0 in stage 9.0 (TID 208, amp-bdg-54.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 17.0 in stage 9.0 (TID 209, amp-bdg-02.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 18.0 in stage 9.0 (TID 210, amp-bdg-47.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 19.0 in stage 9.0 (TID 211, amp-bdg-20.amp, partition 19,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 20.0 in stage 9.0 (TID 212, amp-bdg-18.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 21.0 in stage 9.0 (TID 213, amp-bdg-27.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 22.0 in stage 9.0 (TID 214, amp-bdg-41.amp, partition 22,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 23.0 in stage 9.0 (TID 215, amp-bdg-15.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 24.0 in stage 9.0 (TID 216, amp-bdg-10.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 25.0 in stage 9.0 (TID 217, amp-bdg-22.amp, partition 25,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 26.0 in stage 9.0 (TID 218, amp-bdg-46.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 27.0 in stage 9.0 (TID 219, amp-bdg-29.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 28.0 in stage 9.0 (TID 220, amp-bdg-37.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 29.0 in stage 9.0 (TID 221, amp-bdg-51.amp, partition 29,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 30.0 in stage 9.0 (TID 222, amp-bdg-60.amp, partition 30,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 31.0 in stage 9.0 (TID 223, amp-bdg-05.amp, partition 31,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-05.amp:54334 (size: 25.3 KB, free: 339.4 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-44.amp:40528 (size: 25.3 KB, free: 339.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-61.amp:41089 (size: 25.3 KB, free: 339.9 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-49.amp:36486 (size: 25.3 KB, free: 340.2 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-49.amp:40827 (size: 25.3 KB, free: 338.4 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-41.amp:57788 (size: 25.3 KB, free: 339.2 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-03.amp:36085 (size: 25.3 KB, free: 340.8 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-62.amp:46717 (size: 25.3 KB, free: 341.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-47.amp:57360 (size: 25.3 KB, free: 339.8 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-64.amp:32917 (size: 25.3 KB, free: 340.6 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-22.amp:34751 (size: 25.3 KB, free: 339.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-28.amp:48326 (size: 25.3 KB, free: 339.2 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-34.amp:34029 (size: 25.3 KB, free: 340.0 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-57.amp:56334 (size: 25.3 KB, free: 340.0 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-25.amp:57820 (size: 25.3 KB, free: 342.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-56.amp:50396 (size: 25.3 KB, free: 339.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-60.amp:38207 (size: 25.3 KB, free: 338.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-27.amp:47425 (size: 25.3 KB, free: 339.4 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-54.amp:34279 (size: 25.3 KB, free: 339.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-02.amp:49749 (size: 25.3 KB, free: 338.9 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-04.amp:36661 (size: 25.3 KB, free: 339.8 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-36.amp:49131 (size: 25.3 KB, free: 341.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-15.amp:46865 (size: 25.3 KB, free: 339.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-24.amp:36201 (size: 25.3 KB, free: 340.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-29.amp:34585 (size: 25.3 KB, free: 339.9 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-18.amp:46764 (size: 25.3 KB, free: 341.3 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-10.amp:47502 (size: 25.3 KB, free: 340.0 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-37.amp:42720 (size: 25.3 KB, free: 342.2 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-51.amp:46478 (size: 25.3 KB, free: 340.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-35.amp:44956 (size: 25.3 KB, free: 339.6 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-46.amp:38842 (size: 25.3 KB, free: 340.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on amp-bdg-20.amp:58218 (size: 25.3 KB, free: 339.4 MB) 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-56.amp:41787 16/02/11 20:28:01 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 592 bytes 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-44.amp:34888 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-49.amp:51930 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-41.amp:58413 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-47.amp:56604 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-03.amp:46605 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-60.amp:55591 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-28.amp:33300 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-57.amp:53625 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-54.amp:34211 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-62.amp:50643 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-24.amp:41672 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-46.amp:59053 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-35.amp:60229 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-36.amp:48076 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-18.amp:37719 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-10.amp:47549 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-64.amp:52828 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-02.amp:51182 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-25.amp:44820 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-37.amp:43269 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-15.amp:35924 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-29.amp:47528 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-20.amp:57148 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-05.amp:53075 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-04.amp:41693 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-34.amp:51277 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-49.amp:60339 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-22.amp:38208 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-27.amp:46972 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-61.amp:52505 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to amp-bdg-51.amp:34878 16/02/11 20:28:01 WARN TaskSetManager: Lost task 10.0 in stage 9.0 (TID 202, amp-bdg-57.amp): com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:611) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:192) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217) at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:102) at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at scala.collection.convert.Wrappers$MutableMapWrapper.put(Wrappers.scala:217) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:135) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) ... 31 more 16/02/11 20:28:01 INFO TaskSetManager: Lost task 31.0 in stage 9.0 (TID 223) on executor amp-bdg-05.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 1] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 20.0 in stage 9.0 (TID 212) on executor amp-bdg-18.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 2] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 8.0 in stage 9.0 (TID 200) on executor amp-bdg-56.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 3] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 9.0 in stage 9.0 (TID 201) on executor amp-bdg-44.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 4] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 9.1 in stage 9.0 (TID 224, amp-bdg-44.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 8.1 in stage 9.0 (TID 225, amp-bdg-57.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 20.1 in stage 9.0 (TID 226, amp-bdg-35.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 31.1 in stage 9.0 (TID 227, amp-bdg-37.amp, partition 31,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 10.1 in stage 9.0 (TID 228, amp-bdg-18.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 28.0 in stage 9.0 (TID 220) on executor amp-bdg-37.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 5] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 17.0 in stage 9.0 (TID 209) on executor amp-bdg-02.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 6] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 13.0 in stage 9.0 (TID 205) on executor amp-bdg-36.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 7] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 26.0 in stage 9.0 (TID 218) on executor amp-bdg-46.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 8] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 26.1 in stage 9.0 (TID 229, amp-bdg-46.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 13.1 in stage 9.0 (TID 230, amp-bdg-05.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 17.1 in stage 9.0 (TID 231, amp-bdg-56.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 28.1 in stage 9.0 (TID 232, amp-bdg-02.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 12.0 in stage 9.0 (TID 204) on executor amp-bdg-35.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 9] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 12.1 in stage 9.0 (TID 233, amp-bdg-36.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 16.0 in stage 9.0 (TID 208) on executor amp-bdg-54.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 10] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 1.0 in stage 9.0 (TID 193) on executor amp-bdg-64.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 11] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 15.0 in stage 9.0 (TID 207) on executor amp-bdg-49.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 12] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 27.0 in stage 9.0 (TID 219) on executor amp-bdg-29.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 13] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 4.0 in stage 9.0 (TID 196) on executor amp-bdg-24.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 14] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 4.1 in stage 9.0 (TID 234, amp-bdg-03.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 27.1 in stage 9.0 (TID 235, amp-bdg-54.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 15.1 in stage 9.0 (TID 236, amp-bdg-64.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 1.1 in stage 9.0 (TID 237, amp-bdg-29.amp, partition 1,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 16.1 in stage 9.0 (TID 238, amp-bdg-24.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 0.0 in stage 9.0 (TID 192) on executor amp-bdg-03.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 15] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 0.1 in stage 9.0 (TID 239, amp-bdg-49.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 23.0 in stage 9.0 (TID 215) on executor amp-bdg-15.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 16] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 23.1 in stage 9.0 (TID 240, amp-bdg-10.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 24.0 in stage 9.0 (TID 216) on executor amp-bdg-10.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 17] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 30.0 in stage 9.0 (TID 222) on executor amp-bdg-60.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 18] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 18.0 in stage 9.0 (TID 210) on executor amp-bdg-47.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 19] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 18.1 in stage 9.0 (TID 241, amp-bdg-41.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 30.1 in stage 9.0 (TID 242, amp-bdg-60.amp, partition 30,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 24.1 in stage 9.0 (TID 243, amp-bdg-15.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 22.0 in stage 9.0 (TID 214) on executor amp-bdg-41.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 20] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 22.1 in stage 9.0 (TID 244, amp-bdg-47.amp, partition 22,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 3.0 in stage 9.0 (TID 195) on executor amp-bdg-28.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 21] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 2.0 in stage 9.0 (TID 194) on executor amp-bdg-62.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 22] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 2.1 in stage 9.0 (TID 245, amp-bdg-28.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 3.1 in stage 9.0 (TID 246, amp-bdg-62.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 7.0 in stage 9.0 (TID 199) on executor amp-bdg-34.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 23] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 14.0 in stage 9.0 (TID 206) on executor amp-bdg-25.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 24] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 14.1 in stage 9.0 (TID 247, amp-bdg-34.amp, partition 14,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 7.1 in stage 9.0 (TID 248, amp-bdg-25.amp, partition 7,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 11.0 in stage 9.0 (TID 203) on executor amp-bdg-04.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 25] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 11.1 in stage 9.0 (TID 249, amp-bdg-04.amp, partition 11,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 21.0 in stage 9.0 (TID 213) on executor amp-bdg-27.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 26] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 21.1 in stage 9.0 (TID 250, amp-bdg-61.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 6.0 in stage 9.0 (TID 198) on executor amp-bdg-61.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 27] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 6.1 in stage 9.0 (TID 251, amp-bdg-27.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 25.0 in stage 9.0 (TID 217) on executor amp-bdg-22.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 28] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 25.1 in stage 9.0 (TID 252, amp-bdg-22.amp, partition 25,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 19.0 in stage 9.0 (TID 211) on executor amp-bdg-20.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 29] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 19.1 in stage 9.0 (TID 253, amp-bdg-20.amp, partition 19,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 26.1 in stage 9.0 (TID 229) on executor amp-bdg-46.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 30] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 26.2 in stage 9.0 (TID 254, amp-bdg-46.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 13.1 in stage 9.0 (TID 230) on executor amp-bdg-05.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 31] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 17.1 in stage 9.0 (TID 231) on executor amp-bdg-56.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 32] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 17.2 in stage 9.0 (TID 255, amp-bdg-56.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 13.2 in stage 9.0 (TID 256, amp-bdg-05.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 28.1 in stage 9.0 (TID 232) on executor amp-bdg-02.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 33] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 27.1 in stage 9.0 (TID 235) on executor amp-bdg-54.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 34] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 27.2 in stage 9.0 (TID 257, amp-bdg-02.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 28.2 in stage 9.0 (TID 258, amp-bdg-54.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 16.1 in stage 9.0 (TID 238) on executor amp-bdg-24.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 35] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 16.2 in stage 9.0 (TID 259, amp-bdg-24.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 8.1 in stage 9.0 (TID 225) on executor amp-bdg-57.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 36] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 10.1 in stage 9.0 (TID 228) on executor amp-bdg-18.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 37] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 10.2 in stage 9.0 (TID 260, amp-bdg-49.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 0.1 in stage 9.0 (TID 239) on executor amp-bdg-49.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 38] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 0.2 in stage 9.0 (TID 261, amp-bdg-57.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 8.2 in stage 9.0 (TID 262, amp-bdg-18.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 4.1 in stage 9.0 (TID 234) on executor amp-bdg-03.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 39] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 4.2 in stage 9.0 (TID 263, amp-bdg-03.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 12.1 in stage 9.0 (TID 233) on executor amp-bdg-36.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 40] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 20.1 in stage 9.0 (TID 226) on executor amp-bdg-35.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 41] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 9.1 in stage 9.0 (TID 224) on executor amp-bdg-44.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 42] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 9.2 in stage 9.0 (TID 264, amp-bdg-44.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 20.2 in stage 9.0 (TID 265, amp-bdg-36.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 12.2 in stage 9.0 (TID 266, amp-bdg-35.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 2.1 in stage 9.0 (TID 245) on executor amp-bdg-28.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 43] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 2.2 in stage 9.0 (TID 267, amp-bdg-28.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 1.1 in stage 9.0 (TID 237) on executor amp-bdg-29.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 44] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 1.2 in stage 9.0 (TID 268, amp-bdg-29.amp, partition 1,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 15.1 in stage 9.0 (TID 236) on executor amp-bdg-64.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 45] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 15.2 in stage 9.0 (TID 269, amp-bdg-64.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 22.1 in stage 9.0 (TID 244) on executor amp-bdg-47.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 46] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 23.1 in stage 9.0 (TID 240) on executor amp-bdg-10.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 47] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 23.2 in stage 9.0 (TID 270, amp-bdg-47.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 22.2 in stage 9.0 (TID 271, amp-bdg-10.amp, partition 22,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 24.1 in stage 9.0 (TID 243) on executor amp-bdg-15.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 48] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 3.1 in stage 9.0 (TID 246) on executor amp-bdg-62.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 49] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 31.1 in stage 9.0 (TID 227) on executor amp-bdg-37.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 50] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 31.2 in stage 9.0 (TID 272, amp-bdg-15.amp, partition 31,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 3.2 in stage 9.0 (TID 273, amp-bdg-37.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 24.2 in stage 9.0 (TID 274, amp-bdg-62.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 18.1 in stage 9.0 (TID 241) on executor amp-bdg-41.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 51] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 18.2 in stage 9.0 (TID 275, amp-bdg-41.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 21.1 in stage 9.0 (TID 250) on executor amp-bdg-61.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 52] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 21.2 in stage 9.0 (TID 276, amp-bdg-61.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 5.0 in stage 9.0 (TID 197) on executor amp-bdg-49.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 53] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 5.1 in stage 9.0 (TID 277, amp-bdg-49.amp, partition 5,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 6.1 in stage 9.0 (TID 251) on executor amp-bdg-27.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 54] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 19.1 in stage 9.0 (TID 253) on executor amp-bdg-20.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 55] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 19.2 in stage 9.0 (TID 278, amp-bdg-20.amp, partition 19,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 6.2 in stage 9.0 (TID 279, amp-bdg-27.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 30.1 in stage 9.0 (TID 242) on executor amp-bdg-60.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 56] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 29.0 in stage 9.0 (TID 221) on executor amp-bdg-51.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 57] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 29.1 in stage 9.0 (TID 280, amp-bdg-60.amp, partition 29,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 30.2 in stage 9.0 (TID 281, amp-bdg-51.amp, partition 30,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 26.2 in stage 9.0 (TID 254) on executor amp-bdg-46.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 58] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 26.3 in stage 9.0 (TID 282, amp-bdg-46.amp, partition 26,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 14.1 in stage 9.0 (TID 247) on executor amp-bdg-34.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 59] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 14.2 in stage 9.0 (TID 283, amp-bdg-34.amp, partition 14,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 7.1 in stage 9.0 (TID 248) on executor amp-bdg-25.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 60] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 27.2 in stage 9.0 (TID 257) on executor amp-bdg-02.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 61] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 27.3 in stage 9.0 (TID 284, amp-bdg-05.amp, partition 27,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 7.2 in stage 9.0 (TID 285, amp-bdg-02.amp, partition 7,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 13.2 in stage 9.0 (TID 256) on executor amp-bdg-05.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 62] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 13.3 in stage 9.0 (TID 286, amp-bdg-25.amp, partition 13,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 25.1 in stage 9.0 (TID 252) on executor amp-bdg-22.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 63] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 25.2 in stage 9.0 (TID 287, amp-bdg-22.amp, partition 25,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 11.1 in stage 9.0 (TID 249) on executor amp-bdg-04.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 64] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 16.2 in stage 9.0 (TID 259) on executor amp-bdg-24.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 65] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 16.3 in stage 9.0 (TID 288, amp-bdg-24.amp, partition 16,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 11.2 in stage 9.0 (TID 289, amp-bdg-04.amp, partition 11,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 0.2 in stage 9.0 (TID 261) on executor amp-bdg-57.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 66] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 17.2 in stage 9.0 (TID 255) on executor amp-bdg-56.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 67] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 20.2 in stage 9.0 (TID 265) on executor amp-bdg-36.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 68] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 20.3 in stage 9.0 (TID 290, amp-bdg-56.amp, partition 20,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 17.3 in stage 9.0 (TID 291, amp-bdg-36.amp, partition 17,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 0.3 in stage 9.0 (TID 292, amp-bdg-57.amp, partition 0,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 4.2 in stage 9.0 (TID 263) on executor amp-bdg-03.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 69] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 4.3 in stage 9.0 (TID 293, amp-bdg-49.amp, partition 4,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 2.2 in stage 9.0 (TID 267) on executor amp-bdg-28.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 70] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 28.2 in stage 9.0 (TID 258) on executor amp-bdg-54.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 71] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 10.2 in stage 9.0 (TID 260) on executor amp-bdg-49.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 72] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 10.3 in stage 9.0 (TID 294, amp-bdg-28.amp, partition 10,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 28.3 in stage 9.0 (TID 295, amp-bdg-03.amp, partition 28,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 2.3 in stage 9.0 (TID 296, amp-bdg-54.amp, partition 2,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 15.2 in stage 9.0 (TID 269) on executor amp-bdg-64.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 73] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 15.3 in stage 9.0 (TID 297, amp-bdg-64.amp, partition 15,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 24.2 in stage 9.0 (TID 274) on executor amp-bdg-62.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 74] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 24.3 in stage 9.0 (TID 298, amp-bdg-62.amp, partition 24,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 8.2 in stage 9.0 (TID 262) on executor amp-bdg-18.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 75] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 9.2 in stage 9.0 (TID 264) on executor amp-bdg-44.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 76] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 12.2 in stage 9.0 (TID 266) on executor amp-bdg-35.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 77] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 12.3 in stage 9.0 (TID 299, amp-bdg-18.amp, partition 12,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 9.3 in stage 9.0 (TID 300, amp-bdg-35.amp, partition 9,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 8.3 in stage 9.0 (TID 301, amp-bdg-44.amp, partition 8,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 3.2 in stage 9.0 (TID 273) on executor amp-bdg-37.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 78] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 3.3 in stage 9.0 (TID 302, amp-bdg-61.amp, partition 3,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 18.2 in stage 9.0 (TID 275) on executor amp-bdg-41.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 79] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 18.3 in stage 9.0 (TID 303, amp-bdg-41.amp, partition 18,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 23.2 in stage 9.0 (TID 270) on executor amp-bdg-47.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 80] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 21.2 in stage 9.0 (TID 276) on executor amp-bdg-61.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 81] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 21.3 in stage 9.0 (TID 304, amp-bdg-27.amp, partition 21,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Starting task 23.3 in stage 9.0 (TID 305, amp-bdg-37.amp, partition 23,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 6.2 in stage 9.0 (TID 279) on executor amp-bdg-27.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 82] 16/02/11 20:28:01 INFO TaskSetManager: Starting task 6.3 in stage 9.0 (TID 306, amp-bdg-46.amp, partition 6,PROCESS_LOCAL, 1978 bytes) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 31.2 in stage 9.0 (TID 272) on executor amp-bdg-15.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 83] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 26.3 in stage 9.0 (TID 282) on executor amp-bdg-46.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 84] 16/02/11 20:28:01 ERROR TaskSetManager: Task 26 in stage 9.0 failed 4 times; aborting job 16/02/11 20:28:01 INFO TaskSetManager: Lost task 22.2 in stage 9.0 (TID 271) on executor amp-bdg-10.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 85] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 14.2 in stage 9.0 (TID 283) on executor amp-bdg-34.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 86] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 19.2 in stage 9.0 (TID 278) on executor amp-bdg-20.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 87] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 25.2 in stage 9.0 (TID 287) on executor amp-bdg-22.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 88] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 1.2 in stage 9.0 (TID 268) on executor amp-bdg-29.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 89] 16/02/11 20:28:01 INFO YarnScheduler: Cancelling stage 9 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on 192.168.247.253:53930 in memory (size: 3.1 KB, free: 530.0 MB) 16/02/11 20:28:01 INFO YarnScheduler: Stage 9 was cancelled 16/02/11 20:28:01 INFO DAGScheduler: ResultStage 9 (saveAsNewAPIHadoopFile at VariationRDDFunctions.scala:140) failed in 0.232 s 16/02/11 20:28:01 INFO TaskSetManager: Lost task 5.1 in stage 9.0 (TID 277) on executor amp-bdg-49.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 90] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 20.3 in stage 9.0 (TID 290) on executor amp-bdg-56.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 91] 16/02/11 20:28:01 INFO DAGScheduler: Job 3 failed: saveAsNewAPIHadoopFile at VariationRDDFunctions.scala:140, took 3.066610 s Command body threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 26 in stage 9.0 failed 4 times, most recent failure: Lost task 26.3 in stage 9.0 (TID 282, amp-bdg-46.amp): com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:611) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:192) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217) at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:102) at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at scala.collection.convert.Wrappers$MutableMapWrapper.put(Wrappers.scala:217) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:135) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) ... 31 more Driver stacktrace: 16/02/11 20:28:01 INFO TaskSetManager: Lost task 28.3 in stage 9.0 (TID 295) on executor amp-bdg-03.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 92] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 16.3 in stage 9.0 (TID 288) on executor amp-bdg-24.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 93] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 29.1 in stage 9.0 (TID 280) on executor amp-bdg-60.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 94] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 27.3 in stage 9.0 (TID 284) on executor amp-bdg-05.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 95] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 10.3 in stage 9.0 (TID 294) on executor amp-bdg-28.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 96] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 30.2 in stage 9.0 (TID 281) on executor amp-bdg-51.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 97] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 17.3 in stage 9.0 (TID 291) on executor amp-bdg-36.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 98] 16/02/11 20:28:01 INFO ADAM2Vcf: Overall Duration: 52.16 secs 16/02/11 20:28:01 INFO TaskSetManager: Lost task 11.2 in stage 9.0 (TID 289) on executor amp-bdg-04.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 99] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 7.2 in stage 9.0 (TID 285) on executor amp-bdg-02.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 100] Exception in thread "main" 16/02/11 20:28:01 INFO TaskSetManager: Lost task 15.3 in stage 9.0 (TID 297) on executor amp-bdg-64.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 101] org.apache.spark.SparkException: Job aborted due to stage failure: Task 26 in stage 9.0 failed 4 times, most recent failure: Lost task 26.3 in stage 9.0 (TID 282, amp-bdg-46.amp): com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:611) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:192) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217) at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:102) at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at scala.collection.convert.Wrappers$MutableMapWrapper.put(Wrappers.scala:217) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:135) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) ... 31 more Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1294) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1282) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1281) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1281) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1507) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1469) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1055) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:306) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:306) at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930) at org.bdgenomics.adam.rdd.variation.VariantContextRDDFunctions.saveAsVcf(VariationRDDFunctions.scala:140) at org.bdgenomics.adam.cli.ADAM2Vcf.run(ADAM2Vcf.scala:76) at org.bdgenomics.utils.cli.BDGSparkCommand$class.run(BDGCommand.scala:54) at org.bdgenomics.adam.cli.ADAM2Vcf.run(ADAM2Vcf.scala:58) at org.bdgenomics.adam.cli.ADAMMain.apply(ADAMMain.scala:136) at org.bdgenomics.adam.cli.ADAMMain$.main(ADAMMain.scala:76) at org.bdgenomics.adam.cli.ADAMMain.main(ADAMMain.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:626) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:699) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:611) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:192) at org.apache.spark.serializer.DeserializationStream.readValue(Serializer.scala:171) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:201) at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:198) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71) at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32) at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217) at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:102) at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297) at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at scala.collection.convert.Wrappers$MutableMapWrapper.put(Wrappers.scala:217) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:135) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648) at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605) ... 31 more 16/02/11 20:28:01 INFO TaskSetManager: Lost task 0.3 in stage 9.0 (TID 292) on executor amp-bdg-57.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 102] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-44.amp:40528 in memory (size: 3.1 KB, free: 339.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-36.amp:49131 in memory (size: 3.1 KB, free: 341.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-20.amp:58218 in memory (size: 3.1 KB, free: 339.4 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-22.amp:34751 in memory (size: 3.1 KB, free: 339.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-03.amp:36085 in memory (size: 3.1 KB, free: 340.8 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-24.amp:36201 in memory (size: 3.1 KB, free: 340.1 MB) 16/02/11 20:28:01 INFO SparkContext: Invoking stop() from shutdown hook 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-34.amp:34029 in memory (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-61.amp:41089 in memory (size: 3.1 KB, free: 339.9 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-62.amp:46717 in memory (size: 3.1 KB, free: 341.1 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 24.3 in stage 9.0 (TID 298) on executor amp-bdg-62.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 103] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 8.3 in stage 9.0 (TID 301) on executor amp-bdg-44.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 104] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-04.amp:36661 in memory (size: 3.1 KB, free: 339.8 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-05.amp:54334 in memory (size: 3.1 KB, free: 339.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-18.amp:46764 in memory (size: 3.1 KB, free: 341.3 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-35.amp:44956 in memory (size: 3.1 KB, free: 339.6 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 9.3 in stage 9.0 (TID 300) on executor amp-bdg-35.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 105] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-27.amp:47425 in memory (size: 3.1 KB, free: 339.4 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-57.amp:56334 in memory (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 13.3 in stage 9.0 (TID 286) on executor amp-bdg-25.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 106] 16/02/11 20:28:01 INFO TaskSetManager: Lost task 2.3 in stage 9.0 (TID 296) on executor amp-bdg-54.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 107] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-46.amp:38842 in memory (size: 3.1 KB, free: 340.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-02.amp:49749 in memory (size: 3.1 KB, free: 338.9 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 4.3 in stage 9.0 (TID 293) on executor amp-bdg-49.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 108] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-47.amp:57360 in memory (size: 3.1 KB, free: 339.8 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-51.amp:46478 in memory (size: 3.1 KB, free: 340.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-25.amp:57820 in memory (size: 3.1 KB, free: 342.1 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-28.amp:48326 in memory (size: 3.1 KB, free: 339.2 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 12.3 in stage 9.0 (TID 299) on executor amp-bdg-18.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 109] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-56.amp:50396 in memory (size: 3.1 KB, free: 339.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-54.amp:34279 in memory (size: 3.1 KB, free: 339.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-49.amp:36486 in memory (size: 3.1 KB, free: 340.2 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-60.amp:38207 in memory (size: 3.1 KB, free: 338.5 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-10.amp:47502 in memory (size: 3.1 KB, free: 340.0 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-29.amp:34585 in memory (size: 3.1 KB, free: 339.9 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-64.amp:32917 in memory (size: 3.1 KB, free: 340.7 MB) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-49.amp:40827 in memory (size: 3.1 KB, free: 338.4 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 3.3 in stage 9.0 (TID 302) on executor amp-bdg-61.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 110] 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-37.amp:42720 in memory (size: 3.1 KB, free: 342.2 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 6.3 in stage 9.0 (TID 306) on executor amp-bdg-46.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 111] 16/02/11 20:28:01 WARN TaskSetManager: Lost task 23.3 in stage 9.0 (TID 305, amp-bdg-37.amp): TaskKilled (killed intentionally) 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-15.amp:46865 in memory (size: 3.1 KB, free: 339.5 MB) 16/02/11 20:28:01 INFO TaskSetManager: Lost task 21.3 in stage 9.0 (TID 304) on executor amp-bdg-27.amp: com.esotericsoftware.kryo.KryoException (java.lang.NullPointerException Serialization trace: attributes (htsjdk.variant.variantcontext.CommonInfo) commonInfo (htsjdk.variant.variantcontext.VariantContext) vc (org.seqdoop.hadoop_bam.VariantContextWritable)) [duplicate 112] 16/02/11 20:28:01 WARN TaskSetManager: Lost task 18.3 in stage 9.0 (TID 303, amp-bdg-41.amp): TaskKilled (killed intentionally) 16/02/11 20:28:01 INFO YarnScheduler: Removed TaskSet 9.0, whose tasks have all completed, from pool 16/02/11 20:28:01 INFO BlockManagerInfo: Removed broadcast_6_piece0 on amp-bdg-41.amp:57788 in memory (size: 3.1 KB, free: 339.2 MB) 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 16/02/11 20:28:01 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 16/02/11 20:28:01 INFO SparkUI: Stopped Spark web UI at http://192.168.247.253:4041 16/02/11 20:28:01 INFO DAGScheduler: Stopping DAGScheduler 16/02/11 20:28:01 INFO YarnClientSchedulerBackend: Shutting down all executors 16/02/11 20:28:01 INFO YarnClientSchedulerBackend: Interrupting monitor thread 16/02/11 20:28:01 INFO YarnClientSchedulerBackend: Asking each executor to shut down 16/02/11 20:28:01 INFO YarnClientSchedulerBackend: Stopped 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-15.amp:35924]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-15.amp:35924] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-15.amp:35924 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-51.amp:34878]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-51.amp:34878] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-51.amp:34878 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-10.amp:47549]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-10.amp:47549] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-10.amp:47549 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-04.amp:41693]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-04.amp:41693] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-04.amp:41693 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-34.amp:51277]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-34.amp:51277] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-34.amp:51277 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-22.amp:38208]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-22.amp:38208] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-22.amp:38208 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-29.amp:47528]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-29.amp:47528] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-29.amp:47528 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-57.amp:53625]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-57.amp:53625] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-57.amp:53625 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-56.amp:41787]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-56.amp:41787] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-56.amp:41787 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-41.amp:58413]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-41.amp:58413] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-41.amp:58413 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-24.amp:41672]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-24.amp:41672] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-24.amp:41672 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-35.amp:60229]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-35.amp:60229] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-35.amp:60229 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-60.amp:55591]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-60.amp:55591] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-60.amp:55591 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-62.amp:50643]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-62.amp:50643] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-62.amp:50643 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-36.amp:48076]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-36.amp:48076] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-36.amp:48076 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-44.amp:34888]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-44.amp:34888] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-44.amp:34888 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-61.amp:52505]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-61.amp:52505] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-61.amp:52505 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-27.amp:46972]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-27.amp:46972] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-27.amp:46972 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-47.amp:56604]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-47.amp:56604] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-47.amp:56604 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-64.amp:52828]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-64.amp:52828] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-64.amp:52828 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-25.amp:44820]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-25.amp:44820] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-25.amp:44820 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-20.amp:57148]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-20.amp:57148] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-20.amp:57148 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-37.amp:43269]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-37.amp:43269] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-37.amp:43269 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-46.amp:59053]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-46.amp:59053] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-46.amp:59053 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-54.amp:34211]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-54.amp:34211] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-54.amp:34211 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-28.amp:33300]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-28.amp:33300] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-28.amp:33300 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-49.amp:60339]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-49.amp:60339] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-49.amp:60339 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-03.amp:46605]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-03.amp:46605] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-03.amp:46605 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-18.amp:37719]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-18.amp:37719] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-18.amp:37719 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-05.amp:53075]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-05.amp:53075] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-05.amp:53075 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-49.amp:51930]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-49.amp:51930] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-49.amp:51930 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@192.168.247.253:35234] <- [akka.tcp://sparkExecutor@amp-bdg-02.amp:51182]: Error [Shut down address: akka.tcp://sparkExecutor@amp-bdg-02.amp:51182] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://sparkExecutor@amp-bdg-02.amp:51182 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ] akka.event.Logging$Error$NoCause$ 16/02/11 20:28:01 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/02/11 20:28:01 INFO MemoryStore: MemoryStore cleared 16/02/11 20:28:01 INFO BlockManager: BlockManager stopped 16/02/11 20:28:01 INFO BlockManagerMaster: BlockManagerMaster stopped 16/02/11 20:28:01 INFO SparkContext: Successfully stopped SparkContext 16/02/11 20:28:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/02/11 20:28:01 INFO ShutdownHookManager: Shutdown hook called 16/02/11 20:28:01 INFO ShutdownHookManager: Deleting directory /tmp/spark-c279357b-23d1-47f5-aaf2-70fe449809a2