Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In memory shuffle #135

Open
wants to merge 2 commits into
base: vldb
Choose a base branch
from
Open

In memory shuffle #135

wants to merge 2 commits into from

Conversation

jegonzal
Copy link
Contributor

@jegonzal jegonzal commented Mar 2, 2014

No description provided.

ankurdave added a commit that referenced this pull request Mar 28, 2014
@ankurdave
In memory shuffle (cherry-picked from #135)
5ec645d
@mateiz mateiz closed this in dc96546 Apr 14, 2014
@ankurdave ankurdave reopened this Apr 14, 2014
ankurdave added a commit to ankurdave/graphx that referenced this pull request Apr 20, 2014
@ankurdave
In memory shuffle (cherry-picked from amplab#135)
5b65b1f 
(cherry picked from commit 5ec645d)

Conflicts:
	core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala
	core/src/main/scala/org/apache/spark/storage/BlockManager.scala
	graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
ankurdave added a commit to ankurdave/graphx that referenced this pull request Apr 20, 2014
@ankurdave
Revert "In memory shuffle (cherry-picked from amplab#135)"
8822037 
This reverts commit 5b65b1f.

Conflicts:
	graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
dcrankshaw pushed a commit to dcrankshaw/graphx that referenced this pull request Apr 28, 2014
@ankurdave @dcrankshaw
In memory shuffle (cherry-picked from amplab#135)
565fc6d
ankurdave added a commit to amplab/graphx2 that referenced this pull request Apr 30, 2014
@ankurdave
Revert "In memory shuffle (cherry-picked from amplab/graphx#135)"
bfe40f7 
This reverts commit 5ec645d.

Conflicts:
	graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
ankurdave added a commit to ankurdave/spark that referenced this pull request Aug 19, 2014
@ankurdave
GraphX benchmarking changes for VLDB
faa4073 
1. In-memory shuffle
2. Checking Iterator.hasNext
3. In-place/destructive updates in vertex replication
4. WikiPipelineBenchmark
5. K-Core
6. Dataflow (naive Spark) PageRank

Squashed commit of the following:

commit ad6590c
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Mon Apr 21 17:56:24 2014 -0700

    Add unpersist option to Pregel

commit 3bcaa2f
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Wed Apr 30 05:58:11 2014 +0000

    Added faster version of kcore and fixed compile issue with dataflow pagerank

commit 03f5d76
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Wed Apr 30 02:22:13 2014 +0000

    Added more connected components.

commit cfd209b
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Wed Apr 30 00:01:24 2014 +0000

    Initial CC dataflow implementation.

commit 255db45
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Tue Apr 29 19:38:26 2014 +0000

    Updated pregel logging.

commit 5032924
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Tue Apr 29 08:52:07 2014 +0000

    Added dataflow pagerank. Having issues with it and in-memory shuffle.

commit f483ca4
Merge: 8d22500 9e59642
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Tue Apr 29 00:05:49 2014 -0700

    Merge pull request apache#3 from dcrankshaw/osdi_with_kcore_for_merge

    Osdi with kcore for merge

commit 9e59642
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Mon Apr 28 20:51:24 2014 +0000

    Added more logging.

    Conflicts:
    	graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala

commit 76a6a54
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Apr 13 03:00:43 2014 +0000

    Fixed kcore. now works.

commit 1e924df
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Apr 13 00:06:40 2014 +0000

    Changed kcore filenames.

commit a641dc1
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sat Apr 12 01:59:17 2014 +0000

    Kcore debugging.

commit dbe5180
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Fri Apr 11 22:49:38 2014 +0000

    Fixed results of cherry-pick

commit 4ddc552
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Tue Jan 7 13:44:46 2014 -0800

    Added kcore implementation.

    Conflicts:
    	graph/src/main/scala/org/apache/spark/graph/Pregel.scala

commit 250199a
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Mon Apr 28 22:23:49 2014 +0000

    Minor logging changes.

commit 8d22500
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Apr 25 16:34:53 2014 -0700

    EdgePartition.size should be val

commit bb36cc8
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Apr 25 03:08:27 2014 -0700

    Set locality wait

commit ad4c874
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Sat Apr 19 20:08:57 2014 -0700

    In GraphLoader, coalesce to minEdgePartitions

commit 2898126
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Sat Apr 19 20:10:36 2014 -0700

    In Analytics, take PageRank numIter

commit 719b04a
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Sat Apr 19 20:09:24 2014 -0700

    Log current Pregel iteration

commit 8609184
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:15:44 2014 -0700

    Clean up WikiPipelineBenchmark

commit e8be08e
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:14:50 2014 -0700

    for-loop to while-loop in WikiArticle

commit 540c267
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:13:56 2014 -0700

    Make mutable RDDs optional

commit e27da44
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:12:58 2014 -0700

    Check for empty iterators

commit 5ec645d
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:10:41 2014 -0700

    In memory shuffle (cherry-picked from amplab/graphx#135)

commit f36e576
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:09:23 2014 -0700

    Log warning on partition recompute

commit c289e91
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 28 15:08:06 2014 -0700

    Untrack conf files

commit a61be7a
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Tue Mar 25 07:03:41 2014 +0000

    More timing logging and fixing the iterator issue.

commit 0645183
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Mon Mar 24 21:34:00 2014 +0000

    Fixed IO bugs.

commit a13f3b9
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Mon Mar 24 19:52:46 2014 +0000

    Fixed compile errors

commit e6fc93b
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Mon Mar 24 03:29:19 2014 +0000

    Updated pipeline benchmark to handle other systems.

commit 0a04cb9
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Mar 23 20:57:08 2014 +0000

    Renamed prepostprocess

commit a262b07
Merge: 470a950 3342751
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 21 11:39:02 2014 -0700

    Merge remote-tracking branch 'dcrankshaw-incubator-spark/gx-vldb-bench' into vldb

    Conflicts:
    	project/SparkBuild.scala

commit 470a950
Merge: e09139d 3407a8f
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Fri Mar 21 11:30:12 2014 -0700

    Merge branch 'mutable-rdd' into vldb

commit 3342751
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Mar 2 02:14:46 2014 +0000

    Fixed partitioning issues.

commit 3407a8f
Author: Ankur Dave <ankurdave@gmail.com>
Date:   Sat Mar 1 16:39:09 2014 -0800

    Try mutating old RDDs for delta updates

commit fad630f
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Wed Feb 26 02:33:01 2014 +0000

    more debugging. Seems like we have empty partitions.

commit b788bfb
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Tue Feb 25 11:10:16 2014 +0000

    added more debug logging.

commit 75b2da7
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Tue Feb 25 09:47:32 2014 +0000

    Fixed pipeline. Now seems to work well.

commit 7799b3e
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Mon Feb 24 22:31:47 2014 +0000

    updated slaves and conf.

commit 95e19ea
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Feb 23 06:19:21 2014 +0000

    Readadded prepost

commit da93886
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Feb 23 06:13:24 2014 +0000

    Stringify still not working. FML.

commit 6a4e492
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sun Feb 23 03:34:00 2014 +0000

    Added debugging. Issue with stringify.

commit 99d2713
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Sat Feb 22 04:40:45 2014 +0000

    Added new conf

commit 7a036cb
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Fri Feb 21 20:35:37 2014 -0800

    Fixed compilation issues.

commit d3bbfd0
Author: Dan Crankshaw <dscrankshaw@gmail.com>
Date:   Wed Feb 19 06:15:14 2014 +0000

    Beginning of new pipeline, not compiled yet.

commit 2983132
Author: root <root@ip-10-78-213-7.ec2.internal>
Date:   Wed Feb 19 01:23:15 2014 +0000

    Initial vldb benchmark commmit. Added slaves file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jegonzal @ankurdave