Release 0.13.0

[busbey]

We're at 6+ months from a release, so let's get caught up.

I'll get a change list together and get together a test set by the end of next week. If someone else has more time and can get this going faster, feel free to take over.

[manolama]

ping @busbey I can likely handle this release if you like. Want me to go through the PRs? Might take a week or two for me to get it all done though.

[busbey]

Oh geez. I totally forgot about this.

It'd be great if you could take it over, @manolama. Thanks!

[manolama]

Thanks @busbey, would you mind emailing me the release process notes again please? Wanna make sure I get it right :)

[busbey]

Sure, it's on a wiki page now: https://github.com/brianfrankcooper/YCSB/wiki/Managing-a-Release

[jasontedor]

I have spare cycles for the first time in awhile and would like to work on getting #925 in for this release. I can do this this upcoming week.

[jasontedor]

I've pushed commits to #925, it just needs a review now.

[manolama]

Thanks @jasontedor I'll take a stab at a review for it tomorrow :)

[risdenk]

@manolama - An FYI I was trying the Github milestone features to track things that could go into 0.13.0.

https://github.com/brianfrankcooper/YCSB/milestone/1

[jasontedor]

Thanks @manolama, @risdenk reviewed so it's integrated now and will not hold up the 0.13.0 release.

[manolama]

@risdenk Thanks! I'll check the rest of the 13's and see if I can clean them up and merge em along with some others.

@jasontedor Thanks for the merge. Next time could you try squashing though so we don't have tons of little commits? And would you mind looking at #1008 ?

[jasontedor]

Sure, I'll squash next time, sorry about that! Maybe we should set the options on the repository to allow squash merges from the GitHub UI only? We have that set on the open source elastic/elasticsearch repository so I'm use to just mashing the merge button. 😄

I'll look at #1008 early next week.

[manolama]

Awesome, thanks! Yeah the squash merges only would be great! We'll have to see if one of the repo owners can do that.

[busbey]

how's the release coming? what are we blocked on?

[busbey]

ping?

[manolama]

Arg sorry, blocked on me getting off my duff. Working on it again now.

[manolama]

OK got some fixes ready for the issues @risdenk set for this release: https://github.com/brianfrankcooper/YCSB/milestone/1. Please take a look, then I'll start filling out the schedule.

[risdenk]

@manolama changes look good! Thanks for putting the effort in.

[busbey]

@manolama you mind if I push the changes that move master to 0.14-SNAPSHOT? I want to start working on some changes that I don't think should be included in the 0.13 release.

[manolama]

Go for it @busbey, thanks!

[manolama]

Here is the list of components/bindings that have changed in this release:

• [azuredocumentdb] Added support for Azure DocumentDB #838 [azuredocumentdb] - Added Azure Document DB binding
• [solr,solr6] Add support for Kerberized Solr #887 [solr] - Add support for Kerberos protected Solr installations
• [arangodb3] added ArangoDB3 binding (ArangoDB 3.1, Java driver 4.1.7) #903 [arangodb3] - Added binding for Arango DB V3
• [core, dbs] - Implement a checkstyle config for code consistency and enforce it on all bindings and components
• [elasticsearch,elasticsearch5] Add Elasticsearch 5.x binding #925 [elasticsearch5] - Add bindings for Elasticsearch Version 5.x
• Change the write policy to REPLACE_ONLY #937 [aerospike] - Change the write policy to REPLACE_ONLY
• [cloudspanner] Add YCSB Client for Google's Cloud Spanner. #939 [cloudspanner] - Added binding for Google's CloudSpanner
• A general "refresh" to the Accumulo binding #947 [accumulo] - Refresh of the binding
• [memcached] support binary protocol #965 [memcache] - Added support for the binary protocol
• Refresh Apache licence text #969 [distribution] - Update the APLV2 license text in the release
• [hbase12] Add HBase 1.2+ specific client that relies on the shaded client artifact provided by those versions. #970 [hbase12] - Added a binding for HBase version 1.2
• [core] Add a reset() method to the ByteIterator abstract and implemen… #999 [core] - Added a reset method to the ByteIterator abstract and implementations
• [geode] Update to apache-geode 1.2.0 release #995 [geode] - Updated binding to version 1.2
• [googlebigtable] Upgrading googlebigtable to the latest version. #978 [bigtable] - Updated to the V2 client in 0.9.7
• [core] Fix an issue where the threadid and threadCount were not passed #1000 [core] - Pass the thread ID and count to workload clients
• [core] Add a Fisher-Yates array shuffle to the Utils class. #1001 [core] - Added a Fisher-Yates shuffle implementation for randomizing arrays
• [core] Add an operation enum to the Workload class. This can eventually #1002 [core] - Added an operation enum to the Workload class for eventual simplification of operation selection
• Export totalHistogram for HdrHistogram measurement #983 [core] - Exporting the histogram details for HdrHistogram to mimic the standard histogram output
• [core] Use longs instead of ints to support larger key spaces. #911 [core] - Changed integers to longs for generators to enable wider ranges of keys
• Declarations should use Java collection interfaces such as "List" rather than specific implementation classes such as "LinkedList" #553 [core] - Using the generic collections interfaces for the DB API instead of HashMaps. NOTE Breaking change if you've implemented your own DB client. Simple fix though.
• [core] Add the NumericByteIterator to store byte encoded longs or dou… #1003 [core] - Added the NumericByteIterator to store encoded longs or double values
• [core] Add 'basicdb.count' as a config option to track and print #1005 [core] - Modified the basic db to be able to track the counts of operations for validation via basicdb.count
• [core] Add the RandomDiscreteTimestampGenerator to generate a range of #1004 [core] - Added the RandomDiscreteTimestampGenerator generator
• [s3] S3Client could not estimate read perfomance correctly (#994) #1010, [s3] Use AWS IOUtils to read object data #1021 [s3] - Fixed the s3 client to parse the entire object from the data store
• [googledatastore] Google Datastore to use default credentials. #984 [googledatastore] - Use the default credentials
• [core] Add the TimeSeriesWorkload class, a new type of workload that … #1008 [core] - Added a time series workload for testing time series focused data stores
• [jdbc] Use TEXT for fields instead of VARCHAR #910 [jdbc] - Changed the fields to TEXT instead of varchar
• [core] Change the RemainingFormatter to reflect non-plural times, fix… #1029 [core] - Changed the output of the time remaining to reflect non-plural times
• [hbase098] Fix #923 by interrupting the current thread. #1025 [hbase98] - Properly interrupt the client threads when interrupted downstream
• [core] Add a CONTRIBUTING.md file to fix #245. #1026 [core] - Added a mark down doc on contributing to YCSB
• [hbase10] Address #701 by mimicking the same locks from the HBase 0.9… #1028 [hbase10] - Use the same locking structure as the 0.98 HBase client for closer comparison
• CLI help lists several currently removed datastores #328 [core] - Clean up some old bindings in the CLI list

[busbey]

Is #705 far enough along for us to call jdk9 "supported" yet?

[risdenk]

@busbey regarding jdk9 support. YCSB compiles but I'd be very surprised if any of our supported bindings work with jdk9. For Solr I know we would have to update to 6.6.x for jdk9 support. I'm not sure out the other bindings.

[manolama]

0.13.0-RC1 Release available at: https://github.com/brianfrankcooper/YCSB/releases/tag/0.13.0-RC1

Please give it a test and let us know what works/doesn't work.
I also confirmed it builds and runs on Windows 10.

[twblamer]

I tried a fresh clone of the 1.13.0-RC1 tag (should be 0.13.0-RC1?) and found a couple display issues.

• No line break between "Command line:" string and "YCSB Client 0.13.0-RC1". On 0.12.0 it looked like this:

YCSB Client 0.12.0
Command line: -db com.yahoo.ycsb.BasicDB -s -P workloads/workloada -p basicdb.verbose=false -p measurementtype=hdrhistogram -load

• There is extra output I don't expect to see after the histogram measurements. What is "[INSERT], 0, 994.0" supposed to be? It looks like it could be some kind of verbose histogram output? This was not there in previous releases. I just want the avg/min/max and percentiles.

$ ./bin/ycsb load basic -s -P workloads/workloada -p basicdb.verbose=false
[WARN]  Running against a source checkout. In order to get our runtime dependencies we'll have to invoke Maven. Depending on the state of your system, this may take ~30-45 seconds
[DEBUG]  Running 'mvn -pl com.yahoo.ycsb:core -am package -DskipTests dependency:build-classpath -DincludeScope=compile -Dmdep.outputFilterFile=true'
java -cp /tmp/ycsb/core/conf:/tmp/ycsb/core/target/core-0.13.0-RC1.jar:/var/tmp/tblamer/maven/repository/org/apache/htrace/htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar:/var/tmp/tblamer/maven/repository/org/hdrhistogram/HdrHistogram/2.1.4/HdrHistogram-2.1.4.jar:/var/tmp/tblamer/maven/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.4/jackson-mapper-asl-1.9.4.jar:/var/tmp/tblamer/maven/repository/org/codehaus/jackson/jackson-core-asl/1.9.4/jackson-core-asl-1.9.4.jar com.yahoo.ycsb.Client -db com.yahoo.ycsb.BasicDB -s -P workloads/workloada -p basicdb.verbose=false -load
Command line: -db com.yahoo.ycsb.BasicDB -s -P workloads/workloada -p basicdb.verbose=false -loadYCSB Client 0.13.0-RC1

Loading workload...
Starting test.
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2017-10-16 15:00:03:146 0 sec: 316 operations; ∞ current ops/sec; est completion in 0 second [INSERT: Count=539, Max=514, Min=0, Avg=0.97, 90=0, 99=0, 99.9=4, 99.99=514]
2017-10-16 15:00:03:151 0 sec: 1000 operations; 31090.91 current ops/sec; [CLEANUP: Count=1, Max=2, Min=2, Avg=2, 90=2, 99=2, 99.9=2, 99.99=2] [INSERT: Count=461, Max=8, Min=0, Avg=0.02, 90=0, 99=0, 99.9=8, 99.99=8]
[OVERALL], RunTime(ms), 21
[OVERALL], Throughput(ops/sec), 47619.04761904762
[TOTAL_GCS_PS_Scavenge], Count, 0
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 0
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.0
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 0
[TOTAL_GC_TIME], Time(ms), 0
[TOTAL_GC_TIME_%], Time(%), 0.0
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 2.0
[CLEANUP], MinLatency(us), 2
[CLEANUP], MaxLatency(us), 2
[CLEANUP], 95thPercentileLatency(us), 2
[CLEANUP], 99thPercentileLatency(us), 2
[CLEANUP], 2, 1.0
[INSERT], Operations, 1000
[INSERT], AverageLatency(us), 0.533
[INSERT], MinLatency(us), 0
[INSERT], MaxLatency(us), 514
[INSERT], 95thPercentileLatency(us), 0
[INSERT], 99thPercentileLatency(us), 0
[INSERT], Return=OK, 1000
[INSERT], 0, 994.0
[INSERT], 1, 1.0
[INSERT], 2, 1.0
[INSERT], 4, 2.0
[INSERT], 8, 1.0
[INSERT], 514, 1.0

[manolama]

Arg, thanks for catching the 1.13 bit. I'll fix that up and take a look at the other issues.

[busbey]

Let's cut our losses and get this release out the door. We can warn users that this is a "experimental / untested" release or some such in the release notes, since essentially none of the bindings that changed have gotten a work out.

I'd propose we shortly get working on 0.14 and give bindings a mulligan for this release so far as supported / not status goes. (that is, any binding that changed in this release or a proposed 0.14 release needs to have someone vouch for it to continue to be called "supported", but no one loses that status due to not being tested in the 0.13 release)

anyone opposed?

[twblamer]

I'll try to kick the tires a bit more on the 0.13.0-staging tag just for my own satisfaction, but don't let that hold up anything.

[twblamer]

I built 0.13.0-staging from source and ran a few simple tests with basicdb and mongodb. The output issues I mentioned from 10/17 are gone. The measurement.histogram.verbose property works. No more complaints here.

[busbey]

thanks @twblamer! If anyone got some testing done for other bindings, put a note here and I'll try to keep track when I'm putting together the testing footprint for 0.14.

[busbey]

Proposed release notes in this gist:

https://gist.github.com/busbey/5f7033335a31484aebcea0843970c005

Note that in the release notes all of the "see #XXX" will actually link to the relevant issue/pr. Also note that the link to the CONTRIBUTING.md file will point to that file on the 0.13 release branch.

[busbey]

added a known issue for #877

[busbey]

release posted: https://github.com/brianfrankcooper/YCSB/releases/tag/0.13.0

[manolama]

Thanks for getting this out and sorry I dropped the ball!