Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doubt meaning "return = 1" #314

Closed
cgara opened this issue Jun 20, 2015 · 17 comments
Closed

Doubt meaning "return = 1" #314

cgara opened this issue Jun 20, 2015 · 17 comments

Comments

@cgara
Copy link

cgara commented Jun 20, 2015

Hi,

The "return = 1" means response to the absence crud operation or error in crud operation?

In mongodb it is happening error by twenty percent of the procedure. I disregard the results?

Thank you

@allanbank
Copy link
Collaborator

For the MongoDB client it means that the operation did not complete successfully. For a read that means the document we are searching for was not found. For an insert or update it means the operation failed.

Can you give more details on:

  1. The workload you are running.
  2. The configuration of the MongoDB instance you are running.
  3. The configuration you are using with the client (command line)
  4. The distribution of the results you are seeing.

Thanks,
Rob.

@cgara
Copy link
Author

cgara commented Jun 25, 2015

The MongoDB Version 1:2.0.6-1.1 configuration default, no modified.
The workload is a workloada modification for 100% update and another 100% read.
50 executions failed

100% insert does not display error

Hardware 2Core processor 2GB RAM

Command line: -db com.yahoo.ycsb.db.MongoDbClient -p mongodb.url=mongodb://ip/27017 -p mongodb.database=ycsb -p mongodb.writeConcern=normal -threads 10 -P /YCSB/workloads/workload_update_10000 -s -t
Loading workload...
Starting test.
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
new database url = ip/27017/ycsb
0 sec: 0 operations;
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
mongo connection created with ip/27017/ycsb
10 sec: 2817 operations; 277,78 current ops/sec; [UPDATE AverageLatency(us)=32389,63]
20 sec: 6632 operations; 379,75 current ops/sec; [UPDATE AverageLatency(us)=26158,38]
28 sec: 10000 operations; 407,21 current ops/sec; [UPDATE AverageLatency(us)=23781,22]
[OVERALL], RunTime(ms), 28457.0
[OVERALL], Throughput(ops/sec), 351.407386583266
[UPDATE], Operations, 10000
[UPDATE], AverageLatency(us), 27113.0939
[UPDATE], MinLatency(us), 2728
[UPDATE], MaxLatency(us), 292203
[UPDATE], 95thPercentileLatency(ms), 49
[UPDATE], 99thPercentileLatency(ms), 74
[UPDATE], Return=0, 1746
[UPDATE], Return=1, 8254
...

@allanbank
Copy link
Collaborator

Would you mind posting the modified workloada?

Can you tell me the version of YCSB? Have you tried the master branch?

Just to confirm you ran the ycsb load prior to the actual run correct?

I have recently seen some 1 returns using an older version of MongoDB and the vanilla workloadd but that was < 1% of requests. You are seeing a majority of requests which is definitely not the expected behaviour.

@busbey
Copy link
Collaborator

busbey commented Jun 26, 2015

please try with the current release candidate from #266 rather than the
master branch.

On Thu, Jun 25, 2015 at 6:36 PM, allanbank notifications@github.com wrote:

Would you mind posting the modified workloada?

Can you tell me the version of YCSB? Have you tried the master branch?

Just to confirm you ran the ycsb load prior to the actual run correct?

I have recently seen some 1 returns using an older version of MongoDB and
the vanilla workloadd but that was < 1% of requests. You are seeing a
majority of requests which is definitely not the expected behaviour.


Reply to this email directly or view it on GitHub
#314 (comment)
.

Sean

@cgara
Copy link
Author

cgara commented Jun 26, 2015

Yes I ran the load prior
In Cassandra and Redis DBMS no problem

Workload modified

Copyright (c) 2010 Yahoo! Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you

may not use this file except in compliance with the License. You

may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or

implied. See the License for the specific language governing

permissions and limitations under the License. See accompanying

LICENSE file. # Yahoo! Cloud System Benchmark

Workload A: Update heavy workload

Application example: Session store recording recent actions

Read :1

Default data size: 1 KB records (10 fields, 100 bytes each, plus key)

Request distribution: zipfian

recordcount=10000
operationcount=10000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0
requestdistribution=zipfian

Copyright (c) 2010 Yahoo! Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you

may not use this file except in compliance with the License. You

may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or

implied. See the License for the specific language governing

permissions and limitations under the License. See accompanying

LICENSE file.

Yahoo! Cloud System Benchmark

Workload A: Update heavy workload

Application example: Session store recording recent actions

Update :1

Default data size: 1 KB records (10 fields, 100 bytes each, plus key)

Request distribution: zipfian

recordcount=10000
operationcount=10000
workload=com.yahoo.ycsb.workloads.CoreWorkload
readallfields=true
readproportion=0
updateproportion=1
scanproportion=0
insertproportion=0

requestdistribution=zipfian

@allanbank
Copy link
Collaborator

@cgara -

I have tried to reproduce the large number of non-zero return values and can't figure out what is going on.

Did you have a chance to test wit a 0.2.0-RC?

I did notice from your original post that the workloads are taking a lot longer for you than me. What is the ping time from your YCSB client to the MongoDB server?

Thanks,
Rob.

P.S. FYI - I am pretty sure that the small numbers of non-zero returns for workloadd are caused by #327.

@cgara
Copy link
Author

cgara commented Jul 7, 2015

the return 1 changes the result of runtime?

I not tested the 0.2.0-RC.

Actually performed these workloads to two DBMS is only mongo showed return 1.

I will measure the ping and inform you.

Thanks,

@cgara
Copy link
Author

cgara commented Jul 7, 2015

PING 192.168.0.104 (192.168.0.104) 56(84) bytes of data.
64 bytes from 192.168.0.104: icmp_seq=1 ttl=64 time=8.54 ms
64 bytes from 192.168.0.104: icmp_seq=2 ttl=64 time=6.40 ms
64 bytes from 192.168.0.104: icmp_seq=3 ttl=64 time=2.25 ms
64 bytes from 192.168.0.104: icmp_seq=4 ttl=64 time=1.36 ms
64 bytes from 192.168.0.104: icmp_seq=5 ttl=64 time=2.09 ms
64 bytes from 192.168.0.104: icmp_seq=6 ttl=64 time=3.07 ms
64 bytes from 192.168.0.104: icmp_seq=7 ttl=64 time=2.65 ms
64 bytes from 192.168.0.104: icmp_seq=8 ttl=64 time=2.21 ms
64 bytes from 192.168.0.104: icmp_seq=9 ttl=64 time=2.28 ms
64 bytes from 192.168.0.104: icmp_seq=10 ttl=64 time=1.96 ms
64 bytes from 192.168.0.104: icmp_seq=11 ttl=64 time=2.34 ms
64 bytes from 192.168.0.104: icmp_seq=12 ttl=64 time=2.39 ms
64 bytes from 192.168.0.104: icmp_seq=13 ttl=64 time=2.59 ms
64 bytes from 192.168.0.104: icmp_seq=14 ttl=64 time=6.44 ms
64 bytes from 192.168.0.104: icmp_seq=15 ttl=64 time=2.33 ms
64 bytes from 192.168.0.104: icmp_seq=16 ttl=64 time=1.51 ms

@allanbank
Copy link
Collaborator

Those ping times explain the lower throughput (mine are 0.1 ms to between my test machines) but not the non-zero return values.

I have made a number of significant changes to the MongoDB client for YCSB over the last few months including making sure that the return values mean something for each operation. I can tell from the output you posted that the version of YCSB you are using is older than those changes.

If you can test with the 0.2.0-RC3 release or even the current master I suspect the non-zero return values will disappear.

Rob.

@busbey
Copy link
Collaborator

busbey commented Jul 8, 2015

0.2.0 has a release now: https://github.com/brianfrankcooper/YCSB/releases/tag/0.2.0

If it makes things easier, that means you could grab the convenience binary specific to mongodb: https://github.com/brianfrankcooper/YCSB/releases/download/0.2.0/ycsb-mongodb-binding-0.2.0.tar.gz

@cgara
Copy link
Author

cgara commented Jul 9, 2015

This new version 0.2.0-RC3 showed no return 1.
The problem now is that it doubled all runtime values for my tests.
The [CLEANUP] is changing the runtime? Before did not [CLEANUP].

I'm testing three DBMS, Redis, MongoDB and Cassandra, now each DBMS is with a version of ycsb, this hinders trust the results?

Thanks

@allanbank
Copy link
Collaborator

When you say runtime values are you referring to the average latency of the requests or the wall-clock time for running the YCSB command?

I would not expect the latency for each operation to have changed very much but if you can provide details on which operation and what you saw before and now I will have a look.

The wall-clock time for short runs (a few 10's of seconds) is probably #316.

Rob.

@cgara
Copy link
Author

cgara commented Jul 9, 2015

before:
#[OVERALL], RunTime(ms), 28457.0
[OVERALL], Throughput(ops/sec), 351.407386583266
[UPDATE], Operations, 10000
[UPDATE], AverageLatency(us), 27113.0939
[UPDATE], MinLatency(us), 2728
[UPDATE], MaxLatency(us), 292203
[UPDATE], 95thPercentileLatency(ms), 49
[UPDATE], 99thPercentileLatency(ms), 74
[UPDATE], Return=0, 1746

[UPDATE], Return=1, 8254

now:
#[OVERALL], RunTime(ms), 45976.0
[OVERALL], Throughput(ops/sec), 217.5047851052723
[CLEANUP], Operations, 10
[CLEANUP], AverageLatency(us), 4614.8
[CLEANUP], MinLatency(us), 4
[CLEANUP], MaxLatency(us), 46086
[CLEANUP], 95thPercentileLatency(ms), 46
[CLEANUP], 99thPercentileLatency(ms), 46

@allanbank
Copy link
Collaborator

Right - That "[OVERALL], RunTime" delta is the ~20 second time I saw added to the wall clock time for #316. The problem is that instead of triggering the status thread to exit as soon as the operations complete the application is waiting for the status thread to wake up on its own and then exit.

If you look further down in the new results you should find the [UPDATE] throughput and latency numbers.

For MongoDB the CLEANUP is the time it takes to close the MongoClient. Only the last CLEANUP actually does the close operation which is why the MinLatency is 4 us and the max is 46 ms.

Rob.

@busbey
Copy link
Collaborator

busbey commented Aug 4, 2015

Did all your questions get answered @cgara?

@cgara
Copy link
Author

cgara commented Aug 4, 2015

Yes, all questions were answered.

I am performing exautivos tests with the tool, and I wonder if the runtime takes into account also the connection and disconnection time with the DBMS

@busbey
Copy link
Collaborator

busbey commented Aug 6, 2015

I believe that gets covered in the startup and cleanup times.

@busbey busbey closed this as completed Aug 6, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants