Merge branch 'release/1.2.16'

leo-project · Oct 9, 2015 · ef10236 · ef10236
2 parents 4255340 + 1e640fe
commit ef10236
Show file tree

Hide file tree

Showing 16 changed files with 1,237 additions and 854 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,49 @@
 CHANGELOG
 =========
 
+1.2.16 (Octorber 9, 2015)
+===========================
+
+* Fixed Bugs
+    * [#422](https://github.com/leo-project/leofs/issues/422) ``leo_storage`` A LeoStorage node crashed due to executing consecutive the "ls" command
+    * [#423](https://github.com/leo-project/leofs/issues/423)``v1.2.11-`` ``leo_watchdog`` Did not clear combination of for errors - "cpu_util" and "load_avg_1m"
+
+* Used libraries
+    * leo project
+        * [leo_backend-db v1.1.13](https://github.com/leo-project/leo_backend_db/releases/tag/1.1.13)
+        * [leo_cache v0.6.5](https://github.com/leo-project/leo_cache/releases/tag/0.6.5)
+        * [leo_commons v1.1.5](https://github.com/leo-project/leo_commons/releases/tag/1.1.4)
+        * [leo_dcerl v0.4.0](https://github.com/leo-project/leo_dcerl/releases/tag/0.4.0)
+        * [leo_logger v1.1.11](https://github.com/leo-project/leo_logger/releases/tag/1.1.11)
+        * [leo_mcerl v0.6.0](https://github.com/leo-project/leo_mcerl/releases/tag/0.6.0)
+        * [leo_mq v1.3.16](https://github.com/leo-project/leo_mq/releases/tag/1.3.16)
+        * [leo_object_storage v1.2.10](https://github.com/leo-project/leo_object_storage/releases/tag/1.2.10)
+        * [leo_ordning_reda v1.1.3](https://github.com/leo-project/leo_ordning_reda/releases/tag/1.1.3)
+        * [leo_redundant_manager 1.9.19](https://github.com/leo-project/leo_redundant_manager/releases/tag/1.9.19)
+        * [leo_rpc v0.10.6](https://github.com/leo-project/leo_rpc/releases/tag/0.10.6)
+        * [leo_pod v0.6.6](https://github.com/leo-project/leo_pod/releases/tag/0.6.6)
+        * [leo_s3_libs v1.1.9](https://github.com/leo-project/leo_s3_libs/releases/tag/1.1.9)
+        * [leo_statistics v1.1.9](https://github.com/leo-project/leo_statistics/releases/tag/1.1.9)
+        * [leo_watchdog v0.12.0](https://github.com/leo-project/leo_watchdog/releases/tag/0.12.0)
+        * [savanna_agent v0.4.12](https://github.com/leo-project/savanna_agent/releases/tag/0.4.12)
+        * [savanna_commons v0.8.15](https://github.com/leo-project/savanna_commons/releases/tag/0.8.15)
+        * [erpcgen v0.2.3](https://github.com/leo-project/erpcgen/releases/tag/0.2.3)
+        * [nfs_rpc_server v0.2.3](https://github.com/leo-project/nfs_rpc_server/releases/tag/0.2.3)
+        * [leo_gateway v1.2.16](https://github.com/leo-project/leo_gateway/releases/tag/1.2.16)
+        * [leo_manager v1.2.16](https://github.com/leo-project/leo_manager/releases/tag/1.2.16)
+        * [leo_storage v1.2.16](https://github.com/leo-project/leo_storage/releases/tag/1.2.16)
+    * others
+        * [bitcask v2.0.0](https://github.com/lbasho/bitcask/releases/tag/2.0.0)
+        * [cowboy v1.0.1](https://github.com/leo-project/cowboy/releases/tag/for-leofs-1.2.11)
+        * [cowlib v1.0.0](https://github.com/extend/cowboy/releases/tag/1.0.0)
+        * [elarm v0.3.0](https://github.com/leo-project/elarm/releases/tag/0.3.0)
+        * [eleveldb v2.1.1](https://github.com/basho/eleveldb/releases/tag/2.1.1)
+        * [folsom v0.8.2-for-leofs](https://github.com/leo-project/folsom/releases/tag/0.8.2-for-leofs)
+        * [jiffy v0.13.3](https://github.com/davisp/jiffy/releases/tag/0.13.3)
+        * [lz4 v0.2.2](https://github.com/leo-project/erlang-lz4/releases/tag/0.2.2)
+        * [recon v0.8.5](https://github.com/ferd/recon/releases/tag/2.2.1)
+
+
 1.2.14 (September 11, 2015)
 ===========================
 

diff --git a/Makefile b/Makefile
@@ -3,11 +3,9 @@
 all: deps compile
 
 compile:
-	make patch
 	@./rebar compile
 deps:
 	@./rebar get-deps
-	make patch
 clean:
 	@./rebar clean
 	make -C pkg clean
@@ -17,9 +15,6 @@ xref:
 eunit:
 	@./rebar eunit
 
-patch:
-	(cd deps/leo_commons/ && git checkout refs/tags/1.1.4)
-
 generate:
 	rm -rf rel/leo_manager/leo_manager/
 	rm -rf rel/leo_storage/leo_storage/

diff --git a/README.md b/README.md
@@ -1,8 +1,9 @@
-LeoFS - The Lion of Storage Systems
-===================================
+# LeoFS - The Lion of Storage Systems
 
 ![LeoFS Logo](http://leo-project.net/leofs/docs/_static/leofs-logo-small.png)
 
+## Overview
+
 LeoFS is a highly available, distributed, eventually consistent object/blob store.
 If you are searching a storage system that is able to store huge amount and various kind of files such as photo, movie, log data and so on, LeoFS is suitable for that.
 
@@ -21,15 +22,26 @@ LeoFS is supporting the following features:
   * LeoFS's cluster can be viewed as ONE-HUGE storage. It consists of a set of loosely connected nodes.
   * We can build a global scale storage system with easy operations
 
+## Architecture
+
+![leofs-architecture-1](http://leo-project.net/leofs/docs/_images/leofs-architecture.0012.jpg)
+
+LeoFS consists of 3 applications - [LeoFS Storage](https://github.com/leo-project/leo_storage), [LeoFS Gateway](https://github.com/leo-project/leo_gateway) and [LeoFS Manager](https://github.com/leo-project/leo_manager) which depend on Erlang.
+
+[LeoFS Gateway](https://github.com/leo-project/leo_gateway) handles http-request and http-response from any clients when using REST-API OR S3-API. Also, it is already built in the object-cache mechanism (memory and disk cache).
+
+[LeoFS Storage](https://github.com/leo-project/leo_storage) handles GET, PUT and DELETE objects as well as metadata. Also, it has replicator, recoverer and queueing mechanism in order to keep running a storage node and realise eventual consistency.
+
+[LeoFS Manager](https://github.com/leo-project/leo_manager) always monitors LeoFS Gateway and LeoFS Storage nodes. The main monitoring status are Node status and RING’s checksum in order to realise to keep high availability and keep data consistency.
+
 We can access LeoFS server using <a target="_blank" href="http://www.leofs.org/docs/s3_client.html">S3 clients and S3 client libries of each programming language</a>.
 
-Slide
--------
+
+## Slide
 
 The presentation - <a href="https://www.slideshare.net/rakutentech/scaling-and-high-performance-storage-system-leofs" title="Scaling and High Performance Storage System: LeoFS" target="_blank">Scaling and High Performance Storage System: LeoFS</a>  was given at Erlang User Conference 2014 in Stockholm on June 2014
 
-GOALs
--------
+## GOALs
 
 * LeoFS aims to provide all of 3-HIGHs as follow:
   * HIGH Reliability
@@ -41,14 +53,12 @@ GOALs
      * A lower cost than other storage
      * Provide easy management and easy operation
 
-Further Reference
--------------------
+## Further Reference
 
 * <a target="_blank" href="http://leo-project.net/leofs/docs/">LeoFS Documentation</a>.
 
 
-Build LeoFS with LeoFS Packages
--------------------------------
+## Build LeoFS with LeoFS Packages
 
 LeoFS packages have been already provided on the Web. You're able to easily install LeoFS on your environments.
 
@@ -62,8 +72,7 @@ LeoFS packages have been already provided on the Web. You're able to easily inst
 <a target="_blank" href="http://leo-project.net/leofs/docs/getting_started.html">Here</a> is the installation manual.
 
 
-Build LeoFS From Source (For Developers)
-----------------------------------------
+## Build LeoFS From Source (For Developers)
 
 Here, we explain how to build LeoFS from source code.
 
@@ -204,60 +213,58 @@ $ ./leofs-adm status
 
 ```
 
-Build a LeoFS Cluster
----------------------
+## Build a LeoFS Cluster
 
 You can easily build a LeoFS cluster.
 
 Please refer <a target="_blank" href="http://www.leofs.org/docs/getting_started.html#quick-start-2-cluster">here</a>.
 
-Configure LeoFS
----------------
+## Configure LeoFS
 
 About the configuration of LeoFS, please refer <a target="_blank" href="http://www.leofs.org/docs/configuration.html">here</a>.
 
-Benchmarking
-------------
+## Benchmarking
 
 You can benchmark LeoFS with <a target="_blank" href="https://github.com/basho/basho_bench">Basho Bench</a>.
 
 <a target="_blank" href="http://www.leofs.org/docs/benchmark.html">Here</a> is a documentation to benchmark LeoFS.
 
 
-Integration Test
-----------------
+## Integration Test
 
 We're able to easily check LeoFS with <a target="_blank" href="https://github.com/leo-project/leofs_test2">leofs_test</a> whether LeoFS has issues or not before getting installed LeoFS in your dev/staging/production environment(s).
 
 
-Milestones
------------
+## Milestones
 
 * *DONE* - v1.0 (Nov 2013 - May 2014)
     * Multi Data Center Replication
     * Increase compatibility S3-APIs#5
         * Other bucket operations
 * *DONE* - v1.1
-    * NFS Support (Alpha)
+    * NFS Support *(alpha)*
     * Improve Web GUI Console (Option)
 * *DONE* - v1.2
-    * NFS Support (Beta-1)
+    * NFS Support *(beta)*
     * Watchdog
     * Auto-compaction
 * *On Going* - v1.4
-    * NFS Support (Beta-2)
-    * Improve performance of the list objects
-    * Improve Web GUI Console (Option)
+    * NFS Support *(stable)*
+        * Improve performance of the list objects
+        * NFS lock-option support
+    * Erasure Code
+    * Improve Web GUI console *(option)*
     * Improve compatibility S3-APIs#6
-        * Objects Expiration into the bucket
+        * AWS Signature v4 support
+        * Objects Expiration into a bucket
         * Versioning
 * v2.0
-    * NFS Support (for Production)
-    * QoS Support (Option)
+    * NFS Support
+    * QoS Support *(Option)*
     * Erasure Coding
     * Improve compatibility S3-APIs#7
 
 
 ## Sponsors
 
-LeoProject/LeoFS is sponsored by [Rakuten, Inc.](http://global.rakuten.com/corp/) and [Rakuten Institute of Technology](http://rit.rakuten.co.jp/).
+LeoProject/LeoFS is sponsored by [Rakuten, Inc.](http://global.rakuten.com/corp/) and supported by [Rakuten Institute of Technology](http://rit.rakuten.co.jp/).
diff --git a/leofs.mk b/leofs.mk
@@ -1 +1 @@
-LEOFS_VERSION=1.2.14
+LEOFS_VERSION=1.2.16
diff --git a/pkg/leo_gateway/files/leo_gateway.conf b/pkg/leo_gateway/files/leo_gateway.conf
@@ -98,6 +98,7 @@ bucket_prop_sync_interval = 300
 ## NFSd's the number of acceptors
 ## nfs.nfsd.acceptors = 128
 
+
 ## --------------------------------------------------------------------
 ## GATEWAY - Large Object
 ## --------------------------------------------------------------------
@@ -138,7 +139,7 @@ cache.cache_disc_capacity = 524288000
 cache.cache_disc_threshold_len = 1048576
 
 ## Directory for the disk cache data
-cache.cache_disc_dir_data    = /var/db/leo_gateway/cache/data
+cache.cache_disc_dir_data = /var/db/leo_gateway/cache/data
 
 ## Directory for the disk cache journal
 cache.cache_disc_dir_journal = /var/db/leo_gateway/cache/journal
@@ -181,10 +182,10 @@ watchdog.rex.threshold_mem_capacity = 33554432
 ## watchdog.cpu.is_enabled = true
 
 ## cpu - raised error times
-## watchdog.cpu.raised_error_times = 5
+## watchdog.cpu.raised_error_times = 3
 
 ## cpu - watch interval - default:5sec
-## watchdog.cpu.interval = 10
+## watchdog.cpu.interval = 5
 
 ## Threshold CPU load avg for 1min/5min - default:5.0
 ## watchdog.cpu.threshold_cpu_load_avg = 5.0
@@ -241,6 +242,7 @@ log.member_dir = /var/log/leo_gateway/ring
 ## Output log file(s) - ring
 log.ring_dir = /var/log/leo_gateway/ring
 
+
 ## --------------------------------------------------------------------
 ## GATEWAY - Other Directories
 ## --------------------------------------------------------------------

diff --git a/pkg/leo_storage/files/leo_storage.conf b/pkg/leo_storage/files/leo_storage.conf
@@ -59,6 +59,12 @@ obj_containers.num_of_containers = [8]
 ## - default:false
 ## object_storage.is_strict_check = false
 
+## Threshold of slow processing (msec) - default:1000(msec)
+## object_storage.threshold_of_slow_processing = 1000
+
+## Timeout of seeking metadatas per a metadata - default:10(msec)
+## seeking_timeout_per_metadata = 10
+
 
 ## --------------------------------------------------------------------
 ## STORAGE - Watchdog
@@ -80,7 +86,7 @@ watchdog.rex.threshold_mem_capacity = 33554432
 ##  Watchdog.CPU
 ##
 ## Is cpu-watchdog enabled - default:false
-## watchdog.cpu.is_enabled = true
+## watchdog.cpu.is_enabled = false
 
 ## cpu - raised error times
 ## watchdog.cpu.raised_error_times = 5
@@ -99,29 +105,30 @@ watchdog.rex.threshold_mem_capacity = 33554432
 ##  Watchdog.DISK
 ##
 ## Is disk-watchdog enabled - default:false
-## watchdog.disk.is_enabled = true
+## watchdog.disk.is_enabled = false
 
 ## disk - raised error times
 ## watchdog.disk.raised_error_times = 5
 
 ## disk - watch interval - default:1sec
 ## watchdog.disk.interval = 10
 
-## Threshold disk use% - defalut:85%
+## Threshold use(%) of a target disk's capacity - defalut:85%
 ## watchdog.disk.threshold_disk_use = 85
 
-## Threshold disk util% - defalut:100%
-## watchdog.disk.threshold_disk_util = 100
+## Threshold disk utilization(%) - defalut:90%
+## watchdog.disk.threshold_disk_util = 90
 
-## Threshold disk read kb/sec - defalut:262144(KB)
-## watchdog.disk.threshold_disk_rkb = 262144
+## Threshold disk read kb/sec - defalut:98304(KB) = 96MB
+## watchdog.disk.threshold_disk_rkb = 98304
 
-## Threshold disk write kb/sec - defalut:262144(KB)
-## watchdog.disk.threshold_disk_wkb = 262144
+## Threshold disk write kb/sec - defalut:98304(KB) = 96MB
+## watchdog.disk.threshold_disk_wkb = 98304
 
 ## disk target devices for checking disk utilization
 ## watchdog.disk.target_devices = []
 
+
 ##
 ##  Watchdog.Cluster
 ##
@@ -132,6 +139,19 @@ watchdog.rex.threshold_mem_capacity = 33554432
 ## watchdog.cluster.interval = 10
 
 
+##
+##  Watchdog.Error
+##
+## Is error-watchdog enabled - default:false
+## watchdog.error.is_enabled = false
+
+## error - watch interval - default:60sec
+## watchdog.error.interval = 60
+
+## error - threshold error count - default:100
+## watchdog.error.threshold_count = 100
+
+
 ## --------------------------------------------------------------------
 ## STORAGE - Autonomic Operation
 ## --------------------------------------------------------------------
@@ -292,6 +312,10 @@ leo_ordning_reda.temp_stacked_dir = "/var/db/leo_storage/ord_reda/"
 ## RPC-Client's size of connection buffer
 ## rpc.client.connection_buffer_size = 8
 
+## RPC-Client's max number of requests for reconnection to a remote-node
+## * 1..<integer>: specialized value
+## rpc.client.max_requests_for_reconnection = 64
+
 
 ## --------------------------------------------------------------------
 ## MANAGER - Mnesia