Merge pull request #340 from kjvarga/kvarga/pull-326-google-cloud-sto…

…rage

Google Cloud Storage support

CHANGES.md

@@ -1,3 +1,7 @@
+### 6.1.0
+
+* Support uploading files to Google Cloud Storage [#326](https://github.com/kjvarga/sitemap_generator/pull/326) and [#340](https://github.com/kjvarga/sitemap_generator/pull/340)
+
 ### 6.0.2
 
 * Resolve `BigDecimal.new is deprecated` warnings in Ruby 2.5 [#305](https://github.com/kjvarga/sitemap_generator/pull/305).

README.md

@@ -80,6 +80,12 @@ Successful ping of Bing
   + [Sitemaps with no Index File](#sitemaps-with-no-index-file)
   + [Upload Sitemaps to a Remote Host using Adapters](#upload-sitemaps-to-a-remote-host-using-adapters)
     - [Supported Adapters](#supported-adapters)
+      * [`SitemapGenerator::FileAdapter`](#sitemapgeneratorfileadapter)
+      * [`SitemapGenerator::FogAdapter`](#sitemapgeneratorfogadapter)
+      * [`SitemapGenerator::S3Adapter`](#sitemapgenerators3adapter)
+      * [`SitemapGenerator::AwsSdkAdapter`](#sitemapgeneratorawssdkadapter)
+      * [`SitemapGenerator::WaveAdapter`](#sitemapgeneratorwaveadapter)
+      * [`SitemapGenerator::GoogleStorageAdapter`](#sitemapgeneratorgooglestorageadapter)
     - [An Example of Using an Adapter](#an-example-of-using-an-adapter)
   + [Generating Multiple Sitemaps](#generating-multiple-sitemaps)
 * [Sitemap Configuration](#sitemap-configuration)
@@ -329,24 +335,24 @@ directory.
 
 #### Supported Adapters
 
-* `SitemapGenerator::FileAdapter`
+##### `SitemapGenerator::FileAdapter`
 
   Standard adapter, writes out to a file.
 
-* `SitemapGenerator::FogAdapter`
+##### `SitemapGenerator::FogAdapter`
 
   Uses `Fog::Storage` to upload to any service supported by Fog.
 
   You must `require 'fog'` in your sitemap config before using this adapter,
   or `require` another library that defines `Fog::Storage`.
 
-* `SitemapGenerator::S3Adapter`
+##### `SitemapGenerator::S3Adapter`
 
   Uses `Fog::Storage` to upload to Amazon S3 storage.
 
   You must `require 'fog-aws'` in your sitemap config before using this adapter.
 
-* `SitemapGenerator::AwsSdkAdapter`
+##### `SitemapGenerator::AwsSdkAdapter`
 
   Uses `Aws::S3::Resource` to upload to Amazon S3 storage.  Includes automatic detection of your AWS
   credentials using `Aws::Credentials`.
@@ -364,7 +370,7 @@ directory.
   )
   ```
 
-* `SitemapGenerator::WaveAdapter`
+##### `SitemapGenerator::WaveAdapter`
 
   Uses `CarrierWave::Uploader::Base` to upload to any service supported by CarrierWave, for example,
   Amazon S3, Rackspace Cloud Files, and MongoDB's GridF.
@@ -374,6 +380,31 @@ directory.
 
   Some documentation exists [on the wiki page][remote_hosts].
 
+##### `SitemapGenerator::GoogleStorageAdapter`
+
+  Uses [`Google::Cloud::Storage`][google_cloud_storage_gem] to upload to Google Cloud storage.
+
+  You must `require 'google/cloud/storage'` in your sitemap config before using this adapter.
+
+  An example of using this adapter in your sitemap configuration with options:
+
+  ```ruby
+  SitemapGenerator::Sitemap.adapter = SitemapGenerator::GoogleStorageAdapter.new(
+    credentials: 'path/to/keyfile.json',
+    project_id: 'google_account_project_id',
+    bucket: 'name_of_bucket'
+  )
+  ```
+  Also, inline with Google Authentication options, it can also pick credentials from environment variables. All [supported environment variables][google_cloud_storage_authentication] can be used, for example: `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_CREDENTIALS`.  An example of using this adapter with the environment variables is:
+
+  ```ruby
+  SitemapGenerator::Sitemap.adapter = SitemapGenerator::GoogleStorageAdapter.new(
+    bucket: 'name_of_bucket'
+  )
+  ```
+
+  All options other than the `:bucket` option are passed to the `Google::Cloud::Storage.new` initializer giving you maximum configurability.  See the [Google Cloud Storage initializer][google_cloud_storage_initializer] for supported options.
+
 #### An Example of Using an Adapter
 
 1. Please see [this wiki page][remote_hosts] for more information about setting up SitemapGenerator to upload to a
@@ -1120,3 +1151,6 @@ Copyright (c) Karl Varga released under the MIT license
 [iso_4217]:http://en.wikipedia.org/wiki/ISO_4217
 [media]:https://developers.google.com/webmasters/smartphone-sites/details
 [expires]:https://support.google.com/customsearch/answer/2631051?hl=en
+[google_cloud_storage_gem]:https://rubygems.org/gems/google-cloud-storage
+[google_cloud_storage_authentication]:https://googleapis.dev/ruby/google-cloud-storage/latest/file.AUTHENTICATION.html
+[google_cloud_storage_initializer]:https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-storage/lib/google/cloud/storage.rb

Rakefile

@@ -25,7 +25,7 @@ def gem_file;     "#{name}-#{version}.gem" end
 #
 
 desc "Build #{gem_file} into the pkg/ directory"
-task :build do
+task :build => [:prepare] do
   sh "mkdir -p pkg"
   sh "gem build #{gemspec_file}"
   sh "mv #{gem_file} pkg"
@@ -38,11 +38,11 @@ task :prepare do
 end
 
 desc "Create tag v#{version}, build the gem and push to Git"
-task :release => [:prepare, :build] do
+task :release => [:build] do
   unless `git branch` =~ /^\* master$/
     puts "You must be on the master branch to release!"
     exit!
   end
   sh "git tag v#{version}"
   sh "git push origin master --tags"
-end
+end

VERSION

@@ -1 +1 @@
-6.0.2
+6.1.0

lib/sitemap_generator.rb

@@ -7,14 +7,15 @@
 require 'sitemap_generator/sitemap_location'
 
 module SitemapGenerator
-  autoload(:Interpreter,   'sitemap_generator/interpreter')
-  autoload(:FileAdapter,   'sitemap_generator/adapters/file_adapter')
-  autoload(:S3Adapter,     'sitemap_generator/adapters/s3_adapter')
-  autoload(:AwsSdkAdapter, 'sitemap_generator/adapters/aws_sdk_adapter')
-  autoload(:WaveAdapter,   'sitemap_generator/adapters/wave_adapter')
-  autoload(:FogAdapter,    'sitemap_generator/adapters/fog_adapter')
-  autoload(:BigDecimal,    'sitemap_generator/core_ext/big_decimal')
-  autoload(:Numeric,       'sitemap_generator/core_ext/numeric')
+  autoload(:Interpreter,              'sitemap_generator/interpreter')
+  autoload(:FileAdapter,              'sitemap_generator/adapters/file_adapter')
+  autoload(:S3Adapter,                'sitemap_generator/adapters/s3_adapter')
+  autoload(:AwsSdkAdapter,            'sitemap_generator/adapters/aws_sdk_adapter')
+  autoload(:WaveAdapter,              'sitemap_generator/adapters/wave_adapter')
+  autoload(:FogAdapter,               'sitemap_generator/adapters/fog_adapter')
+  autoload(:GoogleStorageAdapter,     'sitemap_generator/adapters/google_storage_adapter')
+  autoload(:BigDecimal,               'sitemap_generator/core_ext/big_decimal')
+  autoload(:Numeric,                  'sitemap_generator/core_ext/numeric')
 
   SitemapError          = Class.new(StandardError)
   SitemapFullError      = Class.new(SitemapError)

lib/sitemap_generator/adapters/google_storage_adapter.rb

@@ -0,0 +1,37 @@
+if !defined?(Google::Cloud::Storage)
+  raise "Error: `Google::Cloud::Storage` is not defined.\n\n"\
+        "Please `require 'google/cloud/storage'` - or another library that defines this class."
+end
+
+module SitemapGenerator
+  # Class for uploading sitemaps to a Google Storage using `google-cloud-storage` gem.
+  class GoogleStorageAdapter
+    # Requires Google::Cloud::Storage to be defined.
+    #
+    # @param [Hash] opts Google::Cloud::Storage configuration options.
+    # @option :bucket [String] Required. Name of Google Storage Bucket where the file is to be uploaded.
+    #
+    # All options other than the `:bucket` option are passed to the `Google::Cloud::Storage.new`
+    # initializer.  See https://googleapis.dev/ruby/google-cloud-storage/latest/file.AUTHENTICATION.html
+    # for all the supported environment variables and https://github.com/googleapis/google-cloud-ruby/blob/master/google-cloud-storage/lib/google/cloud/storage.rb
+    # for supported options.
+    #
+    # Suggested Options:
+    # @option :credentials [String] Path to Google service account JSON file, or JSON contents.
+    # @option :project_id [String] Google Accounts project id where the storage bucket resides.
+    def initialize(opts = {})
+      opts = opts.clone
+      @bucket = opts.delete(:bucket)
+      @storage_options = opts
+    end
+
+    # Call with a SitemapLocation and string data
+    def write(location, raw_data)
+      SitemapGenerator::FileAdapter.new.write(location, raw_data)
+
+      storage = Google::Cloud::Storage.new(@storage_options)
+      bucket = storage.bucket(@bucket)
+      bucket.create_file(location.path, location.path_in_public, acl: 'public')
+    end
+  end
+end

sitemap_generator.gemspec

@@ -18,5 +18,6 @@ Gem::Specification.new do |s|
   s.add_development_dependency 'rake'
   s.add_development_dependency 'aws-sdk-core'
   s.add_development_dependency 'aws-sdk-s3'
+  s.add_development_dependency 'google-cloud-storage'
   s.files = Dir.glob('{lib,rails,templates}/**/*') + %w(CHANGES.md MIT-LICENSE README.md VERSION)
 end

spec/sitemap_generator/adapters/google_storage_adapter_spec.rb

@@ -0,0 +1,33 @@
+# encoding: UTF-8
+require 'spec_helper'
+require 'google/cloud/storage'
+
+describe SitemapGenerator::GoogleStorageAdapter do
+  subject(:adapter) { SitemapGenerator::GoogleStorageAdapter.new(options) }
+
+  let(:options) { { credentials: 'abc', project_id: 'project_id', bucket: 'bucket' } }
+
+  describe 'write' do
+    let(:location) { SitemapGenerator::SitemapLocation.new }
+
+    it 'writes the raw data to a file and then uploads that file to Google Storage' do
+      bucket = double(:bucket)
+      storage = double(:storage)
+      bucket_resource = double(:bucket_resource)
+      expect(Google::Cloud::Storage).to receive(:new).with(credentials: 'abc', project_id: 'project_id').and_return(storage)
+      expect(storage).to receive(:bucket).with('bucket').and_return(bucket_resource)
+      expect(location).to receive(:path_in_public).and_return('path_in_public')
+      expect(location).to receive(:path).and_return('path')
+      expect(bucket_resource).to receive(:create_file).with('path', 'path_in_public', acl: 'public').and_return(nil)
+      expect_any_instance_of(SitemapGenerator::FileAdapter).to receive(:write).with(location, 'raw_data')
+      adapter.write(location, 'raw_data')
+    end
+  end
+
+  describe '.new' do
+    it "doesn't modify the original options" do
+      adapter
+      expect(options.size).to be(3)
+    end
+  end
+end