Auto merge of #6661 - eregon:consistent-encoding-for-reading-files, r…

…=deivid-rodriguez

Use UTF-8 for reading files including Gemfile

### What was the end-user problem that led to this PR?

See #6660 and oracle/truffleruby#1410.

### What was your diagnosis of the problem?

The above issue details the problem.

### What is your fix for the problem, implemented in this PR?

To read the Gemfile and other files in Bundler with the default source encoding of Ruby, UTF-8, instead of the binary encoding which cannot interpret non-US-ASCII characters.

### Why did you choose this fix out of the possible options?

Because it's what Ruby does for other source files.

Fixes #6660.

(cherry picked from commit e71418e)

lib/bundler.rb

@@ -410,7 +410,7 @@ def sudo(str)
 
     def read_file(file)
       SharedHelpers.filesystem_access(file, :read) do
-        File.open(file, "rb", &:read)
+        File.open(file, "r:UTF-8", &:read)
       end
     end
 
@@ -431,7 +431,7 @@ def load_gemspec(file, validate = false)
 
     def load_gemspec_uncached(file, validate = false)
       path = Pathname.new(file)
-      contents = File.open(file, "r:UTF-8", &:read)
+      contents = read_file(file)
       spec = if contents.start_with?("---") # YAML header
         eval_yaml_gemspec(path, contents)
       else

lib/bundler/env.rb

@@ -61,7 +61,7 @@ def self.report(options = {})
     end
 
     def self.read_file(filename)
-      File.read(filename.to_s).strip
+      Bundler.read_file(filename.to_s).strip
     rescue Errno::ENOENT
       "<No #{filename} found>"
     rescue => e

spec/install/gemfile_spec.rb

@@ -110,4 +110,33 @@
       end
     end
   end
+
+  context "with a Gemfile containing non-US-ASCII characters" do
+    it "reads the Gemfile with the UTF-8 encoding by default" do
+      skip "Ruby 1.8 has no encodings" if RUBY_VERSION < "1.9"
+
+      install_gemfile <<-G
+        str = "Il était une fois ..."
+        puts "The source encoding is: " + str.encoding.name
+      G
+
+      expect(out).to include("The source encoding is: UTF-8")
+      expect(out).not_to include("The source encoding is: ASCII-8BIT")
+      expect(out).to include("Bundle complete!")
+    end
+
+    it "respects the magic encoding comment" do
+      skip "Ruby 1.8 has no encodings" if RUBY_VERSION < "1.9"
+
+      # NOTE: This works thanks to #eval interpreting the magic encoding comment
+      install_gemfile <<-G
+        # encoding: iso-8859-1
+        str = "Il #{"\xE9".b}tait une fois ..."
+        puts "The source encoding is: " + str.encoding.name
+      G
+
+      expect(out).to include("The source encoding is: ISO-8859-1")
+      expect(out).to include("Bundle complete!")
+    end
+  end
 end