Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Unterminated string literal" when building using mono. #1167

Closed
rbutcher opened this issue Aug 17, 2016 · 6 comments
Closed

"Unterminated string literal" when building using mono. #1167

rbutcher opened this issue Aug 17, 2016 · 6 comments
Labels

Comments

@rbutcher
Copy link

rbutcher commented Aug 17, 2016

What You Are Seeing?

When building with mono, a string like "" or @"" causes the tokenizer to throw "Unterminated string literal"

This does not happen when building using the .net runtime.

What is Expected?

The tokenizer should not throw up on escaping the literal ""

What version of Cake are you using?

0.15.2+Branch.main.Sha.c2ca4f7ca51c59dc9c47c9830125b4e66cf14e7a

Are you running on a 32 or 64 bit system?

x64

What environment are you running on? Windows? Linux? Mac?

Windows 10 and Ubuntu 16.04
Mono 4.4.2

Are you running on a CI Server? If so, which one?

Yes and no.
Yes for Ubuntu 16.04 using Bamboo.
No for Windows 10.

How Did You Get This To Happen? (Steps to Reproduce)

run "build.ps1 -mono" on a script that includes this function

string CombinePath(params string[] args)
{
    var result = args.Aggregate((r, c) => System.IO.Path.Combine(r, c));
    if(IsRunningOnWindows()) return result;
    else return result.Replace("\\", "/"); // This line is the problem.
}

Output Log

Module directory does not exist.
Analyzing build script...
Analyzing M:/Workspaces/sharebase-web/build.cake...
Analyzing M:/Workspaces/sharebase-web/utilities.cake...
Processing build script...
Installing addins...
Package Cake.Npm has already been installed.
The addin Cake.Npm will reference Cake.Npm.dll.
Package Cake.Gulp has already been installed.
The addin Cake.Gulp will reference Cake.Gulp.dll.
Package Newtonsoft.Json has already been installed.
The addin Newtonsoft.Json will reference Newtonsoft.Json.dll.
Creating script session...
Adding reference to System.Data.dll...
Adding reference to System.Xml.dll...
Adding reference to System.Xml.Linq.dll...
Adding reference to Cake.Core.dll...
Adding reference to Cake.Common.dll...
Adding reference to Cake.exe...
Adding reference to Cake.Npm.dll...
Adding reference to Cake.Gulp.dll...
Adding reference to Newtonsoft.Json.dll...
Importing namespace Cake.Common...
Importing namespace Cake.Common.Build...
Importing namespace Cake.Common.Build.AppVeyor...
Importing namespace Cake.Common.Build.AppVeyor.Data...
Importing namespace Cake.Common.Build.BitbucketPipelines...
Importing namespace Cake.Common.Build.BitbucketPipelines.Data...
Importing namespace Cake.Common.Build.Bitrise...
Importing namespace Cake.Common.Build.Bitrise.Data...
Importing namespace Cake.Common.Build.ContinuaCI...
Importing namespace Cake.Common.Build.ContinuaCI.Data...
Importing namespace Cake.Common.Build.Jenkins...
Importing namespace Cake.Common.Build.Jenkins.Data...
Importing namespace Cake.Common.Build.TravisCI...
Importing namespace Cake.Common.Build.TravisCI.Data...
Importing namespace Cake.Common.Diagnostics...
Importing namespace Cake.Common.IO...
Importing namespace Cake.Common.IO.Paths...
Importing namespace Cake.Common.Net...
Importing namespace Cake.Common.Security...
Importing namespace Cake.Common.Solution...
Importing namespace Cake.Common.Solution.Project...
Importing namespace Cake.Common.Solution.Project.Properties...
Importing namespace Cake.Common.Solution.Project.XmlDoc...
Importing namespace Cake.Common.Text...
Importing namespace Cake.Common.Tools...
Importing namespace Cake.Common.Tools.Cake...
Importing namespace Cake.Common.Tools.Chocolatey...
Importing namespace Cake.Common.Tools.Chocolatey.ApiKey...
Importing namespace Cake.Common.Tools.Chocolatey.Config...
Importing namespace Cake.Common.Tools.Chocolatey.Features...
Importing namespace Cake.Common.Tools.Chocolatey.Install...
Importing namespace Cake.Common.Tools.Chocolatey.Pack...
Importing namespace Cake.Common.Tools.Chocolatey.Pin...
Importing namespace Cake.Common.Tools.Chocolatey.Push...
Importing namespace Cake.Common.Tools.Chocolatey.Sources...
Importing namespace Cake.Common.Tools.Chocolatey.Upgrade...
Importing namespace Cake.Common.Tools.DNU...
Importing namespace Cake.Common.Tools.DNU.Build...
Importing namespace Cake.Common.Tools.DNU.Pack...
Importing namespace Cake.Common.Tools.DNU.Restore...
Importing namespace Cake.Common.Tools.DotCover...
Importing namespace Cake.Common.Tools.DotCover.Analyse...
Importing namespace Cake.Common.Tools.DotCover.Cover...
Importing namespace Cake.Common.Tools.DotNetCore...
Importing namespace Cake.Common.Tools.DotNetCore.Build...
Importing namespace Cake.Common.Tools.DotNetCore.Execute...
Importing namespace Cake.Common.Tools.DotNetCore.Pack...
Importing namespace Cake.Common.Tools.DotNetCore.Publish...
Importing namespace Cake.Common.Tools.DotNetCore.Restore...
Importing namespace Cake.Common.Tools.DotNetCore.Run...
Importing namespace Cake.Common.Tools.DotNetCore.Test...
Importing namespace Cake.Common.Tools.DupFinder...
Importing namespace Cake.Common.Tools.Fixie...
Importing namespace Cake.Common.Tools.GitLink...
Importing namespace Cake.Common.Tools.GitReleaseManager...
Importing namespace Cake.Common.Tools.GitReleaseManager.AddAssets...
Importing namespace Cake.Common.Tools.GitReleaseManager.Close...
Importing namespace Cake.Common.Tools.GitReleaseManager.Create...
Importing namespace Cake.Common.Tools.GitReleaseManager.Export...
Importing namespace Cake.Common.Tools.GitReleaseManager.Publish...
Importing namespace Cake.Common.Tools.GitReleaseNotes...
Importing namespace Cake.Common.Tools.GitVersion...
Importing namespace Cake.Common.Tools.ILMerge...
Importing namespace Cake.Common.Tools.ILRepack...
Importing namespace Cake.Common.Tools.InspectCode...
Importing namespace Cake.Common.Tools.MSBuild...
Importing namespace Cake.Common.Tools.MSTest...
Importing namespace Cake.Common.Tools.NSIS...
Importing namespace Cake.Common.Tools.NuGet...
Importing namespace Cake.Common.Tools.NuGet.Install...
Importing namespace Cake.Common.Tools.NuGet.Pack...
Importing namespace Cake.Common.Tools.NuGet.Push...
Importing namespace Cake.Common.Tools.NuGet.Restore...
Importing namespace Cake.Common.Tools.NuGet.SetApiKey...
Importing namespace Cake.Common.Tools.NuGet.SetProxy...
Importing namespace Cake.Common.Tools.NuGet.Sources...
Importing namespace Cake.Common.Tools.NuGet.Update...
Importing namespace Cake.Common.Tools.NUnit...
Importing namespace Cake.Common.Tools.OctopusDeploy...
Importing namespace Cake.Common.Tools.OpenCover...
Importing namespace Cake.Common.Tools.ReportGenerator...
Importing namespace Cake.Common.Tools.ReportUnit...
Importing namespace Cake.Common.Tools.Roundhouse...
Importing namespace Cake.Common.Tools.SignTool...
Importing namespace Cake.Common.Tools.SpecFlow...
Importing namespace Cake.Common.Tools.SpecFlow.StepDefinitionReport...
Importing namespace Cake.Common.Tools.SpecFlow.TestExecutionReport...
Importing namespace Cake.Common.Tools.TextTransform...
Importing namespace Cake.Common.Tools.VSTest...
Importing namespace Cake.Common.Tools.WiX...
Importing namespace Cake.Common.Tools.WiX.Heat...
Importing namespace Cake.Common.Tools.XBuild...
Importing namespace Cake.Common.Tools.XUnit...
Importing namespace Cake.Common.Xml...
Importing namespace Cake.Core...
Importing namespace Cake.Core.Diagnostics...
Importing namespace Cake.Core.IO...
Importing namespace Cake.Core.Scripting...
Importing namespace Cake.Gulp...
Importing namespace Cake.Npm...
Importing namespace Newtonsoft.Json.Linq...
Importing namespace System...
Importing namespace System.Collections.Generic...
Importing namespace System.IO...
Importing namespace System.Linq...
Importing namespace System.Text...
Importing namespace System.Threading.Tasks...
Running build script...
Error: System.InvalidOperationException: Unterminated string literal.
   at Cake.Scripting.Mono.CodeGen.Parsing.ScriptTokenizer.ParseString()
   at Cake.Scripting.Mono.CodeGen.Parsing.ScriptTokenizer.GetNextToken()
   at Cake.Scripting.Mono.CodeGen.Parsing.ScriptParser.SkipBlock(ScriptTokenType start, ScriptTokenType stop)
   at Cake.Scripting.Mono.CodeGen.Parsing.ScriptParser.ParseScope()
   at Cake.Scripting.Mono.CodeGen.MonoScriptProcessor.ParseBlocks(Script script)
   at Cake.Scripting.Mono.CodeGen.MonoScriptProcessor.Process(Script script, IReadOnlyList`1& blocks)
   at Cake.Scripting.Mono.CodeGen.MonoCodeGenerator.Generate(Script script)
   at Cake.Scripting.Mono.MonoScriptSession.Execute(Script script)
   at Cake.Core.Scripting.ScriptRunner.Run(IScriptHost host, FilePath scriptPath, IDictionary`2 arguments)
   at Cake.Commands.BuildCommand.Execute(CakeOptions options)
   at Cake.CakeApplication.Run(CakeOptions options)
   at Cake.Program.Main()
@devlead
Copy link
Member

devlead commented Aug 17, 2016

Not investigated the specific issue yet, so just FYI
DirectoryPath has Combine(DirectoryPath path) and CombineWithFilePath(FilePath path) if you want to combine paths and it'll normalize slashes for you.

@devlead
Copy link
Member

devlead commented Aug 17, 2016

I can confirm I get the same error, not gone to depth to see if in Mono Scripting engine or Cake yet, meanwhile this workaround should work for you

string CombinePath(params string[] args)
{
    var result = args.Aggregate((r, c) => System.IO.Path.Combine(r, c));
    if(IsRunningOnWindows()) return result;
    else return result.Replace("\x005C", "/"); // using unicode escape sequence does not trigger this issue
}

Running

Information("{0}", CombinePath("src", "Cake", "Diagnostics", "Formatting"));

Will on Windows output

src\Cake\Diagnostics\Formatting

and on posix

src/Cake/Diagnostics/Formatting

@devlead devlead added the Bug label Aug 17, 2016
@rbutcher
Copy link
Author

rbutcher commented Aug 17, 2016

Awesome! New to cake didn't notice those methods. Are they in the DSL reference somewhere? The links point to the methods on DirectoryPath. Do all methods get projected into typeless function-like calls? Anyways, good to know the hacky implementation is un-needed.

A cursory investigation into the exception points to a bug in the ScriptTokenizer. I think that it is mis-reading "\" for escaping the trailing quotes causing the unterminated exception.

I think the ScriptTokenizer.ParseString() is causing the issue here. But, I haven't pinpointed it yet.

@devlead
Copy link
Member

devlead commented Aug 17, 2016

@rbutcher yeah think your on the right track there, confirmed mono scripting doesn't have the issue so definitely something in Cake causing it.

@rbutcher
Copy link
Author

100% sure that ScriptTokenizer.ParseString() is causing this.
It only takes into account the previous token not realizing that it has already been escaped by StringReader.

C# Interactive

> var reader = new StringReader("\"\\\""); ;
> Console.WriteLine((char)reader.Read()); ;
"
> Console.WriteLine((char)reader.Read()); ;
\
> Console.WriteLine((char)reader.Read()); ;
"
> 

The " " surround was to mimic reading the script in for tokenization.

  • StringReader is initialized with ""
  • The stream StringReader returns (var peek and read) is \ and "
  • Last token was \ and current token is "so the exception is thrown

Tough fix though. Still need to support the actual escaping of " if it appears in the middle of the string.

kekekeks added a commit to kekekeks/Perspex that referenced this issue May 26, 2017
@patriksvensson
Copy link
Member

Closed since #1645 been merged to develop. Feel free to reopen this if it's still a problem after 0.22.0 have been released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants