-
Notifications
You must be signed in to change notification settings - Fork 554
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch to Scala 2.13 #25190
Switch to Scala 2.13 #25190
Conversation
|
||
builder.result | ||
} | ||
// def distinctBy[S](hash: T => S)(implicit cbf: CanBuildFrom[C[T], T, C[T]]): C[T] = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't know how to fix this so I commented it out for now to get the next errors
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is code where Frontend devs are extending the functionality of Scala's collection classes. Unfortunately there was a big improving refactor in Scala colllections going from Scala 2.12 to 2.13, and consequently the code would no longer be extended in the same way. See the migration guide here:
https://docs.scala-lang.org/overviews/core/collections-migration-213.html
In particular, the implicit cbf: CanBuildFrom
stuff went away:
Transformation methods no longer have an implicit
CanBuildFrom
parameter. This makes the library easier to understand (in source code, Scaladoc, and IDE code completion). It also makes compiling user code more efficient.
Extending Scala collections is now done in a different way - see the guide here : https://docs.scala-lang.org/overviews/core/custom-collection-operations.html - eg the part where a new .sumBy()
method is added to existing collections. Another example of this is in https://github.com/rtyley/scala-collection-plus/blob/master/collection-plus/src/main/scala/com/madgag/scala/collection/decorators/package.scala !
What we want to do here, I'm not sure - it would depend on how often and where these methods are called, if we still want to implement them - maybe there are even better methods of providing this functionality?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe there are even better methods of providing this functionality?
LOL, yes there is! Turns out there is a brand new method in Scala 2.13 that is called the exact same name - distinctBy()
- and is called in exactly the same way. This means that all of Seq2Distinct
can be deleted with Scala 2.13 ✨
In fact, it looks like we can delete this entire file - see #25226
c9ae1e3
to
e6e3621
Compare
8b5abeb
to
7bc1127
Compare
In the course of upgrading the Frontend codebase to Scala 2.13 (see #25190) we came across the `implicits.Collections` class, and had trouble upgrading it: #25190 (comment) It turns out that we will be able to _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we're removing superfluous use of the `implicits.Collections` class in this pre-upgrade PR. `implicits.Collections` is currently used to add two additional methods to standard Scala collections: * `distinctBy()` - introduced November 2012 with #263. Perhaps surprisingly, Scala 2.13 has a new built-in method that is called the same thing and is called the same way! So the Frontend implementation can be deleted when the Scala 2.13 upgrade occurs. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy[B](f:A=%3EB):C * `safeDropWhile()` - introduced January 2015 with #7706 to handle a problem with the Scala 2.11 compiler: scala/bug#7529 The issue is no longer present in Scala 2.12, so the method could have been removed when the upgrade to Scala 2.12 was performed in November 2017 with #18218 This pre-upgrade PR deletes the unnecessary `safeDropWhile()` method, and removes several unnecessary references to `implicits.Collections`.
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
7bc1127
to
74aa1bc
Compare
In the course of upgrading the Frontend codebase to Scala 2.13 (see #25190) we came across the `implicits.Collections` class, and had trouble upgrading it: #25190 (comment) It turns out that we will be able to _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we're removing superfluous use of the `implicits.Collections` class in this pre-upgrade PR. `implicits.Collections` is currently used to add two additional methods to standard Scala collections: * `distinctBy()` - introduced November 2012 with #263. Perhaps surprisingly, Scala 2.13 has a new built-in method that is called the same thing and is called the same way! So the Frontend implementation can be deleted when the Scala 2.13 upgrade occurs. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy[B](f:A=%3EB):C * `safeDropWhile()` - introduced January 2015 with #7706 to handle a problem with the Scala 2.11 compiler: scala/bug#7529 The issue is no longer present in Scala 2.12, so the method could have been removed when the upgrade to Scala 2.12 was performed in November 2017 with #18218 This pre-upgrade PR deletes the unnecessary `safeDropWhile()` method, and removes several unnecessary references to `implicits.Collections`.
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
74aa1bc
to
9e05a7f
Compare
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
9e05a7f
to
cb96cd0
Compare
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
cb96cd0
to
a92eaf0
Compare
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
a92eaf0
to
4bddfbb
Compare
`LiveBlogCurrentPage.findPageWithBlock()` is responsible for making sure that when a user clicks on a live-blog permalink to a **block**, eg: https://www.theguardian.com/politics/live/2022/aug/01/tory-leadership-race-rishi-sunak-lizz-truss-vote-keir-starmer-uk-politics-live?page=with:block-62e7e89d8f08730a1f7f5579#block-62e7e89d8f08730a1f7f5579 ...the user is taken to the correct 'page' of the live blog - page 3 of 5 in the example above, for instance. The code was was first introduced in January 2016 with #11700, subsequently updated with #13566, etc. The existing code compiles without warnings under Scala 2.12, but the Scala 2.13 compiler gives a warning that not _all_ conceivable cases in the pattern-match are handled: ``` frontend/common/app/model/LiveBlogCurrentPage.scala:190:12: match may not be exhaustive. [warn] It would fail on the following inputs: List(_), Nil [warn] .map { [warn] ^ ``` In this particular case, the warning's unnecessary - the preceding `.sliding(3)`: https://github.com/guardian/frontend/blob/f90c8a58e6f0941c96392d14fe27e61a4dbaf25b/common/app/model/LiveBlogCurrentPage.scala#L188 ...means that the List supplied to the case match expression will _always_ have 3 elements, and that's precisely the case that's handled - but there's no way for the compiler to know that - so we as devs need to do _something_ to make the warning go away! See also: * The Scala 2.13 upgrade PR: #25190
`LiveBlogCurrentPage.findPageWithBlock()` is responsible for making sure that when a user clicks on a live-blog permalink to a **block**, eg: https://www.theguardian.com/politics/live/2022/aug/01/tory-leadership-race-rishi-sunak-lizz-truss-vote-keir-starmer-uk-politics-live?page=with:block-62e7e89d8f08730a1f7f5579#block-62e7e89d8f08730a1f7f5579 ...the user is taken to the correct 'page' of the live blog - page 3 of 5 in the example above, for instance. The code was was first introduced in January 2016 with #11700, subsequently updated with #13566, etc. The existing code compiles without warnings under Scala 2.12, but the Scala 2.13 compiler gives a warning that not _all_ conceivable cases in the pattern-match are handled: ``` frontend/common/app/model/LiveBlogCurrentPage.scala:190:12: match may not be exhaustive. [warn] It would fail on the following inputs: List(_), Nil [warn] .map { [warn] ^ ``` In this particular case, the warning's unnecessary - the preceding `.sliding(3)`: https://github.com/guardian/frontend/blob/f90c8a58e6f0941c96392d14fe27e61a4dbaf25b/common/app/model/LiveBlogCurrentPage.scala#L188 ...means that the `List` supplied to the case match expression will _always_ have 3 elements, and that's precisely the case that is handled - but there's no way for the compiler to know that - so we as devs need to do _something_ to make the warning go away! Some options: * Use the @unchecked annotation (see https://www.scala-lang.org/api/2.12.7/scala/unchecked.html) This is an option of last resort - turning off compiler checks leaves us exposed to runtime errors and is generally a bad idea! * Use `collect()` rather than `map()` - this accepts a partial function, rather than a total one, so the compiler error will go away. https://www.scala-lang.org/api/2.12.7/scala/collection/immutable/Seq.html#collectFirst[B](pf:PartialFunction[A,B]):Option[B] * Refactor so that the code no longer needs to assume that a `List` type (which can have any length) has length `3`. In the end I decided to go with the refactor, because there were a few things about the existing code that could be tweaked: * Unnecessary work: The method was creating a `LiveBlogCurrentPage` for _every_ page in the liveblog, even though it only ever needed one (the single page that the block actually exists on!), and would eventually throw away the rest. * The logic around padding the front & end of the `pages` List with two `None` entries, to allow extracting the `newer` & `older` pages, totally worked but added an extra step to the code (and therefore a bit of complexity for humans to understand: "what are endedPages?") and could be replaced by just incrementing/decrementing the page index for the one `LiveBlogCurrentPage` we're now creating. * Various variables could be inlined to the point of creation on the `LiveBlogCurrentPage` case class. By using named parameters in constructing the case class, no clarity is lost, and the code is more concise. See also: * The Scala 2.13 upgrade PR: #25190
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
4bddfbb
to
3b418b3
Compare
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. https://www.scala-lang.org/api/2.13.x/scala/collection/Seq.html#distinctBy%5BB%5D(f:A=%3EB):C
7f4e3a4
to
4cb721e
Compare
@@ -15,7 +16,7 @@ import scala.concurrent.Future | |||
|
|||
private object TestModel | |||
|
|||
class ModelOrResultTest extends AnyFlatSpec with Matchers with WithTestExecutionContext { | |||
class ModelOrResultTest extends AnyFlatSpec with Matchers with WithTestExecutionContext with EitherValues { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"ScalaTest's EitherValues
trait provides an implicit conversion that adds left.value
and right.value
methods to Either
, which will return the selected value of the Either
if defined": https://www.scalatest.org/user_guide/using_EitherValues
@@ -20,7 +20,7 @@ class CompetitionStage(competitions: Seq[Competition]) { | |||
competition: Competition, | |||
orderings: Map[String, List[ZonedDateTime]] = Map.empty, | |||
): List[CompetitionStageLike] = { | |||
val stagesWithMatches = competition.matches.toList.groupBy(_.stage).toList | |||
val stagesWithMatches = competition.matches.groupBy(_.stage).toList.sortBy(_._1.stageNumber.toIntOption) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code previously relied on groupBy
producing a Map that had the keys in the 'natural' order of stage 1, stage 2 - but didn't have any sorting to enforce that. Maps are unordered collections, and so there was never a guarantee that Scala 2.13 will put Map keys in the same order as Scala 2.12.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Massive effort! Well done on both of you.
@@ -52,6 +52,7 @@ object Dependencies { | |||
val rome = "rome" % "rome" % romeVersion | |||
val romeModules = "org.rometools" % "rome-modules" % romeVersion | |||
val scalaCheck = "org.scalacheck" %% "scalacheck" % "1.16.0" % Test | |||
val scalaCollectionPlus = "com.madgag" %% "scala-collection-plus" % "0.11" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Who’s this madgag fellow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know but I have often wondered myself 😆
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I purchased madgag.com back in 2003 (as the domain for a comedy listings site I was trying to create as a side-project - I didn't realise there was already the Chortle website doing the same thing!). As the website was for getting to comedy gigs in obscure places, I wanted it to be accessible by mobile phone, which at the time was at the very edge of what phones could do - the iPhone wouldn't launch for another 4 years. Phones had screens, but not touchscreens, just numeric keypads, like this:
At the time the fastest way of typing text - eg a url - into a phone was using T9 (predictive text), so I wanted my domain name to be not only comedy-related, but easy to type in - that meant finding a name that only used the first letters of the numeric keypad (A
, D
, G
, J
, M
, P
, T
, W
). I came up with madgag
, which is very easy to type using a T9 keypad...!
Although I had a fun time writing the code for madgag.com, it was never fully developed, and I turned it off a few years later. However, in writing the site, I happened to learn a bunch of stuff about databases and object-relational-mapping, which - guess what - came up when I had my phone interview at the Guardian! So the interview went well, and helped me get the job here!
@@ -32,7 +32,7 @@ object ProjectSettings { | |||
Compile / packageDoc / publishArtifact := false, | |||
Compile / doc / sources := Seq.empty, | |||
Compile / doc := target.map(_ / "none").value, | |||
scalaVersion := "2.12.13", | |||
scalaVersion := "2.13.8", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎉
Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
…collection.JavaConverters._ `scala.collection.JavaConverters` is deprecated. Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
…apValues` Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
In the course of upgrading the Frontend codebase to Scala 2.13 we came across the `implicits.Collections` class, and had trouble updating it for Scala 2.13: #25190 (comment) It turns out that we can _delete_ this class with the Scala 2.13 upgrade, rather than have to update it, which is nice! To reduce the size of the Scala 2.13 upgrade PR, we first removed every use of `implicits.Collections` that we could while still under Scala 2.12: #25226 This commit gets rid of `implicits.Collections` altogether. At this point, all `implicits.Collections` has in it is the code to add a `distinctBy()` method onto standard Scala collections - this was introduced November 2012 with #263, but Scala 2.13 has a new built-in method that works the exact same way! So we can delete the Frontend implementation. Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
> TupleNZipped has been replaced with LazyZipN: > (xs, ys).zipped -> xs.lazyZip(ys) https://docs.scala-lang.org/overviews/core/collections-migration-213.html#breaking-changes-with-old-syntax-still-supported scala/collection-strawman#223 Although .zipped is only deprecated, we have deprecation warnings set as fatal errors in the frontend project. Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
Great explanation in the Ophan repo Scala 2.13 upgrade about why this change is needed: guardian/ophan@e9ff516 Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
Method right in class Either is deprecated (since 2.13.0): Either is now right-biased and we can use methods directly on it Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
We are trying to call `toIntOption`, a method of `StringOps` on a `String` so we're getting the error: ``` value toIntOption is not a member of String ``` Normally, we would expect implicit conversions to have worked and change String to StringOps but in this case they don't and we're getting: ``` [error] Note that implicit conversions are not applicable because they are ambiguous: [error] both method augmentString in object Predef of type (x: String): scala.collection.StringOps [error] and method String2ToOptions in trait Strings of type (s: String): common.package.String2ToOptions [error] are possible conversion functions from x$4.type to ?{def toIntOption: ?} ``` So we have to explictly call `augmentString()` to trun `String` to `StringOps` and have access to `toIntOption` method. Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
Otherwise we're getting this error: ``` [error] /Users/Ioanna_Kokkini/Projects/frontend/common/app/views/support/JavaScriptPage.scala:103:7: type mismatch [error] found : scala.collection.immutable.Iterable[(String, play.api.libs.json.JsValue)] [error] required: Map[String,play.api.libs.json.JsValue] [error] ) ++ commercialBundleUrl [error] ^ ``` Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
The code previously relied on `groupBy` producing a Map that had the keys in the 'natural' order of stage 1, stage 2 - but didn't have any sorting to enforce that. Maps are unordered collections, and so there was never a guarantee that Scala 2.13 put Map keys in the same order as Scala 2.12. Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
Co-authored-by: Roberto Tyley <roberto.tyley@guardian.co.uk>
e3b3ed2
to
53b68a9
Compare
Seen on PROD (merged by @ioannakok 15 minutes and 31 seconds ago)
|
Massive congratulations on getting this out @ioannakok ! |
Switch to Scala 2.13. This PR is big but also fixes the following syntax errors that could not get while we were on Scala 2.12:
scala.collection.JavaConverters._
withscala.jdk.CollectionConverters._
becauseJavaConverters
is deprecated.scala-collection-plus
library in order to be able to replace the deprecatedmapValues
method with the library'smapV
.implicits.Collections
class.zipped
withlazyZip()
.filterKeys()
being aview
method.Either
is now right biased.EitherValues
in tests that are accessingEither
values.toMap
to ensure we're not returning anIterable
.Co-authored-by: Roberto Tyley roberto.tyley@guardian.co.uk
see Google doc of tricky errors
Previous PRs preparing for this upgrade:
Dependencies
TagsQuery
content-api-scala-client#359Syntax errors
weaker access privileges
compile error #25191implicits.Collections
#25226LiveBlogCurrentPage.findPageWithBlock()
#25338ParseResult
toParseBlockResult
to avoid shadowing a nested class of a parent class #25374See also:
Tested