Support Scala 3

[DieBauer]

Now that Scala 3 is in Milestone releases (https://dotty.epfl.ch/blog/2020/11/09/scala3-m1.html) it would be great if we can support Scala 3.

I tried to make it work with the dottyCompat mode in the sbt-dotty plugin (https://scalacenter.github.io/scala-3-migration-guide/docs/tutorials/sbt-migration.html) but then things start to break :)

The following snippet works on Scala 2.x, but fails with Scala 3.0.0-M1.

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.module.scala.DefaultScalaModule

case class T(x: String)

val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
val result: T = mapper.readValue("""{"x": "Hello"}""", classOf[T])

fails with:

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot construct instance of `T` (although at least one Creator exists): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (String)"{"x": "Hello"}"; line: 1, column: 2]
	at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
	at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1455)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1081)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1332)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:331)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:164)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4526)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3468)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3436)
	at Json$.run(main.scala:12)
	at run.main(main.scala:4)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sbt.Run.invokeMain(Run.scala:115)
	at sbt.Run.execute$1(Run.scala:79)
	at sbt.Run.$anonfun$runWithLoader$4(Run.scala:92)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at sbt.util.InterfaceUtil$$anon$1.get(InterfaceUtil.scala:10)
	at sbt.TrapExit$App.run(TrapExit.scala:257)
	at java.lang.Thread.run(Thread.java:748)

This is the scastie to play with: https://scastie.scala-lang.org/2dYNr8u9TciEOU9rOCQJPQ

[pjfanning]

I created https://github.com/FasterXML/jackson-module-scala/tree/dotty but there are some remaining compile problems to solve

[DieBauer]

I'll have a look at that branch.

For now the above error is thrown, I think, because of the way Jackson determines if a class is a Scala class (https://github.com/FasterXML/jackson-module-scala/blob/master/src/main/scala/com/fasterxml/jackson/module/scala/util/Classes.scala#L10-L11) and this check always evaluates to false, since scala-reflect is not available when compiling with Scala 3.
Long-story short, the propertycreator will be null because of that.

[pjfanning]

@DieBauer if you could find an alternative way to differentiate a Scala class from a plain Java class, we can support that too. We do need to avoid having jackson-module-scala try to work with non-Scala classes because it leads to unexpected behaviours when users have Scala and plain Java classes in their projects.

[pjfanning]

@DieBauer which version if jackson-module-scala did you test with? v2.12.0 increases the reliance on the check for the ScalaSignature annotations that you highlighted.

[DieBauer]

@pjfanning I checked that with 2.11.3, but same issue occurs on 2.12.0-rc2. https://scastie.scala-lang.org/UOWvdJ7xQkS64MV94WuJEQ

Relying on the ScalaSignature runtime annotation will just not work with Scala 3, since there is no scala-reflect. I haven't heard of a viable alternative in the dotty gitter channel yet..

What kind of unexpected behaviours with non-scala classes are you talking about, if it turns out the runtime check cannot be done on scala 3?
What could be a way forward :)

[cowtowncoder]

I may not have enough context here, but it might be easy enough to make "Plain Scala Object or not" detection pluggable, provide dynamically loaded default but allow override (or more specifically I suppose it should be provider for that detector).

The reason there needs to be detection is that logic for figuring out construction of Plain Old Scala Objects is bit more advanced than that for POJOs -- more ways to construct things in Java, can not rely on certain things that Scala (and some other JVM langs like Kotlin) guarantee.
Similarly Scala matches fields and accessors more tightly (although Jackson does not yet have optimal support for these variations).

[pjfanning]

I started a topic at https://users.scala-lang.org/t/recognising-scala-3-classes/6922

@DieBauer would you have any pointers to specific comments of the Dotty gitter channel relating to this? Otherwise, I might being the topic above to the gitter channel to see if I can see if the Dotty team would be amenable to adding a new annotation or a marker trait/interface.

[DieBauer]

This is the question I asked https://gitter.im/lampepfl/dotty?at=5fbf67a566dcfa77e2a3cb08, only 1 person responded though.

Indeed as mentioned in the topic, the classfile contains scala references. But that doesn't help you at runtime :)

[pjfanning]

The dotty branch has a number of compile issues associated with higher-kinds - affects 3 classes, in the main.

I might have time over the Christmas period to look at trying to support dotty compiled classes in the 2.13 build. I think this is a more useful starting point than supporting a scala 3 build.

[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericFactoryDeserializerResolver.scala:21:26 
[error] 21 |  val CLASS_DOMAIN: Class[Collection[_]]
[error]    |                          ^^^^^^^^^^^^^
[error]    |unreducible application of higher-kinded type [A] =>> CC[A] to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericFactoryDeserializerResolver.scala:52:30 
[error] 52 |    def setInitialValue(init: Collection[_]): Unit = init.asInstanceOf[Iterable[A]].foreach(add)
[error]    |                              ^^^^^^^^^^^^^
[error]    |unreducible application of higher-kinded type [A] =>> CC[A] to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericFactoryDeserializerResolver.scala:65:38 
[error] 65 |    extends ContainerDeserializerBase[CC[_]](collectionType)
[error]    |                                      ^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericFactoryDeserializerResolver.scala:81:76 
[error] 81 |    override def deserialize(jp: JsonParser, ctxt: DeserializationContext): CC[_] = {
[error]    |                                                                            ^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericFactoryDeserializerResolver.scala:87:86 
[error] 87 |    override def deserialize(jp: JsonParser, ctxt: DeserializationContext, intoValue: CC[_]): CC[_] = {
[error]    |                                                                                      ^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericFactoryDeserializerResolver.scala:87:94 
[error] 87 |    override def deserialize(jp: JsonParser, ctxt: DeserializationContext, intoValue: CC[_]): CC[_] = {
[error]    |                                                                                              ^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericMapFactoryDeserializerResolver.scala:19:26 
[error] 19 |  val CLASS_DOMAIN: Class[Collection[_, _]]
[error]    |                          ^^^^^^^^^^^^^^^^
[error]    |unreducible application of higher-kinded type [K, V] =>> CC[K, V] to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericMapFactoryDeserializerResolver.scala:53:30 
[error] 53 |    def setInitialValue(init: Collection[_, _]): Unit = {
[error]    |                              ^^^^^^^^^^^^^^^^
[error]    |unreducible application of higher-kinded type [K, V] =>> CC[K, V] to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericMapFactoryDeserializerResolver.scala:67:38 
[error] 67 |    extends ContainerDeserializerBase[CC[_, _]](mapType) with ContextualDeserializer {
[error]    |                                      ^^^^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericMapFactoryDeserializerResolver.scala:82:76 
[error] 82 |    override def deserialize(jp: JsonParser, ctxt: DeserializationContext): CC[_, _] = {
[error]    |                                                                            ^^^^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericMapFactoryDeserializerResolver.scala:88:86 
[error] 88 |    override def deserialize(jp: JsonParser, ctxt: DeserializationContext, intoValue: CC[_, _]): CC[_, _] = {
[error]    |                                                                                      ^^^^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/deser/GenericMapFactoryDeserializerResolver.scala:88:97 
[error] 88 |    override def deserialize(jp: JsonParser, ctxt: DeserializationContext, intoValue: CC[_, _]): CC[_, _] = {
[error]    |                                                                                                 ^^^^^^^^
[error]    |  unreducible application of higher-kinded type CC to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/util/FactorySorter.scala:11:64 
[error] 11 |class FactorySorter[CM[_], CF[+X[_]]] extends TopologicalSorter[CM[_], CF[CM]] {
[error]    |                                                                ^^^^^
[error]    |  unreducible application of higher-kinded type CM to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/util/FactorySorter.scala:12:42 
[error] 12 |  type HKClassManifest[CC2[_]] = ClassTag[CC2[_]]
[error]    |                                          ^^^^^^
[error]    | unreducible application of higher-kinded type CC2 to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/util/FactorySorter.scala:20:73 
[error] 20 |class MapFactorySorter[CM[_, _], CF[+X[_, _]]] extends TopologicalSorter[CM[_, _], CF[CM]] {
[error]    |                                                                         ^^^^^^^^
[error]    |  unreducible application of higher-kinded type CM to wildcard arguments
[error] -- [E043] Type Error: /Users/pj.fanning/code/jackson-module-scala/src/main/scala/com/fasterxml/jackson/module/scala/util/FactorySorter.scala:21:45 
[error] 21 |  type HKClassManifest[CC2[_, _]] = ClassTag[CC2[_, _]]
[error]    |                                             ^^^^^^^^^
[error]    | unreducible application of higher-kinded type CC2 to wildcard arguments
[error] 16 errors found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 11 s, completed 5 Dec 2020, 23:49:47

[pjfanning]

@DieBauer I created a test that shows that jackson-module-scala can handle a scala 3.0.0-M1 case class. See https://github.com/FasterXML/jackson-module-scala/tree/dotty-test -- the equivalent case object test would not compile though -- seems like the TastyReader doesn't support case objects in the way I expected.

[SethTisue]

seems like the TastyReader doesn't support case objects in the way I expected

summon[@bishabosha]

[bishabosha]

@DieBauer I created a test that shows that jackson-module-scala can handle a scala 3.0.0-M1 case class. See https://github.com/FasterXML/jackson-module-scala/tree/dotty-test -- the equivalent case object test would not compile though -- seems like the TastyReader doesn't support case objects in the way I expected.

It seems the problem here is that Scala3CaseObject is a class not a case object, so it can't be used as a value.
https://github.com/pjfanning/scala3-sample-classes/blob/d499620aeefc87d42e7f7956b8973af8975e0770/src/main/scala/com/github/pjfanning/scala3/Scala3CaseObject.scala#L3

I get the same error if I paste the following in scala 2 repl:

scala> :paste
// Entering paste mode (ctrl-D to finish)

package com.github.pjfanning.scala3

class Scala3CaseObject {
  val field1: String = "test"
  val field2: Int = 42
}



// Exiting paste mode, now interpreting.


scala> val x = com.github.pjfanning.scala3.Scala3CaseObject
                                           ^
       error: object Scala3CaseObject is not a member of package com.github.pjfanning.scala3
       note: class Scala3CaseObject exists, but it has no companion object.

[pjfanning]

@bishabosha thanks for checking that - I have now fixed the Scala3CaseObject

[pjfanning]

The 'dotty' branch now compiles (with a few test cases removed - ones related to ScalaObjectMapper which will not work in Scala3 - due to Manifest support being removed along with other scala-reflect features).

Many tests fail due to scala/scala3#6349