Question: How to serialise/parse large xml file with KorIO

[multiport]

Hello,

I'm using KMM with Okio, and i'm triying to serialise/parse a large xml file, I see on https://docs.korge.org/korio/serialization/ how to parse a String but since my file is too large, im triying to parse the stream :

  val fileSystem = FileSystem.SYSTEM_TEMPORARY_DIRECTORY
  val path = fileSystem / "myfile.xml"
  // open a source stream
  val sourceFileRead= fileSystem2.source(path)
  // wrap stream with a buffer
  val fileopened = sourceFileRead.buffer()
  // read until end of file
  var line: String?
  var numLigne :Int =0
  while (fileopened.readUtf8Line().also { line = it } != null) {
      println(line)
      var xmlLine = line?.reader(line!!,numLigne)
      Xml.Stream.parse(xmlLine!!)  // <-- Not gonna work but i dont know how to do ?
  }

Thanks for your help if you have a solution

[soywiz]

It is not posible right now. Xml streaming uses StrReader that was designed to use a String in memory. It was like that to be as fast as possible (no virtual methods, no suspension points, all in-memory).
The thing is that K/N have evolved a lot and now maybe the performance is much better, so might make sense now to create a String Reader that support streams without having the whole file in memory and make Xml to use it.
I have created a PR with a baby step towards it

[multiport]

Thank you very muh for looking into it . What i'm triying to do it's too parse an xml file with korio on common main of kmm to avoid specific code for android and ios , currently on android i'm using a java library to serialise like this :

actual fun parserXml(input :String): MutableList<Employee> {
    val fileSystem = platformFileSystem()
    val chemin :Path =input.toPath()  //val fichier = File(chemin.toString())
    val sourceFichier= fileSystem.source(chemin)
    val targetStream = sourceFichier.buffer().inputStream()     //val targetStream :InputStream = FileInputStream(fichier)
    //creating a XmlPull parse Factory instance
    val parserFactory = XmlPullParserFactory.newInstance()
    val parser = parserFactory.newPullParser()
    // setting the namespaces feature to false
    parser.setFeature(XmlPullParser.FEATURE_PROCESS_NAMESPACES, false)
    // setting the input to the parser
    parser.setInput(targetStream, null)
    // working with the input stream
    var tag: String?
    var event = parser.eventType
    var text: String? = ""
    var firstname = ""
    var familyname  = ""
    var job =""
    var counter=0
    val listEmployee: MutableList<Employee> = mutableListOf()
    while (event != XmlPullParser.END_DOCUMENT) {tag = parser.name
        when (event) {
            XmlPullParser.START_TAG ->
                if (tag == "employee") {
                    counter++
                    firstname = parser.getAttributeValue(null, "firstname")
                    familyname = parser.getAttributeValue(null, "familyname")
                }
            XmlPullParser.TEXT -> {
                text = parser.text
            }
            XmlPullParser.END_TAG -> when (tag) {
                "job" -> job = text!!
                "employee" -> {
                    val employee = Employee(counter.toLong(),firstname, familyname, job)
                    listEmployee.add(employee)
                }
            }
        }
        event = parser.next()
    }
    println("List Has "+listEmployee.size+" employees ")
    return listEmployee
}

PS: I'm also facing the same kindof issue when extracting a "large" .zip file with file with korio :
java.lang.OutOfMemoryError: Failed to allocate a 98976136 byte allocation with 25165824 free bytes and 75MB

        val sourceFile= fileSystem2.source(path)
        val targetStream = sourceFile.buffer().readByteArray()
        val sink2: BufferedSink = fileSystem2.appendingSink(path2, false).buffer()
        sink2.write(targetStream.uncompress(GZIP))

[soywiz]

Thanks for sharing your use-case.
Yes that should be supported once the StrReader support streams.

Regarding to zip file with korio: korio should already support doing that as a stream. In that sample you are reading everything as a ByteArray.
In KorIO you can open a file as an asynchronous stream, then you can open it as a zip file and access the zip file like as a virtual file system, then you can open a file inside the zip as an asynchronous stream, that will uncompress data as required, though seeking won't be supported, since gzip/zlib streams are not designed for seeking.

[soywiz]

Some work:

• Prepare StrReader to be extensible and support streams #1169
• Adds CharReader & CharReaderFromSyncStream & allows Charset.decode to support streams #1177
• Add extra Deque variants #1178
• Support creating SyncStream from a sequence of ByteArray #1179
• Supports XML streaming #1180

[soywiz]

Supported since this PR: #1180

Example:

    @Test
    fun testNamedDescendantStreamInfinite() {
        val xml = Xml.Stream.parse(sequenceSyncStream {
            while (true) {
                yield("<xml>".toByteArray())
            }
        }.toCharReader(UTF8))
        assertEquals(
            """
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
                OpenTag(name=xml, attributes={})
            """.trimIndent(),
            xml.take(10).toList().joinToString("\n")
        )
    }

[multiport]

Thanks seems easy and intuitive to use but unfortunately after I followed your instructions, i wasnt able to use it ... (my fault not yours )

For using gradlew command on windws i had to install another JDK for path variable, compile after that ended well then i had to turn off all ios setup on the KMM build.gradle.kts.
Once I done that my android studio became weird (do not autocomplete) and when i use the code you quote : I cant compile the app with this error message :

Unresolved reference. None of the following candidates is applicable because of receiver type mismatch: 
public inline fun <C : io.ktor.utils.io.core.Closeable /* = java.io.Closeable */, R> TypeVariable(C).use(block: (TypeVariable(C)) -> TypeVariable(R)): TypeVariable(R) defined in io.ktor.utils.io.core

[soywiz]

I'm not sure why did you need to remove KMM from build.gradle.kts, CI is compiling it on windows without modifying it. But feel free to share the error you got to try to figure out the root cause and improve it.

Regarding to the unresolved reference. Try this instead:

val stream = fileToOpen.readAsSyncStream()
try {
  val xml = Xml.Stream.parse(stream.toCharReader(UTF8))
  for (node in xml) {
    // ...
  }
} finally {
  stream.close()
}

[multiport]

Thanks , thats great all seems working without crash (even the previous part who wasnt working earlier): I was able to print all the content of the file (100MB) from val xml.

Just I dont see spécial method on the object Xml.Stream.Element , I guess I've to create it on my own to extract the data contained like getTag or getText getAttribute ?

A println(node) shows 3 kindof :

OpenTag(name=Person, attributes={firstname="John", familyname="Smith", })
Text(text=Lawyer, cdata=false)
CloseTag(name=Person)

[soywiz]

Glad to hear it worked!

In Kotlin you can access specific properties by using smart casts: https://kotlinlang.org/docs/typecasts.html

You can use a when. For example:

korge/korio/src/commonMain/kotlin/com/soywiz/korio/serialization/xml/Xml.kt

Lines 351 to 358 in 8820c03

	sealed class Element {
	data class ProcessingInstructionTag(val name: String, val attributes: Map<String, String>) : Element()
	data class OpenCloseTag(val name: String, val attributes: Map<String, String>) : Element()
	data class OpenTag(val name: String, val attributes: Map<String, String>) : Element()
	data class CommentTag(val text: String) : Element()
	data class CloseTag(val name: String) : Element()
	data class Text(val text: String, var cdata: Boolean = false) : Element()
	}

when (node) {
  is Xml.Stream.Element.OpenTag -> {
    node.attributes // <- you can access node.attributes and node.name because of the smart cast
  }
  is Xml.Stream.Element.Text -> {
    node.text // <- you can access node.text because of the smart cast
  }
}

In this case since Element is sealed, the IDE will suggest you to autocomplete all the other nodes. So you can easily put all of them.

Hope it serves!

[multiport]

Thanks you for all, your patience, explanation and works. All working well