2024-09-23-phi3.5_mini_4k_instruct_q4_gguf_en

docs/_posts/DevinTDHa/2024-09-23-phi3.5_mini_4k_instruct_q4_gguf_en.md

@@ -0,0 +1,120 @@
+---
+layout: model
+title: Phi-3.5-mini Q4_K_M GGUF
+author: John Snow Labs
+name: phi3.5_mini_4k_instruct_q4_gguf
+date: 2024-09-23
+tags: [gguf, phi, open_source, en, tensorflow]
+task: Text Generation
+language: en
+edition: Spark NLP 5.5.0
+spark_version: 3.0
+supported: true
+engine: tensorflow
+annotator: AutoGGUFModel
+article_header:
+  type: cover
+use_language_switcher: "Python-Scala-Java"
+---
+
+## Description
+
+Phi-3.5-mini is a lightweight, state-of-the-art open model built upon datasets used for Phi-3 - synthetic data and filtered publicly available websites - with a focus on very high-quality, reasoning dense data. The model belongs to the Phi-3 model family and supports 128K token context length.
+
+Original model from https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF.
+
+## Predicted Entities
+
+
+
+{:.btn-box}
+<button class="button button-orange" disabled>Live Demo</button>
+<button class="button button-orange" disabled>Open in Colab</button>
+[Download](https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/models/phi3.5_mini_4k_instruct_q4_gguf_en_5.5.0_3.0_1727109802829.zip){:.button.button-orange.button-orange-trans.arr.button-icon}
+[Copy S3 URI](s3://auxdata.johnsnowlabs.com/public/models/phi3.5_mini_4k_instruct_q4_gguf_en_5.5.0_3.0_1727109802829.zip){:.button.button-orange.button-orange-trans.button-icon.button-copy-s3}
+
+## How to use
+
+
+
+<div class="tabs-box" markdown="1">
+{% include programmingLanguageSelectScalaPythonNLU.html %}
+```python
+import sparknlp
+from sparknlp.base import *
+from sparknlp.annotator import *
+from pyspark.ml import Pipeline
+
+document = DocumentAssembler() \
+    .setInputCol("text") \
+    .setOutputCol("document")
+
+autoGGUFModel = AutoGGUFModel.pretrained() \
+    .setInputCols(["document"]) \
+    .setOutputCol("completions") \
+    .setBatchSize(4) \
+    .setNPredict(20) \
+    .setNGpuLayers(99) \
+    .setTemperature(0.4) \
+    .setTopK(40) \
+    .setTopP(0.9) \
+    .setPenalizeNl(True)
+
+pipeline = Pipeline().setStages([document, autoGGUFModel])
+data = spark.createDataFrame([["Hello, I am a"]]).toDF("text")
+result = pipeline.fit(data).transform(data)
+result.select("completions").show(truncate = False)
+```
+```scala
+import com.johnsnowlabs.nlp.base._
+import com.johnsnowlabs.nlp.annotator._
+import org.apache.spark.ml.Pipeline
+import spark.implicits._
+
+val document = new DocumentAssembler()
+  .setInputCol("text")
+  .setOutputCol("document")
+
+val autoGGUFModel = AutoGGUFModel
+  .pretrained()
+  .setInputCols("document")
+  .setOutputCol("completions")
+  .setBatchSize(4)
+  .setNPredict(20)
+  .setNGpuLayers(99)
+  .setTemperature(0.4f)
+  .setTopK(40)
+  .setTopP(0.9f)
+  .setPenalizeNl(true)
+
+val pipeline = new Pipeline().setStages(Array(document, autoGGUFModel))
+
+val data = Seq("Hello, I am a").toDF("text")
+val result = pipeline.fit(data).transform(data)
+result.select("completions").show(truncate = false)
+```
+</div>
+
+## Results
+
+```bash
++-----------------------------------------------------------------------------------------------------------------------------------+
+|completions                                                                                                                        |
++-----------------------------------------------------------------------------------------------------------------------------------+
+|[{document, 0, 78,  new user.  I am currently working on a project and I need to create a list of , {prompt -> Hello, I am a}, []}]|
++-----------------------------------------------------------------------------------------------------------------------------------+
+```
+
+{:.model-param}
+## Model Information
+
+{:.table-model}
+|---|---|
+|Model Name:|phi3.5_mini_4k_instruct_q4_gguf|
+|Compatibility:|Spark NLP 5.5.0+|
+|License:|Open Source|
+|Edition:|Official|
+|Input Labels:|[document]|
+|Output Labels:|[completions]|
+|Language:|en|
+|Size:|2.4 GB|