Skip to content

Commit

Permalink
[SPARKNLP-1027] Fix issue with pretrained model (#14413)
Browse files Browse the repository at this point in the history 
* [SPARKNLP-1027] Change Default AutoGGUF pretrained model

* [SPARKNLP-1027] Change default model in example

* [SPARKNLP-1027] Fix issue with pretrained model
  • Loading branch information
@DevinTDHa
DevinTDHa authored Sep 24, 2024
1 parent 1a112cd commit cc38757
Show file tree
Hide file tree
Showing 3 changed files with 21 additions and 21 deletions.
26 changes: 13 additions & 13 deletions examples/python/llama.cpp/llama.cpp_in_Spark_NLP_AutoGGUFModel.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
"source": [
"## Download a GGUF Model\n",
"\n",
"Lets download a GGUF model to test it out. For this, we will use [microsoft/Phi-3-mini-4k-instruct-gguf](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf). It is a 3.8B parameter model which also is available in 4-bit quantization. \n",
"Lets download a GGUF model to test it out. For this, we will use [bartowski/Phi-3.5-mini-instruct-GGUF](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF). It is a 3.8B parameter model which also is available in 4-bit quantization. \n",
"\n",
"We can download the model by selecting the q4 GGUF file from the \"Files and versions\" tab.\n",
"\n",
Expand All @@ -38,28 +38,28 @@
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-07-20 11:11:30-- https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf?download=true\n",
"--2024-07-20 11:11:30-- https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true\n",
"Resolving huggingface.co (huggingface.co)... 2600:9000:275f:7600:17:b174:6d00:93a1, 2600:9000:275f:3800:17:b174:6d00:93a1, 2600:9000:275f:6e00:17:b174:6d00:93a1, ...\n",
"Connecting to huggingface.co (huggingface.co)|2600:9000:275f:7600:17:b174:6d00:93a1|:443... connected.\n",
"HTTP request sent, awaiting response... 302 Found\n",
"Location: https://cdn-lfs-us-1.huggingface.co/repos/41/c8/41c860f65b01de5dc4c68b00d84cead799d3e7c48e38ee749f4c6057776e2e9e/8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27Phi-3-mini-4k-instruct-q4.gguf%3B+filename%3D%22Phi-3-mini-4k-instruct-q4.gguf%22%3B&Expires=1721725890&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTcyNTg5MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzQxL2M4LzQxYzg2MGY2NWIwMWRlNWRjNGM2OGIwMGQ4NGNlYWQ3OTlkM2U3YzQ4ZTM4ZWU3NDlmNGM2MDU3Nzc2ZTJlOWUvOGE4M2M3ZmI5MDQ5YTliMmU5MjI2NmZhN2FkMDQ5MzNiYjUzYWExZTg1MTM2YjdiMzBmMWI4MDAwZmYyZWRlZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=joXQf4QRpEhtFeQ3r3gJ0zyJ3bXReb9OxM%7EZit3GJ3355ycKQzemJ%7E6eD-J7%7EkphnsPpRpUDhQkCr2-Oidqo7dgltmFsWAX4SmQLn65R1yjO%7EsMvi%7E4vOUpaRPYlSMCyWWJpiZZjQYVH4Uk0o-G62ALFXKGaDfr627kvahP-fJYwNNP1riTrH8hbbah28ZKRAQjUGI1aNqerG0jojudnGOagawISAnudkAOFZfxnN7Qw3CoMySZLj9Euu02RBv2A5Yy0uSjG7b8rilx-tU5HDR3ECohdQQ8yPXjYFU-LZi-zcG1wwBDF-S01qb%7EgPWsTorenxfRM2cG6J%7EvSziGCzA__&Key-Pair-Id=K24J24Z295AEI9 [following]\n",
"--2024-07-20 11:11:30-- https://cdn-lfs-us-1.huggingface.co/repos/41/c8/41c860f65b01de5dc4c68b00d84cead799d3e7c48e38ee749f4c6057776e2e9e/8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27Phi-3-mini-4k-instruct-q4.gguf%3B+filename%3D%22Phi-3-mini-4k-instruct-q4.gguf%22%3B&Expires=1721725890&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTcyNTg5MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzQxL2M4LzQxYzg2MGY2NWIwMWRlNWRjNGM2OGIwMGQ4NGNlYWQ3OTlkM2U3YzQ4ZTM4ZWU3NDlmNGM2MDU3Nzc2ZTJlOWUvOGE4M2M3ZmI5MDQ5YTliMmU5MjI2NmZhN2FkMDQ5MzNiYjUzYWExZTg1MTM2YjdiMzBmMWI4MDAwZmYyZWRlZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=joXQf4QRpEhtFeQ3r3gJ0zyJ3bXReb9OxM%7EZit3GJ3355ycKQzemJ%7E6eD-J7%7EkphnsPpRpUDhQkCr2-Oidqo7dgltmFsWAX4SmQLn65R1yjO%7EsMvi%7E4vOUpaRPYlSMCyWWJpiZZjQYVH4Uk0o-G62ALFXKGaDfr627kvahP-fJYwNNP1riTrH8hbbah28ZKRAQjUGI1aNqerG0jojudnGOagawISAnudkAOFZfxnN7Qw3CoMySZLj9Euu02RBv2A5Yy0uSjG7b8rilx-tU5HDR3ECohdQQ8yPXjYFU-LZi-zcG1wwBDF-S01qb%7EgPWsTorenxfRM2cG6J%7EvSziGCzA__&Key-Pair-Id=K24J24Z295AEI9\n",
"Location: https://cdn-lfs-us-1.huggingface.co/repos/41/c8/41c860f65b01de5dc4c68b00d84cead799d3e7c48e38ee749f4c6057776e2e9e/8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27Phi-3.5-mini-instruct-Q4_K_M.gguf%3B+filename%3D%22P Phi-3.5-mini-instruct-Q4_K_M.gguf%22%3B&Expires=1721725890&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTcyNTg5MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzQxL2M4LzQxYzg2MGY2NWIwMWRlNWRjNGM2OGIwMGQ4NGNlYWQ3OTlkM2U3YzQ4ZTM4ZWU3NDlmNGM2MDU3Nzc2ZTJlOWUvOGE4M2M3ZmI5MDQ5YTliMmU5MjI2NmZhN2FkMDQ5MzNiYjUzYWExZTg1MTM2YjdiMzBmMWI4MDAwZmYyZWRlZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=joXQf4QRpEhtFeQ3r3gJ0zyJ3bXReb9OxM%7EZit3GJ3355ycKQzemJ%7E6eD-J7%7EkphnsPpRpUDhQkCr2-Oidqo7dgltmFsWAX4SmQLn65R1yjO%7EsMvi%7E4vOUpaRPYlSMCyWWJpiZZjQYVH4Uk0o-G62ALFXKGaDfr627kvahP-fJYwNNP1riTrH8hbbah28ZKRAQjUGI1aNqerG0jojudnGOagawISAnudkAOFZfxnN7Qw3CoMySZLj9Euu02RBv2A5Yy0uSjG7b8rilx-tU5HDR3ECohdQQ8yPXjYFU-LZi-zcG1wwBDF-S01qb%7EgPWsTorenxfRM2cG6J%7EvSziGCzA__&Key-Pair-Id=K24J24Z295AEI9 [following]\n",
"--2024-07-20 11:11:30-- https://cdn-lfs-us-1.huggingface.co/repos/41/c8/41c860f65b01de5dc4c68b00d84cead799d3e7c48e38ee749f4c6057776e2e9e/8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27Phi-3.5-mini-instruct-Q4_K_M.gguf%3B+filename%3D%22P Phi-3.5-mini-instruct-Q4_K_M.gguf%22%3B&Expires=1721725890&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTcyNTg5MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzQxL2M4LzQxYzg2MGY2NWIwMWRlNWRjNGM2OGIwMGQ4NGNlYWQ3OTlkM2U3YzQ4ZTM4ZWU3NDlmNGM2MDU3Nzc2ZTJlOWUvOGE4M2M3ZmI5MDQ5YTliMmU5MjI2NmZhN2FkMDQ5MzNiYjUzYWExZTg1MTM2YjdiMzBmMWI4MDAwZmYyZWRlZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=joXQf4QRpEhtFeQ3r3gJ0zyJ3bXReb9OxM%7EZit3GJ3355ycKQzemJ%7E6eD-J7%7EkphnsPpRpUDhQkCr2-Oidqo7dgltmFsWAX4SmQLn65R1yjO%7EsMvi%7E4vOUpaRPYlSMCyWWJpiZZjQYVH4Uk0o-G62ALFXKGaDfr627kvahP-fJYwNNP1riTrH8hbbah28ZKRAQjUGI1aNqerG0jojudnGOagawISAnudkAOFZfxnN7Qw3CoMySZLj9Euu02RBv2A5Yy0uSjG7b8rilx-tU5HDR3ECohdQQ8yPXjYFU-LZi-zcG1wwBDF-S01qb%7EgPWsTorenxfRM2cG6J%7EvSziGCzA__&Key-Pair-Id=K24J24Z295AEI9\n",
"Resolving cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)... 2600:9000:275f:7a00:17:9a40:4dc0:93a1, 2600:9000:275f:fc00:17:9a40:4dc0:93a1, 2600:9000:275f:4800:17:9a40:4dc0:93a1, ...\n",
"Connecting to cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)|2600:9000:275f:7a00:17:9a40:4dc0:93a1|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 2393231072 (2.2G) [binary/octet-stream]\n",
"Saving to: ‘Phi-3-mini-4k-instruct-q4.gguf?download=true’\n",
"Saving to: ‘Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true’\n",
"\n",
"Phi-3-mini-4k-instr 100%[===================>] 2.23G 22.5MB/s in 96s \n",
"\n",
"2024-07-20 11:13:06 (23.7 MB/s) - ‘Phi-3-mini-4k-instruct-q4.gguf?download=true’ saved [2393231072/2393231072]\n",
"2024-07-20 11:13:06 (23.7 MB/s) - ‘Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true’ saved [2393231072/2393231072]\n",
"\n"
]
}
],
"source": [
"EXPORT_PATH = \"Phi-3-mini-4k-instruct-q4.gguf\"\n",
"! wget \"https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf/resolve/main/Phi-3-mini-4k-instruct-q4.gguf?download=true\" -O {EXPORT_PATH}"
"EXPORT_PATH = \"Phi-3.5-mini-instruct-Q4_K_M.gguf\"\n",
"! wget \"https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true\" -O {EXPORT_PATH}"
]
},
{
Expand Down Expand Up @@ -359,7 +359,7 @@
"metadata": {},
"outputs": [],
"source": [
"autoGGUFModel.write().overwrite().save(f\"Phi-3-mini-4k-instruct-q4_spark_nlp\")"
"autoGGUFModel.write().overwrite().save(f\"Phi-3.5-mini-instruct-Q4_K_M_spark_nlp\")"
]
},
{
Expand Down Expand Up @@ -398,12 +398,12 @@
"text": [
"total 2337168\n",
"drwxr-xr-x 2 ducha ducha 4096 Jul 21 16:24 metadata\n",
"-rwxrwxr-x 1 ducha ducha 2393231072 Jul 21 16:24 Phi-3-mini-4k-instruct-q4.gguf\n"
"-rwxrwxr-x 1 ducha ducha 2393231072 Jul 21 16:24 Phi-3.5-mini-instruct-Q4_K_M.gguf\n"
]
}
],
"source": [
"! ls -l Phi-3-mini-4k-instruct-q4_spark_nlp"
"! ls -l Phi-3.5-mini-instruct-Q4_K_M_spark_nlp"
]
},
{
Expand Down Expand Up @@ -437,7 +437,7 @@
"name": "stderr",
"output_type": "stream",
"text": [
"llama_model_loader: loaded meta data with 24 key-value pairs and 195 tensors from /tmp/spark-bbad4f64-91a7-4b6e-8242-7f91e6abca54/userFiles-f7d4e4e9-c02d-46e4-81b5-bf5a26d70930/Phi-3-mini-4k-instruct-q4.gguf (version GGUF V3 (latest))\n",
"llama_model_loader: loaded meta data with 24 key-value pairs and 195 tensors from /tmp/spark-bbad4f64-91a7-4b6e-8242-7f91e6abca54/userFiles-f7d4e4e9-c02d-46e4-81b5-bf5a26d70930/Phi-3.5-mini-instruct-Q4_K_M.gguf (version GGUF V3 (latest))\n",
"llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
"llama_model_loader: - kv 0: general.architecture str = phi3\n",
"llama_model_loader: - kv 1: general.name str = Phi3\n",
Expand Down Expand Up @@ -583,7 +583,7 @@
"\n",
"document_assembler = DocumentAssembler().setInputCol(\"text\").setOutputCol(\"document\")\n",
"\n",
"auto_gguf_model = AutoGGUFModel.load(\"Phi-3-mini-4k-instruct-q4_spark_nlp\")\n",
"auto_gguf_model = AutoGGUFModel.load(\"Phi-3.5-mini-instruct-Q4_K_M_spark_nlp\")\n",
"\n",
"pipeline = Pipeline().setStages([document_assembler, auto_gguf_model])\n",
"\n",
Expand Down
11 changes: 4 additions & 7 deletions python/test/annotator/seq2seq/auto_gguf_model_test.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,13 @@ def runTest(self):
DocumentAssembler().setInputCol("text").setOutputCol("document")
)

modelPath = "models/codellama-7b.Q2_K.gguf"
model = (
AutoGGUFModel.loadSavedModel(modelPath, self.spark)
AutoGGUFModel.pretrained()
.setInputCols("document")
.setOutputCol("completions")
.setBatchSize(4)
.setNPredict(20)
.setNGpuLayers(99)
.setNGpuLayers(5)
.setTemperature(0.4)
.setTopK(40)
.setTopP(0.9)
Expand Down Expand Up @@ -79,9 +78,8 @@ def runTest(self):
DocumentAssembler().setInputCol("text").setOutputCol("document")
)

modelPath = "models/codellama-7b.Q2_K.gguf"
model = (
AutoGGUFModel.loadSavedModel(modelPath, self.spark)
AutoGGUFModel.pretrained()
.setInputCols("document")
.setOutputCol("completions")
.setBatchSize(4)
Expand Down Expand Up @@ -182,9 +180,8 @@ def setUp(self):
self.spark = SparkContextForTest.spark

def runTest(self):
modelPath = "models/codellama-7b.Q2_K.gguf"
model = (
AutoGGUFModel.loadSavedModel(modelPath, self.spark)
AutoGGUFModel.pretrained()
.setInputCols("document")
.setOutputCol("completions")
)
Expand Down
5 changes: 4 additions & 1 deletion src/main/scala/com/johnsnowlabs/nlp/annotators/seq2seq/AutoGGUFModel.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,10 @@ trait ReadAutoGGUFModel {

def readModel(instance: AutoGGUFModel, path: String, spark: SparkSession): Unit = {
def findGGUFModelInFolder(): String = {
val folder = new java.io.File(path)
val folder =
new java.io.File(
path.replace("file:", "")
) // File should be local at this point. TODO: Except if its HDFS?
if (folder.exists && folder.isDirectory) {
folder.listFiles
.filter(_.isFile)
Expand Down

0 comments on commit cc38757

Please sign in to comment.