updating RAG example to use IBM granite model #745
Conversation
Signed-off-by: Sujee Maniyam <sujee@node51.com>
@sujee I am getting an error while running the ray version of the Notebook (rag_1A_dpk_process_ray.ipynb) about not finding pdf2parquet_transform_ray. I had no problem with the Python version of this notebook. Here is the error:ModuleNotFoundError Traceback (most recent call last) ModuleNotFoundError: No module named 'pdf2parquet_transform_ray' |
|
@sujee One more thing: In rag_1D_query_replicate.ipynb file, in "Step-6: Initialize LLM", please add the choice of IBM Granite as one of the models in the comments (I know that you are picking the Granite model for the actual execution). |
If you have created a custom kernel when you setup the python env, like this
Choose this kernel in jupyter when running this notebook. See if it fixes the error. |
|
@sujee Thanks for the suggestion. I was using a conda env. So, I removed that env and created and activated a new env. That fixed the problem! In any case, since I have now tested successfully by going all the way through the rag_1D notebook by starting from either Python or Ray versions, I am ready to approve this, after you make the "one more thing" change above. Thanks. |
Signed-off-by: Sujee Maniyam <sujee@node51.com>
| "| meta/meta-llama-3-70b-instruct | Meta | 70 B | $ 0.65 | $ 2.75 | |\n", | ||
| "\n", | ||
| "\n", | ||
| "(Prices are as of Oct 2024)\n", | ||
| "\n", |
There was a problem hiding this comment.
@sujee Since the prices keep changing all the time, even though you have commented that these are Oct. 24 prices, please remove the two price columns from this table (and the comment about Oct. 2024). Thanks.
BTW, the Llama 8b price number for the Output in the table is wrong. It should have been 0.25 (as shown on the replicate.com) and not 0.05. Same as Granite 8b numbers.
There was a problem hiding this comment.
fixed model table @shahrokhDaijavad
Signed-off-by: Sujee Maniyam <sujee@node51.com>
There was a problem hiding this comment.
Thanks for making the change, @sujee. I tested the whole set of notebooks through 1D again after the change.
| @@ -303,7 +303,8 @@ | |||
| " \"data_files_to_use\": ast.literal_eval(\"['.pdf']\"),\n", | |||
| " # orchestrator\n", | |||
| " \"runtime_worker_options\": ParamsUtils.convert_to_ast(worker_options),\n", | |||
| " \"runtime_num_workers\": MY_CONFIG.RAY_RUNTIME_WORKERS,\n", | |||
| " # \"runtime_num_workers\": MY_CONFIG.RAY_RUNTIME_WORKERS,\n", | |||
There was a problem hiding this comment.
Why is this commented out. Please remove or go back to using MY._CONFIG.RAY_RuNTIME_WORKERS = 1
There was a problem hiding this comment.
Setting MY._CONFIG.RAY_RUNTIME_WORKERS = 1 breaks some other transforms: ededupe and fuzzy-dedupe.
They seem to need at least 2 workers.
So I am setting workers = 1 only for PDF2PQ step (so we don't run into the race condition of cleaning out the downloaded models.
This is a temp fix until other worker issues are resolved.
| "| ibm-granite/granite-3.0-8b-instruct | IBM | 8 B | IBM's newest Granite Model v3.0 (default) |\n", | ||
| "| ibm-granite/granite-3.0-2b-instruct | IBM | 2 B | IBM's newest Granite Model v3.0 |\n", | ||
| "| meta/meta-llama-3.1-405b-instruct | Meta | 405 B | Meta's flagship 405 billion parameter language model |\n", | ||
| "| meta/meta-llama-3-8b-instruct | Meta | 8 B | |\n", |
There was a problem hiding this comment.
Missing description for the last 2 rows
There was a problem hiding this comment.
@sujee I think we can add these descriptions for the last 2 rows:
Meta's 8 billion parameter language model
Meta's 70 billion parameter language model
Signed-off-by: Sujee Maniyam <sujee@node51.com>
Why are these changes needed?
Updating RAG example to use IBM Granite 3.0 model
Related issue number (if any).