Changed and added files according to Telegram Updated Repo #4
Conversation
tests/test_scraper.py
Outdated
| @@ -0,0 +1,15 @@ | |||
| import uuid | |||
| import telegram_scraper | |||
There was a problem hiding this comment.
Please make it command line executable, fix the requirements, leave the server.py as it is ( on second thought, I will fix it for all the data pipelines later). Think of how to ensure the pipelines scrape all data, handle rate limits, etc. Use geo features if possible.
app/scraper.py
Outdated
| messages=[ | ||
| { | ||
| "role": "user", | ||
| "content": f"Examine this tweet: {tweet['snippet']}. Is this tweet describing {filter_keyword}? " |
| @@ -0,0 +1,154 @@ | |||
| import logging | |||
| from logging import Logger | |||
There was a problem hiding this comment.
you need to include instructor, pandas, openai, pydantic, requests in your requirements.txt
app/scraper.py
Outdated
| current_data = media_data | ||
| output_data_list.append(media_data) | ||
|
|
||
| scraped_data_repository.register_scraped_data( |
|
We still have doubts regarding the usage and importing of scraping api key. Also this version does version is not yet updated with openai. We are facing trouble in replicating what is shown in the video. |
|
Merging as is to unblock integration into kubeflow pipelines. Please test this and send follow up PRs ASAP. |
|
Make sure requirements.txt is up to date |
No description provided.