Use LLM to convert unstructured data to structured data using sagemaker

[PriyaBasker23]

Idea Description -

This is POC to transform unstructured data into structured formats.

Dataset - Will be open source publicly available data
Timeperiod- Dec 11th - Till 4th Jan. All data and environment specific stuff will be cleared after demo
Environment - Dataengineering Sandbox

Some of the use case to consider .

**Text to structured data:**
Extracting structured information from unstructured documents 
1.  PDFs.
2. Word documents
3. HTML files.
4. Comments 
5. Converting unstructured log files from applications, servers, or devices into a structured format which will be useful for monitoring and altering . (https://gist.github.com/mwestwood/2473fb7ebf9f12b716dd56e4b8764b8e)
we  can sentiment analysis, named entity recognition, or topic modelling for suitable usecase such as comments

**Speech-to-Text Conversion:**
Transcribing spoken words into structured text data for further analysis or documentation.

**Image Data Annotation:**
Converting unstructured image data into structured formats for tasks such as object detection, image classification with some automatic description generated

Why Should We Do This -

Transforming unstructured data into structured formats within a data platform is essential for maximizing the utility of the data, supporting analytics and decision-making processes, and ensuring compatibility with various tools and technologies.

This POC will help to provide data to downstream ETL pipelines (data warehousing, and reporting systems) are optimised for structured data.

Definition of Done

1. Pick a one or more use case and provide end to end output

• Upload to s3
• Feature extraction
• Validation or tranformation
• Output catered via api end point from sagemaker

3. The output should have the following factors

• Explainability
• Confidence score
• Compare output from few models ( optional)

3. Capture any indications of security considerations and the environemt issues from the perspective of a data engineer.

What skills do we need?

1. General Knowledge about LLM or NLP (we can upskill if dont have any prior knowlegde)
2. Technical understanding about sagemaker
3. Python knowledge

Useful links

• LLM Mini-Series E3 - From Unstructured Data to Structured Data with Open-Source LLMs. https://www.youtube.com/watch?v=AATvXw7ECRo
• https://positivethinking.tech/insights/llm-mini-series-from-unstructured-data-to-structured-data-with-open-source-llms/ ( Contains sample dataset and model)
• https://github.com/aws/amazon-sagemaker-examples/tree/main/advanced_functionality/scikit_bring_your_own

[tom-webber]

LLMs struggle with consistent structuring of data, this video from the Pydantic guy is incredibly insightful and valuable information on getting consistently structured outputs