feat: PostgreSQL implementation for Chat History and Vectorization

[gpickett]

Purpose

• This pull request introduces PostgreSQL as a core feature for CWYD, allowing the platform to utilize PostgreSQL for chat history, search indexing, and vector-based similarity queries. It replaces the default database configuration with PostgreSQL and ensures all related infrastructure and backend components are updated accordingly.

Does this introduce a breaking change?

• Yes
• No

How to Test

• Get the code

• Test the code

1. Set up PostgreSQL as the default database:
   • Configure the necessary environment variables:

     export DATABASE_TYPE=PostgreSQL
     export POSTGRES_URI=your_postgres_uri
     export POSTGRES_USER=your_postgres_user
     export POSTGRES_PASSWORD=your_postgres_password
     

   • Ensure pgvector extension is installed in your PostgreSQL database:

     CREATE EXTENSION vector;
     

2. Run the infrastructure deployment script to create the necessary tables
3. Test chat history and search functionalities by interacting with the CWYD platform.

What to Check

Verify that the following are valid:

• Chat history is stored and retrieved successfully from PostgreSQL.
• Search index creation and vector similarity queries work as expected using PostgreSQL.
• Advanced image processing and integrated vectorization features are disabled when DATABASE_TYPE=PostgreSQL.
• Switching back to CosmosDB (DATABASE_TYPE=CosmosDB) retains existing functionality.

Other Information

• Ensure that PostgreSQL credentials are securely managed in the environment or using Azure Key Vault.
• For any issues during testing or deployment, check logs for connection and query errors.
• This change replaces CosmosDB as the default database, which may affect deployments relying on previous configurations.

[github-actions]

Coverage Report •

File	Stmts	Miss	Cover	Missing
code/backend/api
chat_history.py	247	43	82%	26–28, 40, 46–48, 65, 116, 140–141, 144, 173, 211–212, 215, 252, 301, 310, 317, 319, 324, 328, 330, 333, 342–343, 346, 350–352, 370, 374, 379, 399, 408, 414, 428, 467, 505, 507–508, 510
code/backend/batch/utilities/chat_history
cosmosdb.py	111	93	16%	18–24, 27–29, 31, 33–34, 37–38, 40–41, 44–45, 48, 51, 54, 59–63, 68–71, 73, 76, 86–88, 90, 93–95, 97, 100, 103–104, 107, 109, 113–117, 120–121, 124–127, 129–130, 133, 135, 138, 142–144, 147, 150–151, 153, 156, 167–168, 170–171, 173–178, 180, 183, 186–189, 191, 194, 198–200, 203, 205
database_client_base.py	39	12	69%	9, 14, 19, 26, 31, 36, 43, 50, 57, 68, 75, 82
database_factory.py	23	1	95%	59
postgresdbservice.py	85	65	23%	16–20, 23–25, 28, 36–38, 41–42, 45–47, 50–52, 57, 60, 63, 71, 81, 84–86, 89–91, 94–97, 99, 105–108, 110–112, 115–116, 119–121, 124–127, 132–133, 144–147, 149, 152–154, 157–159
code/backend/batch/utilities/helpers
azure_postgres_helper.py	130	13	90%	83–86, 103, 109–110, 114–118, 120
env_helper.py	184	16	91%	118–119, 122–125, 127–128, 130, 334–336, 358, 363–365
code/backend/batch/utilities/helpers/config
config_helper.py	176	2	98%	87, 90
database_type.py	4	0	100%
code/backend/batch/utilities/helpers/embedders
embedder_factory.py	14	1	92%	15
postgres_embedder.py	48	1	97%	74
code/backend/batch/utilities/orchestrator
orchestrator_base.py	50	1	98%	33
code/backend/batch/utilities/parser
output_parser_tool.py	39	0	100%
code/backend/batch/utilities/search
postgres_search_handler.py	66	5	92%	48, 51, 65, 92, 97
search.py	18	1	94%	17
code/backend/pages
02_Explore_Data.py	33	33	0%	1–8, 10–11, 13, 21, 29, 32–34, 38, 41–42, 45–49, 51, 55–56, 58–61, 64–65
03_Delete_Data.py	46	46	0%	1–9, 11–13, 15, 23–25, 29, 33, 41, 43, 45–46, 48–50, 54–55, 57, 59–61, 65, 69–73, 75, 78–79, 83–88
04_Configuration.py	178	178	0%	1–11, 13–14, 16, 24–26, 30, 32, 37–46, 49–50, 53–62, 65–74, 77–78, 80–82, 86–87, 99–103, 106–107, 111–112, 115, 119–120, 124–125, 130–131, 135–137, 140–141, 144–145, 148–149, 172, 174–175, 177–181, 183–186, 189–194, 206–209, 223–224, 234–238, 258–263, 270, 272, 277, 285, 293, 300–301, 308, 310–311, 315, 323, 329, 336–337, 357–361, 367–368, 389–390, 392–393, 398, 403, 409–412, 418–419, 422, 441, 482–483, 487, 490–492, 494, 498–499, 502–505, 507–508, 510–512, 514–517, 519–520
scripts/data_scripts
create_postgres_tables.py	56	56	0%	1–5, 7–13, 16, 27–28, 35–36, 42, 45, 48, 57–58, 61, 64–65, 68–69, 71, 80–81, 84–85, 87, 98–99, 103–104, 106–107, 109, 122–123, 126–127, 129–130, 132–133, 135–136, 138–141, 143–144
TOTAL	3644	945	74%

Tests	Skipped	Failures	Errors	Time
370	0 💤	0 ❌	0 🔥	1m 0s ⏱️

[Prajwal-Microsoft]

Changes approved for PostgreSQL implementation

[github-actions]

🎉 This PR is included in version 1.13.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀