How to processing big data with pandasAI

[code4indo]

I'm facing challenges in retrieving data from a database table that has a very large size and a huge amount of data into a dataframe. Do you have any recommendations for optimizing the analysis process?

[gventuri]

@code4indo are you using PandasAI connectors? One thing could be to filter the data before it gets converted into a df (which PandasAI allows)

[code4indo]

I am using SQLAlchemy to connect with an Oracle database, and I need to analyze several tables from the database. Here is an example of my code. Is there an alternative way to optimize performance? For instance, the transaction data contains about 7 million rows

import pandas as pd
from sqlalchemy import create_engine
from pandasai import SmartDatalake

# Create a connection engine for Oracle
# Replace 'user', 'password', 'host', 'port', and 'dbname' with your Oracle database credentials
conn_str = 'oracle+cx_oracle://user:password@host:port/dbname'
engine = create_engine(conn_str)

# Define the table names
# These should be the names of the tables in your Oracle database
nama_tabel = ['Customers', 'Accounts', 'Transactions']

# Read each table and store them in a list
# This loop goes through each table name, reads the table using pandas, and appends the DataFrame to the list
dataframes = [pd.read_sql_table(tabel, con=engine) for tabel in nama_tabel]

# Insert the dataframes into the Datalake
# SmartDatalake is used here to manage and process the data
df = SmartDatalake(dataframes)

# Query the Datalake
# This line sends a query to the SmartDatalake to get specific information
# It asks for the top 5 customers with the highest balance and their details like balance, email, and last transaction
response = df.chat("Display the first 5 customers with the highest balance and provide information on their balance, email, and 5 last transaction")

print(response)