composite functions and main function improvements

README.md

@@ -1,75 +1,77 @@
 ![PyPI - Version](https://img.shields.io/pypi/v/eptr2) ![PyPI - Downloads](https://img.shields.io/pypi/dm/eptr2) 
 
-> [!IMPORTANT]  
-> 🇬🇧 You will need username and password credentials from EPIAS to access Transparency Platform data. Register through [EPIAS Registration Platform](https://kayit.epias.com.tr/home) and get your username (your email) and password. English version is available. `eptr2` is still in active development. Breaking changes can be expected. Fill an [issue](https://github.com/tideseed/eptr2/issues) if you encounter any problem.
-
-> [!ÖNEMLİ]  
-> 🇹🇷 Şeffaflık Platformu verilerine erişmek için EPİAŞ üzerinden kayıt yaparak kullanıcı adı ve şifre almanız gerekmektedir. [EPİAŞ Kayıt Platformu](https://kayit.epias.com.tr/home) üzerinden kullanıcı adınızı (kayıt e-postası) ve şifrenizi alabilirsiniz. `eptr2` hala aktif olarak geliştirilmektedir. Büyük değişiklikler beklenebilir. Herhangi bir sorunda, [issue](https://github.com/tideseed/eptr2) kısmından istek açabilirsiniz.
-
-
-# EPIAS Transparency Platform v2.0 Python client by Robokami Data
-
-🇬🇧 `eptr2` (**EP**IAS **Tr**ansparency **2**.0) package is a thin wrapper around [EPIAS Transparency Platform v2.0](https://seffaflik.epias.com.tr/home) API brought to you by [Robokami](https://robokami.com). It is an unofficial package with Apache License 2.0 (free and permissable use for commercial applications, [see details](https://www.tldrlegal.com/license/apache-license-2-0-apache-2-0)). `eptr2` accesses currently more than 213 services with convenience methods.
-
-
-🇹🇷 `eptr2` (**EP**İAŞ **Tr**ansparency **2**.0) paketi [Robokami](https://robokami.com) tarafından [EPİAŞ Şeffaflık Platformu 2.0](https://seffaflik.epias.com.tr/home) API'si üzerine geliştirilmiş bir Python paketidir. Apache License 2.0 ile lisanslanmıştır ([ücretsiz ve büyük ölçüde serbest kullanım](https://www.tldrlegal.com/license/apache-license-2-0-apache-2-0)). `eptr2` 213'ten fazla veri servisine erişim sağlar.
+# Quickstart
 
+This document is a quickstart guide for `eptr2` package. It is a Python client for [EPIAS Transparency Platform v2.0](https://seffaflik.epias.com.tr/home) API. It is an unofficial package with Apache License 2.0.
 
 ## Installation
 
-You can simply use PyPI to install `eptr2` package or directly through GitHub. See [eptr2demo](https://eptr2demo.streamlit.app) page for available calls and examples.
+You can easily install it from PyPI with the following commmand.
 
 ```bash
 pip install eptr2
 ```
 
-NOTE: Starting from v0.4.0, data frame returns will be optional. If pandas is not installed, data frames will not be returned. You can install "dataframe" version with the following command. _(Not implemented yet)_
-
-```bash
-pip install "eptr2[dataframe]"
-```
+If you want to the additional features, it is recommended to install it with the extras. Extras currently include `pandas` and `streamlit`. You can install the package with the following command.
 
 ```bash
-pip install git+https://github.com/Tideseed/eptr2.git
+pip install "eptr2[allextras]"
 ```
 
 ## Usage
 
-You can simply use `EPTR2` class to call services with convenience methods. 
+You can simply use `EPTR2` class to call services with convenience methods. You need to [register](https://kayit.epias.com.tr/epias-transparency-platform-registration-form) with the [EPIAS Transparency Platform](https://seffaflik.epias.com.tr/) to get your username (i.e. registration email) and password. The platform also accommodates an English version.
+
+Below is an example of getting Market Clearing Price (MCP) / Piyasa Takas Fiyatı (PTF). All services use the same pattern.
 
 ```python
 from eptr2 import EPTR2
 
-cred_d = {
-    "username": "YOUR_USERNAME",
-    "password": "YOUR_PASSWORD",
-    "is_test": False, ## (optional) Default: False. Set only to True for transparency test servers.
-}
-
 eptr = EPTR2(
-    username=cred_d["username"], password=cred_d["password"], is_test=cred_d["is_test"]
+    username="YOUR_USERNAME", password="YOUR_PASSWORD"
 )
 
 res = eptr.call("mcp", start_date="2024-07-29", end_date="2024-07-29")
 ```
 
-You can search for available calls with `eptr.get_available_calls()` function. We plan to include all transparency services in the future.
+There are more than 213 calls available. You can search for available calls with `eptr.get_available_calls()` function. This is almost an exhaustive list of available calls in the platform currently. 
+
+### Live Tutorial
+
+Starting from version 1.0.0, `eptr2` package includes a live tutorial feature as a Streamlit app (p.s. You need to have Streamlit installed). You can run the following code to start the tutorial. Its functionality is almost the same as [eptr2demo app](https://eptr2demo.streamlit.app/).
 
 ```python
-available_calls = eptr.get_available_calls()
-print(available_calls)
+from eptr2.tutorials import run_demo_app
+
+run_demo_app(username="YOUR_USERNAME",password="YOUR_PASSWORD")
 ```
 
+_More tutorials are expected to be added in the future._
+
+# About EPIAS Transparency Platform v2.0 Python client by Robokami Data
+
+🇬🇧 `eptr2` (**EP**IAS **Tr**ansparency **2**.0) package is a thin wrapper around [EPIAS Transparency Platform v2.0](https://seffaflik.epias.com.tr/home) API brought to you by [Robokami](https://robokami.com). It is an unofficial package with Apache License 2.0 (free and permissable use for commercial applications, [see details](https://www.tldrlegal.com/license/apache-license-2-0-apache-2-0)). `eptr2` accesses currently more than 213 services with convenience methods.
+
+
+🇹🇷 `eptr2` (**EP**İAŞ **Tr**ansparency **2**.0) paketi [Robokami](https://robokami.com) tarafından [EPİAŞ Şeffaflık Platformu 2.0](https://seffaflik.epias.com.tr/home) API'si üzerine geliştirilmiş bir Python paketidir. Apache License 2.0 ile lisanslanmıştır ([ücretsiz ve büyük ölçüde serbest kullanım](https://www.tldrlegal.com/license/apache-license-2-0-apache-2-0)). `eptr2` 213'ten fazla veri servisine erişim sağlar.
+
+
+## Advanced Topics
+
 ### Aliases
 
-Starting from `v0.7.0` you can create aliases for your calls. Just prepare an alias dictionary and add it to the `EPTR2` object. 
+There are default aliases for the calls. For instance, "ptf" is an alias for "mcp". You can use aliases to call services. 
+
+```python
+res = eptr.call("ptf", start_date="2024-07-29", end_date="2024-07-29")
+```
+
+You can also create aliases for your calls. Just prepare an alias dictionary and add it to the `EPTR2` object. 
 
 ```python
 custom_aliases = {"market-clearing-price": "mcp", "system-marginal-price": "smp"}
 
-eptr = EPTR2(
-    username=cred_d["username"], password=cred_d["password"], is_test=cred_d["is_test"], custom_aliases=custom_aliases
-)
+eptr = EPTR2(username="YOUR_USERNAME",password="YOUR_PASSWORD", custom_aliases=custom_aliases)
 ```
 
 As a warning aliases may overwrite the default keys and default aliases. For instance if your alias is "mcp" pointing to "smp", now default "mcp" call is overwritten with "mcp" alias pointing to "smp".
@@ -82,10 +84,19 @@ eptr.get_aliases(include_custom_aliases = True)
 eptr.get_available_calls(include_aliases = True)
 ```
 
-## Notes
+### Composite Functions
+
+_New feature in version 1.0.0_
 
-Main object call has some parameters to control the behavior of the package. 
+Composite functions are combinations of multiple calls under a single table for a purpose. That purpose might be to gather reporting data or training data for forecast models. You can create your own composite functions with `eptr2` package or use already available ones. 
+
+Our first composite function is `get_hourly_consumption_and_forecast_data`. It returns a data frame with a combination of Load Plan, UECM and Real Time Consumption.
+
+```python
+from eptr2 import EPTR2
+from eptr2.composite import get_hourly_consumption_and_forecast_data
 
-+ You can set `ssl_verify` to `False` if you have SSL verification problems. 
-+ You can set `postprocess` to `False` if you don't want to get data frames as response. 
-+ You can set `get_raw_response` to `True` if you want to get raw urllib3 response object.
+eptr = EPTR2(username="YOUR_USERNAME",password="YOUR_PASSWORD")
+df = get_hourly_consumption_and_forecast_data(eptr, start_date="2024-07-29", end_date="2024-07-29")
+print(df)
+```

eptr2/composite/__init__.py

@@ -0,0 +1 @@
+from eptr2.composite.consumption import get_hourly_consumption_and_forecast_data

eptr2/composite/consumption.py

@@ -0,0 +1,57 @@
+from eptr2 import EPTR2
+import pandas as pd
+
+
+def get_hourly_consumption_and_forecast_data(
+    eptr: EPTR2, start_date: str, end_date: str, verbose: bool = False
+):
+    """
+    This composite function gets load plan, UECM (settlement consumption), real time and consumption data. If end date is after the last settlement data, UECM is filled with real time consumption under consumption column.
+    """
+
+    if verbose:
+        print("Loading load plan...")
+
+    lp_df = eptr.call("load-plan", start_date=start_date, end_date=start_date)
+
+    df = lp_df[["date", "lep"]].rename(columns={"lep": "load_plan", "date": "dt"})
+
+    if verbose:
+        print("Loading UECM...")
+
+    uecm_df = eptr.call("uecm", start_date=start_date, end_date=end_date)
+
+    df = df.merge(
+        uecm_df[["period", "swv"]].rename(columns={"period": "dt", "swv": "uecm"}),
+        on="dt",
+        how="outer",
+    )
+
+    if verbose:
+        print("Loading real time consumption...")
+
+    rt_cons = eptr.call("rt-cons", start_date=start_date, end_date=end_date)
+
+    df = df.merge(
+        rt_cons[["date", "consumption"]].rename(
+            columns={"date": "dt", "consumption": "rt_cons"}
+        ),
+        on="dt",
+        how="outer",
+    )
+
+    df["consumption"] = df.apply(
+        lambda x: x["rt_cons"] if pd.isnull(x["uecm"]) else x["uecm"], axis=1
+    )
+
+    return df
+
+
+if __name__ == "__main__":
+    eptr = EPTR2(credentials_file_path="creds/eptr_credentials.json")
+
+    df = get_hourly_consumption_and_forecast_data(
+        eptr=eptr, start_date="2024-01-01", end_date="2024-12-23", verbose=True
+    )
+
+    print("End")

eptr2/main.py

@@ -1,6 +1,7 @@
 from typing import Any
 import urllib3
 import re
+import os
 import json
 from urllib.parse import urljoin
 import copy
@@ -36,16 +37,13 @@ def __init__(
         self.ssl_verify = kwargs.get("ssl_verify", True)
         self.check_postprocess(postprocess=kwargs.get("postprocess", True))
         self.get_raw_response = kwargs.get("get_raw_response", False)
+
+        ### Credentials and Login
         self.username = username
         self.password = password
-        self.is_test = kwargs.get("is_test", False)
-        self.skip_credentials = kwargs.get("skip_credentials", False)
-        root_phrase_test = "-prp" if self.is_test else ""
-        root_phrase_default = f"https://seffaflik{root_phrase_test}.epias.com.tr"
-        self.root_phrase = kwargs.get("root_phrase", root_phrase_default)
-        self.skip_login_warning = kwargs.get("skip_login_warning", False)
-        self.path_map_keys = get_path_map(just_call_keys=True)
-        self.custom_aliases = kwargs.get("custom_aliases", {})
+        self.is_test = kwargs.get("is_test", False)  ## Currently not used
+        self.credentials_file_path = kwargs.get("credentials_file_path", None)
+        self.login(custom_root_phrase=kwargs.get("root_phrase", None))
 
         if tgt_d is not None:
             self.import_tgt_info(tgt_d)
@@ -56,6 +54,31 @@ def __init__(
 
         self.check_renew_tgt()
 
+        ## Path map keys and custom aliases
+        self.path_map_keys = get_path_map(just_call_keys=True)
+        self.custom_aliases = kwargs.get("custom_aliases", {})
+
+    def login(self, custom_root_phrase: str | None = None):
+        if self.username is None or self.password is None:
+            if self.credentials_file_path is not None:
+                with open(self.credentials_file_path, "r") as f:
+                    credentials_d = json.load(f)
+                    self.username = credentials_d["EPTR_USERNAME"]
+                    self.password = credentials_d["EPTR_PASSWORD"]
+            else:
+                self.username = os.environ.get("EPTR_USERNAME", None)
+                self.password = os.environ.get("EPTR_PASSWORD", None)
+
+        if self.username is None or self.password is None:
+            raise Exception(
+                "Username and password must be provided for login. If you do not have the necessary credentials, you can get them from EPIAS Transparency Platform website."
+            )
+
+        if not custom_root_phrase:
+            root_phrase_test = "-prp" if self.is_test else ""
+            root_phrase_default = f"https://seffaflik{root_phrase_test}.epias.com.tr"
+            self.root_phrase = root_phrase_default
+
     ## Ref: https://stackoverflow.com/a/62303969/3608936
     def __getattr__(self, __name: str) -> Any:
 
@@ -85,21 +108,7 @@ def check_renew_tgt(self):
 
     def get_tgt(self, **kwargs):
         if self.username is None or self.password is None:
-            if self.skip_credentials:
-                if not self.skip_login_warning:
-                    print(
-                        "Warning: You chose to skip the credentials and your calls may fail due to authentication requirements. Username and password will be required in the EPIAS Transparency API after August 26 (check EPIAS Transparency website for the latest and detailed information). This warning is shown once per session. If you want to disable it set 'skip_login_warning' parameter to True when calling EPTR2 class."
-                    )
-                    self.skip_login_warning = True
-
-                self.tgt = None
-                self.tgt_exp = 0
-                self.tgt_exp_0 = 0
-                return None
-            else:
-                raise Exception(
-                    "Username and password must be provided for tgt renewal."
-                )
+            raise Exception("Username and password must be provided for tgt renewal.")
 
         test_suffix = "-prp" if self.is_test else ""
         login_url = f"""https://giris{test_suffix}.epias.com.tr/cas/v1/tickets"""
@@ -231,18 +240,6 @@ def call(self, key: str, **kwargs):
         ### There are some calls requiring special handling, they have a special function to process them
         call_body_raw = process_special_calls(key, call_body_raw)
 
-        ## Parameter change
-        # if key in ["bpm-orders", "bpm-orders-w-avg"]:
-        #     if "date_time" in call_body_raw.keys():
-        #         raise Exception(
-        #             f"date_time parameter is not supported for {key}. Use 'date' instead."
-        #         )
-
-        # if key in ["ng-vgp-contract-price-summary-period"]:
-        #     call_body_raw["is_txn_period"] = False
-        # elif key in ["ng-vgp-contract-price-summary-se"]:
-        #     call_body_raw["is_txn_period"] = True
-
         optional_body_params = get_optional_parameters(key)
         all_params = required_body_params + optional_body_params
 

pyproject.toml

@@ -11,9 +11,11 @@ exclude = ["runs", "data", "helpdocs", "creds"]
 [tool.poetry.dependencies]
 python = ">=3.9.6"
 pandas = { version = ">=2.1.3", optional = true }
+streamlit = { version = ">=1.36.0", optional = true }
 
 [tool.poetry.extras]
 dataframe = ["pandas"]
+allextras = ["pandas", "streamlit"]
 
 [build-system]
 requires = ["poetry-core"]