Merge pull request #1288 from JosepSampe/lithops-dev

Update runtime docs and changelog
lithops-cloud · Mar 27, 2024 · b199505 · b199505
2 parents 1af5574 + d6fa474
commit b199505
Show file tree

Hide file tree

Showing 14 changed files with 122 additions and 189 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,18 @@
 # Changelog
 
-## [v3.1.3.dev1]
+## [v3.2.1.dev0]
+
+### Added
+- 
+
+### Changed
+- 
+
+### Fixed
+-
+
+
+## [v3.2.0]
 
 ### Added
 - [Lithops] Addded support for Python 3.12
@@ -17,6 +29,7 @@
 - [SSH Cli] Fixed minor error with the "err" variable
 - [Cli] Fixed job status on "lithops job list" for standalone backends
 - [Standalone] Fixed issue in the "lithops image build" that appears when the vpc is already created
+- [Future] Fixed issue with missing 'worker_end_tstamp' variable
 
 ## [v3.1.2]
 

diff --git a/config/README.md b/config/README.md
@@ -54,7 +54,6 @@ Storage Backends
 - [Aliyun Functions](../docs/source/compute_config/aliyun_functions.md)
 - [Oracle Cloud Functions](../docs/source/compute_config/oracle_functions.md)
 - [OpenWhisk](../docs/source/compute_config/openwhisk.md)
-- [IBM Cloud Functions](../docs/source/compute_config/ibm_cf.md)
 
 <b>Serverless (CaaS) Backends:</b>
 - [IBM Code Engine](../docs/source/compute_config/code_engine.md)
@@ -113,7 +112,7 @@ if __name__ == '__main__':
 ```
 
 ### Providing configuration in runtime
-Example of providing configuration keys for IBM Cloud Functions and IBM Cloud Object Storage
+Example of providing configuration keys for IBM Code Engine and IBM Cloud Object Storage
 
 ```python
 import lithops

diff --git a/runtime/aliyun_fc/README.md b/runtime/aliyun_fc/README.md
@@ -1,27 +1,10 @@
 # Lithops runtime for Aliyun Functions Compute
 
-The runtime is the place where your functions are executed.
+The runtime is the place where your functions are executed. The default runtime is automatically created the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it.
 
-The default runtime is created the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it.
+Currently, Aliyun Functions Compute supports Python 3.6, 3.9 and 3.10. You can find the list of pre-installed modules [here](https://www.alibabacloud.com/help/en/function-compute/latest/python-event-functions). In addition, the Lithops default runtimes are built with the packages included in this [requirements.txt](requirements.txt) file:
 
-Currently, Aliyun Functions Compute supports Python 3.6, and 3.9, and it provides the following default runtimes with some packages already preinstalled:
-
-
-| Runtime name | Python version | Packages included |
-| ----| ----| ---- |
-| lithops-default-runtime-v36 | 3.6 | [list of packages](https://www.alibabacloud.com/help/en/function-compute/latest/python-event-functions) |
-| lithops-default-runtime-v38 | 3.9 | [list of packages](https://www.alibabacloud.com/help/en/function-compute/latest/python-event-functions) |
-
-Lithops default runtimes are also ship with the following packages:
-```
-pika
-tblib
-cloudpickle
-ps-mem
-```
-
-
-To run a function with the default runtime you don't need to specify anything in the code, since everything is managed internally by Lithops:
+To run a function with the default runtime you don't need to specify anything in the code, since everything is handled internally by Lithops:
 
 ```python
 import lithops
@@ -45,44 +28,14 @@ pw = lithops.FunctionExecutor(runtime_memory=512)
 
 **Build your own Lithops runtime for Aliyun Functions Compute**
 
-If you need some Python modules which are not included in the default runtime, it is possible to build your own Lithops runtime with all of them.
-
-To build your own runtime, you have to collect all necessary modules in a `requirements.txt` file.
+If you require additional Python modules not included in the default runtime, you can create your own custom Lithops runtime incorporating them. To create a custom runtime, compile all the necessary modules into a `requirements.txt` file.
 
-For example, we want to add module `matplotlib` to our runtime, since it is not provided in the default runtime.
+For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory pakcges required by lithops, so you don't have to remove any of them from the list, but just add your packages at the end.
 
-First, we need to extend the default `requirements.txt` file provided with Lithops with all the modules we need. For our example, the `requirements.txt` will contain the following modules:
-```
-numpy
-scikit-learn
-scipy
-pandas
-google-cloud
-google-cloud-storage
-google-cloud-pubsub
-certifi
-chardet
-docutils
-httplib2
-idna
-jmespath
-kafka-python
-lxml
-pika==0.13.0
-python-dateutil
-redis
-requests
-simplejson
-six
-urllib3
-virtualenv
-PyYAML
-matplotlib
-```
+After updating the file accordingly, you can proceed to build the custom runtime by specifying the modified `requirements.txt` file along with a chosen runtime name:
 
-Then, we will build the runtime, specifying the modified `requirements.txt` file and a runtime name:
 ```
-$ lithops runtime build -f requirements.txt my_matplotlib_runtime -b aliyun_fc
+$ lithops runtime build -b aliyun_fc -f requirements.txt my_matplotlib_runtime 
 ```
 
 This command will built and deploy a runtime called `my_matplotlib_runtime` to the available Aliyun FUnctions Compute runtimes.

diff --git a/runtime/aliyun_fc/requirements.txt b/runtime/aliyun_fc/requirements.txt
@@ -1,3 +1,6 @@
+# Requirements.txt contains a list of dependencies for the Python Application #
+
+# Mandatory Lithops packages
 aliyun-fc2
 oss2
 pika
@@ -11,4 +14,8 @@ numpy
 cloudpickle
 ps-mem
 tblib
-psutil
+psutil
+
+# Optional packages (Add your packages below)
+#pandas
+#matplotlib
diff --git a/runtime/aws_lambda/README.md b/runtime/aws_lambda/README.md
@@ -71,14 +71,14 @@ lithops runtime build -f MyDockerfile -b aws_lambda my-container-runtime-name
 For example:
 
 ```
-lithops runtime build -f MyDockerfile -b aws_lambda lithops-ndvi-v310:01
+lithops runtime build -f MyDockerfile -b aws_lambda lithops-ndvi-v312:01
 ```
 
 Finally, we can specify this new runtime in the lithops config:
 
 ```yaml
 aws_lambda:
-    runtime: lithops-ndvi-v310:01
+    runtime: lithops-ndvi-v312:01
 ```
 
 or when creating a Lithops Function Executor:
@@ -89,7 +89,7 @@ import lithops
 def test():
     return 'hello'
 
-lith = lithops.FunctionExecutor(runtime='lithops-ndvi-v310:01')
+lith = lithops.FunctionExecutor(runtime='lithops-ndvi-v312:01')
 lith.call_async(test, data=())
 res = lith.get_result()
 print(res)  # Prints 'hello'

diff --git a/runtime/azure_containers/README.md b/runtime/azure_containers/README.md
@@ -51,27 +51,27 @@ azure_containers:
         $ docker login
 
     Update the Dockerfile that better fits to your requirements with your required system packages and Python modules.
-    If you need another Python version, for example Python 3.9, you must change the initial line of the Dockefile:
+    If you need another Python version, for example Python 3.12, you must change the initial line of the Dockefile:
 
-        $ lithops runtime build docker_username/runtimename:tag -b azure_containers
+        $ lithops runtime build -b azure_containers docker_username/runtimename:tag 
 
     Note that Docker hub image names look like *"docker_username/runtimename:tag"* and must be all lower case, for example:
 
-        $ lithops runtime build -b azure_containers myaccount/lithops-ca-custom-v39:01
+        $ lithops runtime build -b azure_containers myaccount/lithops-ca-custom-v312:01
 
     By default the Dockerfile should be located in the same folder from where you execute the **lithops runtime** command. If your Dockerfile is located in another folder, or the Dockerfile has another name, you can specify its location with the **-f** parameter, for example:
 
-        $ lithops runtime build -b azure_containers -f azure_containers/Dockerfile myaccount/lithops-ca-custom-v39:01
+        $ lithops runtime build -b azure_containers -f azure_containers/Dockerfile myaccount/lithops-ca-custom-v312:01
 
     Once you have built your runtime with all of your necessary packages, you can already use it with Lithops.
     To do so, you have to specify the full docker image name in the configuration or when you create the **FunctionExecutor** instance, or directly in the config file, for example:
 
     ```python
     import lithops
-    fexec = lithops.FunctionExecutor(runtime='myaccount/lithops-ca-custom-v39:01')
+    fexec = lithops.FunctionExecutor(runtime='myaccount/lithops-ca-custom-v312:01')
     ```
 
-    *NOTE: In this previous example shows how to build a Docker image based on Python 3.9, this means that now you also need Python 3.9 in the client machine.*
+    *NOTE: In this previous example shows how to build a Docker image based on Python 3.12, this means that now you also need Python 3.12 in the client machine.*
 
 2. **Use an already built runtime from a public repository**
 
@@ -80,29 +80,29 @@ azure_containers:
 
     ```python
     import lithops
-    fexec = lithops.FunctionExecutor(runtime='lithopscloud/ca-conda-v39:01')
+    fexec = lithops.FunctionExecutor(runtime='lithopscloud/ca-conda-v312:01')
     ```
 
     Alternatively, you can create a Lithops runtime based on already built Docker image by executing the following command, which will deploy all the necessary information to use the runtime with your Lithops.
 
         $ lithops runtime deploy -b azure_containers -s azure_storage docker_username/runtimename:tag
 
-    For example, you can use an already created runtime based on Python 3.9 and with the *matplotlib* and *nltk* libraries by running:
+    For example, you can use an already created runtime based on Python 3.12 and with the *matplotlib* and *nltk* libraries by running:
 
-        $ lithops runtime deploy -b azure_containers -s azure_storage lithopscloud/ca-matplotlib-v39:01
+        $ lithops runtime deploy -b azure_containers -s azure_storage lithopscloud/ca-matplotlib-v312:01
 
     Once finished, you can use the runtime in your Lithops code:
 
     ```python
     import lithops
-    fexec = lithops.FunctionExecutor(runtime='lithopscloud/ca-matplotlib:v39:01')
+    fexec = lithops.FunctionExecutor(runtime='lithopscloud/ca-matplotlib:v312:01')
     ```
 
 ## Runtime Management
 
 1. **Update an existing runtime**
 
-    If you are a developer, and modified the PyWeen source code, you need to deploy the changes before executing Lithops.
+    If you are a developer, and modified the Lithops source code, you need to deploy the changes before executing Lithops.
 
     You can update default runtime by:
 
@@ -112,9 +112,9 @@ azure_containers:
 
         $ lithops runtime update docker_username/runtimename:tag -b azure_containers -s azure_storage
 
-    For example, you can update an already created runtime based on the Docker image `lithopscloud/ca-matplotlib-v39:01` by:
+    For example, you can update an already created runtime based on the Docker image `lithopscloud/ca-matplotlib-v312:01` by:
 
-        $ lithops runtime update lithopscloud/ca-matplotlib-v39:01 -b azure_containers -s azure_storage
+        $ lithops runtime update lithopscloud/ca-matplotlib-v312:01 -b azure_containers -s azure_storage
 
     Alternatively, you can update all the deployed runtimes at a time by:
 
@@ -132,9 +132,9 @@ azure_containers:
 
         $ lithops runtime delete docker_username/runtimename:tag -b azure_containers -s azure_storage
 
-    For example, you can delete runtime based on the Docker image `lithopscloud/ca-conda-v39:01` by:
+    For example, you can delete runtime based on the Docker image `lithopscloud/ca-conda-v312:01` by:
 
-        $ lithops runtime delete lithopscloud/ca-conda-v39:01 -b azure_containers -s azure_storage
+        $ lithops runtime delete lithopscloud/ca-conda-v312:01 -b azure_containers -s azure_storage
 
     You can delete all the runtimes at a time by:
 

diff --git a/runtime/azure_functions/README.md b/runtime/azure_functions/README.md
@@ -1,37 +1,10 @@
 # Lithops runtime for Azure Functions
 
-The runtime is the place where your functions are executed.
+The runtime is the place where your functions are executed. The default runtime is automatically created the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it.
 
-The default runtime is created the first time you execute a function. Lithops automatically detects the Python version of your environment and deploys the default runtime based on it.
+Currently, Azure Functions supports Python 3.6, 3.7, 3.8 and 3.9. You can find the list of pre-installed modules [here](https://github.com/Azure/azure-functions-python-worker/wiki/Preinstalled-Python-Libraries). In addition, the Lithops default runtimes are built with the packages included in this [requirements.txt](requirements.txt) file
 
-Currently, Azure Functions supports Python 3.6, 3.7, 3.8 and 3.9, and it provides the following default runtimes with some packages already preinstalled:
-
-| Runtime name | Python version | Packages included |
-| ----| ----| ---- |
-| lithops-runtime-v36 | 3.6 | [list of packages](https://github.com/Azure/azure-functions-python-worker/wiki/Preinstalled-Python-Libraries) |
-| lithops-runtime-v38 | 3.7 | [list of packages](https://github.com/Azure/azure-functions-python-worker/wiki/Preinstalled-Python-Libraries) |
-| lithops-runtime-v38 | 3.8 | [list of packages](https://github.com/Azure/azure-functions-python-worker/wiki/Preinstalled-Python-Libraries) |
-| lithops-runtime-v38 | 3.9 | [list of packages](https://github.com/Azure/azure-functions-python-worker/wiki/Preinstalled-Python-Libraries) |
-
-Lithops default runtimes are also ship with the following packages:
-```
-azure-functions
-azure-storage-blob
-azure-storage-queue
-pika
-flask
-gevent
-redis
-requests
-PyYAML
-kubernetes
-numpy
-cloudpickle
-ps-mem
-tblib
-```
-
-To run a function with the default runtime you don't need to specify anything in the code, since everything is managed internally by Lithops:
+To run a function with the default runtime you don't need to specify anything in the code, since everything is handled internally by Lithops:
 
 ```python
 import lithops
@@ -50,32 +23,13 @@ result = lithops.get_result()
 
 **Build your own Lithops runtime for Azure Functions**
 
-If you need some Python modules which are not included in the default runtime, it is possible to build your own Lithops runtime with all of them.
-
-To build your own runtime, you have to collect all necessary modules in a `requirements.txt` file. For example, if you want to add the modules `numpy` and `matplotlib` to our runtime, since they are not provided in the default runtime.
+If you require additional Python modules not included in the default runtime, you can create your own custom Lithops runtime incorporating them. To create a custom runtime, compile all the necessary modules into a `requirements.txt` file.
 
-First, we need to extend the `requirements.txt` file provided with Lithops with all the modules we need. For our example, the `requirements.txt` should contain the following modules (note that we added `numpy` and `matplotlib` at the end):
-```
-azure-functions
-azure-storage-blob
-azure-storage-queue
-pika
-flask
-gevent
-redis
-requests
-PyYAML
-kubernetes
-cloudpickle
-ps-mem
-tblib
-numpy
-matplotlib
-```
+For instance, if you wish to integrate the `matplotlib` module into your runtime, which isn't part of the default setup, you need to append it to the existing [requirements.txt](requirements.txt) file. Note that this `requirements.txt` contains the mandatory pakcges required by lithops, so you don't have to remove any of them from the list, but just add your packages at the end.
 
 **IMPORTANT**: Note that the runtime is built using your local machine, and some libraries, like Numpy, compile some *C* code based on the Operating System you are using. Azure functions run on a Linux machine, this mean that if you use **MacOS** or **Windows** for building the runtime, those libraries that compiled *C* code cannot be executed from within the function. In this case, you must use a Linux machine for building the runtime.
 
-Then, we will build the runtime, specifying the modified `requirements.txt` file and a runtime name:
+After updating the file accordingly, you can proceed to build the custom runtime by specifying the modified `requirements.txt` file along with a chosen runtime name:
 ```
 $ lithops runtime build -b azure_functions -f requirements.txt matplotlib-runtime 
 ```

diff --git a/runtime/azure_functions/requirements.txt b/runtime/azure_functions/requirements.txt
@@ -1,3 +1,6 @@
+# Requirements.txt contains a list of dependencies for the Python Application #
+
+# Mandatory Lithops packages
 azure-functions
 azure-storage-blob
 azure-storage-queue
@@ -12,4 +15,8 @@ kubernetes
 cloudpickle
 ps-mem
 tblib
-psutil
+psutil
+
+# Optional packages (Add your packages below)
+#pandas
+#matplotlib