Here we can find more technical details about the project.
There are two applications in this project:
fiware- responsible for fetching mobility data from OST and inserting it into Orion Context Broker. It gets the full GTFS dataset and puts it as JSON models (CAUTION: it's obsolete because we've decided to use CKAN instead of Orion);ckan- this CKAN connector is able to integrate OST's mobility data in two ways, both using OST APIs:- fetch GTFS files (CSV as specified by the standard) and push them to the DataStore (CKAN extension), using another project's capabilities;
- fetch GTFS Stops located in the Lisbon district, convert them to a specific JSON format (specified by the project's consortium) and push them to CKAN DataStore, using its APIs directly.
utils is not an application, it contains the exceptions, constants and other random stuff needed by the other two apps. Notice that the capability to import GTFS data from CKAN back to OST was also implemented. It's also ckan who's responsible for that, although it only serves for demonstration purposes.
This is a Python project that can be run as Celery workers.
YES. It's strongly recommended to use a virtual environment tool to control the libraries' versions. We suggest pyenv with pyenv-virtualenv, but other solutions may suit you.
After setting that up and creating a virtualenv for this project, install its libraries:
git clone git@github.com:OneStopTransport/OST-FiWare-Lisbon.git
cd OST-FiWare-Lisbon/
git submodule update --init
pip install -r fiware_lisbon/requirements.pip
...and that's it.
As stated before, this project fetches data from OST and inserts it into a CKAN or Context Broker instance. In order for it to work, we need a CKAN instance or a Context Broker instance running on FI-LAB (or on your server) and a suitable API Key to be used with OST.
- OST developer account (Sign up > Login > Settings > Developer Settings) ;
- OST Server API Key;
- Orion Context Broker instance running on FI-LAB with a public IP (check the documentation for details) or on your custom server;
- CKAN instance with DataStore extension. We'll try to provide you a Vagrant for easy deployment;
- CKAN API Key which you can retrieve from your profile, after creating an account on the CKAN instance;
- RabbitMQ instance if you want this being run as a Celery Beat Worker.
For security reasons, all the confidential data (such as the API Keys or the Context Broker Host) were not added to the project's repository, since they're retrieved from the system's environment. If you'll be running this project in an Unix system, you can put the following lines on the ~/.exports file or in one of the following: ~/.bash_profile, ~/.bashrc or ~/.profile:
# One.Stop.Transport API Keys
export OST_SERVER_KEY="<INSERT_OST_API_KEY_HERE>"
export FIWARE_HOST="<INSERT_CONTEXTBROKER_IP_HERE>"
export CKAN_API_KEY="<INSERT_CKAN_API_KEY_HERE>"
export CKAN_HOST="<INSERT_CKAN_HOST_HERE>"
FIWARE_HOST can be an IP address or an URL, but don't forget the http:// protocol and the 1026 port, or else the ContextBroker operations will not work.
If you'll be running this as a Celery worker, you'll need this too:
# RabbitMQ credentials and URLs
export MQ_HOST_PROD="messaging.vhost.ost.pt" # Only if you're running this on the OST architecture
export MQ_HOST_TEST="<INSERT_RABBITMQ_HOST_HERE>"
export MQ_USER="<INSERT_MQ_USER_HERE>"
export MQ_PASSWORD="<INSERT_MQ_PASSWORD_HERE>"
export MQ_VHOST="/"
There are two procedures, one if you'll use this as a regular Python project or as a periodic Celery Beat worker.
Just run the following on the project's directory (fiware_lisbon):
python -m fiware.tasks # to import GTFS data to ContextBroker
python -m ckan.tasks # to import GTFS data to CKAN
Just run the following on the project's directory (fi-ware-lisbon):
celery worker -B -Q fiware_queue
And it will transfer the data at 6:00 AM to the Context Broker and at 6:30 AM to the CKAN DataStore, every day!