fixing bugs when some, but not all, custom API URLs are specified

README.md

@@ -45,6 +45,7 @@ edfi_api:
   base_url: https://api.schooldistrict.org/v5.3/api
   oauth_url: https://api.schooldistrict.org/v5.3/api/oauth/token 
   dependencies_url: https://api.schooldistrict.org/v5.3/api/metadata/data/v3/2024/dependencies
+  open_api_metadata_url: https://api.schooldistrict.org/v5.3/api/metadata/
   descriptors_swagger_url: https://api.schooldistrict.org/v5.3/api/metadata/data/v3/2024/descriptors/swagger.json
   resources_swagger_url: https://api.schooldistrict.org/v5.3/api/metadata/data/v3/2024/resources/swagger.json
   version: 3
@@ -72,11 +73,12 @@ show_stacktrace: True
 * (optional) Specify the `namespace` to use when accessing the Ed-Fi API. The default is `ed-fi` but others include `tpdm` or custom values. To send data to multiple namespaces, you must use a YAML configuration file and `lightbeam send` for each.
 * Specify the details of the `edfi_api` to which to connect including
   * (optional) The `base_url` which serves a JSON object specifying the paths to data endpoints, Swagger, and dependencies. The default is `https://localhost/api` (the address of an Ed-Fi API [running locally in Docker](https://techdocs.ed-fi.org/display/EDFITOOLS/Docker+Deployment)), but the location varies depending on how Ed-Fi is deployed.
-  * If the metadata for a particular API is not located in the "default" location (at the root of the base_url), then ALL the following urls should be explicitly specified.  These can normally be left blank, unless you are encountering errors indicating that the metadata files cannot be found (such as "Could not parse response from [base_url]").
+  * Most Ed-Fi APIs publish other endpoints they provide in a JSON document at the base URL - `lighbteam` attempts to discover these. If, however, your API does not publish these URLs at the base, or you want to manually override any of them for any reason, you can specify the following:
     * (optional) `oauth_url` (usually [base_url]/oauth/token)
     * (optional) `dependencies_url` (usually [base_url]/metadata/data/v3/dependencies)
     * (optional) `descriptors_swagger_url` (usually [base_url]/metadata/data/v3/descriptors/swagger.json)
     * (optional) `resources_swagger_url` (usually [base_url]/metadata/data/v3/resources/swagger.json)
+    * (optional) `open_api_metadata_url` (usually [base_url]/metadata/) - this is ignored if both `descriptors_swagger_url` and `resources_swagger_url` are specified
   * The `version` as one of `3` or `2` (`2` is currently unsupported).
   * (optional) The `mode` as one of `shared_instance`, `sandbox`, `district_specific`, `year_specific`, or `instance_year_specific`.
   * (required if `mode` is `year_specific` or `instance_year_specific`) The `year` used to build the resource URL. The default is the current year.

lightbeam/api.py

@@ -42,16 +42,22 @@ def prepare(self):
         # then they don't need to be pulled from api metadata, so this section can be skipped.
         # This is most common if the api metadata json files are not in the "default" location.
         # Otherwise, pull urls from api metadata.
-        if self.config["oauth_url"] == "" or self.config["dependencies_url"]== "":
+        if (
+            self.config.get("oauth_url", "")==""
+            or self.config.get("dependencies_url", "")==""
+            or self.config.get("open_api_metadata_url", "")==""
+        ):
             try:
                 api_base = api_base.json()
+                if self.config.get("oauth_url", "")=="":
+                    self.config["oauth_url"] = api_base["urls"]["oauth"]
+                if self.config.get("dependencies_url", "")=="":
+                    self.config["dependencies_url"] = api_base["urls"]["dependencies"]
+                if self.config.get("open_api_metadata_url", "")=="":
+                    self.config["open_api_metadata_url"] = api_base["urls"]["openApiMetadata"]
             except Exception as e:
                 self.logger.critical("could not parse response from {0} ({1})".format(self.config["base_url"], str(e)))
 
-            self.config["oauth_url"] = api_base["urls"]["oauth"]
-            self.config["dependencies_url"] = api_base["urls"]["dependencies"]
-            self.config["open_api_metadata_url"] = api_base["urls"]["openApiMetadata"]
-
         # load all endpoints in dependency-order
         self.lightbeam.all_endpoints = self.get_sorted_endpoints()
 
@@ -168,8 +174,20 @@ def get_sorted_endpoints(self):
     # Loads the Swagger JSON from the Ed-Fi API
     def load_swagger_docs(self):
 
+        # If Swagger URLs are explicitly set, use them
+        if self.config.get("descriptors_swagger_url", "")!="" and self.config.get("resources_swagger_url", ""):
+            openapi = [
+                {
+                    "name": "descriptors",
+                    "endpointUri": self.config["descriptors_swagger_url"],
+                }, 
+                {
+                    "name": "resources",
+                    "endpointUri": self.config["resources_swagger_url"],
+                }
+            ]
         # If the metadata URL is set (pulled from root metadata file earlier), then pull endpoint urls from metadata
-        if "open_api_metadata_url" in self.config.keys() and self.config["open_api_metadata_url"] != "":
+        elif self.config.get("open_api_metadata_url", "")!="":
             # grab Descriptors and Resources swagger URLs
             try:
                 self.logger.debug("fetching swagger docs...")
@@ -180,20 +198,11 @@ def load_swagger_docs(self):
                 openapi = response.json()
 
             except Exception as e:
-                self.logger.critical("Unable to load Swagger docs from API... terminating. Check API connectivity.")
-
-        # If metadata URL is not found, set endpoint URLs from config file
+                self.logger.critical("Unable to load Swagger docs from API... terminating. Check your `edfi_api.open_api_metadata_url` and/or manually specify `edfi_api.descriptors_swagger_url` and `edfi_api.resources_swagger_url`.")
+        # We can only get here if the API doesn't publish its metadata AND the user didn't configure Swagger URLs
         else:
-            openapi = [
-                {
-                    "name": "descriptors",
-                    "endpointUri": self.config["descriptors_swagger_url"],
-                }, 
-                {
-                    "name": "resources",
-                    "endpointUri": self.config["resources_swagger_url"],
-                }
-            ]
+            self.logger.critical("Swagger docs for the API were not discoverable... please manually specify them in `lightbeam.yaml` as `edfi_api.descriptors_swagger_url` and `edfi_api.resources_swagger_url`.")
+
 
         # load (or re-use cached) Descriptors and Resources swagger
         self.descriptors_swagger = None