diff --git a/docs/data-operations-manual/Explanation/Key-Concepts/Endpoint-types.md b/docs/data-operations-manual/Explanation/Key-Concepts/Endpoint-types.md new file mode 100644 index 0000000..7aa9c12 --- /dev/null +++ b/docs/data-operations-manual/Explanation/Key-Concepts/Endpoint-types.md @@ -0,0 +1,32 @@ +# Endpoint URL Types and Plugins + +The pipeline can collect data published in a wide range of different formats, which means there is a lot of variety in the types of URLs we might add as endpoints. Broadly, however, endpoints typically fall into one of the following two categories: +- Hosted file - these will usually be URL which ends in something like `.json` or `.csv` +- Standards compliant web server - these will usually be identifiable by parts of the URL like `MapServer` or `FeatureServer`, or sections that look like query parameters, like `?service=WFS&version=1.0.0` + +## Data formats of resources that can be processed** +- Geojson (preferred for geospatial data because this mandates Coordinate Reference System WGS84) +- CSV text files containing WKT format geometry +- PDF-A +- Shapefiles +- Excel files containing WKT format geometry (xls, xlsx, xlsm, xlsb, odf, ods, odt) +- Mapinfo +- Zip files containing Mapinfo or Shapefiles +- GML +- Geopackage +- OGC Web Feature Service +- ESRI ArcGIS REST service output in GeoJSON format + +**Hosted files** +These can typically be added as they are with no problems. The pipeline can read most common formats and will transform them into the csv format it needs if they’re not already supplied as csv. + +**Web servers** +Web server endpoints usually provide some flexibility around the format that data is returned in. The data provider may have shared a correctly configured URL which returns valid data, or they may have just provided a link to the server service directory, which does not itself contain data we can process. + +E.g. this URL from Canterbury provides information on a number of different planning-related layers available from their ArcGIS server: +`https://mapping.canterbury.gov.uk/arcgis/rest/services/External/Planning_Constraints_New/MapServer` + +Depending on the endpoint, it may be necessary to either **edit the URL** to return valid data, or **use a plugin** to make sure the data is processed correctly. A plugin is typically needed for an API endpoint if the collector needs to paginate (e.g. the ArcGIS API typically limits to 1,000 records per fetch) or strip unnecessary content from the response (e.g. WFS feeds can sometimes contain access timestamps which can result in a new resource being created each day the collector runs). + +>[!NOTE] +> Wherever possible, we prefer to collect from a URL which returns data in an open standard such as geojson or WFS, instead of the ArcGIS service directory page. \ No newline at end of file diff --git a/docs/data-operations-manual/Explanation/Operational-Procedures.md b/docs/data-operations-manual/Explanation/Operational-Procedures.md index 1c5ea04..c159356 100644 --- a/docs/data-operations-manual/Explanation/Operational-Procedures.md +++ b/docs/data-operations-manual/Explanation/Operational-Procedures.md @@ -6,15 +6,15 @@ These procedures can vary based on whether a dataset is national or compiled, wh To help with this complexity, we've got a few levels of documentation to help: -1. This explanatory overview is at the highest level. -2. Below that, the tutorials section covers a range of different scenarios that can occur when [adding](Adding-Data) and [maintaining](Maintaining-Data) data and explain the procedure that should be followed in each one. +1. This explanatory overview is at the highest level, and gives a basic explanation of some of the key steps in the data lifecycle. +2. Below that, the tutorials section documents some of the standard processes we follow for each of these steps, and also explains some different scenarios that can occur when [adding](Adding-Data) and [maintaining](Maintaining-Data) data to make it clear how processes can vary slightly in different situations. 3. The procedure steps in the scenarios link to the most detailed level of documentation - the [how-to guides](How-to-guides) - which give step by step instructions for how to complete particular tasks. ## Validating data When receiving data from the LPA, we need to first validate the data to check that it conforms to our data requirements. -Depending on the dataset, the LPAs usually use the [planning form](https://submit.planning.data.gov.uk/check/) to check if the data is good to go. They don't do that all the time though, so we still need to manually validate the data. However, the check tool does not yet work for Brownfield-land/site datasets so we always need to validate the data on our end. +Depending on the dataset, the LPAs usually use the [check service](https://submit.planning.data.gov.uk/check/) to check whether their data meets the specifications. But in most cases we still carry out validation checks ourselves before adding data. Read the [how to validate an endpoint guide](Validate-an-endpoint) to see the steps we follow. @@ -22,18 +22,13 @@ Read the [how to validate an endpoint guide](Validate-an-endpoint) to see the st There are two main scenarios for adding data: -- Adding an endpoint for a new dataset and/or collection (e.g. we don't have a the dataset on file at all) -- Adding a new endpoint to an existing dataset +- Adding a new endpoint to an existing dataset. This will usually be for a compiled, ODP dataset (e.g. adding a new endpoint from a Local Planning Authority to the `article-4-direction-area` dataset). -Based on this, the process is slightly different. +- Adding an endpoint for a new dataset and/or collection. This is usually for a national-scale dataset which is being added to the platform (e.g. adding `flood-storage-area` data from the Environment Agency to the platform for the first time). -A how-to on adding a new dataset and collection can be found [here](Add-a-new-dataset-and-collection). +The [adding data](../Tutorials/Adding-Data.md) page in the tutorials section explains the process we follow for each of these scenarios. -A how-to on adding a new endpoint to an existing dataset can be found [here](Add-an-endpoint). Endpoints can come in a variety of types. The format can differ from endpoint to endpoint as can the required plugins needed to process the endpoint correctly. - -More information on types can be found [here](Endpoint-URL-Types-And-Plugins#data-formats-of-resources-that-can-be-processed) - -More information on plugins can be found [here](Endpoint-URL-Types-And-Plugins#adding-query-parameters-to-arcgis-server-urls) +You may find it useful to read some of the Key Concepts documentation, in particular on [pipeline processes and the data model](../Explanation/Key-Concepts/pipeline-processes.md) and [endpoint types](../Explanation/Key-Concepts/Endpoint-types.md). ## Maintaining data @@ -43,13 +38,14 @@ Maintaining data means making sure that the changes a data provider makes to the All entries on the platform must be assigned an entity number in the `lookup.csv` for the collection. This usually happens automatically when adding a new endpoint through the `add-endpoints-and-lookups` script. However, when an endpoint is already on the platform but the LPA has indicated that the endpoint has been updated with a new resource and new entries, we can’t just re-add the endpoint. Instead, we assign the new entries their entity numbers differently. -A how-to on assigning entities can be found [here](Assign-entities) +The [maintaining data](../Tutorials/Maintaining-Data.md) page in Tutorials covers some of these different scenarios and the steps that should be followed for each. + ### Merging entities -There can be duplicates present in a dataset. This primarily takes place where multiple organisations are providing data against the same object (or entity). We do not automatically detect and remove these, the old-entity table is used to highlight these duplications and provide them under a single entity number. +There can be duplicates present in a dataset. This primarily takes place where multiple organisations are providing data about the same entity. We do not automatically detect and remove these. Instead, the `lookup.csv` for a dataset can be used to map data from different organisations to the same entity, or the `old-entity.csv` can be used to redirect information from one entity to another. -A how-to on merging entities can be found [here](Merge-entities) +Read the duplicate scenario in the [adding data](../Tutorials/Adding-Data.md) tutorial page, and the [how-to merge entities](../How-To-Guides/Maintaining/Merge-entities.md) page to learn more. ## Retiring data @@ -59,23 +55,27 @@ When an endpoint consistently fails, or LPAs give us a different endpoint (as op When we retire an endpoint, we also need to retire the source(s) associated with it as sources are dependent on endpoints. -Read [how-to retire an endpoint](Retire-endpoints) to learn more. +Read [how-to retire an endpoint](../How-To-Guides/Retiring/Retire-endpoints.md) to learn more. ### Retiring resources It won’t be necessary to do this step often, however, sometimes a resource should not continue to be processed and included in the platform. This can be for multiple reasons, and in most cases will occur when it has been found that the resource contains significant errors. -A how-to on retiring resources can be found [here](Retire-resources) +Read [how-to retire a resource](../How-To-Guides/Retiring/Retire-resources.md) to learn more. ### Retiring entities -**Note:** We may want to keep old entities on our platform as historical data. There are two reasons an entity might be removed: +**Note:** We usually want to keep old entities on our platform as historical data. + +> **For example** a World Heritage Site was added as an entity to our platform. Although it is no longer a World Heritage Site, we want to retain the entity to indicate that it held this status during a specific period. + +However, there are two situations when an entity might be removed: 1. It was added in error. In this case, we should remove it from our system. 2. It has been stopped for some reason. In this scenario, we should retain the entity. - Ideally, we would mark such entities with end-dates to indicate they have been stopped, but implementing this requires additional work. -For example, a World Heritage Site was added as an entity to our platform. Although it is no longer a World Heritage Site, we want to retain the entity to indicate that it held this status during a specific period. + +Ideally, we would mark such entities with end-dates to indicate they have been stopped, but implementing this requires additional work. In a given scenario, determine the reason why the entities are no longer present. Check with Swati before deleting entities. diff --git a/docs/data-operations-manual/How-To-Guides/Adding/Add-an-endpoint.md b/docs/data-operations-manual/How-To-Guides/Adding/Add-an-endpoint.md index b597357..eca5672 100644 --- a/docs/data-operations-manual/How-To-Guides/Adding/Add-an-endpoint.md +++ b/docs/data-operations-manual/How-To-Guides/Adding/Add-an-endpoint.md @@ -78,7 +78,7 @@ make add-data COLLECTION=conservation-area INPUT_CSV=import.csv ``` -1. **(Optiona) Update entity-organisation.csv** +1. **(Optional) Update entity-organisation.csv** If the data that has been added is part of the `conservation-area` collection e.g `conservation-area` and `conservation-area-document`, the entity range must be added as a new row. This is done using the entities generated in `lookup`. Use the first and the last of the entity numbers of the newly generated lookups e.g if `44012346` is the first and `44012370` the last, use these as `entity-minimum` and `entity-maximum`. @@ -105,11 +105,15 @@ 1. **Test locally** Once the changes have been made and pushed, the next step is to test locally if the changes have worked. Follow the steps in [building a collection locally](..\Testing\Building-a-collection-locally.md) -1. **Push changes** - Use git to push changes up to the repository, each night when the collection runs the files are downloaded from here. It is a good idea to name the commit after the organisation you are importing. +1. **Push changes** + Commit your changes to a new branch that is named after the organisation whose endpoints are being added (use the 3 letter code for succinct names, e.g. `add-LBH-data`). + + Push the changes on your branch to remote and create a new PR. This should be reviewed and approved by a colleague in the Data Management team before being merged into `main`. + + Once the chages are merged they will be picked up by the nightly Airflow jobs which will build an updated dataset. 1. **Run action workflow (optional)** - Optionally, you can manually execute the workflow that usually runs overnight yourself - if you don’t want to wait until the next day - to check if the data is actually on the platform. Simply follow the instructions in the [guide for triggering a collection manually](/data-operations-manual/How-To-Guides/Maintaining/Trigger-collection-manually). + Optionally, if you don’t want to wait until the next day, you can manually execute the workflow that usually runs overnight yourself in order to be able to check if the data is actually on the platform. Simply follow the instructions in the [guide for triggering a collection manually](/data-operations-manual/How-To-Guides/Maintaining/Trigger-collection-manually). ## Endpoint edge-cases diff --git a/docs/data-operations-manual/How-To-Guides/Adding/Endpoint-URL-Types-And-Plugins.md b/docs/data-operations-manual/How-To-Guides/Adding/Adjust-arcgis-query-parameters.md similarity index 100% rename from docs/data-operations-manual/How-To-Guides/Adding/Endpoint-URL-Types-And-Plugins.md rename to docs/data-operations-manual/How-To-Guides/Adding/Adjust-arcgis-query-parameters.md diff --git a/docs/data-operations-manual/How-To-Guides/Validating/Validate-an-endpoint.md b/docs/data-operations-manual/How-To-Guides/Validating/Validate-an-endpoint.md index 67d38ad..6ab1482 100644 --- a/docs/data-operations-manual/How-To-Guides/Validating/Validate-an-endpoint.md +++ b/docs/data-operations-manual/How-To-Guides/Validating/Validate-an-endpoint.md @@ -1,74 +1,104 @@ # Validating an endpoint 1. **Check required information is available** - Before we can continue with anything, we need to make sure that we have the following information: _the organisation, the endpoint_url, the documentation_url,_ and _the licence_. If even one of them is missing, we can’t continue with the process and require the LPA to give us the correct information. - - There are certain fields that we require to be in the data (while some fields are required in all datasets, additional fields might be required depending on the dataset). In the first instance, this is what we check. _If any other fields are present, the data should still be added. Any fields that are missing should be reported in the provided [feedback form](https://drive.google.com/drive/folders/1YEKupdAVcdCt6YVG6NC3iUftAsQhk9Zq)._ - - We don’t need to do this manually, instead, we can use [_CHECK_](https://check.staging.digital-land.info/) or the [_endpoint_checker_](https://github.com/digital-land/jupyter-analysis/blob/main/endpoint_checker/endpoint_checker.ipynb). The endpoint checker is the most up to date and trustworthy check we have at the moment, so use it in most cases. CHECK is just a quick and easier way to find out if there is something wrong with the data but just because it does not flag anything up does not mean that the data is perfect. It also does not work for brownfield data. Hence we use the endpoint checker. - -2. **Update endpoint checker variables** - The endpoint checker works as follows: - In most cases, we just need to add the correct _collection_name_, _dataset_, _organisation, endpoint_url, documentation_url, and licence_. The script will perform a variety of checks. - _The endpoint checker works by running the pipeline on the given endpoint to check if it will be successful while also highlighting useful information as a summary as to whether the endpoint would be successful on our platform. It essentially downloads the resource through the endpoint and attempts to run the pipeline locally. It checks:_ - -- _Can the resource be downloaded successfully?_ -- _Are there any unassigned entities? (If so, these are what will later be added in the `lookup.csv`)_ -- _Checks if the column mapping was done correctly (If, in this stage, a particular column is empty or has missing values, the column mapping likely needs investigating)_ -- _Checks if there are any issues ( internal issues with an ‘error’ severity need to be fixed before moving on)_ -- _Checks for duplicate entities between current database and new endpoint entities_ - - _Any entities that are duplicates, we need to merge them (check merge entities section for this)_ - -The correct variables need to be provided to the endpoint checker in order for it to work. The following is an example of updated variables: - -``` -collection_name = 'article-4-direction-collection' -dataset = 'article-4-direction' -organisation = 'local-authority-eng:SAW' -endpoint_url = 'https://www.sandwell.gov.uk/downloads/file/2894/article-4-direction-dataset' -documentation_url = "https://www.sandwell.gov.uk/downloads/download/868/article-4-directions-planning-data" -start_date="" -plugin = '' -licence = "ogl3" -reference_column = "" -additional_column_mappings=None -additional_concats=None -data_dir = '../data/endpoint_checker' -``` - -In most cases, the only variables that need to be changed are _collection-name, dataset, organisation, enpoint_url, documentation_url,_ and _licence_. - -**Cases when to use other variables:** - Certain endpoint_urls types might require a plugin \[I don’t know which ones, ask later\]. The required licence will be indicated in the documentation_url. At times, additional column mappings need to be done. This is the case when the headers in the data we have been provided do not match the headers we need. Here we are mapping the provided `fid` to our `reference` field. - -``` -additional_column_mappings=[{'dataset':'conservation-area','endpoint':'6dad8060cbdf75cf7c28c511b107a367e7c0435e04d130793b8464454bfd384c','column' : 'fid', 'field':'reference'}] -``` - -If additional_concats needs to be used, it can be used like so \[again, unsure of when this is actually used\]: - -``` -additional_concats = [{ - 'dataset':'tree-preservation-zone', - 'endpoint':'de1eb90a8b037292ef8ae14bfabd1184847ef99b7c6296bb6e75379e6c1f9572', - 'resource':'e6b0ccaf9b50a7f57a543484fd291dbe43f52a6231b681c3a7cc5e35a6aba254', - 'field':'reference', - 'fields':'REFVAL;LABEL', - 'separator':'/' -}] -``` - -3. **Check for issues at ‘Issues Logs’** - Make sure there are no internal errors showing up. If they do, these need to be fixed before continuing. The fix depends on the error warning. It could be a simple one that can be fixed with some mapping. - -4. **Check for duplicates at ‘Duplicates with different entity numbers’** - If there are duplicates printed out, and the duplicates have different entity numbers, then you will need to merge these duplicates after adding the endpoint. Follow the instructions in [Merging Entities](../../Maintaining/Merge-entities) for this. - -5. **Check for unique counts in ‘Duplicate Values in Reference Columns’** - In the printed out table, the amount of _nunique_ must be the same as _count_ and _size_. This tells us that all the data is unique. If they are not unique, we need to get back to the LPA and require them to update their endpoint to only have unique references. - In an ideal world, no issues will be flagged up. This means that: - -- Unassigned entities will be listed out -- The final dataset will be shown -- The amount of unique rows will be equal to the number of rows in the dataset - If everything worked correctly, we will have successfully validated the data and can now move on to adding the data. This is done by adding to the endpoint, source, and lookup files. More on that in the [Adding an Endpoint](../../Adding/Add-an-endpoint) step. + In order to add an endpoint we require the following information: + + * the organisation + * the endpoint_url + * the documentation_url + * the licence + + If any are missing you will need to ask the LPA to give more information. + +1. **Use check service to test whether the data matches the specifications** + + For ODP datasets the [check service](https://check.planning.data.gov.uk/) will test most of the things we need. Select the required dataset and submit the endpoint URL to see the results. + + If there are serious issues such as missing `reference` or `geometry` fields, or the `reference` field isn't unique, these should be raised with the LPA before adding the data. + + NOTE the check service: + + * does not work with WFS data + * does not check for geometry duplicates in conservation-area data + + In these situations you should also use the endpoint checker (see steps below). + +1. **Use the endpoint checker to check for any issues** + + The endpoint checker works by downloading a resource from the endpoint and running the pipeline locally to check if it can be processed while also highlighting any other quality issues. It checks: + + - Can the resource be downloaded successfully? + - Are there any unassigned entities? + - Checks if column mapping works correctly + - Checks if there are any issues + - Checks for duplicate entities between current database and new endpoint entities + + To run the endpoint checker populate the key variables in the notebook, like in this example: + + ``` + # required + collection_name = 'article-4-direction-collection' + dataset = 'article-4-direction' + organisation = 'local-authority-eng:SAW' + endpoint_url = 'https://www.sandwell.gov.uk/downloads/file/2894/article-4-direction-dataset' + documentation_url = "https://www.sandwell.gov.uk/downloads/download/868/article-4-directions-planning-data" + licence = "ogl3" + data_dir = '../data/endpoint_checker' + + # optional + start_date= "" + plugin = "" # "wfs" or "arcgis" + reference_column = "" + additional_column_mappings = None + additional_concats = None + ``` + + **Cases when to use other variables:** + When the columns supplied don't match the fields in the data specifications they can be remapped using the `additional_column_mappings` variable, like so: + + ``` + additional_column_mappings=[{'dataset':'conservation-area','endpoint':'6dad8060cbdf75cf7c28c511b107a367e7c0435e04d130793b8464454bfd384c','column' : 'fid', 'field':'reference'}] + ``` + + Or concat config can be supplied with `additional_concats`, like below. (see [configure an endpoint](../Adding/Configure-an-endpoint.md) for an explanation.) + + ``` + additional_concats = [{ + 'dataset':'tree-preservation-zone', + 'endpoint':'de1eb90a8b037292ef8ae14bfabd1184847ef99b7c6296bb6e75379e6c1f9572', + 'resource':'e6b0ccaf9b50a7f57a543484fd291dbe43f52a6231b681c3a7cc5e35a6aba254', + 'field':'reference', + 'fields':'REFVAL;LABEL', + 'separator':'/' + }] + ``` + + Once the endpoint checker has run you can carry out the following checks. + + i. **Check for unassigned entities** + There should be the same number as the number of new records in the endpoint. + + ii. **Check the expected columns have been mapped** + After the "Check logs collated from the pipeline process" cell you'll see a printout of the columns that have been mapped. If there are any missing you should use the `additional_column_mappings` to test making additional mappings. You should map as many other fields as possible to the specifications. + + iii. **Check for issues at ‘Issues Logs’** + Make sure there are no internal errors showing up. If they do, these need to be fixed before continuing. The fix depends on the error warning. It could be a simple one that can be fixed with some mapping. + + iv. **Check for duplicates at ‘Duplicates with different entity numbers’** + If there are duplicates printed out, and the duplicates have different entity numbers, then you will need to merge these duplicates after running the `add-endpoint-and-lookups` command. This should be done by changing entity number of newly generated entity to the older entity it is a duplicate of. + + v. **Check for unique counts in ‘Duplicate Values in Reference Columns’** + In the printed out table, the amount of _nunique_ must be the same as _count_ and _size_. This tells us that all the data is unique. If they are not unique, we need to get back to the LPA and require them to update their endpoint to only have unique references. + + +Once the steps above have been followed you should be certain of the following: + +* The endpoint contains the expected data +* The dataset has a unique `reference` field (or an equivalent which can be mapped to `reference`) +* Any other fields supplied which can be mapped to the specification +* Any error-level issues have been fixed +* *If it's a spatial dataset*, that it has a `geometry` field, or a `point` field in the case of `tree` data +* *If it's a conservation-area dataset*, whether it contains any spatial duplicates with existing datasets which will need to be merged + + +Now use the [adding an Endpoint](../../Adding/Add-an-endpoint) guide to add the data. diff --git a/docs/data-operations-manual/Tutorials/Adding-Data.md b/docs/data-operations-manual/Tutorials/Adding-Data.md index 77c2b4c..a5fe396 100644 --- a/docs/data-operations-manual/Tutorials/Adding-Data.md +++ b/docs/data-operations-manual/Tutorials/Adding-Data.md @@ -1,6 +1,60 @@ -# Adding Data +# Standard process -This section covers situations when we add new data onto the platform. + +The trigger for this process should be a new Jira ticket on the [data managment board](https://mhclgdigital.atlassian.net/jira/software/projects/DATA/boards/229) for adding data. This ticket should link to a [Digital Land Service Desk](https://mhclgdigital.atlassian.net/jira/servicedesk/projects/DLSD) ticket which contains the correspondence with the customer. + +When you pick up the ticket, follow the steps below. + +### 1. Validate endpoint + +Follow the [validate an endpoint](../How-To-Guides/Validating/Validate-an-endpoint.md) process to check whether the data meets the specifications. If you find any issues you should respond the the data provider and ask if they can fix them. + +Before adding new data you should also **check whether there is already data for this provision on the platform**. You can do this using the [LPA dashboard in the publish service](https://submit.planning.data.gov.uk/organisations), our [config manager reports](https://config-manager-prototype.herokuapp.com/reporting/odp-summary/status), or by using the [search page on planning.data](https://www.planning.data.gov.uk/entity/). + +If there is existing data you may need to retire an old endpoint alongside adding the new one. The scenarios in the [maintaining data](Maintaining-Data.md) tutorials will help you work out the right process to follow. + +> **NOTE** - If adding a **national dataset**: +> The validation process will not be so standard. We should receive a description of the dataset from data design in the ticket. Check that the data on the endpoint matches the description. + +### 2. Add endpoint + +Follow the [add an endpoint](../How-To-Guides/Adding/Add-an-endpoint.md) process to set up the configuration for the new endpoint in the config repo. + +Push your changes but **do not merge them before moving on to the next step**. + +### 3. QA endpoint configuration + +In order to make sure the configuration for the new endpoint is checked properly you should raise a PR for your changes and fill out the template that is automatically generated. This will make it clear what sort of change is being made, and also give you a QA checklist to fill out. + +You should share your PR with a colleague in data management team to review, and they should follow the same checks in the checklist. + + +### 4. Merge changes (and run workflow) + +Once your PR is approved the changes can be merged. At this point you could also run the action workflow to build the updated dataset (see the last step of the [add an endpoint](../How-To-Guides/Adding/Add-an-endpoint.md) process). + +> **NOTE** - If adding a **national dataset**: +> You should first run the action workflow in the development Airflow environment. This will publish the new data on the development planning data site so that the data design team can review it before going live. +> +> Currently, the live Airflow workflow runs each night and will pick up your changes, so you should run the dev workflow as early as possible in the day and let design know they have the rest of the day to review them. If you need to make significant changes, revert your commit from config `main` so they don't go live. + +### 5. Review new data on platform +Once the workflow has been run (either manually, or in the automated overnight process) you should carry out some last QA checks: + +- Check that there are the expected number of new entities on the platform (you can use the search page to do this, either with the [location parameter](https://www.planning.data.gov.uk/entity/?dataset=conservation-area&geometry_curie=statistical-geography%3AE09000022) or the [`organisation_entity` parameter](https://www.planning.data.gov.uk/entity/?dataset=conservation-area&organisation_entity=192) if the new data doesn't have a location). +- Check if the new data is on the [LPA dashboard in the publish service](https://submit.planning.data.gov.uk/organisations). + +### 6. Close ticket + +Once the final checks are complete you can close the tickets: + +- Reply to the customer in [Digital Land Service Desk](https://mhclgdigital.atlassian.net/jira/servicedesk/projects/DLSD) using the canned responses to let them know data is live. +- Move Jira tickets to done + +# New endpoint scenarios +When adding data from a new endpoint you may need to follow slightly different steps based on the context, like exactly what data is being provided, how it's being provided, and whether data for the provision already exists. + +The sections below aim to explain some common scenarios to make it clear what steps should be followed, and what the expected outcome should be. ## New endpoint for a _single_, new ODP provision diff --git a/docs/data-operations-manual/Tutorials/Maintaining-Data.md b/docs/data-operations-manual/Tutorials/Maintaining-Data.md index ad6e811..aca4812 100644 --- a/docs/data-operations-manual/Tutorials/Maintaining-Data.md +++ b/docs/data-operations-manual/Tutorials/Maintaining-Data.md @@ -1,6 +1,6 @@ # Maintaining Data -The scenarios are split into data being added and data being removed. In reality, endpoint updates could be a combination of both, in which case you will need to follow each process for entities which have been added or removed. +These scenarios are split into data being added and data being removed. In reality, endpoint updates could be a combination of both, in which case you will need to follow each process for entities which have been added or removed. ## New endpoint for existing provision