Initial data guide 36 #141

fyliu · 2023-03-23T21:35:47Z

Add guide to create initial data scripts.

A team member should be able to create data insertion scripts using this guide.

closes Document how to create initial data for tables #36

fyliu · 2023-03-23T21:37:02Z

I'm leaving this as draft until the sphinx PR is merged. This documentation is using the sphinx structure with an added entry in the index page.

From the blog https://thenewstack.io/how-to-convert-google-spreadsheet-to-json-formatted-text/

fyliu · 2023-04-06T22:31:09Z

I removed everything sphinx-specific from the commits in this merge. This is ready for review.

joey-ma · 2023-04-07T23:13:58Z

This is cool! I can try to create the SOCMajor model next, and we can resume reviewing this PR later.

fyliu · 2023-04-10T19:07:47Z

@yoyoyojoe no need to implement the table from the guide although you can. Issue #24 is also good and is in the prioritized backlog. It should work for all tables although I only tried SOCMajor and UserStatusType. I don't think guides need to be perfect. Their purpose is to get developers most of the way there.

Create Table: permission_type #24

fyliu · 2023-04-12T17:52:17Z

@yoyoyojoe It turns out there's a blocker for #24. I ran into more problems trying to work on that. I think it'll be a while before this PR is review-able.

joey-ma

As mentioned by Fang, we're converting this to Draft and will put this in the Ice Box for the time being.

joey-ma · 2023-04-07T21:57:46Z

docs/how-to/create-initial-data-migrations.md

+   1. Export the data from the Google [spreadsheet][pd-data-spreadsheet]
+      1. Find the sheet in the document containing the data to export. Let's use the `SOC Major - Data` data as our example. It's
+      1. Make sure that the first row (column names) is frozen. Otherwise, freeze it by selecting the first row in the sheet, then Menu > View > Freeze > Up to row 1
+      1. Export to JSON. Menu > Export JSON > Export JSON for this sheet


As part of note-taking:

at this step, I encountered Authorization Required:

which then brings me to Choose an Account:

which then, upon selecting my account, brings me to Google hasn't verified this app:

by clicking on Advanced, then going to Go to Untitled project (unsafe):

I can then give authorization when prompted "Untitled project wants to access your Google Account":

after authorization, I perform the same actions again, and now I get the Exported JSON:

This is cool, but a few comments / observations:

How much of the above process should be documented?

What would be a better way to handle this Google hasn't verified this app pop-up?

What is the benefit of this custom script?

For example, I'm not sure how Export JSON is a Menu item for Google Sheet PD: Table and field explanations, but I'm aware of the Extensions, and I found this Export Sheet Data Add-on after a brief search on the Google Workspace Marketplace (Extensions > Add-ons > Get Add-ons), and it seems we can customize it to do the same thing.

Being able to export to file is neat as well, and if we name the Sheets in PascalCase to start, and we customize further, we might be able to customize it to do more and export all Sheets (tables) at once.

Thanks for the documentation and great comments/questions. But the answers are not going to be great.

How much of the above process should be documented?

Documentation is great, but this is normal general google apps scripts behavior so it's probably not up to us (as a project) to host this documentation. Maybe someone has the documentation that we can link to or we can put it in a hfla general developer docs place.

The other thing is that this is a temporary solution that we'll get rid of soon after this is merged. We won't need/want this script at all. But this is a working solution and I want to get something working before doing the optimizing step.

What would be a better way to handle this Google hasn't verified this app pop-up?

There is no better way to do this. This is normal and the warning serves to tell users to examine the script closely before running it.

What is the benefit of this custom script?

Good question. The benefit it that it works to get us JSON from the spreadsheet. Actually, Bonnie found it so it's "free" for me. The drawback is that it's a manual step and copying the JSON into a file is also manual. That's why it only serves as a stop gap solution, and because there's a much better method, I didn't spend time to look for other similar things. The one you found does go one step further than the current script, so I would have gone with that if I had known.

docs/how-to/create-initial-data-migrations.md

joey-ma · 2023-04-07T22:18:46Z

docs/how-to/create-initial-data-migrations.md

+   1. Go to the project root and run this command
+
+      ```bash
+      docker-compose exec web python scripts/convert.py core/initial_data/SOCMajor_export.json


I'm not seeing the convert.py.

I found that we can update this part: docker-compose exec web python scripts/json_data_to_python.py core/initial_data/SOCMajor_export.json

Yes, that's what the convert script became. The docs need updating.

Has the document been updated?

joey-ma · 2023-04-07T23:13:12Z

docs/how-to/create-initial-data-migrations.md

+         1. Run this command to run the script
+
+            ```bash
+            docker-compose exec web python manage.py runscript socmajor_seed


In order to test this further, we'll need to have the SOCMajor model created.

Yes, or some other model that has initial data. Ones like SOCMajor have lots of initial data which demonstrates why certain choices were made on the specific methods we chose here to generate the initial data.

I did create the model locally to test that this works. I also did the user_status model and it was okay too.

Can you change to use ProgramArea (or other table which exists)?

ethanstrominger · 2023-08-18T15:32:37Z

docs/how-to/create-initial-data-migrations.md

+   1. Go to the project root and run this command
+
+      ```bash
+      docker-compose exec web python scripts/convert.py core/initial_data/SOCMajor_export.json


Has the document been updated?

ethanstrominger · 2023-08-18T15:33:13Z

docs/how-to/create-initial-data-migrations.md

+         1. Run this command to run the script
+
+            ```bash
+            docker-compose exec web python manage.py runscript socmajor_seed


Can you change to use ProgramArea (or other table which exists)?

ethanstrominger · 2023-08-18T15:36:00Z

docs/how-to/create-initial-data-migrations.md

+
+            ```bash
+            docker-compose exec web python manage.py runscript socmajor_seed
+            ```


The spreadsheet uses "id" while the base abstract model has uuid. IMO simplest is to change uuid to id. Anyone who had created a db would need to recreate - I can provide steps on how to do that. Alternatively, instructions could be added to manually change this.

ethanstrominger · 2023-08-18T15:38:57Z

docs/how-to/create-initial-data-migrations.md

+            ```bash
+            docker-compose exec web python manage.py runscript socmajor_seed
+            ```
+


Not related to doc: IMO it would be better for id to be populated in the spreadsheet for all tables in case we add joins to that data in the seed data. If you agree, I can make that change.

ethanstrominger · 2023-08-18T15:58:29Z

docs/how-to/create-initial-data-migrations.md

+      1. Find the sheet in the document containing the data to export. Let's use the `SOC Major - Data` data as our example. It's
+      1. Make sure that the first row (column names) is frozen. Otherwise, freeze it by selecting the first row in the sheet, then Menu > View > Freeze > Up to row 1
+      1. Export to JSON. Menu > Export JSON > Export JSON for this sheet
+   1. Save the JSON into a file


The current steps do not indicate there is a dialog box, it reads better with sub steps, the last step is a note. If these changes are made for lines 27-30 it would look like this:

Export to JSON. Menu > Export JSON > Export JSON for this sheet. A dialog box with JSON will appear.

Select all the JSON and copy into the clipboard.
==> Add this line,

Save the JSON as [ModelNameInPascalCase]_export.json under app/core/initial_data/

==> Change to add NOTE:. It is not a step.
NOTE: The Pascal case is important in the next step to generate a python script to insert the data. It must match the model's class name for this to work.

ethanstrominger · 2023-08-18T16:02:25Z

docs/how-to/create-initial-data-migrations.md

+            ```
+
+[pd-data-spreadsheet]: https://docs.google.com/spreadsheets/d/1x_zZ8JLS2hO-zG0jUocOJmX16jh-DF5dccrd_OEGNZ0/
+[apps-script]: https://thenewstack.io/how-to-convert-google-spreadsheet-to-json-formatted-text/#:~:text=To%20do%20this,%20click%20Extensions,save%20your%20work%20so%20far.


Instructions are needed on how to include the generated script either as part of a seed data script or include in migration. I like the option of including in migration scripts and I figured out how to do this.

ethanstrominger · 2023-11-06T19:59:44Z

Created a new pull request with additional changes from ethanstrominger initial-data-migration-36 to peopledepot main

fyliu · 2023-11-27T19:07:53Z

Replaced by PR Obsolete: Initial data migration 36 #223

fyliu force-pushed the initial-data-guide-36 branch from 93b50d7 to 819ca2c Compare March 24, 2023 04:08

fyliu mentioned this pull request Mar 30, 2023

Document how to create initial data for tables #36

Closed

4 tasks

fyliu and others added 3 commits April 6, 2023 15:19

Add GA script for exporting spreadsheet to JSON

8dda7c4

From the blog https://thenewstack.io/how-to-convert-google-spreadsheet-to-json-formatted-text/

fix: handle underscore in column names in exportjson script

5f569ec

script: convert json to python for loading initial data

c609754

fyliu force-pushed the initial-data-guide-36 branch from 819ca2c to ddfef5f Compare April 6, 2023 22:23

fyliu marked this pull request as ready for review April 6, 2023 22:25

docs: how to create initial data scripts

bb4ae18

fyliu force-pushed the initial-data-guide-36 branch from ddfef5f to bb4ae18 Compare April 6, 2023 22:29

fyliu mentioned this pull request Apr 6, 2023

People Depot Team Meeting Agenda 2024 #103

Open

ExperimentsInHonesty requested a review from joey-ma April 6, 2023 23:17

fyliu marked this pull request as draft April 7, 2023 23:14

fyliu mentioned this pull request Apr 10, 2023

Eliminate external JSON export script #144

Closed

6 tasks

fyliu mentioned this pull request Apr 14, 2023

Create Table: user_status_type #35

Closed

19 tasks

joey-ma reviewed Apr 27, 2023

View reviewed changes

ethanstrominger requested changes Aug 18, 2023

View reviewed changes

ethanstrominger closed this Nov 6, 2023

This was referenced May 30, 2024

Create Table: user_status_type fyliu/peopledepot#93

Closed

Create Table: user_status_type fyliu/peopledepot#135

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial data guide 36 #141

Initial data guide 36 #141

fyliu commented Mar 23, 2023 •

edited

Loading

fyliu commented Mar 23, 2023

fyliu commented Apr 6, 2023

joey-ma commented Apr 7, 2023

fyliu commented Apr 10, 2023

fyliu commented Apr 12, 2023

joey-ma left a comment

joey-ma Apr 7, 2023

fyliu Jul 24, 2023

joey-ma Apr 7, 2023

joey-ma Apr 7, 2023

fyliu Jul 24, 2023

ethanstrominger Aug 18, 2023

joey-ma Apr 7, 2023

fyliu Jul 24, 2023 •

edited

Loading

ethanstrominger Aug 18, 2023

ethanstrominger Aug 18, 2023

ethanstrominger Aug 18, 2023

ethanstrominger Aug 18, 2023

ethanstrominger Aug 18, 2023

ethanstrominger Aug 18, 2023

ethanstrominger Aug 18, 2023

ethanstrominger commented Nov 6, 2023

fyliu commented Nov 27, 2023

Initial data guide 36 #141

Initial data guide 36 #141

Conversation

fyliu commented Mar 23, 2023 • edited Loading

fyliu commented Mar 23, 2023

fyliu commented Apr 6, 2023

joey-ma commented Apr 7, 2023

fyliu commented Apr 10, 2023

fyliu commented Apr 12, 2023

joey-ma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fyliu Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ethanstrominger commented Nov 6, 2023

fyliu commented Nov 27, 2023

fyliu commented Mar 23, 2023 •

edited

Loading

fyliu Jul 24, 2023 •

edited

Loading