Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Xparse fix #89

Merged
merged 4 commits into from
Nov 20, 2023
Merged

Xparse fix #89

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7ff0e96
fix vproc policies
alec-bertoy Nov 16, 2023
70c2483
fix ingest x file parsing and work type identification
alec-bertoy Nov 17, 2023
4d357ba
update ingest readme
alec-bertoy Nov 17, 2023
ac7e1b1
readme fix
alec-bertoy Nov 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions nulrdcscripts/ingest/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,19 @@ poetry run ingest -i INPUT_PATH -o OUTPUT_FILEPATH
poetry run ingest -i INPUT_PATH -l INVENTORY_PATH
```

### Skip ".md5" files
### Parse x files by filename
```
poetry run ingest -i INPUT_PATH -s .md5
poetry run ingest -i INPUT_PATH -x parse
```

### Custom description
### Skip ".md5" files
```
poetry run ingest -i INPUT_PATH -d "Date/Time" "Barcode"
poetry run ingest -i INPUT_PATH -s .md5
```

### Parse x files by filename
### Custom description
```
poetry run ingest -i INPUT_PATH -x parse
poetry run ingest -i INPUT_PATH -d "date/time" "barcode"
```

### Example File Structure
Expand Down Expand Up @@ -64,15 +64,15 @@ input_folder
|A|P|S|X|
|:-------:|:-------:|:-------:|:-------:|
|-a or _a|-p or _p|-s or _s|-x or _x|
|-am or _am|-pm or _pm|spectrogram|-Asset or _Asset|
|-am- or _am\_|-pm- or _pm\_|.json|Back.|
|||.log|-Can or _Can|
|||.pdf|-Ephemera or _Ephemera|
|||.xml|Front.|
|||.xml.gz|
|||.framemd5|
|||.qctool.mkv|
|||dpx.txt|
|-am or _am|-pm or _pm|spectrogram|-asset or _asset|
|-am- or _am\_|-pm- or _pm\_|.json|-Asset or _Asset|
|||.log|back or Back|
|||.pdf|front or Front|
|||.xml|-can or _can|
|||.xml.gz|-Can or _Can|
|||.framemd5|-ephemera or _ephemera|
|||.qctool.mkv|-Ephemera or _Ephemera|
|||dpx.txt|.jpg|

Anything that can't be identified will be set to 'S'

Expand Down
21 changes: 14 additions & 7 deletions nulrdcscripts/ingest/ingest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,40 +157,47 @@ def main():
elif "parse" in args.aux_parse:
aux_dict = {
"front": {
"identifiers": ["Front."],
"identifiers": ["front", "Front"],
"type": "pattern",
"role": "X",
"label": "asset front",
"file_builder": "_auxiliary_",
},
"back": {
"identifiers": ["Back."],
"identifiers": ["back", "Back"],
"type": "pattern",
"role": "X",
"label": "asset back",
"file_builder": "_auxiliary_",
},
"asset": {
"identifiers": ["_Asset", "-Asset"],
"identifiers": ["_asset", "-asset", "_Asset", "-Asset"],
"type": "pattern",
"role": "X",
"label": "asset",
"file_builder": "_auxiliary_",
},
"can": {
"identifiers": ["_Can", "-Can"],
"identifiers": ["_can", "-can", "_Can", "-Can"],
"type": "pattern",
"role": "X",
"label": "can",
"file_builder": "_auxiliary_",
},
"ephemera": {
"identifiers": ["_Ephemera", "-Ephemera"],
"identifiers": ["_ephemera", "-ephemera", "_Ephemera", "-Ephemera"],
"type": "pattern",
"role": "X",
"label": "ephemera",
"file_builder": "_auxiliary_",
},
"auxiliary": {
"identifiers": ["_x", "-x"],
"type": "pattern",
"role": "X",
"label": "image",
"file_builder": "_auxiliary_",
}
}
# add the aux_dict to the beginning of the role_dict
# this will catch X files that also have a/p identifiers in the filename
Expand Down Expand Up @@ -439,9 +446,9 @@ def import_inventories(source_inventories):
# work type is assumed by the presence of format-specific column headers
if "Width (cm.)" in reader.fieldnames:
work_type = "IMAGE"
elif "Speed IPS" in reader.fieldnames:
elif "speed IPS" or "Speed IPS" in reader.fieldnames:
work_type = "AUDIO"
elif "Region" or "Stock" in reader.fieldnames:
elif "region" or "Region" or "stock" or "Stock" in reader.fieldnames:
work_type = "VIDEO"
else:
print(
Expand Down
3 changes: 0 additions & 3 deletions nulrdcscripts/vproc/data/mediaconch_policies/AJA_NTSC_VHS-2SAS-MKV.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@
<rule name="Video/CodecID is V_MS/VFW/FOURCC / FFV1" value="CodecID" tracktype="Video" occurrence="*" operator="=">V_MS/VFW/FOURCC / FFV1</rule>
<rule name="Video/Width is 720" value="Width" tracktype="Video" occurrence="*" operator="=">720</rule>
<rule name="Video/Height is 486" value="Height" tracktype="Video" occurrence="*" operator="=">486</rule>
<rule name="Video/PixelAspectRatio is 0.909" value="PixelAspectRatio" tracktype="Video" occurrence="*" operator="=">0.909</rule>
<rule name="Video/DisplayAspectRatio is 1.347" value="DisplayAspectRatio" tracktype="Video" occurrence="*" operator="=">1.347</rule>
<rule name="Video/FrameRate_Mode is CFR" value="FrameRate_Mode" tracktype="Video" occurrence="*" operator="=">CFR</rule>
<rule name="Video/FrameRate is 29.970" value="FrameRate" tracktype="Video" occurrence="*" operator="=">29.970</rule>
<rule name="Video/Standard is NTSC" value="Standard" tracktype="Video" occurrence="*" operator="=">NTSC</rule>
Expand All @@ -28,7 +26,6 @@
<rule name="Video/ScanType is Interlaced" value="ScanType" tracktype="Video" occurrence="*" operator="=">Interlaced</rule>
<rule name="Video/ScanOrder is BFF" value="ScanOrder" tracktype="Video" occurrence="*" operator="=">BFF</rule>
<rule name="Video/Compression_Mode is Lossless" value="Compression_Mode" tracktype="Video" occurrence="*" operator="=">Lossless</rule>
<rule name="Video/TimeCode_Source is Matroska tags" value="TimeCode_Source" tracktype="Video" occurrence="*" operator="=">Matroska tags</rule>
<rule name="Video/Default is Yes" value="Default" tracktype="Video" occurrence="*" operator="=">Yes</rule>
<rule name="Video/Forced is No" value="Forced" tracktype="Video" occurrence="*" operator="=">No</rule>
<rule name="Video/colour_description_present is Yes" value="colour_description_present" tracktype="Video" occurrence="*" operator="=">Yes</rule>
Expand Down
27 changes: 4 additions & 23 deletions nulrdcscripts/vproc/data/mediaconch_policies/AJA_NTSC_VHS-2SAS-MOV.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,17 @@
<policy type="and" name="P0371-GRNG-F566_v01.mov">
<rule name="General/VideoCount is 1" value="VideoCount" tracktype="General" occurrence="*" operator="=">1</rule>
<rule name="General/AudioCount is 2" value="AudioCount" tracktype="General" occurrence="*" operator="=">2</rule>
<rule name="General/OtherCount is 1" value="OtherCount" tracktype="General" occurrence="*" operator="=">1</rule>
<rule name="General/FileExtension is mov" value="FileExtension" tracktype="General" occurrence="*" operator="=">mov</rule>
<rule name="General/Format is QuickTime" value="Format" tracktype="General" occurrence="*" operator="=">QuickTime</rule>
<rule name="General/Format_Profile is QuickTime" value="Format_Profile" tracktype="General" occurrence="*" operator="=">QuickTime</rule>
<rule name="General/FrameRate is 29.970" value="FrameRate" tracktype="General" occurrence="*" operator="=">29.970</rule>
<rule name="General/Encoded_Library is Apple QuickTime" value="Encoded_Library" tracktype="General" occurrence="*" operator="=">Apple QuickTime</rule>
<rule name="General/Encoded_Library_Name is Apple QuickTime" value="Encoded_Library_Name" tracktype="General" occurrence="*" operator="=">Apple QuickTime</rule>
<rule name="Video/StreamOrder is 0" value="StreamOrder" tracktype="Video" occurrence="*" operator="=">0</rule>
<rule name="Video/ID is 1" value="ID" tracktype="Video" occurrence="*" operator="=">1</rule>
<rule name="Video/Format is YUV" value="Format" tracktype="Video" occurrence="*" operator="=">YUV</rule>
<rule name="Video/CodecID is v210" value="CodecID" tracktype="Video" occurrence="*" operator="=">v210</rule>
<rule name="Video/BitRate_Mode is CBR" value="BitRate_Mode" tracktype="Video" occurrence="*" operator="=">CBR</rule>
<rule name="Video/Width is 720" value="Width" tracktype="Video" occurrence="*" operator="=">720</rule>
<rule name="Video/Width_CleanAperture is 704" value="Width_CleanAperture" tracktype="Video" occurrence="*" operator="=">704</rule>
<rule name="Video/Width_CleanAperture is 704" value="Width_CleanAperture" tracktype="Video" occurrence="*" operator="=">720</rule>
<rule name="Video/Height is 486" value="Height" tracktype="Video" occurrence="*" operator="=">486</rule>
<rule name="Video/PixelAspectRatio is 0.909" value="PixelAspectRatio" tracktype="Video" occurrence="*" operator="=">0.909</rule>
<rule name="Video/DisplayAspectRatio is 1.347" value="DisplayAspectRatio" tracktype="Video" occurrence="*" operator="=">1.347</rule>
<rule name="Video/Rotation is 0.000" value="Rotation" tracktype="Video" occurrence="*" operator="=">0.000</rule>
<rule name="Video/FrameRate_Mode is CFR" value="FrameRate_Mode" tracktype="Video" occurrence="*" operator="=">CFR</rule>
<rule name="Video/FrameRate is 29.970" value="FrameRate" tracktype="Video" occurrence="*" operator="=">29.970</rule>
Expand All @@ -28,13 +23,12 @@
<rule name="Video/ScanType is Interlaced" value="ScanType" tracktype="Video" occurrence="*" operator="=">Interlaced</rule>
<rule name="Video/ScanOrder is BFF" value="ScanOrder" tracktype="Video" occurrence="*" operator="=">BFF</rule>
<rule name="Video/Compression_Mode is Lossless" value="Compression_Mode" tracktype="Video" occurrence="*" operator="=">Lossless</rule>
<rule name="Video/Language is en" value="Language" tracktype="Video" occurrence="*" operator="=">en</rule>
<rule name="Video/colour_description_present is Yes" value="colour_description_present" tracktype="Video" occurrence="*" operator="=">Yes</rule>
<rule name="Video/colour_primaries is BT.601 NTSC" value="colour_primaries" tracktype="Video" occurrence="*" operator="=">BT.601 NTSC</rule>
<rule name="Video/transfer_characteristics is BT.709" value="transfer_characteristics" tracktype="Video" occurrence="*" operator="=">BT.709</rule>
<rule name="Video/matrix_coefficients is BT.601" value="matrix_coefficients" tracktype="Video" occurrence="*" operator="=">BT.601</rule>
<rule name="Audio/StreamOrder is 2" value="StreamOrder" tracktype="Audio" occurrence="*" operator="=">2</rule>
<rule name="Audio/ID is 3" value="ID" tracktype="Audio" occurrence="*" operator="=">3</rule>
<rule name="Audio/ID is 2" value="ID" tracktype="Audio" occurrence="*" operator="=">3</rule>
<rule name="Audio/Format is PCM" value="Format" tracktype="Audio" occurrence="*" operator="=">PCM</rule>
<rule name="Audio/Format_Settings_Endianness is Little" value="Format_Settings_Endianness" tracktype="Audio" occurrence="*" operator="=">Little</rule>
<rule name="Audio/Format_Settings_Sign is Signed" value="Format_Settings_Sign" tracktype="Audio" occurrence="*" operator="=">Signed</rule>
Expand All @@ -43,10 +37,7 @@
<rule name="Audio/Channels is 2" value="Channels" tracktype="Audio" occurrence="*" operator="=">2</rule>
<rule name="Audio/SamplingRate is 48000" value="SamplingRate" tracktype="Audio" occurrence="*" operator="=">48000</rule>
<rule name="Audio/BitDepth is 24" value="BitDepth" tracktype="Audio" occurrence="*" operator="=">24</rule>
<rule name="Audio/Delay_DropFrame is No" value="Delay_DropFrame" tracktype="Audio" occurrence="*" operator="=">No</rule>
<rule name="Audio/Delay_Source is Container" value="Delay_Source" tracktype="Audio" occurrence="*" operator="=">Container</rule>
<rule name="Audio/StreamOrder is 3" value="StreamOrder" tracktype="Audio" occurrence="*" operator="=">3</rule>
<rule name="Audio/ID is 4" value="ID" tracktype="Audio" occurrence="*" operator="=">4</rule>
<rule name="Audio/ID is 3" value="ID" tracktype="Audio" occurrence="*" operator="=">3</rule>
<rule name="Audio/Format is PCM" value="Format" tracktype="Audio" occurrence="*" operator="=">PCM</rule>
<rule name="Audio/Format_Settings_Endianness is Little" value="Format_Settings_Endianness" tracktype="Audio" occurrence="*" operator="=">Little</rule>
<rule name="Audio/Format_Settings_Sign is Signed" value="Format_Settings_Sign" tracktype="Audio" occurrence="*" operator="=">Signed</rule>
Expand All @@ -55,14 +46,4 @@
<rule name="Audio/Channels is 2" value="Channels" tracktype="Audio" occurrence="*" operator="=">2</rule>
<rule name="Audio/SamplingRate is 48000" value="SamplingRate" tracktype="Audio" occurrence="*" operator="=">48000</rule>
<rule name="Audio/BitDepth is 24" value="BitDepth" tracktype="Audio" occurrence="*" operator="=">24</rule>
<rule name="Audio/Delay_DropFrame is No" value="Delay_DropFrame" tracktype="Audio" occurrence="*" operator="=">No</rule>
<rule name="Audio/Delay_Source is Container" value="Delay_Source" tracktype="Audio" occurrence="*" operator="=">Container</rule>
<rule name="Other/StreamOrder is 1" value="StreamOrder" tracktype="Other" occurrence="*" operator="=">1</rule>
<rule name="Other/ID is 2" value="ID" tracktype="Other" occurrence="*" operator="=">2</rule>
<rule name="Other/Type is Time code" value="Type" tracktype="Other" occurrence="*" operator="=">Time code</rule>
<rule name="Other/Format is QuickTime TC" value="Format" tracktype="Other" occurrence="*" operator="=">QuickTime TC</rule>
<rule name="Other/TimeCode_FirstFrame is 00:00:00:00" value="TimeCode_FirstFrame" tracktype="Other" occurrence="*" operator="=">00:00:00:00</rule>
<rule name="Other/TimeCode_Striped is Yes" value="TimeCode_Striped" tracktype="Other" occurrence="*" operator="=">Yes</rule>
<rule name="Other/Title is Untitled" value="Title" tracktype="Other" occurrence="*" operator="=">Untitled</rule>
<rule name="Other/Language is en" value="Language" tracktype="Other" occurrence="*" operator="=">en</rule>
</policy>