Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New DublinCore-based metadata implementation #1266

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
997fbb7
Initial implementation (enough to have folks take a look at intentions).
gregchapman-dev Apr 6, 2022
5c63ce5
Add better comments about the new implementation.
gregchapman-dev Apr 7, 2022
ec303fc
Some tweaks.
gregchapman-dev Apr 8, 2022
6dc2c95
Modified xmlToM21.py and m21ToXml.py to use the new Metadata APIs. I…
gregchapman-dev Apr 9, 2022
36dd6c2
Save attempted backward compatibility work before restructuring class…
gregchapman-dev Apr 11, 2022
ab115e8
I think I have all the backward compatibility in place. __setattr__ j…
gregchapman-dev Apr 12, 2022
02038dc
Backward compatibility tested/fixed by importing using my converter21…
gregchapman-dev Apr 12, 2022
20eedb4
Support abbreviations and uniqueNames. Store contributor values as Co…
gregchapman-dev Apr 13, 2022
a6b1f4e
Add valueType member of PROPERTY to support type coercion during sett…
gregchapman-dev Apr 13, 2022
01200d4
Structural rework: MetadataBase and ExtendedMetadata are gone, only a…
gregchapman-dev Apr 16, 2022
79c9c35
Significant rework, moving things around to make it more readable, ma…
gregchapman-dev Apr 18, 2022
ba82b78
pylint, flake8, mypy.
gregchapman-dev Apr 18, 2022
8515731
Regenerate corpus metadata cache, since Metadata's internals are comp…
gregchapman-dev Apr 19, 2022
58f5391
mypy/lint.
gregchapman-dev Apr 19, 2022
edfd0e7
pylint/mypy/flake8
gregchapman-dev Apr 19, 2022
386709b
Get the existing tests passing.
gregchapman-dev Apr 19, 2022
f16ec0b
Some doctests and some fixes.
gregchapman-dev Apr 20, 2022
0ed8607
pylint
gregchapman-dev Apr 20, 2022
4dacd8a
Oops, pylinting broke the test results.
gregchapman-dev Apr 20, 2022
76cc455
More tests, more fixes.
gregchapman-dev Apr 20, 2022
2be9b46
More metadata testing.
gregchapman-dev Apr 21, 2022
3d4ff7b
More testing. A little fixing.
gregchapman-dev Apr 22, 2022
d31f66e
Comment out the unused (except during development testing) code in th…
gregchapman-dev Apr 22, 2022
538a51a
Catch up all the new code with the whole "import typing as t" thing.
gregchapman-dev May 14, 2022
c699732
Fix tests that return None. Fix "A guide to this new...".
gregchapman-dev May 15, 2022
4407487
Fix up the merge.
gregchapman-dev May 22, 2022
9cb5cea
Regenerate the metadata cache (again, because there were changes on m…
gregchapman-dev May 23, 2022
22bef56
Rename the property description stuff, and move the property list to …
gregchapman-dev May 23, 2022
8b75e11
pylint, mypy, flake8
gregchapman-dev May 23, 2022
f3835cd
Lots of renames.
gregchapman-dev May 23, 2022
0a6f498
Make getAll(...) return a Tuple, not a List.
gregchapman-dev May 24, 2022
53c7f18
Tuples and Lists are declared a bit differently (yay, mypy).
gregchapman-dev May 24, 2022
f5c8d9a
Get rid of 'music21' namespace, replace with new 'humdrum' namespace.…
gregchapman-dev May 24, 2022
98b5f69
pylint complained about lines too long.
gregchapman-dev May 24, 2022
a6348a6
Fix a couple tests I missed.
gregchapman-dev May 24, 2022
a3029d6
Redo the main APIs a bit:
gregchapman-dev May 25, 2022
333440c
pylint
gregchapman-dev May 25, 2022
e741e45
Fix the musicxml import/export tests.
gregchapman-dev May 25, 2022
48fd89d
"Modernize" MEI import's use of metadata APIs. Other cleanup (mostly…
gregchapman-dev May 25, 2022
7a21f4b
A little cleanup of unused PropertyDescription fields and associated …
gregchapman-dev May 25, 2022
0d0c92d
If string is not parseable as DateSingle, just return the unconverted…
gregchapman-dev May 25, 2022
d9955c0
pylint
gregchapman-dev May 25, 2022
5951e54
humdrum/spineParser.py now uses new metadata APIs, and imports _all_ …
gregchapman-dev May 26, 2022
771497b
Rewrite __getattr__ to support any uniqueName, workId, or workId abbr…
gregchapman-dev May 26, 2022
7bfa20b
Add __getitem__ a.k.a. dict-key-access for uniqueName and namespace:n…
gregchapman-dev May 26, 2022
9aa4f57
flake8
gregchapman-dev May 26, 2022
1310f45
Oops. Return 'MULTIPLE' from __getattr__ if there is more than one i…
gregchapman-dev May 27, 2022
ee32c42
More cleanup. Make copyright property work like all the others (retu…
gregchapman-dev May 27, 2022
f768051
Add 'subtitle' ('mei:subtitle') property term. Many metadata improve…
gregchapman-dev May 27, 2022
5a1eab7
Test the new accessors.
gregchapman-dev May 27, 2022
eedc849
Enhance/fix ABC and Lilypond metadata processing to use the new metad…
gregchapman-dev May 27, 2022
2901952
Remove md.get(), md.getFirst(), and md.getFirstCustom().
gregchapman-dev May 28, 2022
b4af067
Implement __setitem__ (replaces set()), and __setattr__ (adds propert…
gregchapman-dev May 29, 2022
ebec3ce
RomanText parsing now produces multiple titles etc if seen. Fixed so…
gregchapman-dev May 29, 2022
e9e80a3
More coverage (and a bugfix or two) in metadata/primitives.py.
gregchapman-dev May 30, 2022
a848ad2
Change md.all() and md.search() and md.contributors to include all th…
gregchapman-dev May 31, 2022
3864cb2
Simplification and cleanup. Had to regenerate metadata cache. Fixed …
gregchapman-dev Jun 1, 2022
b715e80
Get past the new pylint restrictions (shakes fist at jacobtylerwalls …
gregchapman-dev Jun 1, 2022
452b553
Respond to various review comments:
gregchapman-dev Jun 7, 2022
e768513
Metadata attributes return constructed strings instead of 'MULTIPLE'.
gregchapman-dev Jun 14, 2022
7576d48
Fix docs/doctests. Use new ValueType instead of t.Any.
gregchapman-dev Jun 15, 2022
3c49227
More review feedback incorporated:
gregchapman-dev Jun 15, 2022
501e13d
properties.py: first cut at removals, set oldMusic21WorkId only if ne…
gregchapman-dev Jun 18, 2022
888b344
Cleanup testMetadata.py (and add some new tests). Fix class Text to …
gregchapman-dev Jun 18, 2022
27f740a
Oops, lost an f''.
gregchapman-dev Jun 18, 2022
3d6547c
Better indenting in testMetadata.py, replace multiple checks in __set…
gregchapman-dev Jun 18, 2022
7cb6a2d
Put Metadata.date and Metadata.setWorkId back in place, deprecated.
gregchapman-dev Jun 18, 2022
e707ae6
Organized public vs private APIs a bit, and cleaned up some of those …
gregchapman-dev Jun 19, 2022
b416f76
Add some imprint-related metadata terms (from the humdrum namespace).
gregchapman-dev Jun 22, 2022
7bad65e
Remove 'performer' from spineParser.py, since I removed it from metad…
gregchapman-dev Jun 23, 2022
676c11f
More review feedback incorporated.
gregchapman-dev Jun 23, 2022
194202b
Store the software list in metadata._contents.
gregchapman-dev Jun 23, 2022
60e10fb
Fix test failures.
gregchapman-dev Jun 24, 2022
2d16b16
Big name change (in code and docs) from nsKey to namespaceName.
gregchapman-dev Jun 24, 2022
cd392f3
Added and improved docs/doctests throughout.
gregchapman-dev Jun 26, 2022
267ee95
Fix small notes; give credit
mscuthbert Aug 3, 2022
59c76c4
tiny change noted in docs
mscuthbert Aug 3, 2022
e23eb15
Improve search performance (and related work, and a few random fixes)…
gregchapman-dev Aug 4, 2022
ed4bd2a
Metadata's _contents are now keyed by uniqueName (or custom name), no…
gregchapman-dev Aug 4, 2022
76bda31
Merge branch 'master' into gregc/metadataDublinCore to pick up a sing…
gregchapman-dev Aug 4, 2022
3f1d5ea
More review responses:
gregchapman-dev Aug 5, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion music21/_version.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
Changing this number invalidates old pickles -- do it if the old pickles create a problem.
'''

__version_info__ = (8, 0, 0, 'a3') # can be 4-tuple: (7, 0, 5, 'a2')
__version_info__ = (8, 0, 0, 'a9') # can be 4-tuple: (7, 0, 5, 'a2')

v = '.'.join(str(x) for x in __version_info__[0:3])
if len(__version_info__) > 3 and __version_info__[3]: # type: ignore
Expand Down
20 changes: 11 additions & 9 deletions music21/abcFormat/translate.py
Original file line number Diff line number Diff line change
Expand Up @@ -402,24 +402,26 @@ def abcToStreamScore(abcHandler, inputM21=None):
if isinstance(t, abcFormat.ABCMetadata):
if t.isTitle():
if titleCount == 0: # first
md.title = t.data
# environLocal.printDebug(['got metadata title', md.title])
md.add('title', t.data)
# environLocal.printDebug(['got metadata title', t.data])
titleCount += 1
# all other titles go in alternative field
else:
md.alternativeTitle = t.data
# environLocal.printDebug(['got alternative title', md.alternativeTitle])
md.add('alternativeTitle', t.data)
# environLocal.printDebug(['got alternative title', t.data])
titleCount += 1

elif t.isComposer():
md.composer = t.data
md.add('composer', t.data)
# environLocal.printDebug(['got composer', t.data])

elif t.isOrigin():
md.localeOfComposition = t.data
# environLocal.printDebug(['got local of composition', md.localOfComposition])
md.add('localeOfComposition', t.data)
# environLocal.printDebug(['got locale of composition', t.data])

elif t.isReferenceNumber():
md.number = int(t.data) # convert to int?
# environLocal.printDebug(['got work number', md.number])
md.add('number', int(t.data))
gregchapman-dev marked this conversation as resolved.
Show resolved Hide resolved
# environLocal.printDebug(['got work number', t.data])

partHandlers = []
tokenCollections = abcHandler.splitByVoice()
Expand Down
2 changes: 1 addition & 1 deletion music21/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
<class 'music21.base.Music21Object'>

>>> music21.VERSION_STR
'8.0.0a3'
'8.0.0a9'

Alternatively, after doing a complete import, these classes are available
under the module "base":
Expand Down
3 changes: 2 additions & 1 deletion music21/braille/examples.py
Original file line number Diff line number Diff line change
Expand Up @@ -1176,7 +1176,8 @@ def testVoices(self):

demo = corpus.parse('demos/two-voices')
x = objectToBraille(demo, debug=True)
y = '''Movement Name: two-voices.xml
y = '''Composer: Music21
Movement Name: two-voices.xml
Title: Music21 Fragment
---begin segment---
<music21.braille.segment BrailleSegment>
Expand Down
39 changes: 28 additions & 11 deletions music21/braille/translate.py
Original file line number Diff line number Diff line change
Expand Up @@ -472,24 +472,41 @@ def metadataToString(music21Metadata, returnBrailleUnicode=False):
<class 'music21.metadata.Metadata'>
>>> print(translate.metadataToString(mdObject))
Alternative Title: 3.1
Composer: Claudio Monteverdi
Title: La Giovinetta Pianta

>>> print(translate.metadataToString(mdObject, returnBrailleUnicode=True))
⠠⠁⠇⠞⠑⠗⠝⠁⠞⠊⠧⠑⠀⠠⠞⠊⠞⠇⠑⠒⠀⠼⠉⠲⠁
⠠⠉⠕⠍⠏⠕⠎⠑⠗⠒⠀⠠⠉⠇⠁⠥⠙⠊⠕⠀⠠⠍⠕⠝⠞⠑⠧⠑⠗⠙⠊
⠠⠞⠊⠞⠇⠑⠒⠀⠠⠇⠁⠀⠠⠛⠊⠕⠧⠊⠝⠑⠞⠞⠁⠀⠠⠏⠊⠁⠝⠞⠁
'''
allBrailleLines = []
for key in music21Metadata._workIds:
value = music21Metadata._workIds[key]
if value is not None:
n = ' '.join(re.findall(r'([A-Z]*[a-z]+)', key))
outString = f'{n.title()}: {value}'
if returnBrailleUnicode:
outTemp = []
for word in outString.split():
outTemp.append(wordToBraille(word))
outString = alphabet[' '].join(outTemp)
allBrailleLines.append(outString)
for uniqueName, value in music21Metadata.all(returnPrimitives=True, returnSorted=False):
if value is None:
# we don't put None values in braille output
continue

if uniqueName == 'software':
# we don't put software versions in braille output
continue

namespaceName: t.Optional[str] = music21Metadata.uniqueNameToNamespaceName(uniqueName)
if not namespaceName:
# we don't put custom metadata in braille output
continue

if namespaceName.startswith('m21FileInfo:'):
# we don't put fileInfo in braille output
continue

n = ' '.join(re.findall(r'([A-Z]*[a-z]+)', uniqueName))
outString = f'{n.title()}: {value}'
if returnBrailleUnicode:
outTemp = []
for word in outString.split():
outTemp.append(wordToBraille(word))
outString = alphabet[' '].join(outTemp)
allBrailleLines.append(outString)
return '\n'.join(sorted(allBrailleLines))


Expand Down
2 changes: 1 addition & 1 deletion music21/converter/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1732,7 +1732,7 @@ def testConversionMXMetadata(self):
a = parse(testFiles.binchoisMagnificat)
self.assertEqual(a.metadata.composer, 'Gilles Binchois')
# this gets the best title available, even though this is movement title
self.assertEqual(a.metadata.title, 'Excerpt from Magnificat secundi toni')
self.assertEqual(a.metadata.bestTitle, 'Excerpt from Magnificat secundi toni')

def testConversionMXBarlines(self):
from music21 import bar
Expand Down
2 changes: 1 addition & 1 deletion music21/converter/subConverters.py
Original file line number Diff line number Diff line change
Expand Up @@ -818,7 +818,7 @@ def parseFile(self,
filePath: t.Union[pathlib.Path, str],
number: t.Optional[int] = None,
**keywords):
# noinspection SpellCheckingInspection
# noinspection SpellCheckingInspection,PyShadowingNames
'''
Open Noteworthy data (as nwctxt) from a file path.

Expand Down
Binary file modified music21/corpus/_metadataCache/core.p.gz
Binary file not shown.
32 changes: 24 additions & 8 deletions music21/corpus/manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -362,17 +362,33 @@ def listSearchFields():
>>> for field in corpus.manager.listSearchFields():
... field
...
'actNumber'
'alternativeTitle'
'ambitus'
'associatedWork'
'collectionDesignation'
'commission'
'abstract'
'accessRights'
'accompanyingMaterialWriter'
...
'composer'
'copyright'
'composerAlias'
'composerCorporate'
'conceptor'
'conductor'
...
'dateCreated'
'dateFirstPublished'
'dateIssued'
'dateModified'
'dateSubmitted'
'dateValid'
...
'tempoFirst'
'tempos'
'textLanguage'
'textOriginalLanguage'
'timeSignatureFirst'
'timeSignatures'
'title'
...
'''
return tuple(sorted(metadata.RichMetadata.searchAttributes))
return metadata.bundles.MetadataBundle.listSearchFields()

# -----------------------------------------------------------------------------

Expand Down
156 changes: 120 additions & 36 deletions music21/humdrum/spineParser.py
Original file line number Diff line number Diff line change
Expand Up @@ -776,9 +776,8 @@ def parseMetadata(self, s=None):
grToRemove = []

for gr in s[GlobalReference]:
wasParsed = gr.updateMetadata(md)
if wasParsed:
grToRemove.append(gr)
gr.updateMetadata(md)
grToRemove.append(gr)

if grToRemove:
s.remove(grToRemove, recurse=True)
Expand Down Expand Up @@ -2727,6 +2726,116 @@ def __init__(self, codeOrAll='', valueOrNone=None):
if '@' in self.code:
self.code, self.language = self.code.split('@')

humdrumKeyToUniqueName: dict = {
# dict value is music21's unique name or '' (if there is no supported equivalent)
# Authorship information:
gregchapman-dev marked this conversation as resolved.
Show resolved Hide resolved
'COM': 'composer', # composer's name
'COA': 'attributedComposer', # attributed composer
'COS': 'suspectedComposer', # suspected composer
'COL': 'composerAlias', # composer's abbreviated, alias, or stage name
'COC': 'composerCorporate', # composer's corporate name
'CDT': '', # composer's birth and death dates (**zeit format)
'CBL': '', # composer's birth location
'CDL': '', # composer's death location
'CNT': '', # composer's nationality
'LYR': 'lyricist', # lyricist's name
'LIB': 'librettist', # librettist's name
'LAR': 'arranger', # music arranger's name
'LOR': 'orchestrator', # orchestrator's name
'TXO': 'textOriginalLanguage', # original language of vocal/choral text
'TXL': 'textLanguage', # language of the encoded vocal/choral text
# Recording information (if the Humdrum encodes information pertaining
# to an audio recording)
'TRN': 'translator', # translator of the text
'RTL': '', # album title
'RMM': 'manufacturer', # manufacturer or sponsoring company
'RC#': '', # recording company's catalog number of album
'RRD': 'dateIssued', # release date (**date format)
'RLC': '', # place of recording
'RNP': 'producer', # producer's name
'RDT': '', # date of recording (**date format)
'RT#': '', # track number
# Performance information (if the Humdrum encodes, say, a MIDI performance)
'MGN': '', # ensemble's name
'MPN': '', # performer's name
'MPS': '', # suspected performer
'MRD': '', # date of performance (**date format)
'MLC': '', # place of performance
'MCN': 'conductor', # conductor's name
'MPD': '', # date of first performance (**date format)
'MDT': '', # unknown, but I've seen 'em (another way to say date of performance?)
# Work identification information
'OTL': 'title', # title
'OTP': 'popularTitle', # popular title
'OTA': 'alternativeTitle', # alternative title
'OPR': 'parentTitle', # title of parent work
'OAC': 'actNumber', # act number (e.g. '2' or 'Act 2')
'OSC': 'sceneNumber', # scene number (e.g. '3' or 'Scene 3')
'OMV': 'movementNumber', # movement number (e.g. '4', or 'mov. 4', or...)
'OMD': 'movementName', # movement name
'OPS': 'opusNumber', # opus number (e.g. '23', or 'Opus 23')
'ONM': 'number', # number (e.g. number of song within ABC multi-song file)
'OVM': 'volumeNumber', # volume number (e.g. '6' or 'Vol. 6')
'ODE': 'dedicatedTo', # dedicated to
'OCO': 'commission', # commissioned by
'OCL': 'transcriber', # collected/transcribed by
'ONB': '', # free form note (nota bene) related to title or identity of work
'ODT': 'dateCreated', # date or period of composition (**date or **zeit format)
'OCY': 'countryOfComposition', # country of composition
'OPC': 'localeOfComposition', # city, town, or village of composition
# Group information
'GTL': 'groupTitle', # group title (e.g. 'The Seasons')
'GAW': 'associatedWork', # associated work, such as a play or film
'GCO': 'collectionDesignation', # collection designation (e.g. 'Norton Scores')
# Imprint information
'PUB': '', # publication status 'published'/'unpublished'
'PED': '', # publication editor
'PPR': 'firstPublisher', # first publisher
'PDT': 'dateFirstPublished', # date first published (**date format)
'PTL': 'publicationTitle', # publication (volume) title
'PPP': 'placeFirstPublished', # place first published
'PC#': 'publishersCatalogNumber', # publisher's catalog number (NOT scholarly catalog)
'SCT': 'scholarlyCatalogAbbreviation', # scholarly catalog abbrev/number (e.g. 'BWV 551')
'SCA': 'scholarlyCatalogName', # scholarly catalog (unabbreviated) (e.g. 'Koechel 117')
'SMS': 'manuscriptSourceName', # unpublished manuscript source name
'SML': 'manuscriptLocation', # unpublished manuscript location
'SMA': 'manuscriptAccessAcknowledgement', # acknowledgment of manuscript access
'YEP': 'electronicPublisher', # publisher of electronic edition
'YEC': 'copyright', # date and owner of electronic copyright
'YER': 'electronicReleaseDate', # date electronic edition released
'YEM': '', # copyright message (e.g. 'All rights reserved')
'YEN': '', # country of copyright
'YOR': '', # original document from which encoded document was prepared
'YOO': 'originalDocumentOwner', # original document owner
'YOY': '', # original copyright year
'YOE': 'originalEditor', # original editor
'EED': 'electronicEditor', # electronic editor
'ENC': 'electronicEncoder', # electronic encoder (person)
'END': '', # encoding date
'EMD': '', # electronic document modification description (one per modificiation)
'EEV': '', # electronic edition version
'EFL': '', # file number e.g. '1/4' for one of four
'EST': '', # encoding status (free form, normally eliminated prior to distribution)
'VTS': '', # checksum (excluding the VTS line itself)
# Analytic information
'ACO': '', # collection designation
'AFR': '', # form designation
'AGN': '', # genre designation
'AST': '', # style, period, or type of work designation
'AMD': '', # mode classification e.g. '5; Lydian'
'AMT': '', # metric classification, must be one of eight specific names
'AIN': '', # instrumentation; alphabetically ordered list of *I abbrevs, space-delimited
'ARE': '', # geographical region of origin (list of 'narrowing down' names of regions)
'ARL': '', # geographical location of origin (lat/long)
# Historical and background information
'HAO': '', # aural history (lots of text, stories about the work)
'HTX': '', # freeform translation of vocal text
# Representation information
'RLN': '', # Extended ASCII language code
'RNB': '', # a note about the representation
'RWB': '' # a warning about the representation
}

def updateMetadata(self, md):
'''
update a metadata object according to information in this GlobalReference
Expand All @@ -2735,41 +2844,16 @@ def updateMetadata(self, md):
'''
c = self.code
v = self.value
wasParsed = True

contributorNames = {
'COM': 'composer',
'COA': 'attributed composer',
'COS': 'suspected composer',
'COL': 'composer alias',
'COC': 'corporate composer',
'LYR': 'lyricist',
'LIB': 'librettist',
'LAR': 'arranger',
'LOR': 'orchestrator',
'TRN': 'translator',
'YOO': 'original document owner',
'YOE': 'original editor',
'EED': 'electronic editor',
'ENC': 'electronic encoder'
gregchapman-dev marked this conversation as resolved.
Show resolved Hide resolved
}

if c in contributorNames:
contrib = metadata.Contributor()
contrib.role = contributorNames[c]
contrib.name = v
md.addContributor(contrib)

elif c.lower() in md.workIdAbbreviationDict:
md.setWorkId(c, v)

elif c == 'YEC': # electronic edition copyright.
md.copyright = metadata.Copyright(v)

uniqueName: str = self.humdrumKeyToUniqueName.get(c, '')
if uniqueName:
md.add(uniqueName, v)
elif c in self.humdrumKeyToUniqueName:
# it's a humdrum key, but unsupported
md.addCustom('humdrum:' + c, v)
gregchapman-dev marked this conversation as resolved.
Show resolved Hide resolved
else:
wasParsed = False

return wasParsed
# it's a free-form key
md.addCustom(c, v)

def _reprInternal(self):
return f'{self.code} {self.value!r}'
Expand Down
Loading