-
-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
added json output to songs and podcasts
- Loading branch information
1 parent
7588b32
commit d1e07ea
Showing
4 changed files
with
40 additions
and
4 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,3 +2,4 @@ | |
**/*.csv | ||
**/.vscode | ||
**/.txt | ||
repopack-output.txt |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @joshkenney ! This is a handy feature. What's the use-case behind JSON instead of CSV though?
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey @raleighlittles! Thanks for reaching out. I collect iPods and was looking for a way to grab a representation of the libraries. Your software does this brilliantly and I was stoked to find it! I'm also learning a bit of Rust so I tried to make this simple mod. I plan on uploading the JSON as iTunesDB collections in MongoDB (I know, I could have just used CSV but that would have been too easy lol). I'm planning to do a variety of things with the data including publishing the collections as playlists to Spotify by running it against their API, also, running some analysis to find most common songs, etc.
If you're interested in collaborating lmk!
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks again for taking the time to write this software! it saved me an incredible amount of time
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Spotify API idea is genius, wow. There's already a Rust spotify API: https://github.com/ramsayleung/rspotify
It looks like we could just..
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since you collect iPods, would you mind doing me a favor? I'm looking for some files from an iPod so that I could use that as a test to develop new parsing functionality and add it to the repository. The main ones I care about right now are:
The "DeviceInfo" file, described here http://www.ipodlinux.org/ITunesDB/#DeviceInfo_File
Examples of photos taken on an iPod. They should have a file extension called
.ithmb
. These would be on iPod 4th gen or neweran ITunesDB file for an iTunes library that had movies and TV shows. Right now my parsing code only works with audio - podcasts and songs only.
If you have iPods that would have any of these files, I would appreciate it if you could share. @joshkenney
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very cool! Great find! Let’s do it.
I’ll spin up an express app and let you know the route to access the tracks
Also - I was messing around with your repo last night. I am hoping to append some information about the source iPod (model, name, generation, release year). This would allow me to keep all tracks in the same collection, rather than in separate ones.
I see that, currently, I can pass in “preferences” argument which, I think looks at the Preferences file only if I pass a full iPod_Control directory as the path.
Could we do something similar to grab the iPod stuff out of, I assume, Device/SysInfo?
Happy to hop on a zoom call whenever. I will take another look over what I have done so far and push as a separate branch
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I was writing my response while you were sending your next message. I can absolutely send you the files you’re looking for. What’s the best way to get those to you? I could also send you entire iPod_Control directories sans the Music Directory if that would help you. I can send you one from each generation. I rsync all the pods as part of my preservation process
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another thing we might want to harvest if we can is the firmware version
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know too much about the folder structure on iPods, because the files I have were mostly ones that were sent to me or I found later. But the code should be able to extract the preferences file wherever it is if you give it the full path:
./target/debug/itunesdb_parser ../sample-files/2023-08-29_Preferences "preferences"
Hmm, that might be an option. I don't see anything about the Sysinfo file on the itunesDB wiki but if its a file that has that info, then sure.
Here is a google shared drive, can you post at a device info file there? (also let me know which generation iPod/model it came from)
I can add the code to extract the info from them.
In the meantime, I'll read up on how to setup the spotify API on the rust-side of things using that crate I linked earlier
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I just upload iPod_Control.zip file to your drive.
It contains the one of each i
Pod_Control
directories for each model of iPod in my collection (minus theMusic/
subdirectory).The iPod 5th generation had photos on device and the iPod 5.5 generation had videos on device.
I've also pushed the
deviceInfo
branch to my fork.You can see the format I'd like to use but those things are not found in iTunesDB. I need help referencing the correct file, possibly
iPod_Control/Device/SysInfo
Let me know what you find and how I can help!
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! Those files were super helpful, I added support on my repository now for parsing (experimental-ish) iTunesSD files, equalizer presets, and the DeviceInfo file.
The
SysInfo
file is just a regular text file thankfully, so no parsing needed.d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You said the gen 5.5 had videos on it, do you know if these were videos that you transferred/uploaded to the iPod manually? Or are these videos like movies/TV content that came from iTunes?
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re: Spotify
I couldn't seem to get the spotify api on the rust side working. I'm going to try using the Python one: https://spotipy.readthedocs.io/en/2.24.0/
since it seems to be more popular. And I'll just have the python code read the song info out of the CSV file. That way it's platform-independent
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! glad the directory structures were helpful!
do you know any way to get the name of the iPod off? they name given to the iPod by the user?
if not, maybe your software could grab it from the filename of parent folder?
if you'd like, I can put together a table so your program can parse the SysInfo ModelNumStr
I've listed the most important elements:
i'd really like to add this info the json output as well. should i submit a pull request for that? perhaps we can pass a flag as a command line arg? default can be csv unless json is preferred by the spotiPy api...
additionally, a nice quality of life improvement would be if it could name the file the name of the ipod or the serial number instead of "songs" and store in a subdirectory of the program
the source of the videos on the 5.5 gen seem to be downloaded to the iPod manually because they have the .m4v extension which seems nonstandard. these are indexed the same as the songs, in
iPod_Control/Music/
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, this is included in the DeviceInfo file. https://github.com/raleighlittles/iTunesDB-Parser/blob/main/parser/src/parsers/deviceinfo_parser.rs
Yes to all of these, submit a PR adding it as a command-line argument, I like that idea.
I got the Spotify integration to work finally, using Python though. Branch here:
feature/add-spotify-component
https://github.com/raleighlittles/iTunesDB-Parser/tree/feature/add-spotify-component
See the spotify directory. Has a Python script that you can run (requires a client ID and client secret that you get from the spotify API website), then you run it passing the CSV file that you got from the iTunesDB file..
For example I ran
With this dummy CSV file
It found those 2 songs and created a playlist for them on Spotify using the hostname of the machine in the description
I'm going to clean it up a bit and then merge into the master branch
@joshkenney
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you add some documentation in the readme on how to run that?
I just submitted the PR with this third param (json / csv)
I tested the SpotiPy functionality and it was great! i was able to publish an iPod test playlist as well
In doing so, I noticed it is not grabbing all of the tracks. in my example only 61/133. i tried to do some digging but ran out of time.
I just uploaded GEOFFREY.zip which is a 2nd gen iPod nano with 133 songs - 61 of which are read by the parser
Happy Thanksgiving! 🦃
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're passing an iTunesDB file, not a device info file, thats why its crashing.
Use a real DeviceInfo file and it would work.
I had the same problem too where a lot of songs were missing, unfortunately I checked a few of them and the songs were just not on Spotify at all, so I think that's why it didn't find it.
Run the spotify script again and this time send me the log, I can go through and investigate it further.
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From the iPods I have, it seems
iTunes/DeviceInfo
is only on Firewire iPods (1st, 2nd, and 3rd gen classics)To your knowledge, is there a newer or renamed file on all other i.e. USB iPods?
Yeah - I get that Spotify might not have all the songs. The issue I'm describing is with the parser itself, where it doesn't parse all of the songs on the iPod.
See GEOFFREY.zip as an example:
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahhh I see, okay! I'll look into that, I haven't noticed that before.
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Damn, I'm still not sure why the songs are missing, there's a general discrepancy overall between what the header says the number of songs are and the actual number of track items found.
It says there's 133 track items in the header, but I only find 72 track items. Out of those 72 track items, 3 of them don't have a title or an artist associated with it for some reason, leaving 69 valid track items, and the tool only picks up 61 of those.. However, there are 133 audio files (mp3 or m4a) in the iPod music directory.
I created a tool to rename the song files based on the CSV, so that way I can try to find info about the ones remaining.
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah it's really strange indeed. I can confirm Geoffrey has 113 named songs with titles and artists on the physical iPod and all play
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i also noticed the tracklist only gets printed sometimes, making comparisons difficult if it doesnt get to this part of the code:
d1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Huh, I see that get printed on every iTunesDB file I parse. I created an issue for it here to investigate, will probably have to wait until next week to really dive into it though.
Btw, do you have access to a Linux system and a 4th gen+ iPod? If so, can you try the instructions here:
http://www.ipodlinux.org/Device_Information/
Involves running an SCSI inquiry to extract Device Info from the iPod. You have to install the
sg3_utils
package @joshkenneyd1e07ea
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @joshkenney , just wanted to follow up about the last thing I mentioned, running the SCSI query on the ipod. Have a linux system, and 10 minutes? If so I'd appreciate you testing it, it's something I could add to the repository and share with others.