Skip to content

Commit

Permalink
added json output to songs and podcasts
Browse files Browse the repository at this point in the history
  • Loading branch information
joshkenney committed Nov 13, 2024
1 parent 7588b32 commit d1e07ea
Show file tree
Hide file tree
Showing 4 changed files with 40 additions and 4 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@
**/*.csv
**/.vscode
**/.txt
repopack-output.txt
6 changes: 4 additions & 2 deletions parser/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
chrono = "0.4.31"
csv = "1.3.0"
chrono = { version = "0.4.31", features = ["serde"] } # Add serde feature
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

# Languages parsing
isolang = "2.4.0"
isolang = "2.4.0"
9 changes: 7 additions & 2 deletions parser/src/itunesdb.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,11 @@
use crate::helpers::helpers;
use crate::helpers::itunesdb_helpers;

pub struct Podcast {
use serde::Serialize;


#[derive(Serialize)]
pub struct Podcast {
pub podcast_title : String,
pub podcast_publisher : String,
pub podcast_genre : String,
Expand All @@ -35,7 +39,8 @@
}
}

pub struct Song {
#[derive(Serialize)]
pub struct Song {
pub file_extension: String,
pub bitrate_kbps: u32,
pub sample_rate_hz: u32,
Expand Down
28 changes: 28 additions & 0 deletions parser/src/parsers/itunesdb_parser.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
use std::fmt::Write;
use std::fs::File;
use std::io;

use crate::constants::itunesdb_constants;
use crate::itunesdb;
Expand Down Expand Up @@ -840,6 +842,32 @@ pub fn parse_itunesdb_file(itunesdb_file_as_bytes : Vec<u8>) {

println!("{} songs found", songs_found.len());

// Add JSON output @joshkenney
if !songs_found.is_empty() {
let songs_json = serde_json::to_string_pretty(&songs_found)
.expect("Error serializing songs to JSON");

let mut songs_json_file = File::create("songs.json")
.expect("Error creating songs JSON file");

io::Write::write_all(&mut songs_json_file, songs_json.as_bytes())
.expect("Error writing songs JSON file");

println!("Created songs.json with {} songs", songs_found.len());
}

if !podcasts_found.is_empty() {
let podcasts_json = serde_json::to_string_pretty(&podcasts_found)
.expect("Error serializing podcasts to JSON");

let mut podcasts_json_file = File::create("podcasts.json")
.expect("Error creating podcasts JSON file");

io::Write::write_all(&mut podcasts_json_file, podcasts_json.as_bytes())
.expect("Error writing podcasts JSON file");

println!("Created podcasts.json with {} podcasts", podcasts_found.len());
}

podcast_csv_writer.write_record(&[
"Episode Title",
Expand Down

24 comments on commit d1e07ea

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @joshkenney ! This is a handy feature. What's the use-case behind JSON instead of CSV though?

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @raleighlittles! Thanks for reaching out. I collect iPods and was looking for a way to grab a representation of the libraries. Your software does this brilliantly and I was stoked to find it! I'm also learning a bit of Rust so I tried to make this simple mod. I plan on uploading the JSON as iTunesDB collections in MongoDB (I know, I could have just used CSV but that would have been too easy lol). I'm planning to do a variety of things with the data including publishing the collections as playlists to Spotify by running it against their API, also, running some analysis to find most common songs, etc.

If you're interested in collaborating lmk!

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks again for taking the time to write this software! it saved me an incredible amount of time

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Spotify API idea is genius, wow. There's already a Rust spotify API: https://github.com/ramsayleung/rspotify

It looks like we could just..

  • Use the "Search" endpoint to search for the spotify track ID, using the song's name and artist
  • Save those track Ids to a list
  • Use the "Playlist" endpoint to create a new Playlist
  • Add the song IDs from step 2 to that playlist

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you collect iPods, would you mind doing me a favor? I'm looking for some files from an iPod so that I could use that as a test to develop new parsing functionality and add it to the repository. The main ones I care about right now are:

  1. The "DeviceInfo" file, described here http://www.ipodlinux.org/ITunesDB/#DeviceInfo_File

  2. Examples of photos taken on an iPod. They should have a file extension called .ithmb. These would be on iPod 4th gen or newer

  3. an ITunesDB file for an iTunes library that had movies and TV shows. Right now my parsing code only works with audio - podcasts and songs only.

If you have iPods that would have any of these files, I would appreciate it if you could share. @joshkenney

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! Great find! Let’s do it.

I’ll spin up an express app and let you know the route to access the tracks

Also - I was messing around with your repo last night. I am hoping to append some information about the source iPod (model, name, generation, release year). This would allow me to keep all tracks in the same collection, rather than in separate ones.

I see that, currently, I can pass in “preferences” argument which, I think looks at the Preferences file only if I pass a full iPod_Control directory as the path.

Could we do something similar to grab the iPod stuff out of, I assume, Device/SysInfo?

Happy to hop on a zoom call whenever. I will take another look over what I have done so far and push as a separate branch

@joshkenney
Copy link
Contributor Author

@joshkenney joshkenney commented on d1e07ea Nov 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I was writing my response while you were sending your next message. I can absolutely send you the files you’re looking for. What’s the best way to get those to you? I could also send you entire iPod_Control directories sans the Music Directory if that would help you. I can send you one from each generation. I rsync all the pods as part of my preservation process

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing we might want to harvest if we can is the firmware version

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that, currently, I can pass in “preferences” argument which, I think looks at the Preferences file only if I pass a full iPod_Control directory as the path.

I don't know too much about the folder structure on iPods, because the files I have were mostly ones that were sent to me or I found later. But the code should be able to extract the preferences file wherever it is if you give it the full path:

 ./target/debug/itunesdb_parser ../sample-files/2023-08-29_Preferences "preferences"

Could we do something similar to grab the iPod stuff out of, I assume, Device/SysInfo?

Hmm, that might be an option. I don't see anything about the Sysinfo file on the itunesDB wiki but if its a file that has that info, then sure.

Here is a google shared drive, can you post at a device info file there? (also let me know which generation iPod/model it came from)

I can add the code to extract the info from them.

In the meantime, I'll read up on how to setup the spotify API on the rust-side of things using that crate I linked earlier

@joshkenney
Copy link
Contributor Author

@joshkenney joshkenney commented on d1e07ea Nov 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I just upload iPod_Control.zip file to your drive.

It contains the one of each iPod_Control directories for each model of iPod in my collection (minus the Music/ subdirectory).

The iPod 5th generation had photos on device and the iPod 5.5 generation had videos on device.

.
├── classic
│   ├── gen1
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   └── SysInfo
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── iEKInfo
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPlaylists
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   ├── gen2
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   └── SysInfo
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── iEKInfo
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPlaylists
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   ├── gen3
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   └── SysInfo
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── OTGPlaylistInfo
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPlaylists
│   │       ├── iTunesPodcasts
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   ├── gen4
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   └── SysInfo
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── DeviceInfo
│   │       ├── OTGPlaylistInfo
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesEQPresets
│   │       ├── iTunesPrefs
│   │       ├── iTunesPrefs.plist
│   │       └── winPrefs
│   ├── gen4photo
│   │   ├── Artwork
│   │   │   └── ArtworkDB
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   └── SysInfo
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   ├── gen5
│   │   ├── Artwork
│   │   │   ├── ArtworkDB
│   │   │   ├── F1028_1.ithmb
│   │   │   └── F1029_1.ithmb
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   ├── SysInfo
│   │   │   ├── clock
│   │   │   └── radio
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── ApertureAlbumPrefs
│   │       ├── OTGPlaylistInfo
│   │       ├── iPhotoAlbumPrefs
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   ├── gen5.5
│   │   ├── Artwork
│   │   │   ├── ArtworkDB
│   │   │   ├── F1028_1.ithmb
│   │   │   └── F1029_1.ithmb
│   │   ├── Device
│   │   │   ├── Accessories
│   │   │   ├── Preferences
│   │   │   ├── SysInfo
│   │   │   ├── clock
│   │   │   └── radio
│   │   └── iTunes
│   │       ├── PhotosFolderAlbums
│   │       ├── PhotosFolderName
│   │       ├── PhotosFolderPrefs
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPlaylists
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   ├── gen6
│   │   ├── Artwork
│   │   │   ├── ArtworkDB
│   │   │   ├── F1055_1.ithmb
│   │   │   ├── F1060_1.ithmb
│   │   │   └── F1061_1.ithmb
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   ├── SysInfo
│   │   │   ├── alarms
│   │   │   └── clock
│   │   ├── Games_RO
│   │   │   ├── 11004
│   │   │   └── 11010
│   │   ├── Tones
│   │   │   └── Beep.tone
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── ApertureAlbumPrefs
│   │       ├── PhotosFolderAlbums
│   │       ├── PhotosFolderName
│   │       ├── PhotosFolderPrefs
│   │       ├── Rentals.plist
│   │       ├── iPhotoAlbumPrefs
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   └── gen7
│       ├── Artwork
│       │   ├── ArtworkDB
│       │   ├── F1055_1.ithmb
│       │   ├── F1060_1.ithmb
│       │   ├── F1061_1.ithmb
│       │   ├── F1068_1.ithmb
│       │   ├── Image\ DB\ Temp\ 1.tmp
│       │   ├── Image\ DB\ Temp\ 10.tmp
│       │   ├── Image\ DB\ Temp\ 11.tmp
│       │   ├── Image\ DB\ Temp\ 12.tmp
│       │   ├── Image\ DB\ Temp\ 13.tmp
│       │   ├── Image\ DB\ Temp\ 14.tmp
│       │   ├── Image\ DB\ Temp\ 15.tmp
│       │   ├── Image\ DB\ Temp\ 16.tmp
│       │   ├── Image\ DB\ Temp\ 17.tmp
│       │   ├── Image\ DB\ Temp\ 18.tmp
│       │   ├── Image\ DB\ Temp\ 19.tmp
│       │   ├── Image\ DB\ Temp\ 2.tmp
│       │   ├── Image\ DB\ Temp\ 20.tmp
│       │   ├── Image\ DB\ Temp\ 21.tmp
│       │   ├── Image\ DB\ Temp\ 22.tmp
│       │   ├── Image\ DB\ Temp\ 23.tmp
│       │   ├── Image\ DB\ Temp\ 24.tmp
│       │   ├── Image\ DB\ Temp\ 25.tmp
│       │   ├── Image\ DB\ Temp\ 26.tmp
│       │   ├── Image\ DB\ Temp\ 3.tmp
│       │   ├── Image\ DB\ Temp\ 4.tmp
│       │   ├── Image\ DB\ Temp\ 5.tmp
│       │   ├── Image\ DB\ Temp\ 6.tmp
│       │   ├── Image\ DB\ Temp\ 7.tmp
│       │   ├── Image\ DB\ Temp\ 8.tmp
│       │   ├── Image\ DB\ Temp\ 9.tmp
│       │   └── Image\ DB\ Temp.tmp
│       ├── Device
│       │   ├── PlayCounts
│       │   ├── Preferences
│       │   ├── SysInfo
│       │   ├── Users
│       │   ├── alarms
│       │   └── clock
│       ├── Games_RO
│       │   ├── 11004
│       │   └── 11010
│       ├── Tones
│       │   └── Beep.tone
│       ├── gamedata_RW
│       │   ├── 11004
│       │   └── 12347
│       ├── gamestats_WO
│       │   ├── 11004
│       │   └── 12347
│       ├── iPodPrefs
│       └── iTunes
│           ├── Extras.itdb
│           ├── PhotosFolderAlbums
│           ├── PhotosFolderName
│           ├── PhotosFolderPrefs
│           ├── Rentals.plist
│           ├── iTunesControl
│           ├── iTunesDB
│           ├── iTunesPodcasts
│           ├── iTunesPrefs
│           └── iTunesPrefs.plist
├── mini
│   ├── gen1
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   └── SysInfo
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── iEKInfo
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   └── gen2
│       ├── Device
│       │   ├── ExtendedSysInfoXml
│       │   ├── Preferences
│       │   └── SysInfo
│       ├── iPodPrefs
│       └── iTunes
├── nano
│   ├── gen1
│   │   ├── Accessories
│   │   ├── Artwork
│   │   │   ├── ArtworkDB
│   │   │   ├── F1027_1.ithmb
│   │   │   └── F1031_1.ithmb
│   │   ├── Device
│   │   │   ├── Preferences
│   │   │   ├── SysInfo
│   │   │   ├── clock
│   │   │   ├── radio
│   │   │   └── timer
│   │   ├── iPodPrefs
│   │   └── iTunes
│   │       ├── OTGPlaylistInfo
│   │       ├── PhotosFolderAlbums
│   │       ├── PhotosFolderName
│   │       ├── PhotosFolderPrefs
│   │       ├── iEKInfo
│   │       ├── iTunesControl
│   │       ├── iTunesDB
│   │       ├── iTunesPlaylists
│   │       ├── iTunesPrefs
│   │       └── iTunesPrefs.plist
│   └── gen2
│       ├── Accessories
│       ├── Artwork
│       │   ├── ArtworkDB
│       │   ├── F1027_1.ithmb
│       │   └── F1031_1.ithmb
│       ├── Device
│       │   ├── Preferences
│       │   ├── SysInfo
│       │   ├── clock
│       │   └── radio
│       ├── iPodPrefs
│       └── iTunes
│           ├── iTunesControl
│           ├── iTunesDB
│           ├── iTunesLock
│           ├── iTunesPrefs
│           └── iTunesPrefs.plist
└── shuffle
    └── gen2
        └── iTunes
            ├── iEKInfo
            ├── iTunesControl
            ├── iTunesDB
            ├── iTunesPState
            ├── iTunesPrefs
            ├── iTunesPrefs.plist
            ├── iTunesSD
            └── iTunesShuffle

I've also pushed the deviceInfo branch to my fork.

You can see the format I'd like to use but those things are not found in iTunesDB. I need help referencing the correct file, possibly iPod_Control/Device/SysInfo

  {
    "file_extension": "MP3",
    "bitrate_kbps": 233,
    "sample_rate_hz": 44100,
    "file_size_bytes": 6288256,
    "file_size_friendly": "6.29 MB",
    "song_duration_s": 215,
    "song_duration_friendly": "3 minutes, 35 seconds",
    "num_plays": 6,
    "song_rating_raw": 0,
    "song_added_to_library_epoch": 3403542998,
    "song_added_to_library_ts": "2011-11-07T20:36:38Z",
    "song_year": 2006,
    "song_title": "Descending",
    "song_artist": "Lamb of God",
    "song_composer": "Lamb of God",
    "song_album": "Sacrament",
    "song_genre": "Metal",
    "song_comment": "http://www.knac.com",
    "song_filename": "iPod_Control/Music/F00/ZEUN.mp3",
    "ipod_model": "Unknown",
    "ipod_generation": "Unknown",
    "ipod_name": "Unknown iPod (2GB)",
    "ipod_release_year": null
  },

Let me know what you find and how I can help!

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Those files were super helpful, I added support on my repository now for parsing (experimental-ish) iTunesSD files, equalizer presets, and the DeviceInfo file.

The SysInfo file is just a regular text file thankfully, so no parsing needed.

BoardHwName: iPod Q22
pszSerialNumber: JQ45044RQKK
ModelNumStr: M9435
FirewireGuid: 0x000A270002D17DB4
HddFirmwareRev: 3.07    
RegionCode: LL(0x0001)
PolicyFlags: 0x00000000
buildID: 0x02618000 (2.6.1)
visibleBuildID: 0x01418000 (1.4.1)
boardHwRev: 0x00000000 (0.0 0)
boardHwSwInterfaceRev: 0x00040013 (0.0.4 19)
bootLoaderImageRev: 0x00000000 (0.0 0)
diskModeImageRev: 0x00000000 (0.0 0)
diagImageRev: 0x00000000 (0.0 0)
osImageRev: 0x00000000 (0.0 0)
iPodFamily: 0x00000003
updaterFamily: 0x00000006

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You said the gen 5.5 had videos on it, do you know if these were videos that you transferred/uploaded to the iPod manually? Or are these videos like movies/TV content that came from iTunes?

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: Spotify

I couldn't seem to get the spotify api on the rust side working. I'm going to try using the Python one: https://spotipy.readthedocs.io/en/2.24.0/

since it seems to be more popular. And I'll just have the python code read the song info out of the CSV file. That way it's platform-independent

@joshkenney
Copy link
Contributor Author

@joshkenney joshkenney commented on d1e07ea Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! glad the directory structures were helpful!

do you know any way to get the name of the iPod off? they name given to the iPod by the user?

if not, maybe your software could grab it from the filename of parent folder?

if you'd like, I can put together a table so your program can parse the SysInfo ModelNumStr

I've listed the most important elements:

pszSerialNumber: JQ45044RQKK
ModelNumStr: M9435
ModelNumDesc: iPod Mini 1st Generation * from table I can provide
HddFirmwareRev: 3.07    
visibleBuildID: 0x01418000 (1.4.1)

i'd really like to add this info the json output as well. should i submit a pull request for that? perhaps we can pass a flag as a command line arg? default can be csv unless json is preferred by the spotiPy api...

additionally, a nice quality of life improvement would be if it could name the file the name of the ipod or the serial number instead of "songs" and store in a subdirectory of the program

the source of the videos on the 5.5 gen seem to be downloaded to the iPod manually because they have the .m4v extension which seems nonstandard. these are indexed the same as the songs, iniPod_Control/Music/

@raleighlittles
Copy link
Owner

@raleighlittles raleighlittles commented on d1e07ea Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know any way to get the name of the iPod off? they name given to the iPod by the user?

Yup, this is included in the DeviceInfo file. https://github.com/raleighlittles/iTunesDB-Parser/blob/main/parser/src/parsers/deviceinfo_parser.rs

i'd really like to add this info the json output as well. should i submit a pull request for that? perhaps we can pass a flag as a command line arg?

Yes to all of these, submit a PR adding it as a command-line argument, I like that idea.


I got the Spotify integration to work finally, using Python though. Branch here: feature/add-spotify-component

https://github.com/raleighlittles/iTunesDB-Parser/tree/feature/add-spotify-component

See the spotify directory. Has a Python script that you can run (requires a client ID and client secret that you get from the spotify API website), then you run it passing the CSV file that you got from the iTunesDB file..

For example I ran

$ .venv/bin/python spotify_integration.py -f test.csv                                                             ─╯
[DEBUG] Found 2 songs on Spotify

With this dummy CSV file

Song Title,Artist,Album,Year released,File size,Song Duration,Filename,Genre,File extension,Bitrate (kbps),Sample Rate (Hz),File size (bytes),Song duration (seconds),Play count,Rating,Added to library on (timestamp),Added to library on (epoch),Composer,Comment
Empire State of Mind,Jay-Z,Hôtel Costes - Vol.01,1999,5.45 MB,"5 minutes, 32 seconds",iPod_Control/Music/F02/W0313979.mp3,Hôtel Costes,MP3,128,44100,5453614,332,0,No rating,2021-11-20 00:54:23 UTC,3720214463,,
Iron Man,Black Sabbath,Hôtel Costes - Vol.01,1999,5.45 MB,"5 minutes, 32 seconds",iPod_Control/Music/F02/W0313979.mp3,Hôtel Costes,MP3,128,44100,5453614,332,0,No rating,2021-11-20 00:54:23 UTC,3720214463,,

It found those 2 songs and created a playlist for them on Spotify using the hostname of the machine in the description

Image

I'm going to clean it up a bit and then merge into the master branch

@joshkenney

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, this is included in the DeviceInfo file. https://github.com/raleighlittles/iTunesDB-Parser/blob/main/parser/src/parsers/deviceinfo_parser.rs

➜  itunes-db-parser git:(main) ./parser/target/debug/itunesdb_parser ~/Downloads/GEOFFREY/iPod_Control/iTunes/iTunesDB "deviceinfo"
thread 'main' panicked at src/parsers/deviceinfo_parser.rs:6:9:
Invalid DeviceInfo file size! Expected: 1536 | Got: 211678
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

can you add some documentation in the readme on how to run that?


I just submitted the PR with this third param (json / csv)


I got the Spotify integration to work finally, using Python though. Branch here:

I tested the SpotiPy functionality and it was great! i was able to publish an iPod test playlist as well

In doing so, I noticed it is not grabbing all of the tracks. in my example only 61/133. i tried to do some digging but ran out of time.

I just uploaded GEOFFREY.zip which is a 2nd gen iPod nano with 133 songs - 61 of which are read by the parser

Happy Thanksgiving! 🦃

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're passing an iTunesDB file, not a device info file, thats why its crashing.

Image

Use a real DeviceInfo file and it would work.


In doing so, I noticed it is not grabbing all of the tracks. in my example only 61/133.

I had the same problem too where a lot of songs were missing, unfortunately I checked a few of them and the songs were just not on Spotify at all, so I think that's why it didn't find it.

Run the spotify script again and this time send me the log, I can go through and investigate it further.

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're passing an iTunesDB file, not a device info file, thats why its crashing.

From the iPods I have, it seems iTunes/DeviceInfo is only on Firewire iPods (1st, 2nd, and 3rd gen classics)
To your knowledge, is there a newer or renamed file on all other i.e. USB iPods?

I had the same problem too where a lot of songs were missing, unfortunately I checked a few of them and the songs were just not on Spotify at all, so I think that's why it didn't find it.

Yeah - I get that Spotify might not have all the songs. The issue I'm describing is with the parser itself, where it doesn't parse all of the songs on the iPod.

See GEOFFREY.zip as an example:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.43s
➜  parser git:(main) ./target/debug/itunesdb_parser ~/Downloads/GEOFFREY/iPod_Control/iTunes/iTunesDB itunes     

File is using language: en, and has iTunes version: N/A (244)
133 songs in tracklist
Error! Epoch time converted was 0. Check the slice starting at idx 4560 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 5508 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 5508 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 7456 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 8420 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 8420 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 12252 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 12252 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 14164 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 16016 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 17880 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 17880 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 19728 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 23452 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 24420 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 25356 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 27228 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 29048 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 33672 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 34620 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 34620 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 36500 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 36500 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 37496 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 37496 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 38436 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 38436 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 39484 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 40452 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 45140 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 46160 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 46160 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 48932 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 49872 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 51748 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 51748 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 52700 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 54588 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 54588 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 56464 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 56464 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 56464 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 58308 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 58308 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 61164 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 62140 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 63056 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 63056 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 65032 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 65960 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 65960 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 67848 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 69772 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 69772 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 70704 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 70704 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 71652 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 71652 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 73636 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 73636 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 75564 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 75564 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 77448 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 77448 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 83172 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 84128 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 84128 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 86076 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 87096 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 89264 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 91408 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 92492 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 94668 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 94668 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 95788 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 95788 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 99008 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 99916 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 100820 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 100820 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 101740 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 101740 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Error! Epoch time converted was 0. Check the slice starting at idx 104936 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #26
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Error! Epoch time converted was 0. Check the slice starting at idx 109284 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #26
Unable to decode Data Object with type #26
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Error! Epoch time converted was 0. Check the slice starting at idx 113724 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #26
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Error! Epoch time converted was 0. Check the slice starting at idx 118128 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Error! Epoch time converted was 0. Check the slice starting at idx 122496 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Unable to decode Data Object with type #26
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Unable to decode Data Object with type #26
Unable to decode Data Object with type #37
Unable to decode Data Object with type #39
Error! Epoch time converted was 0. Check the slice starting at idx 135588 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #26
Unable to decode Data Object with type #37
Error! Epoch time converted was 0. Check the slice starting at idx 143224 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 143224 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 144668 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 146124 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 147688 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 149288 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 150660 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 150660 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 152200 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 152200 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 154032 with len 4, actually contains a valid timestamp
Error! Epoch time converted was 0. Check the slice starting at idx 156880 with len 4, actually contains a valid timestamp
Unable to decode Data Object with type #102
Unable to decode Data Object with type #102
Unable to decode Data Object with type #102
Unable to decode Data Object with type #102
Unable to decode Data Object with type #102
0 podcasts found
61 songs found
Created music.csv with 61 songs

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue I'm describing is with the parser itself, where it doesn't parse all of the songs on the iPod.

Ahhh I see, okay! I'll look into that, I haven't noticed that before.

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Damn, I'm still not sure why the songs are missing, there's a general discrepancy overall between what the header says the number of songs are and the actual number of track items found.

It says there's 133 track items in the header, but I only find 72 track items. Out of those 72 track items, 3 of them don't have a title or an artist associated with it for some reason, leaving 69 valid track items, and the tool only picks up 61 of those.. However, there are 133 audio files (mp3 or m4a) in the iPod music directory.

I created a tool to rename the song files based on the CSV, so that way I can try to find info about the ones remaining.

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it's really strange indeed. I can confirm Geoffrey has 113 named songs with titles and artists on the physical iPod and all play

@joshkenney
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i also noticed the tracklist only gets printed sometimes, making comparisons difficult if it doesnt get to this part of the code:

// Parse TrackList
        else if potential_section_heading == itunesdb_constants::TRACKLIST_KEY.as_bytes() {
            let num_songs_in_db = helpers::get_slice_as_le_u32(
                idx,
                &itunesdb_file_as_bytes,
                itunesdb_constants::TRACKLIST_NUM_SONGS_OFFSET,
                itunesdb_constants::TRACKLIST_NUM_SONGS_LEN,
            );

            println!("{} songs in tracklist", num_songs_in_db);

@raleighlittles
Copy link
Owner

@raleighlittles raleighlittles commented on d1e07ea Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I see that get printed on every iTunesDB file I parse. I created an issue for it here to investigate, will probably have to wait until next week to really dive into it though.


Btw, do you have access to a Linux system and a 4th gen+ iPod? If so, can you try the instructions here:

http://www.ipodlinux.org/Device_Information/

Involves running an SCSI inquiry to extract Device Info from the iPod. You have to install the sg3_utils package @joshkenney

@raleighlittles
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @joshkenney , just wanted to follow up about the last thing I mentioned, running the SCSI query on the ipod. Have a linux system, and 10 minutes? If so I'd appreciate you testing it, it's something I could add to the repository and share with others.

Please sign in to comment.