conversion/real_conversion.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LyricsAnnot import LyricsAnnot\n",
    "import json\n",
    "import main\n",
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the functions we used and steps we followed to create our final dataset from DALI and DAMP, enriching it with information from Genius and Spotify. Please note that this notebook if rather to keep track of what we did rather than rerunning it. However, if you need to add another dataset to our dataset, you could use something similar to what we did below to integrate it to our work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conversion of the whole DALI dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path to datasets\n",
    "DAMP_dir = \"../data/DAMP_MVP/sing_300x30x2\"\n",
    "DALI_dir = \"../data/dali_json\" \n",
    "\n",
    "id_file_path = \"./id.txt\" # List of the usable converted songs along with their new id\n",
    "\n",
    "# List of the songs put aside from the conversion for different reasons\n",
    "dali_avoided_songs_file_path = \"./dali_avoided_songs.txt\"\n",
    "damp_avoided_songs_file_path = \"./damp_avoided_songs.txt\"\n",
    "\n",
    "dali_already_converted_path = \"../data/dali_json/dali_already_converted\"\n",
    "damp_already_converted_path = \"../data/DAMP_MVP/damp_already_converted\"\n",
    "\n",
    "# Path of the folder where the final json files get saved once the song gets processed and converted\n",
    "save_path = \"./saved/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "main.create_damp_notations(DAMP_dir, save_path, damp_already_converted_path, id_file_path, damp_avoided_songs_file_path)\n",
    "main.create_dali_notations(DALI_dir, save_path, dali_already_converted_path, id_file_path, dali_avoided_songs_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-conversion steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there some songs that got wrongly converted (because of errors in the dataset content that prevent from right line matching, for example) and hence need to be put aside? Usually, these songs only have one single paragraph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "song_ids = utils.get_single_paragraph_song_info(\"./saved/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "397"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(song_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 397 songs we need to put aside. Let's use their id to perform this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'00000003': ('🄼🄰🄻🄰🄼 TERAKHIR Terakhir Terakhir', '🇳 🇦\\ufeff 🇩 🇦\\ufeff = 🇳 🇦\\ufeff 🇩 \\ufeff🇮 \\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000\\u3000 MALAM'), '00000005': ('Me acostumbre', 'Arcangel feat. Bad Bunny'), '00000016': ('Shape Of You', 'Shape'), '0000001E': ('Muchacha ojos de papel', 'Spinetta'), '0000002F': ('Me Duele Amarte', 'Reik'), '00000039': ('Asi fue', 'Juan Gabriel'), '0000005A': ('Felices los 4', 'Maluma'), '0000008E': ('Let it go', 'James Bay'), '00000096': ('Tomorrow Starts Today - Andi Mack Theme Song', 'Sabrina Carpenter'), '000000A4': (\"How Far I'll Go\", 'Moana'), '000000A6': ('Strip That Down', 'Liam Payne ft. Quavo'), '000000C1': ('We Found Love', 'Rihanna'), '000000D0': ('Hallelujah', 'Jeff Buckley'), '000000E3': ('1Kilo-Deixe me ir', '1Kilo'), '000000E4': ('Bang - Anitta', 'Anitta'), '000000F8': ('Primeiros erros', 'Capital Inicial'), '0000010A': ('O ESCUDO', 'Voz Da Verdade'), '00000118': ('Deu Onda', 'Mc G15'), '00000122': ('nao morrerei', 'marquinhos gomes'), '00000126': ('Ôh Novinha', 'MC Don Juan'), '00000127': ('Te Amo', 'Piranha'), '0000012B': ('Era Uma', 'Vez'), '0000012E': ('Aos Pes Da Cruz', 'Kleber Lucas'), '00000143': ('Te amare', 'Miguel Bose'), '0000014A': ('Cuéntame al oído', 'La Oreja de Van Gogh'), '0000014F': ('NETFLIXXX', 'Brytiago'), '00000150': ('Mi Gente', 'J Balvin'), '00000161': ('Fue dificil', 'Maria José Quintanilla'), '00000165': ('My Love', 'Westlife'), '00000169': ('Vergiss Mich', 'Bushido'), '0000018F': ('Shape Of You', 'Ed Sheeran'), '0000019E': ('Uzun ince bir yoldayim', 'asik veysel'), '000001AE': ('80 Million', 'Max Giesinger'), '000001B2': (\"Amoi seg' ma uns wieder\", 'Andreas Gabalier'), '000001B3': ('ahmet kaya - söyle', 'Ahmet KAYA'), '000001D5': ('Vencer al amor', 'India Martinez'), '000001D6': ('Loco', 'Enrique Iglesias ft. Romeo Santos'), '000001DC': ('Vois sur ton chemin', 'Les Choristes'), '000001F0': ('ça les dérange', 'Vitaa'), '000001F4': ('Soulman', \"Ben l'oncle soul\"), '00000201': ('Ce Soir Ne Sors Pas', 'Lacrim'), '0000020B': (\"Love d'un voyou\", 'Fababy ft. Aya Nakamura'), '00000218': ('Jour 1', 'Louane'), '00000223': ('Millionnaire', 'Soprano'), '00000232': ('Break Free', 'Ariana Grande'), '00000237': ('Yours', 'Ella Henderson'), '00000238': ('Yellow -', 'Coldplay'), '0000023A': ('Chandelier', 'Sia'), '00000240': ('Thinking Out Loud', 'Ed Sheeran'), '00000241': ('Back to Black', 'Amy Winehouse'), '00000255': ('Sign of the Times', 'Harry Styles'), '0000025B': ('Radioactive', 'Imagine Dragons'), '0000025E': ('Wicked Game', 'Chris Isaak'), '00000265': ('Little Things', 'One Direction'), '0000026C': ('Best Of Me', 'BTS'), '00000274': ('Ko Ko Bop', 'EXO'), '00000284': ('Bukan Cinta Biasa', 'Siti Nurhaliza'), '00000292': ('O Mere Dil Ke Chain', 'Kishore Kumar'), '000002BC': ('Allenamento #3', 'CAPO PLAZA'), '000002C0': ('SENZA PAGARE - J AX & FEDEZ', 'J ax'), '000002F0': ('Certe Notti', 'Ligabue'), '000002FD': ('secret base～君がくれたもの～ - ZONE', 'ZONE'), '00000303': ('Lovers Again', 'EXILE'), '00000306': ('アイのシナリオ', 'Honeyworks'), '0000030C': ('小さな恋のうた', 'MONGOL800'), '00000311': ('보고싶다', '김범수'), '00000314': ('이선희 - 인연', '이선희'), '00000322': ('이 소설의 끝을 다시 써보려해', '한동근'), '00000327': ('좋은사람', '박효신'), '00000329': ('시차', '우원재'), '0000032E': ('잊지 말아요 - 백지영', '백지영'), '00000331': ('raindrop', 'Iu (아이유)'), '00000335': ('내가 저지른 사랑', '임창정'), '0000033A': ('나야나', 'PRODUCE 101 season2'), '00000341': ('Que hago yo', 'HA ASH'), '00000342': ('Lamento Boliviano', 'Enanitos Verdes'), '00000343': ('El RECUENTO DE LOS DAÑOS - GLORIA TREVI', 'GLORIA TREVI'), '0000034E': ('Amor Eterno', 'Rocio Durcal'), '00000364': ('Que hago yo?', 'Haash'), '0000036C': ('Culon Culito', 'Cartel De Santa'), '00000374': ('Sayang Maafkan Aku', 'Syafiq Farhain'), '0000037E': ('Kesetiaan', 'Siti Sarah'), '00000392': ('Girls', 'Marcus'), '0000039A': ('Desperado', 'Eagles'), '000003A0': ('Hjerteknuser', 'Kaizers Orchestra'), '000003A5': ('Have You Ever Seen The Rain', 'Creedence Clearwater Revival'), '000003CB': (\"AS IF IT'S YOUR LAST\", 'Blackpink'), '00000404': ('Мелом', 'Пропаганда'), '0000040A': ('Вжух, как это клево', 'Марьяна Ро'), '0000040B': ('скрипач', 'Казан Казиев'), '00000415': ('Я поднимаю руки', 'Лепс Григорий'), '00000428': ('Happy End', 'Пара Нормальных'), '00000430': ('Ультрамариновые танцы', 'Элджей'), '00000436': ('Пина Колада', 'Люся Чеботина'), '0000043A': ('Самая-самая', 'Крид Егор'), '0000043E': ('Дым сигарет с ментолом', 'Нэнси'), '0000044A': ('Березы', 'Безруков Сергей'), '00000464': ('Широка река', 'Кадышева Надежда и ансамбль \"Золотое кольцо\"'), '00000475': ('Санавабич', 'Miyagi'), '0000047A': ('รังเกียจกันไหม - UrboyTJ', 'UrboyTJ'), '00000482': ('No Flockin', 'Kodak Black'), '000004A5': ('Kiss It Better', 'Rihanna'), '000004AC': (\"Don't Judge Me\", 'Chris Brown'), '000004C5': ('Geschwisterliebe', 'Die Ärzte'), '000004D3': ('Adios', 'Rammstein'), '000004D7': ('Kinky Afro', 'Happy Mondays'), '00000508': ('Bad Moon Rising', 'Creedence Clearwater Revival'), '0000052F': ('Hounds Of Love', 'Kate Bush'), '00000547': ('Sweetest Thing', 'U2'), '0000054D': ('The Invisible Man', 'Queen'), '00000558': ('Mine', 'Taylor Swift'), '00000561': ('Totale Finsternis', 'Tanz Der Vampire'), '00000564': ('A Boy Named Sue', 'Johnny Cash'), '00000571': ('Mirror Mirror', 'Blind Guardian'), '00000578': ('Eet', 'Regina Spektor'), '00000579': ('Freiheit', 'Marius Müller-Westernhagen'), '0000058B': ('Stray Heart', 'Green Day'), '00000592': ('Hard Candy Christmas', 'Dolly Parton'), '0000059A': ('Last Man Standing', 'HammerFall'), '000005A6': ('Rosas', 'La Oreja De Van Gogh'), '000005B5': ('Du Entschuldige I Kenn Di', 'Peter Cornelius'), '000005C1': (\"'39\", 'Queen'), '000005D0': ('Piano Man', 'Billy Joel'), '000005DC': ('Tu', 'Noelia'), '000005E0': ('Tra Te E Il Mare', 'Laura Pausini'), '000005FC': (\"Cryin'\", 'Aerosmith'), '0000060B': ('Yo (Excuse Me Miss)', 'Chris Brown'), '00000622': ('Return To Innocence', 'Enigma'), '00000626': ('Mindestens In 1000 Jahren', 'Frittenbude'), '0000062C': ('Breathe Easy', 'Blue'), '00000630': ('Cassius', 'Foals'), '00000649': ('Change The World', 'The Offspring'), '00000651': ('Broken Angel', 'Arash'), '00000658': ('DOA', 'Foo Fighters'), '0000065B': ('Our Song', 'Taylor Swift'), '0000065E': ('Proud Mary', 'Creedence Clearwater Revival'), '0000067D': ('Time Is On My Side', 'The Rolling Stones'), '00000681': (\"Without You I'm Nothing\", 'Placebo'), '0000068A': ('Build A Bridge', 'Limp Bizkit'), '00000695': ('Shining Light', 'Ash'), '0000069D': ('Danny Boy', 'Elvis Presley'), '000006A0': ('Allt På Ett Kort', 'Bob Hund'), '000006A3': ('Touch Of My Hand', 'Britney Spears'), '000006A6': ('Calling You', 'Kat DeLuna'), '000006B6': ('Leave It Alone', 'NOFX'), '000006B7': ('Get Out Alive', 'Three Days Grace'), '000006BB': ('National Anthem', 'LeAnn Rimes'), '000006C2': ('Saturday Night', 'Suede'), '000006C5': ('Bad Case Of Loving You (Doctor, Doctor)', 'Robert Palmer'), '000006CD': ('Take Your Mama', 'Scissor Sisters'), '000006CE': ('Desire', 'U2'), '000006E1': ('Stranger', 'Hilary Duff'), '000006EB': ('(Do You) Get Excited', 'Roxette'), '000006F3': ('7 E 40', 'Lucio Battisti'), '000006FB': ('Yes', 'Demi Lovato'), '00000711': ('Stay', 'Tooji'), '00000715': ('En Algún Lugar', 'Duncan Dhu'), '0000071A': (\"You'll Be Mine\", 'The Pierces'), '00000721': ('Brown Eyed Girl', 'Van Morrison'), '00000724': ('Missing', 'Evanescence'), '00000740': ('Crush', 'David Archuleta'), '00000745': ('Antisocial', 'Anthrax'), '0000074E': ('Woman', 'John Lennon'), '00000765': ('The Greatest Man That Ever Lived (Variations On A Shaker Hymn)', 'Weezer'), '00000770': ('You Sexy Thing', 'Hot Chocolate'), '00000772': ('The Prayer', 'Bloc Party'), '00000777': ('The Waiting One', 'All That Remains'), '00000783': ('The Down Syndrome', 'Grey Daze'), '00000785': ('Walk Like An Egyptian', 'The Bangles'), '0000079E': ('Dramatic Song', 'Toby Turner'), '000007A1': ('Deutschland', 'Die Prinzen'), '000007B0': ('Asereje', 'Las Ketchup'), '000007BB': ('Mr. Sandman', 'Emmylou Harris'), '000007CC': ('Serenata Rap', 'Jovanotti'), '000007DC': ('Sunshine', 'Aerosmith'), '000007E5': ('(Do You Get) Excited?', 'Roxette'), '000007F2': (\"Baba O'Riley\", 'The Who'), '000007F6': ('Vieni A Ballare In Puglia', 'CapaRezza'), '000007F7': ('Una Carezza In Un Pugno', 'Adriano Celentano'), '00000802': ('Broken', 'Seether'), '00000804': ('Would You Love A Monsterman?', 'Lordi'), '00000805': ('Virtual Insanity', 'Jamiroquai'), '0000081F': ('Mil Horas', 'Los Abuelos De La Nada'), '00000827': ('Half A World Away', 'R.E.M.'), '0000082B': ('Time Bomb', 'Rancid'), '00000837': ('There Is A Light That Never Goes Out', 'The Smiths'), '00000838': (\"Weus'd A Herz Hast Wia A Bergwerk\", 'Rainhard Fendrich'), '0000084A': ('You Are The One', 'Shiny Toy Guns'), '00000860': ('Sex Bomb', 'Tom Jones'), '00000875': ('RadioVideo', 'System Of A Down'), '00000883': ('Mentira', 'João Pedro Pais'), '00000884': ('New Soul', 'Yael Naim'), '0000088C': ('Bad Day', 'Darwin Deez'), '000008A8': ('Hostage Of Love', 'Razorlight'), '000008BF': ('Rooftops (A Liberation Broadcast)', 'Lostprophets'), '000008D4': ('Vielleicht', 'Madsen'), '000008D6': ('Down', 'Something Corporate'), '000008DD': (\"Europe's Living A Celebration\", 'Rosa López'), '000008EC': ('Easy To Ignore', 'Sixpence None The Richer'), '000008F6': ('Black Sheep', 'Sonata Arctica'), '000008FC': ('Tourniquet', 'Evanescence'), '0000090E': ('Deutsche Bahn', 'Wise Guys'), '0000091A': (\"I'm Just A Kid\", 'Simple Plan'), '00000923': ('Since U Been Gone', 'Kelly Clarkson'), '00000926': ('Up!', 'Shania Twain'), '00000930': ('Der Graf', 'Die Ärzte'), '0000093A': ('Fast Car', 'Jonas Blue'), '00000948': ('Yellow Ledbetter', 'Pearl Jam'), '00000961': ('Heartbreaker', 'G-Dragon'), '0000097F': ('We Gotta Get Out Of This Place', 'The Animals'), '00000983': ('Omaboy', 'Die Ärzte'), '00000991': ('Three Lions', 'Lightning Seeds'), '00000995': ('Paper Gangsta', 'Lady Gaga'), '0000099D': ('Pocahontas', 'AnnenMayKantereit'), '000009A0': ('Porch', 'Pearl Jam'), '000009B7': ('Double Je', 'Christophe Willem'), '000009C4': ('50 Special', 'Lùnapop'), '000009C6': ('Born To Be Wild', 'Steppenwolf'), '000009DB': ('Dear God', 'Avenged Sevenfold'), '000009DE': ('Lo Que Pasó, Pasó', 'Daddy Yankee'), '000009FB': ('When I Kissed The Teacher', 'ABBA'), '000009FD': ('Yellow', 'Coldplay'), '00000A07': ('Too Young', 'Phoenix'), '00000A09': ('I Am What I Am', 'Gloria Gaynor'), '00000A10': ('Dicono Di Me', 'Cesare Cremonini'), '00000A11': ('First Time', 'Lifehouse'), '00000A18': (\"Jesus Doesn't Want Me For A Sunbeam\", 'Nirvana'), '00000A1F': ('Psycho', 'Puddle Of Mudd'), '00000A3D': ('Invece No', 'Laura Pausini'), '00000A45': ('Tomber La Chemise', 'Zebda'), '00000A5F': ('The Wind Blows', 'The All-American Rejects'), '00000A63': ('Una Donna Per Amico', 'Lucio Battisti'), '00000A70': ('Mama Kin', 'Aerosmith'), '00000A96': ('Around The World (La La La La La)', 'ATC'), '00000A9E': ('Fallen Angels', 'Black Veil Brides'), '00000AA2': ('Lift U Up', 'Gotthard'), '00000AAE': (\"Who's Going Home With You Tonight?\", 'Trapt'), '00000ABB': ('Schickeria', 'Rainhard Fendrich'), '00000AC0': ('Strange World', 'Iron Maiden'), '00000AC1': ('Show Me Your Colours', 'S Club 7'), '00000ACD': ('Never Wake Up', 'Sum 41'), '00000ACE': ('Break Stuff', 'Limp Bizkit'), '00000AD8': (\"C'mon C'mon\", 'The Von Bondies'), '00000ADA': (\"God's Gonna Cut You Down\", 'Johnny Cash'), '00000AE5': ('If I Had A Hammer', 'Trini Lopez'), '00000AEA': ('Por Mujeres Como Tú', 'Pepe Aguilar'), '00000AF5': ('Have I Told You Lately', 'Rod Stewart'), '00000AFB': ('From The Inside', 'Linkin Park'), '00000B07': ('Sultans Of Swing', 'Dire Straits'), '00000B28': ('Großvater', 'STS'), '00000B45': ('500 Miles', 'Peter'), '00000B55': ('Why Still Bother', 'Itchy Poopzkid'), '00000B65': ('On Melancholy Hill', 'Gorillaz'), '00000B7D': ('Walk Away', 'Kelly Clarkson'), '00000B83': ('Inside Of You', 'Hoobastank'), '00000B92': ('Runaround Sue', 'Dion'), '00000B9D': ('These Dreams', 'Heart'), '00000BA2': ('Days Without', 'All That Remains'), '00000BBC': ('Cherry Lips (Go Baby Go!)', 'Garbage'), '00000BD6': ('Just Give Me A Reason', 'P!nk'), '00000BE1': ('(Let Me Be Your) Teddy Bear', 'Elvis Presley'), '00000BF4': ('Moviestar', 'Harpo'), '00000BF7': ('Le Manège', 'Stanislas'), '00000C11': ('Georgia On My Mind', 'Ray Charles'), '00000C23': ('Molinos De Viento', 'Mägo De Oz'), '00000C27': (\"'54, '74, '90, 2006\", 'Sportfreunde Stiller'), '00000C56': ('These Eyes', 'The Guess Who'), '00000C68': ('Go Tell It On The Mountain', 'Simon And Garfunkel'), '00000C6D': ('Raggio Di Sole', 'Le Vibrazioni'), '00000C7F': (\"The Shoop Shoop Song (It's In His Kiss)\", 'Cher'), '00000C8F': (\"I Don't Feel Like Dancin'\", 'Scissor Sisters'), '00000C91': ('Demon Cleaner', 'Kyuss'), '00000C96': ('Free Your Mind', 'En Vogue'), '00000C97': ('We Are Who We Are', 'Little Mix'), '00000C99': (\"Sgt. Pepper's Lonely Hearts Club Band\", 'The Beatles'), '00000C9F': ('Doctor Jones', 'Aqua'), '00000CC3': ('Summertime', 'DJ Jazzy Jeff'), '00000CC6': ('Enjoy The Silence', 'Lacuna Coil'), '00000CDF': ('Élan', 'Nightwish'), '00000CF1': ('Kirche', 'Böhse Onkelz'), '00000D06': (\"It's Hard To Know\", 'Hot Water Music'), '00000D18': ('Hey Tonight', 'Creedence Clearwater Revival'), '00000D1E': (\"Macy's Day Parade\", 'Green Day'), '00000D25': ('Them Bones', 'Alice In Chains'), '00000D2E': ('Dance Of Death', 'Iron Maiden'), '00000D41': (\"Vuonna '85\", 'Eppu Normaali'), '00000D44': (\"Don't Cry For Pain\", 'Ana Johnsson'), '00000D45': ('Pushing Me Away', 'Linkin Park'), '00000D55': ('Spaceman', 'Babylon Zoo'), '00000D6D': ('En Apesanteur', 'Calogero'), '00000D70': ('Tu Seras', 'Emma Daumas'), '00000D7B': ('Dile Al Sol', 'La Oreja De Van Gogh'), '00000D88': ('Cat And Mouse', 'The Red Jumpsuit Apparatus'), '00000DA4': ('Geil, Geil, Geil (Wir Sind Die Größten)', 'Wolfgang Petry'), '00000DA6': ('Leave In Silence', 'Depeche Mode'), '00000DAC': ('Vindicated', 'Dashboard Confessional'), '00000DBD': ('Second Chance', 'Shinedown'), '00000DC5': ('Skandal Im Sperrbezirk', 'Spider Murphy Gang'), '00000DC9': ('Gay Pirates', 'Cosmo Jarvis'), '00000DCB': ('Working For The Weekend', 'Loverboy'), '00000DCD': ('Non Passerai', 'Marco Mengoni'), '00000DD2': ('Pretty Fly (For A White Guy)', 'The Offspring'), '00000DDF': ('Schickeria', 'Spider Murphy Gang'), '00000DE2': ('Half The World Away', 'Oasis'), '00000DE4': ('Oerend Hard', 'Normaal'), '00000DFB': ('Lonely Boy', 'The Black Keys'), '00000E03': ('Going Nowhere', 'Little Mix'), '00000E09': ('Carta', 'Toranja'), '00000E16': ('Friday On My Mind', 'Gary Moore'), '00000E23': ('Take A Chance On Me', 'ABBA'), '00000E32': ('Shapeshifter', 'Celldweller'), '00000E3D': ('Sei Parte Di Me', 'Zero Assoluto'), '00000E49': ('Strong', 'London Grammar'), '00000E5C': (\"Un'emozione Per Sempre\", 'Eros Ramazzotti'), '00000E60': ('Animals', 'Maroon 5'), '00000E76': ('Laura', 'Scissor Sisters'), '00000E80': ('Shine', 'Imogen Heap'), '00000E91': ('Coma', \"Guns N' Roses\"), '00000E93': ('Dieser Weg', 'Xavier Naidoo'), '00000E95': ('Numb', 'Linkin Park'), '00000E98': ('Hit The Lights', 'Metallica'), '00000E9D': ('Cotton-Eye Joe', 'Rednex'), '00000EA2': ('Pure Morning', 'Placebo'), '00000EA8': (\"Ich Möcht' So Gerne Metal Hör'n\", 'J.B.O.'), '00000EAE': (\"Sorry, You're Not A Winner\", 'Enter Shikari'), '00000EB6': ('Plug In Baby', 'Muse'), '00000ED4': ('Girls & Boys', 'Blur'), '00000ED5': ('El Del Medio De Los Chichos', 'Estopa'), '00000EDB': ('Vivir Así Es Morir De Amor', 'Camilo Sesto'), '00000EE8': ('This Will Make You Love Again', 'IAMX'), '00000EF7': ('I Remember You', 'Skid Row'), '00000EFF': ('She Goes Nana', 'The Radios'), '00000F09': ('Story Of A Lonely Guy', 'Blink-182'), '00000F0E': ('...Baby One More Time', 'Britney Spears'), '00000F1E': ('Run To The Hills', 'Iron Maiden'), '00000F32': (\"Give 'Em Hell, Kid\", 'My Chemical Romance'), '00000F35': ('Zoe Jane', 'Staind'), '00000F3D': ('Between Angels And Insects', 'Papa Roach'), '00000F3F': (\"Gangsta's Paradise\", 'Coolio'), '00000F41': ('Ho Hey', 'The Lumineers'), '00000F49': ('Soulweeper 2', 'Volbeat'), '00000F4B': ('Beweg Dein Arsch', 'LaFee'), '00000F53': ('Kein Gerede', 'WIZO'), '00000F56': ('Tu Mi Porti Su', 'Giorgia'), '00000F61': ('Ricominciamo', 'Adriano Pappalardo'), '00000F66': ('Tutto Il Resto È Noia', 'Franco Califano'), '00000F6E': ('Hail To The King', 'Avenged Sevenfold'), '00000F79': ('99 Red Balloons', 'Goldfinger'), '00000F85': ('Return To Sender', 'Elvis Presley'), '00000F89': ('Going Missing', 'Maxïmo Park'), '00000F97': ('The Joker', 'Steve Miller Band'), '00000F99': ('Name', 'Goo Goo Dolls'), '00000F9A': ('Oh Yeah', 'Roxy Music'), '00000FA6': ('Wrapped', 'Gloria Estefan'), '00000FAB': ('Sweet Home Alabama', 'Lynyrd Skynyrd'), '00000FC8': ('Rusted From The Rain', 'Billy Talent'), '00000FD8': ('DeMoliendo Hoteles', 'Charly García'), '00000FE0': ('Do You Want To', 'Franz Ferdinand'), '00000FE6': ('Schwinger', 'Seeed'), '00000FF0': ('Cross My Heart', 'Marianas Trench'), '00000FFD': ('Juke Box Hero', 'Foreigner'), '00001003': ('Say You, Say Me', 'Lionel Richie'), '0000100A': ('I Miss You', 'Miley Cyrus'), '0000101F': ('Celebrity Status', 'Marianas Trench'), '00001025': (\"Parce Qu'on Vient De Loin\", 'Corneille'), '00001026': ('Jump', 'Van Halen'), '00001031': ('Traffic', 'Stereophonics'), '0000103E': ('Brizgalna Brizga', 'Atomik Harmonik'), '00001041': (\"Rome Wasn't Built In A Day\", 'Morcheeba'), '00001042': ('Our Velocity', 'Maxïmo Park'), '0000104A': ('Touch Me', 'The Doors'), '00001050': ('E Ritorno Da Te', 'Laura Pausini'), '00001051': ('Denial', 'Sugababes'), '00001054': ('Hier Kommt Kurt', 'Frank Zander'), '0000105B': ('Eve Of Destruction', 'Barry McGuire'), '00001068': ('Leuchtturm', 'Nena'), '00001092': ('Real Things', 'Javine'), '00001094': ('Cinderela', 'Carlos Paião'), '000010A1': ('Add It Up', 'Violent Femmes'), '000010A6': ('Die Welt Kann Mich Nicht Mehr Verstehen', 'Tocotronic'), '000010AB': ('Crying In The Chapel', 'Elvis Presley'), '000010DA': ('Guilty Of Love', 'Shanadoo'), '00001100': ('Noches De Bohemia', 'Navajita Plateá'), '00001112': ('4000 Rainy Nights', 'Stratovarius'), '00001117': ('The Trooper', 'Iron Maiden'), '00001121': ('Gifts And Curses', 'Yellowcard'), '00001124': ('Tonight', 'Big Bang'), '00001129': ('Haschisch Kakalake', 'Creme De La Creme'), '00001139': ('My Hero', 'Foo Fighters'), '00001142': ('Torero', 'Chayanne'), '0000114F': ('Everyday', 'Buddy Holly'), '00001150': ('Bad Day', 'Daniel Powter'), '00001156': ('Kiss This', 'Aaron Tippin'), '00001161': ('Dirty Deeds Done Dirt Cheap', 'AC/DC'), '0000116E': ('El Alma En Pie', 'Chenoa'), '00001176': ('We Own The Night', 'The Wanted'), '00001179': ('Ghostbusters', 'Ray Parker')}\n"
     ]
    }
   ],
   "source": [
    "print(song_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's remove the 397 songs from the usable converted songs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial dictionary size: 4517\n",
      "Dictionary size after deletions: 4120\n"
     ]
    }
   ],
   "source": [
    "utils.delete_dict_entries(song_ids, './id.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 304 songs have been deleted from the id file. Now, let's update the id of the usable songs (and their names) so that they are all consecutive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Song 1/397 successfully deleted!\n",
      "Song 2/397 successfully deleted!\n",
      "Song 3/397 successfully deleted!\n",
      "Song 4/397 successfully deleted!\n",
      "Song 5/397 successfully deleted!\n",
      "Song 6/397 successfully deleted!\n",
      "Song 7/397 successfully deleted!\n",
      "Song 8/397 successfully deleted!\n",
      "Song 9/397 successfully deleted!\n",
      "Song 10/397 successfully deleted!\n",
      "Song 11/397 successfully deleted!\n",
      "Song 12/397 successfully deleted!\n",
      "Song 13/397 successfully deleted!\n",
      "Song 14/397 successfully deleted!\n",
      "Song 15/397 successfully deleted!\n",
      "Song 16/397 successfully deleted!\n",
      "Song 17/397 successfully deleted!\n",
      "Song 18/397 successfully deleted!\n",
      "Song 19/397 successfully deleted!\n",
      "Song 20/397 successfully deleted!\n",
      "Song 21/397 successfully deleted!\n",
      "Song 22/397 successfully deleted!\n",
      "Song 23/397 successfully deleted!\n",
      "Song 24/397 successfully deleted!\n",
      "Song 25/397 successfully deleted!\n",
      "Song 26/397 successfully deleted!\n",
      "Song 27/397 successfully deleted!\n",
      "Song 28/397 successfully deleted!\n",
      "Song 29/397 successfully deleted!\n",
      "Song 30/397 successfully deleted!\n",
      "Song 31/397 successfully deleted!\n",
      "Song 32/397 successfully deleted!\n",
      "Song 33/397 successfully deleted!\n",
      "Song 34/397 successfully deleted!\n",
      "Song 35/397 successfully deleted!\n",
      "Song 36/397 successfully deleted!\n",
      "Song 37/397 successfully deleted!\n",
      "Song 38/397 successfully deleted!\n",
      "Song 39/397 successfully deleted!\n",
      "Song 40/397 successfully deleted!\n",
      "Song 41/397 successfully deleted!\n",
      "Song 42/397 successfully deleted!\n",
      "Song 43/397 successfully deleted!\n",
      "Song 44/397 successfully deleted!\n",
      "Song 45/397 successfully deleted!\n",
      "Song 46/397 successfully deleted!\n",
      "Song 47/397 successfully deleted!\n",
      "Song 48/397 successfully deleted!\n",
      "Song 49/397 successfully deleted!\n",
      "Song 50/397 successfully deleted!\n",
      "Song 51/397 successfully deleted!\n",
      "Song 52/397 successfully deleted!\n",
      "Song 53/397 successfully deleted!\n",
      "Song 54/397 successfully deleted!\n",
      "Song 55/397 successfully deleted!\n",
      "Song 56/397 successfully deleted!\n",
      "Song 57/397 successfully deleted!\n",
      "Song 58/397 successfully deleted!\n",
      "Song 59/397 successfully deleted!\n",
      "Song 60/397 successfully deleted!\n",
      "Song 61/397 successfully deleted!\n",
      "Song 62/397 successfully deleted!\n",
      "Song 63/397 successfully deleted!\n",
      "Song 64/397 successfully deleted!\n",
      "Song 65/397 successfully deleted!\n",
      "Song 66/397 successfully deleted!\n",
      "Song 67/397 successfully deleted!\n",
      "Song 68/397 successfully deleted!\n",
      "Song 69/397 successfully deleted!\n",
      "Song 70/397 successfully deleted!\n",
      "Song 71/397 successfully deleted!\n",
      "Song 72/397 successfully deleted!\n",
      "Song 73/397 successfully deleted!\n",
      "Song 74/397 successfully deleted!\n",
      "Song 75/397 successfully deleted!\n",
      "Song 76/397 successfully deleted!\n",
      "Song 77/397 successfully deleted!\n",
      "Song 78/397 successfully deleted!\n",
      "Song 79/397 successfully deleted!\n",
      "Song 80/397 successfully deleted!\n",
      "Song 81/397 successfully deleted!\n",
      "Song 82/397 successfully deleted!\n",
      "Song 83/397 successfully deleted!\n",
      "Song 84/397 successfully deleted!\n",
      "Song 85/397 successfully deleted!\n",
      "Song 86/397 successfully deleted!\n",
      "Song 87/397 successfully deleted!\n",
      "Song 88/397 successfully deleted!\n",
      "Song 89/397 successfully deleted!\n",
      "Song 90/397 successfully deleted!\n",
      "Song 91/397 successfully deleted!\n",
      "Song 92/397 successfully deleted!\n",
      "Song 93/397 successfully deleted!\n",
      "Song 94/397 successfully deleted!\n",
      "Song 95/397 successfully deleted!\n",
      "Song 96/397 successfully deleted!\n",
      "Song 97/397 successfully deleted!\n",
      "Song 98/397 successfully deleted!\n",
      "Song 99/397 successfully deleted!\n",
      "Song 100/397 successfully deleted!\n",
      "Song 101/397 successfully deleted!\n",
      "Song 102/397 successfully deleted!\n",
      "Song 103/397 successfully deleted!\n",
      "Song 104/397 successfully deleted!\n",
      "Song 105/397 successfully deleted!\n",
      "Song 106/397 successfully deleted!\n",
      "Song 107/397 successfully deleted!\n",
      "Song 108/397 successfully deleted!\n",
      "Song 109/397 successfully deleted!\n",
      "Song 110/397 successfully deleted!\n",
      "Song 111/397 successfully deleted!\n",
      "Song 112/397 successfully deleted!\n",
      "Song 113/397 successfully deleted!\n",
      "Song 114/397 successfully deleted!\n",
      "Song 115/397 successfully deleted!\n",
      "Song 116/397 successfully deleted!\n",
      "Song 117/397 successfully deleted!\n",
      "Song 118/397 successfully deleted!\n",
      "Song 119/397 successfully deleted!\n",
      "Song 120/397 successfully deleted!\n",
      "Song 121/397 successfully deleted!\n",
      "Song 122/397 successfully deleted!\n",
      "Song 123/397 successfully deleted!\n",
      "Song 124/397 successfully deleted!\n",
      "Song 125/397 successfully deleted!\n",
      "Song 126/397 successfully deleted!\n",
      "Song 127/397 successfully deleted!\n",
      "Song 128/397 successfully deleted!\n",
      "Song 129/397 successfully deleted!\n",
      "Song 130/397 successfully deleted!\n",
      "Song 131/397 successfully deleted!\n",
      "Song 132/397 successfully deleted!\n",
      "Song 133/397 successfully deleted!\n",
      "Song 134/397 successfully deleted!\n",
      "Song 135/397 successfully deleted!\n",
      "Song 136/397 successfully deleted!\n",
      "Song 137/397 successfully deleted!\n",
      "Song 138/397 successfully deleted!\n",
      "Song 139/397 successfully deleted!\n",
      "Song 140/397 successfully deleted!\n",
      "Song 141/397 successfully deleted!\n",
      "Song 142/397 successfully deleted!\n",
      "Song 143/397 successfully deleted!\n",
      "Song 144/397 successfully deleted!\n",
      "Song 145/397 successfully deleted!\n",
      "Song 146/397 successfully deleted!\n",
      "Song 147/397 successfully deleted!\n",
      "Song 148/397 successfully deleted!\n",
      "Song 149/397 successfully deleted!\n",
      "Song 150/397 successfully deleted!\n",
      "Song 151/397 successfully deleted!\n",
      "Song 152/397 successfully deleted!\n",
      "Song 153/397 successfully deleted!\n",
      "Song 154/397 successfully deleted!\n",
      "Song 155/397 successfully deleted!\n",
      "Song 156/397 successfully deleted!\n",
      "Song 157/397 successfully deleted!\n",
      "Song 158/397 successfully deleted!\n",
      "Song 159/397 successfully deleted!\n",
      "Song 160/397 successfully deleted!\n",
      "Song 161/397 successfully deleted!\n",
      "Song 162/397 successfully deleted!\n",
      "Song 163/397 successfully deleted!\n",
      "Song 164/397 successfully deleted!\n",
      "Song 165/397 successfully deleted!\n",
      "Song 166/397 successfully deleted!\n",
      "Song 167/397 successfully deleted!\n",
      "Song 168/397 successfully deleted!\n",
      "Song 169/397 successfully deleted!\n",
      "Song 170/397 successfully deleted!\n",
      "Song 171/397 successfully deleted!\n",
      "Song 172/397 successfully deleted!\n",
      "Song 173/397 successfully deleted!\n",
      "Song 174/397 successfully deleted!\n",
      "Song 175/397 successfully deleted!\n",
      "Song 176/397 successfully deleted!\n",
      "Song 177/397 successfully deleted!\n",
      "Song 178/397 successfully deleted!\n",
      "Song 179/397 successfully deleted!\n",
      "Song 180/397 successfully deleted!\n",
      "Song 181/397 successfully deleted!\n",
      "Song 182/397 successfully deleted!\n",
      "Song 183/397 successfully deleted!\n",
      "Song 184/397 successfully deleted!\n",
      "Song 185/397 successfully deleted!\n",
      "Song 186/397 successfully deleted!\n",
      "Song 187/397 successfully deleted!\n",
      "Song 188/397 successfully deleted!\n",
      "Song 189/397 successfully deleted!\n",
      "Song 190/397 successfully deleted!\n",
      "Song 191/397 successfully deleted!\n",
      "Song 192/397 successfully deleted!\n",
      "Song 193/397 successfully deleted!\n",
      "Song 194/397 successfully deleted!\n",
      "Song 195/397 successfully deleted!\n",
      "Song 196/397 successfully deleted!\n",
      "Song 197/397 successfully deleted!\n",
      "Song 198/397 successfully deleted!\n",
      "Song 199/397 successfully deleted!\n",
      "Song 200/397 successfully deleted!\n",
      "Song 201/397 successfully deleted!\n",
      "Song 202/397 successfully deleted!\n",
      "Song 203/397 successfully deleted!\n",
      "Song 204/397 successfully deleted!\n",
      "Song 205/397 successfully deleted!\n",
      "Song 206/397 successfully deleted!\n",
      "Song 207/397 successfully deleted!\n",
      "Song 208/397 successfully deleted!\n",
      "Song 209/397 successfully deleted!\n",
      "Song 210/397 successfully deleted!\n",
      "Song 211/397 successfully deleted!\n",
      "Song 212/397 successfully deleted!\n",
      "Song 213/397 successfully deleted!\n",
      "Song 214/397 successfully deleted!\n",
      "Song 215/397 successfully deleted!\n",
      "Song 216/397 successfully deleted!\n",
      "Song 217/397 successfully deleted!\n",
      "Song 218/397 successfully deleted!\n",
      "Song 219/397 successfully deleted!\n",
      "Song 220/397 successfully deleted!\n",
      "Song 221/397 successfully deleted!\n",
      "Song 222/397 successfully deleted!\n",
      "Song 223/397 successfully deleted!\n",
      "Song 224/397 successfully deleted!\n",
      "Song 225/397 successfully deleted!\n",
      "Song 226/397 successfully deleted!\n",
      "Song 227/397 successfully deleted!\n",
      "Song 228/397 successfully deleted!\n",
      "Song 229/397 successfully deleted!\n",
      "Song 230/397 successfully deleted!\n",
      "Song 231/397 successfully deleted!\n",
      "Song 232/397 successfully deleted!\n",
      "Song 233/397 successfully deleted!\n",
      "Song 234/397 successfully deleted!\n",
      "Song 235/397 successfully deleted!\n",
      "Song 236/397 successfully deleted!\n",
      "Song 237/397 successfully deleted!\n",
      "Song 238/397 successfully deleted!\n",
      "Song 239/397 successfully deleted!\n",
      "Song 240/397 successfully deleted!\n",
      "Song 241/397 successfully deleted!\n",
      "Song 242/397 successfully deleted!\n",
      "Song 243/397 successfully deleted!\n",
      "Song 244/397 successfully deleted!\n",
      "Song 245/397 successfully deleted!\n",
      "Song 246/397 successfully deleted!\n",
      "Song 247/397 successfully deleted!\n",
      "Song 248/397 successfully deleted!\n",
      "Song 249/397 successfully deleted!\n",
      "Song 250/397 successfully deleted!\n",
      "Song 251/397 successfully deleted!\n",
      "Song 252/397 successfully deleted!\n",
      "Song 253/397 successfully deleted!\n",
      "Song 254/397 successfully deleted!\n",
      "Song 255/397 successfully deleted!\n",
      "Song 256/397 successfully deleted!\n",
      "Song 257/397 successfully deleted!\n",
      "Song 258/397 successfully deleted!\n",
      "Song 259/397 successfully deleted!\n",
      "Song 260/397 successfully deleted!\n",
      "Song 261/397 successfully deleted!\n",
      "Song 262/397 successfully deleted!\n",
      "Song 263/397 successfully deleted!\n",
      "Song 264/397 successfully deleted!\n",
      "Song 265/397 successfully deleted!\n",
      "Song 266/397 successfully deleted!\n",
      "Song 267/397 successfully deleted!\n",
      "Song 268/397 successfully deleted!\n",
      "Song 269/397 successfully deleted!\n",
      "Song 270/397 successfully deleted!\n",
      "Song 271/397 successfully deleted!\n",
      "Song 272/397 successfully deleted!\n",
      "Song 273/397 successfully deleted!\n",
      "Song 274/397 successfully deleted!\n",
      "Song 275/397 successfully deleted!\n",
      "Song 276/397 successfully deleted!\n",
      "Song 277/397 successfully deleted!\n",
      "Song 278/397 successfully deleted!\n",
      "Song 279/397 successfully deleted!\n",
      "Song 280/397 successfully deleted!\n",
      "Song 281/397 successfully deleted!\n",
      "Song 282/397 successfully deleted!\n",
      "Song 283/397 successfully deleted!\n",
      "Song 284/397 successfully deleted!\n",
      "Song 285/397 successfully deleted!\n",
      "Song 286/397 successfully deleted!\n",
      "Song 287/397 successfully deleted!\n",
      "Song 288/397 successfully deleted!\n",
      "Song 289/397 successfully deleted!\n",
      "Song 290/397 successfully deleted!\n",
      "Song 291/397 successfully deleted!\n",
      "Song 292/397 successfully deleted!\n",
      "Song 293/397 successfully deleted!\n",
      "Song 294/397 successfully deleted!\n",
      "Song 295/397 successfully deleted!\n",
      "Song 296/397 successfully deleted!\n",
      "Song 297/397 successfully deleted!\n",
      "Song 298/397 successfully deleted!\n",
      "Song 299/397 successfully deleted!\n",
      "Song 300/397 successfully deleted!\n",
      "Song 301/397 successfully deleted!\n",
      "Song 302/397 successfully deleted!\n",
      "Song 303/397 successfully deleted!\n",
      "Song 304/397 successfully deleted!\n",
      "Song 305/397 successfully deleted!\n",
      "Song 306/397 successfully deleted!\n",
      "Song 307/397 successfully deleted!\n",
      "Song 308/397 successfully deleted!\n",
      "Song 309/397 successfully deleted!\n",
      "Song 310/397 successfully deleted!\n",
      "Song 311/397 successfully deleted!\n",
      "Song 312/397 successfully deleted!\n",
      "Song 313/397 successfully deleted!\n",
      "Song 314/397 successfully deleted!\n",
      "Song 315/397 successfully deleted!\n",
      "Song 316/397 successfully deleted!\n",
      "Song 317/397 successfully deleted!\n",
      "Song 318/397 successfully deleted!\n",
      "Song 319/397 successfully deleted!\n",
      "Song 320/397 successfully deleted!\n",
      "Song 321/397 successfully deleted!\n",
      "Song 322/397 successfully deleted!\n",
      "Song 323/397 successfully deleted!\n",
      "Song 324/397 successfully deleted!\n",
      "Song 325/397 successfully deleted!\n",
      "Song 326/397 successfully deleted!\n",
      "Song 327/397 successfully deleted!\n",
      "Song 328/397 successfully deleted!\n",
      "Song 329/397 successfully deleted!\n",
      "Song 330/397 successfully deleted!\n",
      "Song 331/397 successfully deleted!\n",
      "Song 332/397 successfully deleted!\n",
      "Song 333/397 successfully deleted!\n",
      "Song 334/397 successfully deleted!\n",
      "Song 335/397 successfully deleted!\n",
      "Song 336/397 successfully deleted!\n",
      "Song 337/397 successfully deleted!\n",
      "Song 338/397 successfully deleted!\n",
      "Song 339/397 successfully deleted!\n",
      "Song 340/397 successfully deleted!\n",
      "Song 341/397 successfully deleted!\n",
      "Song 342/397 successfully deleted!\n",
      "Song 343/397 successfully deleted!\n",
      "Song 344/397 successfully deleted!\n",
      "Song 345/397 successfully deleted!\n",
      "Song 346/397 successfully deleted!\n",
      "Song 347/397 successfully deleted!\n",
      "Song 348/397 successfully deleted!\n",
      "Song 349/397 successfully deleted!\n",
      "Song 350/397 successfully deleted!\n",
      "Song 351/397 successfully deleted!\n",
      "Song 352/397 successfully deleted!\n",
      "Song 353/397 successfully deleted!\n",
      "Song 354/397 successfully deleted!\n",
      "Song 355/397 successfully deleted!\n",
      "Song 356/397 successfully deleted!\n",
      "Song 357/397 successfully deleted!\n",
      "Song 358/397 successfully deleted!\n",
      "Song 359/397 successfully deleted!\n",
      "Song 360/397 successfully deleted!\n",
      "Song 361/397 successfully deleted!\n",
      "Song 362/397 successfully deleted!\n",
      "Song 363/397 successfully deleted!\n",
      "Song 364/397 successfully deleted!\n",
      "Song 365/397 successfully deleted!\n",
      "Song 366/397 successfully deleted!\n",
      "Song 367/397 successfully deleted!\n",
      "Song 368/397 successfully deleted!\n",
      "Song 369/397 successfully deleted!\n",
      "Song 370/397 successfully deleted!\n",
      "Song 371/397 successfully deleted!\n",
      "Song 372/397 successfully deleted!\n",
      "Song 373/397 successfully deleted!\n",
      "Song 374/397 successfully deleted!\n",
      "Song 375/397 successfully deleted!\n",
      "Song 376/397 successfully deleted!\n",
      "Song 377/397 successfully deleted!\n",
      "Song 378/397 successfully deleted!\n",
      "Song 379/397 successfully deleted!\n",
      "Song 380/397 successfully deleted!\n",
      "Song 381/397 successfully deleted!\n",
      "Song 382/397 successfully deleted!\n",
      "Song 383/397 successfully deleted!\n",
      "Song 384/397 successfully deleted!\n",
      "Song 385/397 successfully deleted!\n",
      "Song 386/397 successfully deleted!\n",
      "Song 387/397 successfully deleted!\n",
      "Song 388/397 successfully deleted!\n",
      "Song 389/397 successfully deleted!\n",
      "Song 390/397 successfully deleted!\n",
      "Song 391/397 successfully deleted!\n",
      "Song 392/397 successfully deleted!\n",
      "Song 393/397 successfully deleted!\n",
      "Song 394/397 successfully deleted!\n",
      "Song 395/397 successfully deleted!\n",
      "Song 396/397 successfully deleted!\n",
      "Song 397/397 successfully deleted!\n"
     ]
    }
   ],
   "source": [
    "utils.delete_wrong_converted_songs(song_ids, './saved/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "utils.change_ids_and_rename_files('./id.txt', './saved/', 8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistics about the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_file_path = \"./id.txt\"\n",
    "damp_avoided_songs_file_path = \"./damp_avoided_songs.txt\"\n",
    "dali_avoided_songs_file_path = \"./dali_avoided_songs.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DALI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "dali_usable_count = utils.count_sources(id_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3051\n"
     ]
    }
   ],
   "source": [
    "print(dali_usable_count['count_dali'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "dali_avoided_songs = utils.count_avoided_songs(dali_avoided_songs_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'avoided_counts': {'no_language_information': 26, 'wrongly_encoded_asian_song': 54, 'no_paragraphs': 1776, 'not_found_on_Genius': 29, 'total': 1885}}\n"
     ]
    }
   ],
   "source": [
    "print(dali_avoided_songs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61.81118314424635% of the songs from DALI dataset are usable to create the feed the knowledge graph.\n"
     ]
    }
   ],
   "source": [
    "if dali_usable_count != 0:\n",
    "    percentage = dali_usable_count['count_dali']/(dali_usable_count['count_dali']+dali_avoided_songs['avoided_counts']['total'])*100\n",
    "    print(f\"{percentage}% of the songs from DALI dataset are usable to create the feed the knowledge graph.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DAMP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have way less statistics to present regarding our DAMP suset as our avoided_songs file got overwritten. However, if we consider the 19 songs previously discarded (cf. post-conversion steps from DAMP conversion section) as the number of avoided_songs, and our total number of files to be the 264 files we got from the (sadly interrupted midway)conversion, we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "damp_usable_count = utils.count_sources(id_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1115\n"
     ]
    }
   ],
   "source": [
    "print(damp_usable_count['count_damp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "damp_avoided_songs = utils.count_avoided_songs(damp_avoided_songs_file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'avoided_counts': {'no_language_information': 25, 'wrongly_encoded_asian_song': 0, 'no_paragraphs': 1876, 'not_found_on_Genius': 1933, 'total': 3834}}\n"
     ]
    }
   ],
   "source": [
    "print(damp_avoided_songs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that this number is inferior to the number of DAMP files we merges (245). Since it takes into account the overlaps between DAMP and DALI, the only possible reason is that some songs from the DAMP subset were duplicates from each other, and that it was the case for 14 of them. This is probably a consequence of our DAMP partially failed conversion. Fortunately, the steps we followed enabled us to manage to clean this before using the data to create the knowledge graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29.081898800208663% of the songs from our subset of DAMP dataset are usable to create the feed the knowledge graph.\n"
     ]
    }
   ],
   "source": [
    "if damp_usable_count != 0:\n",
    "    percentage = damp_usable_count['count_damp']/(damp_avoided_songs['avoided_counts']['total'])*100\n",
    "    print(f\"{percentage}% of the songs from our subset of DAMP dataset are usable to create the feed the knowledge graph.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Common statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, here is the number of songs that were in common in DALI and in our subset of DAMP datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "46\n"
     ]
    }
   ],
   "source": [
    "print(damp_usable_count['count_double_sources'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is the total number of usable files from our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4120\n"
     ]
    }
   ],
   "source": [
    "print(damp_usable_count['count_dali']+damp_usable_count['count_damp']-damp_usable_count['count_double_sources'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}