Replace LevelDB with faster persistence strategy, add Hogwarts audio …

…demo (#29) * overhaul API and audio demo to remove LevelDB * add validation accuracy, more refactors * add scripts for validation/training * working harry potter demo * make shapes properties, logging updates * respond to PR feedback * fix tests
PAIR-code · Aug 10, 2018 · f3981ad · f3981ad
1 parent 1069a85
commit f3981ad
Show file tree

Hide file tree

Showing 48 changed files with 2,467 additions and 3,229 deletions.
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -21,5 +21,5 @@
   "editor.detectIndentation": false,
   "editor.wrappingIndent": "none",
   "typescript.tsdk": "node_modules/typescript/lib",
-  "clang-format.executable": "node_modules/.bin/clang-format"
+  "clang-format.executable": "${workspaceRoot}/src/server/node_modules/.bin/clang-format"
 }
diff --git a/README.md b/README.md
@@ -1 +1,66 @@
-Federated learning experiment using TensorFlow.js
+# Federated Learning in TensorFlow.js
+
+This is the parent repository for an (experimental and probably only demo-ready) implementation of [Federated Learning](https://ai.googleblog.com/2017/04/federated-learning-collaborative.html) in [Tensorflow.js](https://js.tensorflow.org/). Federated Learning is a method for training machine learning models in a distributed fashion. Although it involves a central server, that server never needs to see any data or even compute a gradient. Instead, _clients_ perform all of the inference and training locally (which they already do in Tensorflow.js), and just periodically send the server updated weights (rather than data). The server's only job is to aggregate and redistribute them, which means it can be extremely lightweight!
+
+## Basic Usage
+
+On the server (NodeJS) side:
+
+```js
+import * as http from 'http';
+import * as federated from 'tfjs-federated-learning-server';
+
+const INITIAL_MODEL_URL = 'file:///initial/model.json';
+const httpServer = http.createServer();
+const fedServer = new federated.Server(httpServer, INITIAL_MODEL_URL);
+
+fedServer.onNewVersion((model, oldVersion, newVersion) => {
+  console.log(`updated model from ${oldVersion} to ${newVersion}`);
+});
+
+fedServer.setup().then(() => {
+  httpServer.listen(8080);
+})
+```
+
+On the client (browser) side:
+
+```js
+import * as federated from 'tfjs-federated-learning-client';
+
+const SERVER_URL = 'https://federated.learning.server'; // points to server above
+const client = new federated.Client(SERVER_URL);
+
+client.onNewVersion((model, oldVersion, newVersion) => {
+  console.log(`updated model from ${oldVersion} to ${newVersion}`);
+});
+
+client.setup().then(() => {
+  // make predictions!
+  const yhat = client.predict(x);
+
+  // train (and asynchronously update the server)!
+  client.fit(x, y);
+});
+```
+
+## Advanced Usage
+
+See specific [server](./src/server/README.md) and [client](./src/client/README.md) docs.
+
+## A Note about Privacy
+
+Federated learning is considered "private" because clients never send their data to the server, only updated weights. In an ideal world,
+1. the server should not be able to meaningfully reconstruct client data from weight updates
+2. one client should not be able to reliably detect the presence of another client from model updates
+
+To help achieve (1), we can:
+- increase `numExamplesPerUpdate` (averages together more individual data points before computing the update)
+- increase `weightNoiseStddev` (adds noise to the update)
+- possibly change `epochs`, `batchSize`, and `learningRate` (clear that modifying these affects reconstructability, but unclear exactly _how_)
+- implement secure aggregation (encrypt update and only allow decryption after averaging with many other users)
+- modify the model architecture (different layer updates contain different amounts of information about inputs)
+
+To help achieve (2), we can:
+- increase `numUpdatesPerVersion` (average together more updates before computing the new version)
+- implement new features that limit the contributions of individual clients to the new version
diff --git a/demo/audio/client/harryp.ttf b/demo/audio/client/harryp.ttf
diff --git a/demo/audio/client/index.css b/demo/audio/client/index.css
@@ -15,94 +15,189 @@
  * =============================================================================
  */
 
-body {
-  font-family: monospace;
+@font-face {
+  font-family: harryp;
+  src: url(harryp.ttf);
+}
+
+html {
+  color: #222;
+  background-color: white;
+  transition: all 500ms ease-in-out;
+}
+
+html.nox {
+  background-color: #222;
+  color: #aaa;
 }
-button {
+
+html.accio {
+  cursor: url(wand.png) 0 0, auto !important;
+}
+
+html.accio.nox {
+  cursor: url(wand-active.png) 0 0, auto !important;
+}
+
+button:not(:disabled) {
   cursor: pointer;
 }
-#record-button {
-  font-family: monospace;
-  margin-top: 5px;
-  background-color: lightgreen;
+
+html.accio button:not(:disabled) {
+  cursor: url(wand-active.png) 0 0, pointer !important;
 }
-.chart {
-  display: inline-block;
-  position: relative;
-  padding-top: 1.2em;
-  vertical-align: top;
+
+body {
+  font-family: harryp;
+  font-size: 24px;
 }
-.chart label {
+
+.header {
+  background-color:black;
   position: absolute;
   top: 0;
   left: 0;
+  height: 60px;
+  width: 100%;
+  border-bottom: 1px solid #ccc;
 }
-.chart canvas {
-  border: 1px solid #ccc;
-}
-.left-col {
-  width: 800px;
+
+.header h1 {
   float: left;
+  margin: 5px;
+  color: yellow;
 }
-.right-col {
+
+.header nav {
   display: inline-block;
-  width: 400px;
 }
-#probs {
-  margin-top: 1em;
-}
-#label-radios {
+
+.header nav ul {
   list-style: none;
-  padding-left: 32px;
+  margin: 0;
+  padding: 0;
+  height: 60px;
 }
-#label-radios li {
-  display: inline;
+
+.header nav li {
+  float: left;
+  margin-left: 1em;
+  line-height: 60px;
 }
-#label-radios label {
-  cursor: pointer;
-  position: relative;
-  font-family: "Open Sans", verdana, arial, sans-serif;
-  padding: 0 7px;
-  padding-bottom: 16px;
-  border: 1px solid #ccc;
-  font-size: 12px;
-  fill: rgb(68, 68, 68);
+
+.header nav li a {
+  color: yellow;
+}
+
+.container {
+  width: 666px;
+  margin: 0 auto;
+  margin-top: 80px;
+}
+
+.results.container {
+  width: 90%;
+  min-width: 1000px;
+}
+
+#results {
+  max-width: 1600px;
+}
+
+legend {
+  font-size: 32px;
+}
+
+#model-version {
+  font-size: 24px;
+}
+
+span.status-number,
+span.last-prediction {
+  font-size: 32px;
+  text-shadow: 2px 2px 2px yellow;
+}
+
+.record-button {
+  font-family: harryp;
+  margin-top: 5px;
+  background-color: lightgreen;
+  font-size: 48px;
+  padding: 0 10px;
+}
+
+.record-button.active {
+  background-color: darkgoldenrod;
 }
-#label-radios label:hover {
-  background: #e8e8e8;
+
+.chart {
+  display: inline-block;
+  position: relative;
+  padding-top: 1.2em;
+  vertical-align: top;
 }
-#label-radios input {
+
+.chart label {
   position: absolute;
-  margin: 0;
-  top: 16px;
-  left: 50%;
-  transform: translateX(-50%);
+  top: 0;
+  left: 0;
+}
+
+.chart canvas {
+  border: 1px solid #ccc;
 }
+
 #example-table {
   width: 100%;
 }
+
 #example-table tbody tr {
   border-top: 1px solid;
 }
-#example-table tbody td {
+
+#example-table td {
   padding-right: 1em;
 }
+
 #example-table th {
   text-align: left;
   padding-right: 1em;
 }
+
 #example-table .incorrect-prediction {
   color: darkred;
   font-weight: bold;
 }
+
 audio {
-  width: 250px;
+  max-width: 250px;
 }
+
 fieldset {
   position: relative;
 }
+
 .version {
   position: absolute;
   right: 1em;
   bottom: 1em;
 }
+
+@media (max-width: 665px) {
+  .spectrum, .header nav, .model-input {
+    display: none;
+  }
+  .container {
+    width: 100%;
+  }
+  .header h1 {
+    width: 100%;
+    text-align: center;
+  }
+  audio {
+    max-width: 100px;
+  }
+  #example-table td, #example-table th {
+    padding-right: 0;
+  }
+}
diff --git a/demo/audio/client/index.html b/demo/audio/client/index.html
@@ -20,35 +20,41 @@
     <link rel="stylesheet" type="text/css" href="index.css">
   </head>
   <body>
-    <div class='right-col'>
-      <fieldset>
-        <legend>Labeled Examples</legend>
-        <table id='example-table'>
-          <thead><tr><th>audio</th><th>image</th><th>label</th><th>prediction</th></tr></thead>
-          <tbody id='labeled-examples'></tbody>
-        </table>
-      </fieldset>
+    <div class='header'>
+      <h1>Federated Hogwarts</h1>
+      <nav>
+        <ul>
+          <li><a href='https://js.tensorflow.org/' target='_blank'>TensorflowJS</a></li>
+          <li><a href='https://ai.googleblog.com/2017/04/federated-learning-collaborative.html' target='_blank'>Federated Learning</a></li>
+          <li><a href='https://github.com/PAIR-code/federated-learning' target='_blank'>Demo Code</a></li>
+        </ul>
+      </nav>
     </div>
-    <div class='left-col'>
-      <fieldset>
-        <legend>Hi!</legend>
-        <span id='intro-text'></span>
-        <h1 id='suggested-label'></h1>
-        <button id='record-button' disabled>Connecting to server&hellip;</button>
+    <div class='container'>
+      <fieldset id='spells'>
+        <legend>Spells</legend>
+        <div id='status'>Connecting to server&hellip;</div>
       </fieldset>
-      <fieldset>
+      <fieldset class='spectrum'>
         <legend>Audio</legend>
         <div class='chart'>
           <label>Spectrum</label>
-          <canvas id="spectrum-canvas" height="90" width="770"></canvas>
+          <canvas id="spectrum-canvas" height="90" width="617"></canvas>
         </div>
       </fieldset>
       <fieldset>
-        <legend>Model (version <span id='model-version'>pending&hellip;</span>)</legend>
+        <legend>Model (<span id='model-version'>version pending&hellip;</span>)</legend>
         <div id='model'>
           Waiting for input&hellip;
         </div>
       </fieldset>
+      <fieldset>
+        <legend>Priori Incantatem</legend>
+        <table id='example-table'>
+          <thead><tr><th>audio</th><th>image</th><th>label</th><th>prediction</th></tr></thead>
+          <tbody id='labeled-examples'></tbody>
+        </table>
+      </fieldset>
     </div>
     <script src="https://cdn.plot.ly/plotly-latest.min.js"></script>
     <script src="index.js"></script>