07_rnn

Recurrent Neural Network (RNN)

• Week 7 RNN Slides

Session A: SketchRNN (RNN with Pre-Trained Models 1)

Objectives:

• Learn about Sequential Data and Markov Chains
• Learn about Recurrent Neural Network architectures
• Become familiar with use cases for RNNs
• Learn ml5’s RNN functions and underlying pre-trained models

Markov Chains

• Markov Chains by Victor Powell and Lewis Lehe
• ITP Course Generator by Allison Parrish
• N-Grams and Markov Chains by Allison Parrish
• Markov Chains from A2Z
• Markov Chain video tutorial Part 1, Markov Chain video tutorial Part 2
• Markov Chain p5.js example

RNNs

• The Unreasonable Effectiveness of RNNs and Visualizing and Understanding Recurrent Networks by by Andrei Karpathy
• Rohan & Lenny #3: Recurrent Neural Networks & LSTMs

Related Projects:

• Double Agent by Simon Biggs (Drawing)
• Four Experiments in Handwriting with a Neural Network (Drawing)
• 10 things artificial intelligence did in 2018 by Janelle Shane (Text)
• Writing with the Machine
• Magenta: Make Music and Art Using Machine Learning
• Handwriting Generation with RNN and p5.js
• RNN for generating Baroque Music video
• Let's Read a Story by Itay Niv

RDP (Ramer-Douglas-Peucker) Algorithm

• p5.js RDP Example

SketchRNN Examples

• Generating Drawings with SketchRNN
• Interactive Drawings with SketchRNN
• Interactive Drawing with SketchRNN and RDP algorithm

Session B: Magenta.js (RNN with Pre-Trained Models 2)

Objectives

• Learn distinction between LSTM and RNN.
• Become familiar with Magenta.js and RNN models for generating music.
• Understand concept of “temperature” as it relates to RNNs.

Neural Networks and Music

• Neural Nets for Generating Music by Kyle McDonald
• MusicRNN Demos

Resources

• Parse a MIDI file into a Tone.js-friendly JSON format
• Note names, MIDI numbers and frequencies
• Tone.js Documentation
• Hello Magenta Tutorial
• Background on MelodyRNN
• Luisa Pereira's Code of Music course
• Lakh MIDI Dataset

Related Projects

• AI Duet using MelodyRNN by Yotam Mann
• Connecting with Music Through Magenta.js by Tero Parviainen
• magenta.js projects
• RUNN by Vibert Thio

MusicRNN Examples

• Melody with Tone.js
• Generate Melody with MusicRNN
• Generate Melody with MusicRNN and Tone.js playback

LSTM

• Understanding LSTM Networks by Christopher Olah

Assignment 6 Due Sunday October 20 at 6pm

Reading

• Markov Chains by Victor Powell and Lewis Lehe
• Neural Nets for Generating Music by Kyle McDonald
• Teaching Machines to Draw by David Ha
• Connecting with Music Through Magenta.js by Tero Parviainen

Instructions

Use ml5.sketchRNN() and/or magenta.js to generate visuals and/or sound in an interactive p5.js sketch. You can invent your own creative idea or here are some suggested exercises:

• As suggested by @hardmaru, implement the RDP Line Simplification algorithm in the Interactive Drawings with SketchRNN example. Does the program perform "better" or "worse" (what does that even mean)? What is the optimal epsilon value? (A video tutorial about RDP algorithm will be shared via the mailing list.)
• Consider the interaction design of Interactive Drawings with SketchRNN example? How can you improve upon the user experience? Can you craft a story around the interaction or make it into a game?
• Visualize the melody generated with MelodyRNN. magenta.js has built in functions for doing this that you can experiment with or try drawing your own visualization with p5.js.
• Create your own version of an "AI Duet", can you build a user interaction around playing a melody that the ML model continues?

Document your coding exercise and how the readings informed your work in a blog post and link from the homework wiki page.